Multi-Precision Policy Enforced Training (MuPPET) : A precision-switching strategy for quantised fixed-point training of CNNs

1 INTRODUCTION . Convolutional neural networks ( CNNs ) have demonstrated unprecedented accuracy in various machine learning tasks , from video understanding ( Gan et al. , 2015 ; He et al. , 2018 ) to drone navigation ( Loquercio et al. , 2018 ; Kouris & Bouganis , 2018 ) . To achieve such high levels of accuracy in inherently complex applications , current methodologies employ the design of large and complex CNN models ( Szegedy et al. , 2017 ; Huang et al. , 2017 ) trained over large datasets ( Deng et al. , 2009 ; Lin et al. , 2014 ) . Nevertheless , the combination of large models and massive datasets results in long training times . This in turn leads to long turn-around times which limits the productivity of deep learning practitioners and prohibits wider experimentation . For instance , automatic tuning and search of neural architectures ( Cai et al. , 2018 ; Zhong et al. , 2018 ) is a rapidly advancing area where accelerated training enables improving the produced networks . To counteract these long turn-around times , substantial research effort has been invested in hyperparameter tuning for the acceleration of training , with a particular focus on batch size and content . One line of work maximises memory and hardware utilisation by changing the batch size , in order to perform CNN training specific prefetching , scheduling and dependency improvement ( Chen et al. , 2019 ; Rhu et al. , 2016 ) . Other works focus on altering batch size or reconstructing minibatches to improve the convergence rate while sustaining high hardware utilisation ( Devarakonda et al. , 2017 ; Johnson & Guestrin , 2018 ; Peng et al. , 2019 ) , demonstrating up to 6.25× training speedup . A number of studies have focused on the use of reduced-precision training schemes . Reducedprecision arithmetic involves the utilisation of data formats that have smaller wordlengths than the conventional 32-bit floating-point ( FP32 ) representation and is an approach for co-optimising pro- cessing speed , memory footprint and communication overhead . Existing literature can be categorised into works that use reduced precision to accelerate only the training stage while targeting an FP32 model , and those that produce networks with quantised weights . Regarding the former , Courbariaux et al . ( 2015 ) and Gupta et al . ( 2015 ) utilise dynamic quantisation and stochastic rounding respectively as a means to combat the accuracy loss due to quantisation . Nevertheless , the effectiveness of the proposed schemes have only been demonstrated on smallscale datasets such as CIFAR-10 and MNIST , and on a limited set of networks . Furthermore , the range of quantisation levels that has been explored varies greatly , with a number of works attacking the problem by focusing on mild quantisation levels such as half-precision floating-point ( FP16 ) ( Micikevicius et al. , 2018 ) , while others focus on lower precisions such as 8-bit floating-point ( FP8 ) ( Wang et al. , 2018 ) . Finally , quantisation has also been used as a means of reducing the memory and communication overhead in distributed training ( De Sa et al. , 2015 ; 2017 ; Alistarh et al. , 2017 ) . At the same time , the characteristics of modern CNN workloads and the trend towards quantised models have led to an emergence of specialised hardware processors , with support for low-precision arithmetic at the hardware level . From custom designs such as Google ’ s TPUs ( Jouppi et al. , 2017 ) and Microsoft ’ s FPGA-based Brainwave system ( Fowers et al. , 2018 ) to commodity devices such as NVIDIA ’ s Turing GPUs , existing platforms offer native support for reduced-precision data types including 16-bit floating-point ( FP16 ) , and 8- ( INT8 ) and 4-bit ( INT4 ) fixed-point , providing increased parallelism for lower bitwidths . Although these platforms have been mainly designed for the inference stage , the low-precision hardware offers significant opportunities for accelerating the time-consuming training stage . In this respect , there is an emerging need to provide training algorithms that can leverage these existing hardware optimisations and provide higher training speed . This work tackles the field of reduced-precision training at an algorithmic level . Independently of the number of quantisation levels chosen , or how extreme the quantisation is , this work proposes a metric that estimates the amount of information each new training step obtains for a given quantisation level , by capturing the diversity of the computed gradients across epochs . This enables the design of a policy that , given a set of quantisation levels , decides at run time appropriate points to increase the precision of the training process at that current instant without impacting the achieved test accuracy compared to training in FP32 . Due to its agnostic nature , it remains orthogonal and complementary to existing low-precision training schemes . Furthermore , by pushing the precision below the 16-bit bitwidth of existing state-of-the-art techniques , the proposed method is able to leverage the lowprecision capabilities of modern processing systems to yield training speedups without penalising the resulting accuracy , significantly improving the time-to-accuracy trade-off . 2 BACKGROUND AND RELATED WORK . The state-of-the-art method in training in reduced precision is mixed-precision training ( Micikevicius et al. , 2018 ) . The authors propose to employ low-precision FP16 computations in the training stage of high-precision CNNs that perform inference in FP32 . Along the training phase , the algorithm maintains a high-precision FP32 copy of the weights of the network , known as a master copy . At each minibatch , the inputs and weights are quantised to FP16 with all computations of the forward and backward pass performed in FP16 , yielding memory footprint and runtime savings . Under this scheme , each stochastic gradient descent ( SGD ) update step entails accumulating FP16 gradients into the FP32 master copy of the weights , with this process performed iteratively throughout the training of the network . Micikevicius et al . ( 2018 ) evaluate their scheme over a set of state-of-theart models on ImageNet , and show that mixed-precision training with FP16 computations achieves comparable accuracy to standard FP32 training . Wang et al . ( 2018 ) also presented a method to train an FP32 model using 8-bit floating-point ( FP8 ) . The authors propose a hand-crafted FP8 data type , together with a chunk-based computation technique , and employ strategies such as stochastic rounding to alleviate the accuracy loss due to training at reduced precision . For AlexNet , ResNet18 and ResNet50 on ImageNet , Wang et al . ( 2018 ) demonstrates comparable accuracy to FP32 training while performing computations in FP8 . Additionally the works presented in ( Zhou et al. , 2016 ; Chen et al. , 2017 ) approach the problem of reduced-precision training employing fixed-point computations . FxpNet ( Chen et al. , 2017 ) was only evaluated on CIFAR-10 , failing to demonstrate performance on more complex datasets such as ImageNet . DoReFa-net ( Zhou et al. , 2016 ) was tested on ImageNet but only ran on AlexNet missing out on state-of-the-art networks such as GoogLeNet and ResNet . All related works focus on accelerating the training of an FP32 model through reduced-precision computations . At the hardware level , 8-bit fixed-point multiplication uses 18.5× less energy and 27.5× less area with up to 4× lower multiplication times than FP32 ( Sze et al. , 2017 ) . Consequently , this work attempts to push the boundaries of reduced-precision training by moving to reducedprecision fixed-point computations while updating an FP32 model . Preliminary tests ( Sec . 4.4 for details ) demonstrated that training solely in 8-bit fixed-point results in a significant degradation of validation accuracy compared to full FP32 training . This work aims to counteract this degradation by progressively increasing the precision of computations throughout training in an online manner determined by the proposed metric inspired by gradient diversity ( Yin et al. , 2018 ) . Additionally by operating in an online fashion , MuPPET tailors the training process to best suit the particular network-dataset pair at each stage of the training process . Gradient diversity was introduced by Yin et al . ( 2018 ) as a metric of measuring the dissimilarity between sets of gradients that correspond to different minibatches . The gradient diversity of a set of gradients is defined as ∆S ( w ) = ∑n i=1 ||∇fi ( w ) ||22 || ∑n i=1∇fi ( w ) ||22 = ∑n i=1 ||∇fi ( w ) ||22∑n i=1 ||∇fi ( w ) ||22 + ∑ i 6=j〈∇fi ( w ) , ∇fj ( w ) 〉 ( 1 ) where∇fi ( w ) represents the gradient of weights w for minibatch i . The key point to note in Eq . ( 1 ) is that the denominator contains the inner product between two gradients from different minibatches . Thus , orthogonal gradients would result in high gradient diversity , while similar gradients would result in low gradient diversity . The proposed framework , MuPPET , enhances this concept by considering gradients between minibatches across epochs and proposes the developed metric as a proxy for the amount of new information gained in each training step . Section 3 further expands on how gradient diversity is incorporated into the MuPPET algorithm . 3 METHODOLOGY . 3.1 MULTILEVEL OPTIMISATION FOR TRAINING CNNS . Conventionally , the training process of a CNN can be expressed as in Eq . ( 2 ) . Given a CNN model f parameterised by a set of weights w ∈ RD , whereD is the number of weights of f , training involves a search for weight values that minimise the task-specific empirical loss , Loss , on the target dataset . Typically , a fixed arithmetic precision is employed across the training algorithm with FP32 currently being the de facto representation used by the deep learning community . min w ( FP32 ) ∈RD Loss ( f ( w ( FP32 ) ) ) ( 2 ) The proposed method follows a different approach by introducing a multilevel optimisation scheme ( Migdalas et al. , 2013 ) that leverages the performance gains of reduced-precision arithmetic . The single optimisation problem of Eq . ( 2 ) is transformed into a series of optimisation problems with each one employing different precision for computations , but maintaining weights storage at FP32 precision . Under this scheme , an N -level formulation comprises N sequential optimisation problems to be solved , with each level corresponding to a “ finer ” model . Overall , this formulation adds a hierarchical structure to the training stage , with an increasing arithmetic precision across the hierarchy of optimisation problems . Starting from the N -th problem , the inputs , weights , and activations of the CNN model f are quantised with precision qN , which is the lowest precision in the system and represents the coarsest version of the model . Each of the N levels progressively employs higher precision until the first level is reached , which corresponds to the original problem of Eq . ( 2 ) . Formally , at the i-th level , the optimisation problem is formulated as min w ( qi ) ∈V Loss ( f ( w ( q i ) ) ) s.t . V = { w ( q i ) ∈ [ LB , UB ] D } ( 3 ) where LB and UB are the lower and upper bound in the representational range of precision qi . The target CNN model f uses a set of weights quantised with precision qi and hence the solution of this optimisation problem can be interpreted as an approximation to the original problem of Eq . ( 2 ) . To transition from one level to the next , the result of each level of optimisation is employed as a starting point for the next level , up to the final outermost optimisation that reduces to Eq . ( 2 ) . 3.2 THE MUPPET ALGORITHM Fig . 1 presents the process of training a CNN using the proposed algorithm . All figures in this paper are shown in the Appendix B at a larger scale for enhanced readability . Within each epoch , MuPPET performs mixed-precision training where the weights are stored in an FP32 master copy and are quantised to the desired fixed-point precision on-the-fly . At epoch j , the computations for the forward and backward passes ( F and B blocks respectively ) are performed at the current quantised precision ( qj ) and the ac- tivations as well as the gradients obtained from each layer are quantised by the quantiser module before being passed on to the next layer , or stored . After each minibatch , the full-precision master copy of the weights is updated using a quantised gradient matrix . As discussed in Section 3.1 , the quantisation level is gradually increased over the period of the training . In MuPPET , switching between these optimisation levels at the correct times is crucial in order not to compromise the final validation accuracy . In this respect , MuPPET introduces a precision switching policy based on an inter-epoch gradient diversity ( Yin et al. , 2018 ) metric that dictates when to switch to the next precision . Details of the switching policy are presented in Section 3.3 .

Summary: This paper proposes a training strategy called Multi-Precision Policy Enforced Training(MUPPET). This strategy aims to reduce training time by low-precision data representation and computations during the training stage. According to the gradient diversity, the authors introduce a precision-switching mechanism which chooses the best epoch to increase the precision. The validation accuracy and training time across several networks and datasets are shown in the experiments. However, the results are not superior enough compared with the state-of-the-art.