CPT: Efficient Deep Neural Network Training via Cyclic Precision

1 INTRODUCTION . The record-breaking performance of modern deep neural networks ( DNNs ) comes at a prohibitive training cost due to the required massive training data and parameters , limiting the development of the highly demanded DNN-powered intelligent solutions for numerous applications ( Liu et al. , 2018 ; Wu et al. , 2018 ) . As an illustration , training ResNet-50 involves 1018 FLOPs ( floating-point operations ) and can take 14 days on one state-of-the-art ( SOTA ) GPU ( You et al. , 2020b ) . Meanwhile , the large DNN training costs have raised increasing financial and environmental concerns . For example , it is estimated that training one DNN can cost more than $ 10K US dollars and emit carbon as high as a car ’ s lifetime emissions . In parallel , recent DNN advances have fueled a tremendous need for intelligent edge devices , many of which require on-device in-situ learning to ensure the accuracy under dynamic real-world environments , where there is a mismatch between the devices ’ limited resources and the prohibitive training costs ( Wang et al. , 2019b ; Li et al. , 2020 ; You et al. , 2020a ) . To address the aforementioned challenges , extensive research efforts have been devoted to developing efficient DNN training techniques . Among them , low-precision training has gained significant attention as it can largely boost the training time/energy efficiency ( Jacob et al. , 2018 ; Wang et al. , 2018a ; Sun et al. , 2019 ) . For instance , GPUs can now perform mixed-precision DNN training with 16-bit IEEE Half-Precision floating-point formats ( Micikevicius et al. , 2017b ) . Despite their promise , existing low-precision works have not yet fully explored the opportunity of leveraging recent findings in understanding DNN training . In particular , existing works mostly fix the model precision during the whole training process , i.e. , adopt a static quantization strategy , while recent works in DNN training optimization suggest dynamic hyper-parameters along DNNs ’ training trajectory . For example , ( Li et al. , 2019 ) shows that a large initial learning rate helps the model to memorize easier-to-fit and more generalizable patterns , which aligns with the common practice to start from a large learning rate for exploration and anneal to a small one for final convergence ; and ( Smith , 2017 ; Loshchilov & Hutter , 2016 ) improve DNNs ’ classification accuracy by adopting cyclical learning rates . In this work , we advocate dynamic precision training , and make the following contributions : • We show that DNNs ’ precision seems to have a similar effect as the learning rate during DNN training , i.e. , low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence , and dynamic precision schedules help DNNs converge to a better minima . This finding opens up a design knob for simultaneously improving the optimization and efficiency of DNN training . • We propose Cyclic Precision Training ( CPT ) which adopts a cyclic precision schedule along DNNs ’ training trajectory for pushing forward the achievable trade-offs between DNNs ’ accuracy and training efficiency . Furthermore , we show that the cyclic precision bounds can be automatically identified at the very early stage of training using a simple precision range test , which has a negligible computational overhead . • Extensive experiments on five datasets and eleven models across a wide spectrum of applications ( including classification and language modeling ) validate the consistent effectiveness of the proposed CPT technique in boosting the training efficiency while leading to a comparable or even better accuracy . Furthermore , we provide loss surface visualization for better understanding CPT ’ s effectiveness and discuss its connection with recent findings in understanding DNNs ’ training optimization . 2 RELATED WORKS . Quantized DNNs . DNN quantization ( Courbariaux et al. , 2015 ; 2016 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ; Li et al. , 2016 ; Jacob et al. , 2018 ; Mishra & Marr , 2017 ; Mishra et al. , 2017 ; Park et al. , 2017 ; Zhou et al. , 2016 ) has been well explored based on the target accuracy-efficiency tradeoffs . For example , ( Jacob et al. , 2018 ) proposes quantization-aware training to preserve the post quantization accuracy ; ( Jung et al. , 2019 ; Bhalgat et al. , 2020 ; Esser et al. , 2019 ; Park & Yoo , 2020 ) strive to improve low-precision DNNs ’ accuracy using learnable quantizers . Mixed-precision DNN quantization ( Wang et al. , 2019a ; Xu et al. , 2018 ; Elthakeb et al. , 2020 ; Zhou et al. , 2017 ) assigns different bitwidths for different layers/filters . While these works all adopt a static quantization strategy , i.e. , the assigned precision is fixed post quantization , CPT adopts a dynamic precision schedule during the training process . Low-precision DNN training . Pioneering works ( Wang et al. , 2018a ; Banner et al. , 2018 ; Micikevicius et al. , 2017a ; Gupta et al. , 2015 ; Sun et al. , 2019 ) have shown that DNNs can be trained with reduced precision . For distributed learning , ( Seide et al. , 2014 ; De Sa et al. , 2017 ; Wen et al. , 2017 ; Bernstein et al. , 2018 ) quantize the gradients to reduce the communication costs , where the training computations still adopt full precision ; For centralized/on-device learning , the weights , activations , gradients , and errors involved in both the forward and backward computations all adopt reduced precision . Our CPT can be applied on top of these low-precision training techniques , all of which adopt a static precision during the whole training trajectory , to further boost the training efficiency . Dynamic-precision DNNs . There exist some dynamic precision works which aim to derive a quantized DNN for inference after the full-precision training . Specifically , ( Zhuang et al. , 2018 ) first trains a full-precision model to reach convergence and then gradually decreases the model precision to the target one for achieving better inference accuracy ; ( Khoram & Li , 2018 ) also starts from a full-precision model and then gradually learns the precision of each layer to derive a mixed-precision counterpart ; ( Yang & Jin , 2020 ) learns a fractional precision of each layer/filter based on the linear interpolation of two consecutive bitwidths which doubles the computation and requires an extra fine-tuning step ; and ( Shen et al. , 2020 ) proposes to adapt the precision of each layer during inference in an input-dependent manner to balance computational cost and accuracy . 3 THE PROPOSED CPT TECHNIQUE . In this section , we first introduce the hypothesis that motivates us to develop CPT using visualization examples in Sec . 3.1 , and then present the CPT concept in Sec . 3.2 followed by the Precision Range Test ( PRT ) method in Sec . 3.3 , where PRT aims to automate the precision schedule for CPT . 3.1 CPT : MOTIVATION . Hypothesis 1 : DNN ’ s precision has a similar effect as the learning rate . Existing works ( Grandvalet et al. , 1997 ; Neelakantan et al. , 2015 ) show that noise can help DNN training theoretically or empirically , motivating us to rethink the role of quantization in DNN training . We conjecture that low precision with large quantization noise helps DNN training exploration with an effect similar to a high learning rate , while high precision with more accurate updates aids model convergence , similar to a low learning rate . Validating Hypothesis 1 . Settings : To empirically justify our hypothesis , we train ResNet-38/74 on the CIFAR-100 dataset for 160 epochs following the basic training setting as in Sec . 4.1 . In particular , we divide the training of 160 epochs into three stages : [ 0-th , 80-th ] , [ 80-th,120-th ] , and [ 120-th , 160-th ] : for the first training stage of [ 0-th , 80-th ] , we adopt different learning rates and precisions for the weights and activations , while using full precision for the remaining two stages with a learning rate of 0.01 for the [ 80-th,120-th ] epochs and 0.001 for the [ 120-th , 160-th ] epochs in all the experiments in order to explore the relationship between the learning rate and precision in the first training stage . Results : As shown in Tab . 1 , we can observe that as the learning rate is sufficiently reduced for the first training stage , adopting a lower precision for this stage will lead to a higher accuracy than training with full precision . In particular , with the standard initial learning rate of 0.1 , full precision training achieves a 1.00 % /0.70 % higher accuracy than the 4-bit one on ResNet-38/74 , respectively ; whereas as the initial learning rate decreases , this accuracy gap gradually narrows and then reverses , e.g. , when the initial learning rate becomes 1e-2 , training with [ 0-th , 80-th ] of 4-bit achieves a 1.40 % /1.57 % higher accuracy than the full precision ones . Insights : This set of experiments show that ( 1 ) when the initial learning rate is low , training with lower initial precisions consistently leads to a better accuracy than training with full precision , indicating that lowering the precision introduces a similar effect of favoring exploration as that of a high learning rate ; and ( 2 ) although a low precision can alleviate the accuracy drop caused by a low learning rate , a high learning rate is in general necessary to maximize the accuracy . Hypothesis 2 : Dynamic precision helps DNN generalization . Recent findings in DNN training have motivated us to better utilize DNN precision to achieve a win-win in both DNN accuracy and efficiency . Specifically , it has been discussed that ( 1 ) DNNs learn to fit different patterns at different training stages , e.g. , ( Rahaman et al. , 2019 ; Xu et al. , 2019 ) reveal that DNN training first learns lowerfrequency components and then high-frequency features , with the former being more robust to perturbations and noises ; and ( 2 ) dynamic learning rate schedules help to improve the optimization in DNN training , e.g. , ( Li et al. , 2019 ) points out that a large initial learning rate helps the model to memorize easier-to-fit and more generalizable patterns while ( Smith , 2017 ; Loshchilov & Hutter , 2016 ) show that cyclical learning rate schedules improve DNNs ’ classification accuracy . These works inspire us to hypothesize that dynamic precision might help DNNs to reach a better optimum in the optimization landscape , especially considering the similar effect between the learning rate and precision validated in our Hypothesis 1 . Validating Hypothesis 2 . Our Hypothesis 2 has been consistently confirmed by various empirical observations . For example , a recent work ( Fu et al. , 2020 ) proposes to progressively increase the precision during the training process , and we follow their settings to validate our hypothesis . Settings : We train a ResNet-74 on CIFAR-100 using the same training setting as ( Wang et al. , 2018b ) except that we quantize the weights , activations , and gradients during training ; for the progressive precision case we uniformly increase the precision of weights and activations from 3-bit to 8-bit in the first 80 epochs and adopt static 8-bit gradients , while the static precision baseline uses 8-bit for all the weights/activations/gradients . Results : Fig . 1 shows that training with progressive precision schedule achieves a slightly higher accuracy ( +0.3 % ) than its static counterpart , while the former can reduce training costs . Furthermore , we visualize the loss landscape ( following the method in ( Li et al. , 2018 ) ) in Fig . 2 ( b ) : interestingly the progressive precision schedule helps to converge to a better local minima with wider contours , indicating a lower generalization error ( Li et al. , 2018 ) over the static 8-bit baseline in Fig . 2 ( a ) . The progressive precision schedule in ( Fu et al. , 2020 ) relies on manual hyper-parameter tuning . As such , a natural following question would be : what kind of dynamic schedules would be effective while being simple to implement for different tasks/models ? In this work , we show that a simple cyclic schedule consistently benefits the training convergence while boosting the training efficiency . 3.2 CPT : THE KEY CONCEPT The key concept of CPT draws inspiration from ( Li et al. , 2019 ) which demonstrates that a large initial learning rate helps the model to learn more generalizable patterns . We thus hypothesize that a lower precision that leads to a shortterm poor accuracy might actually help the DNN exploration during training thanks to its associated larger quantization noise , while it is well known that a higher precision enables the learning of higher-complexity , fine-grained patterns that is critical to better convergence . Together , this combination could improve the achieved ac- curacy as it might better balance coarse-grained exploration and fine-grained optimization during DNN training , which leads to the idea of CPT . Specifically , as shown in Fig . 3 , CPT varies the precision cyclically between two bounds instead of fixing the precision during training , letting the models explore the optimization landscape with different granularities . While CPT can be implemented using different cyclic scheduling methods , here we present as an example an implementation of CPT in a cosine manner : B n t = ⌈Bnmin + 1 2 ( Bnmax −Bnmin ) ( 1 − cos ( t % Tn Tn π ) ) ⌋ ( 1 ) where Bnmin and B n max are the lower and upper precision bound , respectively , in the n-th cycle of precision schedule , ⌈⋅⌋ and % denote the rounding operation and the remainder operation , respectively , and Bnt is the precision at the t-th global epoch which falls into the n-th cycle with a cycle length of Tn . Note that the cycle length Tn is equal to the total number of training epochs divided by the total number of cycles denoted as N , where N is a hyper-parameter of CPT . For example , if N = 2 , then a DNN training with CPT will experience two cycles of cyclic precision schedule during training . As shown in Sec . 4.3 , we find that the benefits of CPT are maintained when adopting different total number of cyclic precision schedule cycles during training , i.e. , CPT is not sensitive to N . A visualization example for the precision schedule can be found in Appendix A. Additionally , we find that CPT is generally effective when using different dynamic precision schedule patterns ( i.e. , not necessarily the cosine schedule in Eq . ( 1 ) . We implement CPT following Eq . ( 1 ) in this work and discuss the potential variants in Sec . 4.3 . We visualize the training curve of CPT on ResNet-74 with CIFAR-100 in Fig . 1 and find that it achieves a 0.91 % higher accuracy paired with a 36.7 % reduction in the required training BitOPs ( bit operations ) , as compared to its static fixed precision counterpart . In addition , Fig . 2 ( c ) visualizes the corresponding loss landscape , showing the effectiveness of CPT , i.e. , such a simple and automated precision schedule leads to a better convergence with lower sharpness . 3.3 CPT : PRECISION RANGE TEST The concept of CPT is simple enough to be plugged into any model or task to boost the training efficiency . One remaining question is how to determine the precision bounds , i.e. , Bimin and B i max in Eq . ( 1 ) , which we find can be automatically decided in the first cycle ( i.e. , Ti = T0 ) of the precision schedule using a simple PRT at a negligible computational cost . Specifically , PRT starts from the lowest possible precision , e.g. , 2-bit , and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several consecutive iterations ; once this training accuracy difference is larger than a preset threshold , indicating that the training can at least partially converge , PRT would claim that the lower bound is identified . While the upper bound can be sim- ilarly determined , there exists an alternative which suggests simply adopting the precision of CPT ’ s static precision counterpart . The remaining cycles use the same precision bounds . Fig . 4 visualizes the PRT for ResNet-152/MobileNetV2 trained on CIFAR-100 . We can see that the lower precision bound identified when the model experiences a notable training accuracy improvement for ResNet-152 is 3-bit while that for MobileNetV2 is 4-bit , aligning with the common observation that ResNet-152 is more robust to quantization than the more compact model MobileNetV2 .

Summary: The authors propose cyclic precision training (CPT), a method to train integer-quantized neural networks with high precision while saving bit operations. CPT alternates the numerical precision of the network during training between low (2-3 bits) and high (the final desired precision, e.g. 8 bits). A total of 32 cycles of low to high are used, and this is only applied to the weights and activations, the backward pass is always in high precision. CPT is able to achieve improved accuracy across a variety of models on CIFAR-10/100 as well as transformers on PTB and WikiText. On ImageNet, CPT achieves accuracy on part with regular quantized training but saves bit ops.