CPT: Efficient Deep Neural Network Training via Cyclic Precision

1 INTRODUCTION . The record-breaking performance of modern deep neural networks ( DNNs ) comes at a prohibitive training cost due to the required massive training data and parameters , limiting the development of the highly demanded DNN-powered intelligent solutions for numerous applications ( Liu et al. , 2018 ; Wu et al. , 2018 ) . As an illustration , training ResNet-50 involves 1018 FLOPs ( floating-point operations ) and can take 14 days on one state-of-the-art ( SOTA ) GPU ( You et al. , 2020b ) . Meanwhile , the large DNN training costs have raised increasing financial and environmental concerns . For example , it is estimated that training one DNN can cost more than $ 10K US dollars and emit carbon as high as a car ’ s lifetime emissions . In parallel , recent DNN advances have fueled a tremendous need for intelligent edge devices , many of which require on-device in-situ learning to ensure the accuracy under dynamic real-world environments , where there is a mismatch between the devices ’ limited resources and the prohibitive training costs ( Wang et al. , 2019b ; Li et al. , 2020 ; You et al. , 2020a ) . To address the aforementioned challenges , extensive research efforts have been devoted to developing efficient DNN training techniques . Among them , low-precision training has gained significant attention as it can largely boost the training time/energy efficiency ( Jacob et al. , 2018 ; Wang et al. , 2018a ; Sun et al. , 2019 ) . For instance , GPUs can now perform mixed-precision DNN training with 16-bit IEEE Half-Precision floating-point formats ( Micikevicius et al. , 2017b ) . Despite their promise , existing low-precision works have not yet fully explored the opportunity of leveraging recent findings in understanding DNN training . In particular , existing works mostly fix the model precision during the whole training process , i.e. , adopt a static quantization strategy , while recent works in DNN training optimization suggest dynamic hyper-parameters along DNNs ’ training trajectory . For example , ( Li et al. , 2019 ) shows that a large initial learning rate helps the model to memorize easier-to-fit and more generalizable patterns , which aligns with the common practice to start from a large learning rate for exploration and anneal to a small one for final convergence ; and ( Smith , 2017 ; Loshchilov & Hutter , 2016 ) improve DNNs ’ classification accuracy by adopting cyclical learning rates . In this work , we advocate dynamic precision training , and make the following contributions : • We show that DNNs ’ precision seems to have a similar effect as the learning rate during DNN training , i.e. , low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence , and dynamic precision schedules help DNNs converge to a better minima . This finding opens up a design knob for simultaneously improving the optimization and efficiency of DNN training . • We propose Cyclic Precision Training ( CPT ) which adopts a cyclic precision schedule along DNNs ’ training trajectory for pushing forward the achievable trade-offs between DNNs ’ accuracy and training efficiency . Furthermore , we show that the cyclic precision bounds can be automatically identified at the very early stage of training using a simple precision range test , which has a negligible computational overhead . • Extensive experiments on five datasets and eleven models across a wide spectrum of applications ( including classification and language modeling ) validate the consistent effectiveness of the proposed CPT technique in boosting the training efficiency while leading to a comparable or even better accuracy . Furthermore , we provide loss surface visualization for better understanding CPT ’ s effectiveness and discuss its connection with recent findings in understanding DNNs ’ training optimization . 2 RELATED WORKS . Quantized DNNs . DNN quantization ( Courbariaux et al. , 2015 ; 2016 ; Rastegari et al. , 2016 ; Zhu et al. , 2016 ; Li et al. , 2016 ; Jacob et al. , 2018 ; Mishra & Marr , 2017 ; Mishra et al. , 2017 ; Park et al. , 2017 ; Zhou et al. , 2016 ) has been well explored based on the target accuracy-efficiency tradeoffs . For example , ( Jacob et al. , 2018 ) proposes quantization-aware training to preserve the post quantization accuracy ; ( Jung et al. , 2019 ; Bhalgat et al. , 2020 ; Esser et al. , 2019 ; Park & Yoo , 2020 ) strive to improve low-precision DNNs ’ accuracy using learnable quantizers . Mixed-precision DNN quantization ( Wang et al. , 2019a ; Xu et al. , 2018 ; Elthakeb et al. , 2020 ; Zhou et al. , 2017 ) assigns different bitwidths for different layers/filters . While these works all adopt a static quantization strategy , i.e. , the assigned precision is fixed post quantization , CPT adopts a dynamic precision schedule during the training process . Low-precision DNN training . Pioneering works ( Wang et al. , 2018a ; Banner et al. , 2018 ; Micikevicius et al. , 2017a ; Gupta et al. , 2015 ; Sun et al. , 2019 ) have shown that DNNs can be trained with reduced precision . For distributed learning , ( Seide et al. , 2014 ; De Sa et al. , 2017 ; Wen et al. , 2017 ; Bernstein et al. , 2018 ) quantize the gradients to reduce the communication costs , where the training computations still adopt full precision ; For centralized/on-device learning , the weights , activations , gradients , and errors involved in both the forward and backward computations all adopt reduced precision . Our CPT can be applied on top of these low-precision training techniques , all of which adopt a static precision during the whole training trajectory , to further boost the training efficiency . Dynamic-precision DNNs . There exist some dynamic precision works which aim to derive a quantized DNN for inference after the full-precision training . Specifically , ( Zhuang et al. , 2018 ) first trains a full-precision model to reach convergence and then gradually decreases the model precision to the target one for achieving better inference accuracy ; ( Khoram & Li , 2018 ) also starts from a full-precision model and then gradually learns the precision of each layer to derive a mixed-precision counterpart ; ( Yang & Jin , 2020 ) learns a fractional precision of each layer/filter based on the linear interpolation of two consecutive bitwidths which doubles the computation and requires an extra fine-tuning step ; and ( Shen et al. , 2020 ) proposes to adapt the precision of each layer during inference in an input-dependent manner to balance computational cost and accuracy . 3 THE PROPOSED CPT TECHNIQUE . In this section , we first introduce the hypothesis that motivates us to develop CPT using visualization examples in Sec . 3.1 , and then present the CPT concept in Sec . 3.2 followed by the Precision Range Test ( PRT ) method in Sec . 3.3 , where PRT aims to automate the precision schedule for CPT . 3.1 CPT : MOTIVATION . Hypothesis 1 : DNN ’ s precision has a similar effect as the learning rate . Existing works ( Grandvalet et al. , 1997 ; Neelakantan et al. , 2015 ) show that noise can help DNN training theoretically or empirically , motivating us to rethink the role of quantization in DNN training . We conjecture that low precision with large quantization noise helps DNN training exploration with an effect similar to a high learning rate , while high precision with more accurate updates aids model convergence , similar to a low learning rate . Validating Hypothesis 1 . Settings : To empirically justify our hypothesis , we train ResNet-38/74 on the CIFAR-100 dataset for 160 epochs following the basic training setting as in Sec . 4.1 . In particular , we divide the training of 160 epochs into three stages : [ 0-th , 80-th ] , [ 80-th,120-th ] , and [ 120-th , 160-th ] : for the first training stage of [ 0-th , 80-th ] , we adopt different learning rates and precisions for the weights and activations , while using full precision for the remaining two stages with a learning rate of 0.01 for the [ 80-th,120-th ] epochs and 0.001 for the [ 120-th , 160-th ] epochs in all the experiments in order to explore the relationship between the learning rate and precision in the first training stage . Results : As shown in Tab . 1 , we can observe that as the learning rate is sufficiently reduced for the first training stage , adopting a lower precision for this stage will lead to a higher accuracy than training with full precision . In particular , with the standard initial learning rate of 0.1 , full precision training achieves a 1.00 % /0.70 % higher accuracy than the 4-bit one on ResNet-38/74 , respectively ; whereas as the initial learning rate decreases , this accuracy gap gradually narrows and then reverses , e.g. , when the initial learning rate becomes 1e-2 , training with [ 0-th , 80-th ] of 4-bit achieves a 1.40 % /1.57 % higher accuracy than the full precision ones . Insights : This set of experiments show that ( 1 ) when the initial learning rate is low , training with lower initial precisions consistently leads to a better accuracy than training with full precision , indicating that lowering the precision introduces a similar effect of favoring exploration as that of a high learning rate ; and ( 2 ) although a low precision can alleviate the accuracy drop caused by a low learning rate , a high learning rate is in general necessary to maximize the accuracy . Hypothesis 2 : Dynamic precision helps DNN generalization . Recent findings in DNN training have motivated us to better utilize DNN precision to achieve a win-win in both DNN accuracy and efficiency . Specifically , it has been discussed that ( 1 ) DNNs learn to fit different patterns at different training stages , e.g. , ( Rahaman et al. , 2019 ; Xu et al. , 2019 ) reveal that DNN training first learns lowerfrequency components and then high-frequency features , with the former being more robust to perturbations and noises ; and ( 2 ) dynamic learning rate schedules help to improve the optimization in DNN training , e.g. , ( Li et al. , 2019 ) points out that a large initial learning rate helps the model to memorize easier-to-fit and more generalizable patterns while ( Smith , 2017 ; Loshchilov & Hutter , 2016 ) show that cyclical learning rate schedules improve DNNs ’ classification accuracy . These works inspire us to hypothesize that dynamic precision might help DNNs to reach a better optimum in the optimization landscape , especially considering the similar effect between the learning rate and precision validated in our Hypothesis 1 . Validating Hypothesis 2 . Our Hypothesis 2 has been consistently confirmed by various empirical observations . For example , a recent work ( Fu et al. , 2020 ) proposes to progressively increase the precision during the training process , and we follow their settings to validate our hypothesis . Settings : We train a ResNet-74 on CIFAR-100 using the same training setting as ( Wang et al. , 2018b ) except that we quantize the weights , activations , and gradients during training ; for the progressive precision case we uniformly increase the precision of weights and activations from 3-bit to 8-bit in the first 80 epochs and adopt static 8-bit gradients , while the static precision baseline uses 8-bit for all the weights/activations/gradients . Results : Fig . 1 shows that training with progressive precision schedule achieves a slightly higher accuracy ( +0.3 % ) than its static counterpart , while the former can reduce training costs . Furthermore , we visualize the loss landscape ( following the method in ( Li et al. , 2018 ) ) in Fig . 2 ( b ) : interestingly the progressive precision schedule helps to converge to a better local minima with wider contours , indicating a lower generalization error ( Li et al. , 2018 ) over the static 8-bit baseline in Fig . 2 ( a ) . The progressive precision schedule in ( Fu et al. , 2020 ) relies on manual hyper-parameter tuning . As such , a natural following question would be : what kind of dynamic schedules would be effective while being simple to implement for different tasks/models ? In this work , we show that a simple cyclic schedule consistently benefits the training convergence while boosting the training efficiency . 3.2 CPT : THE KEY CONCEPT The key concept of CPT draws inspiration from ( Li et al. , 2019 ) which demonstrates that a large initial learning rate helps the model to learn more generalizable patterns . We thus hypothesize that a lower precision that leads to a shortterm poor accuracy might actually help the DNN exploration during training thanks to its associated larger quantization noise , while it is well known that a higher precision enables the learning of higher-complexity , fine-grained patterns that is critical to better convergence . Together , this combination could improve the achieved ac- curacy as it might better balance coarse-grained exploration and fine-grained optimization during DNN training , which leads to the idea of CPT . Specifically , as shown in Fig . 3 , CPT varies the precision cyclically between two bounds instead of fixing the precision during training , letting the models explore the optimization landscape with different granularities . While CPT can be implemented using different cyclic scheduling methods , here we present as an example an implementation of CPT in a cosine manner : B n t = ⌈Bnmin + 1 2 ( Bnmax −Bnmin ) ( 1 − cos ( t % Tn Tn π ) ) ⌋ ( 1 ) where Bnmin and B n max are the lower and upper precision bound , respectively , in the n-th cycle of precision schedule , ⌈⋅⌋ and % denote the rounding operation and the remainder operation , respectively , and Bnt is the precision at the t-th global epoch which falls into the n-th cycle with a cycle length of Tn . Note that the cycle length Tn is equal to the total number of training epochs divided by the total number of cycles denoted as N , where N is a hyper-parameter of CPT . For example , if N = 2 , then a DNN training with CPT will experience two cycles of cyclic precision schedule during training . As shown in Sec . 4.3 , we find that the benefits of CPT are maintained when adopting different total number of cyclic precision schedule cycles during training , i.e. , CPT is not sensitive to N . A visualization example for the precision schedule can be found in Appendix A. Additionally , we find that CPT is generally effective when using different dynamic precision schedule patterns ( i.e. , not necessarily the cosine schedule in Eq . ( 1 ) . We implement CPT following Eq . ( 1 ) in this work and discuss the potential variants in Sec . 4.3 . We visualize the training curve of CPT on ResNet-74 with CIFAR-100 in Fig . 1 and find that it achieves a 0.91 % higher accuracy paired with a 36.7 % reduction in the required training BitOPs ( bit operations ) , as compared to its static fixed precision counterpart . In addition , Fig . 2 ( c ) visualizes the corresponding loss landscape , showing the effectiveness of CPT , i.e. , such a simple and automated precision schedule leads to a better convergence with lower sharpness . 3.3 CPT : PRECISION RANGE TEST The concept of CPT is simple enough to be plugged into any model or task to boost the training efficiency . One remaining question is how to determine the precision bounds , i.e. , Bimin and B i max in Eq . ( 1 ) , which we find can be automatically decided in the first cycle ( i.e. , Ti = T0 ) of the precision schedule using a simple PRT at a negligible computational cost . Specifically , PRT starts from the lowest possible precision , e.g. , 2-bit , and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several consecutive iterations ; once this training accuracy difference is larger than a preset threshold , indicating that the training can at least partially converge , PRT would claim that the lower bound is identified . While the upper bound can be sim- ilarly determined , there exists an alternative which suggests simply adopting the precision of CPT ’ s static precision counterpart . The remaining cycles use the same precision bounds . Fig . 4 visualizes the PRT for ResNet-152/MobileNetV2 trained on CIFAR-100 . We can see that the lower precision bound identified when the model experiences a notable training accuracy improvement for ResNet-152 is 3-bit while that for MobileNetV2 is 4-bit , aligning with the common observation that ResNet-152 is more robust to quantization than the more compact model MobileNetV2 .

Summary: The authors proposed an interesting low-precision training method using a dynamic precision schedule. Their proposed Cyclic Precision Training (CPT) cyclically varies the precision during the training and the boundary of precision values is determined by a precision range test (PRT). As shown in their empirical results, CPT largely reduces time/energy costs during training while maintaining comparable accuracy. While CPT does show a promising empirical performance, the motivation and reason behind the proposed method require further explanations.