CPT: Efficient Deep Neural Network Training via Cyclic Precision
Authors: Yonggan Fu
Han Guo
Meng Li
Xin Yang
Yining Ding
Vikas Chandra
Yingyan Lin
Venue: ICLR 2021 Spotlight (2021)
Keywords: Efficient training
low precision training

Abstract: Low-precision deep neural network (DNN) training has gained tremendous attention as reducing precision is one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper, we attempt to explore low-precision training from a new perspective as inspired by recent findings in understanding DNN training: we conjecture that DNNs' precision might have a similar effect as the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using a simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (1) converge to a wider minima with a lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training.

Published as a conference paper at ICLR 2021
CPT: EFFICIENT DEEP NEURAL NETWORK TRAINING VIA CYCLIC PRECISION
Yonggan Fu, Han Guo, Xin Yang, Yining Ding & Yingyan Lin
Department of Electrical and Computer Engineering
Rice University {yf22, hg31, xy33, yd31, yingyan. edu
Meng Li & Vikas Chandra Facebook Inc {meng. li, vchandra (ft.ndra)@bb.om
ABSTRACT
Low-precision deep neural network (DNN) training has gained tremendous atten- tion as reducing precision one of the most effective knobs for boosting DNNs' training time/energy efficiency. In this paper. attempt to explore low- sow-precision training from a new perspective as inspired by recent findings in understanding DNN training: conjecture that DNNs' precision might have similar effect the learning rate during DNN training, and advocate dynamic precision along the training trajectory for further boosting the time/energy efficiency of DNN training. Specifically, we propose Cyclic Precision Training (CPT) to cyclically vary the precision between two boundary values which can be identified using simple precision range test within the first few training epochs. Extensive simulations and ablation studies on five datasets and eleven models demonstrate that CPT's effectiveness is consistent across various models/tasks (including classification and language modeling). Furthermore, through experiments and visualization we show that CPT helps to (I) converge to wider minima with lower generalization error and (2) reduce training variance which we believe opens up a new design knob for simultaneously improving the optimization and efficiency of DNN training. Our codes are available at:
INTRODUCTION
The record- woord-breaking performance of modern deep neural networks (DNNs) comes at a prohibitive training cost due to the required massive training data and parameters, limiting the development of the highly demanded DNN-powered intelligent solutions for numerous applications (Liu et al.) 2018 Wu ct al. 2018). As an illustration, training ResNet-50 involves 10 FLOPs (flotting-poin) operations) and can take 14 days DNN training costs have raised increasing financial and environmental concerns. For example, estimated that training one DNN can cost more than $10K US dollars and emit carbon as high as a car's is lifetime emissions. In parallel, recent DNN advances have fueled a tremendous need for intelligent edge devices, many of which require on-device in-situ learning ensure the accuracy under dynamic real-world environments, where there is mismatch between the devices' limited
state-of-the-art (SOTA) GPU You al. 2020b). Meanwhile, the large
resources and the prohibitive training costs (Wang et 2019b Li et al. 2020 You et al.) 2020a).
To address the aforementioned challenges, extensive research efforts have been devoted to developing efficient DNN training techniques. Among them, low-precision training has gained significant attention as it can largely boost the training time/energy efficiency (Jacob et al.) 2018) Wang et al.) 2018a; Sun et al. 2019). For instance, GPUs can now perform mixed-precision DNN training with 16-bit IBEE Palf-Precision floating-point formats (Micikevicius al. 2017b). Despite their promise, existing low pww-iiiiioonn works have not yet fully explored the opportunity of leveraging recent findings in understanding DNN training. In particular, existing works mostly fix the model precision during the whole training process, i.c., adopt a static quantization strategy, while recent works in DNN training optimization suggest dynamic syper-pprameters along DNNs' training trajectory. For example, (Li

---

Published as a conference paper at ICLR 2021
al. 2019) shows that a large initial learning rate helps the model to memorize easier-to-Fe fit and more generalizable patterns, which aligns with the common practice to start from a large learning rate for exploration and anneal small one for final convergence; and (Smith 2017 Loshehilov & Hutter 2016) improve DNNs' classification accuracy by adopting cyclical learning rates.
In this work, advocate dynamic precision training, and make the following contributions:
We show that DNNs precision seems to have a similar effect as the learning rate during DNN training, i.e. low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence, and dynamic precision schedules help DNNs converge to a better minima. This finding opens up design knob for simultaneously improving the optimization and efficiency of DNN training.
We propose Cyclic Precision Training (CPT) which adopts cyclic precision schedule along DNNs training trajectory for pushing forward the achievable trade-offs between DNNs' accuracy and training efficiency. Furthermore, we show that the cyelic precision bounds can be automatically identified at the very early stage of training using a simple precision range test, which has negligible computational overhead.
Extensive experiments on five datasets and eleven models across wide spectrum of appli- cations (including classification and language modeling) validate the consistent effectiveness of the proposed CPT technique in boosting the training efficiency while leading to com- parable or even better accuracy. Furthermore, we provide loss surface visualization for better understanding CPT's effectiveness and discuss its connection with recent findings in understanding DNNs' training optimization.
2 RELATED WORKS
Quantized DNNs. DNN quantization (Courbariaux et al., 2015 2016 Rastegari et al. 2016 Zhu
ctt al., 2016 Li et al., 2016 Jacob et al. 2018 [Mshrr & Marr 2017 Mishra et al., 2017: Park
al. 2017| Zhou et al. 2016) has been well explored based on the target eccuracy-efficient
trade-
offs. For example, CJaob et al. 2018) proposes quantization-awwal aware training to preserve the post quantization accuracy: (Jng et al. |019] Bhalgat et al. 2020; Esser et al. 2019 Park Yoo, 2020) strive to improve low-precision DNNs accuracy using learnable quantizers. Mixed-precision DNN quantization (Wang et al.) 2019a Xu et al. 2018 Elthakeb al. 20200 Zhou al., 2017) assigns different bitwidths for different layers/filters. While these works all adopt a static quantization strategy, i.c., the assigned precision is fixed post quantization, CPT adopts a dynamic precision schedule during the training process.
Low-precision DNN training, Pioneering works (Wang et al. 2018a Banner et al. 2018 Micikevi cius et al. 7a Gupta al.) 2013 (un et al. (019) have shown that DNNs be traindd with reduced precision. For distributed learning, (Seide et al. 2014 De Sa et al. 2017 Wen et al. 2017 Bernstein et al.) 2018) quantize the gradients to reduce the communication costs, where the training computations still adopt full precision; For centralizee//on-dvvice learning, the weights, activations, gradients, and errors involved in both the forward and backward computations all adopt reduced precision. Our CPT can be applied on top of these low-precision training techniques, all of which adopt a static precision during the whole training trajectory, to further boost the training efficiency.
prrecison DNNs. There exist some dynamic precision works which aim to derive quantized DNN for inference after the full-precision training. Specifically, (Zhuang et al., 2018)) first trains a full-precision model to reach convergence and then gradually decreases the model precision to the target one for achieving better inference accuracy: (Khoram Lij (008) also starts from a full- full-precision model and then gradually learns the precision of cach layer to derive mixxd-precision counterpart; (Yang& & Jin 2020) learns a fractional precision of each layer/filter based on the linear interpolation of two consecutive bitwidths which doubles the computation and requires extra fine-tuning step; and (Shen et al.] (020) proposes to adapt the precision of each layer during inference in an input-dependent manner to balance computational cost and accuracy.
THE PROPOSED CPT TECHNIQUE
In this section, we first introduce the hypothesis that motivates us to develop CPT using visualization examples in Sec. S.I. and then present the CPT concept in 3.2. followee by the Precision Range Test (PRT) method in Sec, 3.3 where PRT aims to automate the precision schedule for CPT.
2

---

Published as a conference paper at ICLR 2021
Table 1: The test accuracy of ResNeet-38/4 trained on CIFAR-100 with different learning rate and precision combinations in the first stage. Note that the last two stages of all the experiments are trained with full precision and learning rate of 0.01 and 0.001, respectively.
Nes-38
ResNet-74
stage LR
0.1
0.06
0.03
0.01
0.1
0.06
0.03
0.01
4-bit Acc (%)
69.45
68.63
67.69
65.90
70.96
69.54
68.26
67.19
6-bit Acc (%)
70.22
68.87
67.15
66.10
71.62
70.28
68.84
66.16
8-bit Acc (%)
69.96
68.66
66.75
64,99
71.60
70.67
68.45
65.85
FP Acc (%)
70.45
69.53
67.47
64,50
71.66
70.00
68.69
65.62
3.1 CPT: MOTIVATION
Hypothesis 1: DNN's precision has similar effect the learning rate. Existing works (Grand valet et al. 1997 Neelakantan et al. 2013) show that noise can help DNN training theoretically or empirically, motivating us rethink the role of quantization in DNN training. We conjecture that low precision with large quantization noise helps DNN training exploration with an effect similar to a high learning rate, while high precision with more accurate updates aids model convergence, similar to low learning rate.
Validating Hypothesis 1. Settings: To empirically justify our hypothesis, we train ResNet-38/74
on the CIFAR- --0R dataset for 160 epochs following the basic training setting as in Sec. [.1 In particular, we divide the training of 160 epochs into three stages: [0-th, 80-th], (80-th,120-th) and (120-th, 160-th]: for the first training stage of [O-th, 80-th), we adopt different learning rates and precisions for the weights and activations, while using full precision for the remaining two stages with learning rate of 0.01 for the [80-th, 120-th] epochs and 0.001 for the [120-th, 160-th] epochs in all the experiments in order to explore the relationship between the learning rate and precision in the first training stage.
can observe that the learning rate is sufficiently reduced for the first training stage, adopting a Tower precision for this stage will lead to a higher accuracy than training with full precision. In particular, with the standard initial learning rate of 0.1, full precision training achieves a 1.00% 70% higher accuracy than the 4-bit one on ResNet-38/77 as the initial learning rate decreases, this accuracy gap gradually narrows and then reverses, e.g., when the initial learning rate becomes le-2, training with [0-th, 80-th] of 4-bit achieves a 1.40% higher accuracy than the full precision ones.
Results; As shown in Tab.
respectively; whereas
This set of experiments show that (1) when the initial learning rate is low, training with lower initial precisions consistently leads to better accuracy than training with full precision, indicating that lowering the precision introduces a similar effect of favoring exploration as that of high learning rate; and (2) although low precision can alleviate the accuracy drop caused by low learning rate, high learning rate is in general necessary to maximize the accuracy.
Insights:
Hypothesis 2: Dynamic precision helps DNN gen- eralization. Recent findings in DNN training have motivated to better utilize DNN precision to
Stali BAe
Progrestive 1909
Test Accuracy (%)
achieve a win-win in both DNN accuracy and ef- ficiency. Specifically, it has been discussed that (1)
30
DNNs learn to fit different patterns at different train- ing stages, e.g., (Rahaman et al. 2019 Xu al. 2019) reveal that DNN training first learns lower- frequency components and then fighffrequency fea- tures, with the former being more robust to pertur- bations and noises; and (2) dynamic learning rate schedules help to improve the optimization in DNN training, e.g., (Li et al. 2019) points out that a large initial learning rate helps the model to memorize easier-to-fit and more generalizable patterns while (Smith 2017 Loshchilov & Hutter 2016) show that cyclical learning rate schedules improve DNNs' classification accuracy. These works inspire us hypothesize that dynamic precision might help DNNs to reach a better optimum in the optimization landscape, especially considering the similar effect between the learning rate and precision validated in our Hypothesis 1.
20
40
600
20
300
3.20
140
1.5
Epochs
Figure 1: Test accuracy evolution of ResNet-
74 CIFAR-100 under different schedules.
3

---

Published as a conference paper at ICLR 2021
145
1.0
1.0
A36
130
328
0.2
335
-33
-22
+108
-10 -E
use
u.
100
-175 -290 .833
3m
6.1
123
-100
-0.30
-1.
-IM:
05a 0.1
Loss landscape Static
Loss landscape of Progressive 3-8
Loss landscape Cyclic 3-8 bit
(a)
(c)
Figure 2: Loss landscape visualization after convergence of ResNet-74 on CIFAR-100 trained with different precision schedules, where wider contours with larger intervals indicate better local minima and a lower generalization error as analyzed in (Li et al. 2018).
Validating Hypothesis 2. Our Hypothesis 2 has been consistently confirmed by various empirical observations. For example, a recent work (Fu et al. 2020) proposes to progressively increase the precision during the training process, and follow their settings to validate our hypothesis.
Settings: We train a ResNet-74 74 on CIFAR- 100 using the same training setting as (Wang et al. (2018))
except that we quantize the weights, activations, and gradients during training; for the progressive precision case uniformly increase the precision of weights and activations from 3-bit to 8-bit in the first 80 epochs and adopt static 8-bit gradients, while the static precision baseline uses 8-bit for all the brighhs/acciiaatios//rreeeee
Results: Fig. shows that training with progressive precision schedule achieves slightly higher accuracy (+0.3%) than static counterpart, while the former can reduce training costs. Furthermore, we visualize the loss landscape (following the method in (Li et al. 2018) in Fig. [b): interestingly the progressive precision schedule helps to converge to a better local minima with wider contours, indicating a lower generalization error (Li al. 2018 over the static 8-bit baseline in Tig.
The progressive precision schedule (Fu et al. 2020) relies on manual such, a natural following question would be: what kind of dynamic schedules would be effective while being simple to implement for different tasks/models? In this work, we show that a simple eyclic schedule consistently benefits the training convergence while boosting the training efficiency.
tuning. As
3.2 CPT: THE KEY CONCEPT
The key concept of CPT draws inspiration from eLi et al. (019)) which demonstrates that large initial learning rate helps the model to learn more generalizable patterns. We thus hypothe- size that lower precision that leads short- term poor accuracy might actually help the DNN exploration during training thanks to its associ- ated larger quantization noise, while it is well known that a higher precision enables the learn- ing of higher-complexity, fine-grained patterns that is critical to better convergence. Together, this combination could improve the achieved ac- curacy as it might better balance coarse-graine -arrine exploration and fine-grained optimization during DNN training, which leads to the idea of CPT. Specifically, as shown in Fig. 3 CPT varies the precision cyclically between two bounds instead of fixing the precision during training, letting the models explore the optimization landscape with different granularities.
WA-8/ GB-4
WA-3 GE-S Wi-L/ GE-S
Wik-d/ G2-
Static Precision Training
Cyclic Precision Training
WM.0/GE4
Weight Activation Gradient
E:
Figure 3: Static vs. Cyclic Precision Training (CPT), where CPT eyclically schedules the pre- cision of weights and activations during training.
While CPT can be implemented using different cyclic scheduling methods, here we present as an example an implementation of CPT a cosine manner:
B [Bi....
1
(B B cos-
% Tx T, m))
where Bht and B B are the lower and upper precision bound, respectively, in the n-th cycle of precision schedule, [] and % denote the rounding operation and the remainder operation, respectively,
4

---

Published as a conference paper at ICLR 2021
and B, is the precision the t-th global epoch which falls into the n-th cycle with a cycle length of Tn Note that the cycle length T, is equal to the total number of training epochs divided by the total number of cycles denoted N, where N is hyper-parameter of CPT. For example, if N 2, then a DNN training with CPT will experience two cycles of cyclic precision schedule during training. As shown in Sec. 4.3] we find that the benefits of CPT maintained when adopting different total number of cyclic precision schedule cycles during training, i.c., CPT is not sensitive to N. A visualization example for the precision schedule can be found in Appendix A Additionally, we find that CPT is generally effective when using different dynamic precision schedule patterns (i.e., not necessarily the cosine schedule Eq. We implement CPT following Eq. this work and
discuss the potential variants in Sec. 4.3
We visualize the training curve of CPT on ResNet-74 with CIFAR- 100 Fig. D and find that achieves a 0.91% % higher accuracy paired with 36.7% reduction in the required training BitOPs (bit operations), as compared to static fixed precision counterpart. In addition, Fig. (c) visualizes the corresponding loss landscape, showing the effectiveness of CPT, i.e., such simple and automated precision schedule leads to better convergence with lower sharpness.
3.3
CPT: PRECISION RANGE TEST
The concept of CPT is simple enough to be plugged into any model or task to boost the train- ing efficiency. One remaining question is how to determine the precision bounds, i.e., Byii and Bina in Eq. (1). which we find can be automat- ically decided in the first cycle (i.e., T; Th)
ResNee-152
Motileeee2
Training Accuracy (%)
6-b8 748
4HO 10-BIE
40-
of the precision schedule using a simple PRT at negligible computational cost. Specifically, PRT starts from the lowest possible precision, e.g., 2-bit, and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several con- secutive iterations; once this training accuracy difference is larger than preset threshold, in- dicating that the training can at least partially converge. PRT would claim that the lower bound is identified. While the upper bound can be sim- ilarly determined, there exists an alternative which suggests simply adopting the precision of CPT's static precision counterpart. The remaining cycles use the same precision bounds.
3-bit
4-bit
5-bit
10-
Iterations
Figure 4: Illustrating the precision range test for ResNet-152 and MobileNetV2 on CIFAR-100, where the switching point which exceeds the preset threshold is denoted by red circles.
Fig. 4 visualizes the PRT for ResNet- 152/MobileNeet22 trained on CIFAR-100. We can see that the lower precision bound identified when the model experiences a notable training accuracy improvement for ResNet-112 3-bit while that for MobileNerV2 is 4-bit, aligning with the common observation that 152 is more robust to quantization than the more compact model MobileNetV2.
EXPERIMENT RESULTS
In this section, we will first describe the experiment setup in Sec. 4.1 benchmarking results over SOTA training methods across various tasks in Sec. 4.2 and then comprehensive ablation studies of CPT in Sec.
4.1 EXPERIMENT SETUP
Models, datasets and baselines. We consider eleven models (including eight ResNet based mod- els (He et al. 2016), MobileNetV2 (Sandler et al., 2018), Transformer (Vaswani al. 2017). and LSTM (Hochreiter & Schmidhuber, 1997)) and five tasks (including CIFAR-10/100
(Krizhessky 2016), and Penn Tree- 2019b) for implement- ing MobileNetV2 on CIFAR- CIFAR-IO/100. Baselines: We first benchmark CPT over three SOTA static low-precision training techniques: SBM (Banner et al. 2018), DoReFa (Zhou et al.) 2016)), and WAGEUBN (Yang et al.] 22200). each of which adopts a different quantizer. Since SBM is the most competitive baseline among the three based both their reported and our experiment results,
ct al. 2009), ImageNet (Deng et al. 2009). WikiText- 103 (Merity et al.
bank (PTB) (Marcus et al.) 1993)) Specifically, we follow (Wang et al.
5

---

Published as a conference paper at ICLR 2021
apply CPT top of SBM, and all the static precision baselines adopt the SBM quantizer unless specifically stated. Another baseline is the cyclic learning rate (CLR) (Loshchilov & Hutter 2016) on top of static precision training, and we follow the best setting in (Loshchilov & Hutter 2016).
Training settings. We follow the standard training setting in all the experiments. In particular, for classification tasks, we follow SOTA settings in (Wang et al. (018b) for CIFAR-10/100 and (He et al. 2016) for ImageNet experiments, respectively: and for language modeling tasks, we follow (Baevski & Auli 2018 for Transformer on WikiText-103 and (Merity al., 2017 for LSTM on PTB.
Precision settings. The lower precision bounds in all the experiments are set using the PRT in Sec 3.3 and the upper bound is the same as the precision of the corresponding static precision baselines. We only apply CPT to the weights and activations (together annotated as FW) and use static precision for the errors and gradients (together annotated as BW), the latter of which is to ensure the stability of the gradients (Wang et al. 201 8a) (more discussion in Appendix C). In particular, CPT from 3-bit to 8-bit with 8-bit gradient annotated as FVB.88 The total number of periodic precision cycles, i.e. N in Sec 3.3 for all the experiments is fixed to be 32 (see the ablation studies in Sec. 4.3
Hardware settings and metrics. To validate the real hardware efficiency of the proposed CPT, we adopt standard FPGA implementation flows. Specifically, we employ the Vivado HIx design flow to implement FPGA-based accelerators on Xilinx development board called ZC706 (Xilinx). To better evaluate the training cost, we consider both calculated GBitOPs (Giga bit operations) and real-measured latency on the ZC706 FPGA board.
4.2 BENCHMARK WITH SOTA STATIC PRECISION TRAINING METHODS
Benchmark on CIEAR-10/100, Benchmark over SOTA quantizers: We benchmark CPT with three
SOTA static low-precision training methods, as summarized in Tab. 2 when training ResNet-74/1 64 and MobileNetV2 CIFAR-10/100, covering both deep and compact DNNs which are represen- tative difficult cases of oowprecisioo DNN training. Note that the accuracy improvement is the difference between CPT and the strongest baseline under the same setting. Tab. shows that (I) our CPT consistently achieves a win-win with both a higher accuracy (+0.19% ~ +1.25% and lower training cost (-21.0% ~ -37.1% computational cost and -1..7% -21.4% latency) under all the cases on 1FFAR-10/100 even for the compact model MobileNetV2, and (2) CPT notably outperforms the baselines in terms of accuracy under extremely low precision, which is one of the most useful scenarios for oowpprecisoon DNN training. In particular, CPT with periodic precision between 4-bit and 6-bit boosts the accuracy by + 1.25% and +1 +1.0% on ResNett74/164 respectively, as compared to the static 6-bit training CIFAR-10, 10, verifying that CPT leads better convergence.
CPT FW(334)) / BW6 Static FW4/ BW6
* CPT FW(3,6) / BW6
CPT FW(3,8) BW8
Static FW6/ BW6
Static FW8 BW8 Static FW8 / BW8 CLR
Static FW4/ BW6 + CLR
Static FW6/ BW6 + CLR
ResNet-38
ResNet-74
ResNet- 110
Test Accuracy (%)
Test Accuracy (%)
Test Accuracy (%)
-0.4% Aco. 36 67% Ops
x x
10.1%% Ace. 36 88%
2
1.73% Acc, 38,8% Ope
+0.52% Arc
10.35% Ace
+1
.3.04% Ops
-0.04% Ops
38.00% OPs
+0.35% Ace
+077% Ace
-0.22% Ace
65
GBitOPs
815
120
GBitOPs
100
170
2.00
2.5
230
2%
20
GBitOPs
ResNet-152
ResNet-164
MobileNetV2
Test Accuracy (%)
Test Accuracy (%)
Test Accuracy (%)
Aod 67%
+0 3887% O3s
+0.37%
+0.75%
Act
Ops
-32.04%
+0.77% Acc
+0.78% Ac:
GBRtOPs
GBItOPs
GBltOPs
Figure 5: Test accuracy vs. the required GBitOPs when training ResNet-38/74/ 1001522/11// and MobileNetV2 on CIFAR-100 using static precision, static precision plus CLR, and CPT methods.

---

Published as conference paper at ICLR 2021
Table 2: The test aceuracy, computational cost, and latency of CPT, DoReFa (Zou et al. 2016).
WAGEUBN (Yngg et al. 2020), and SBM (Banner et al. 2018) for training the ResNet- 74/1 64 and
MobileNetV2 models CIFAR-10/100.
Model
Method
Precision (FW/BW)
CIFAR-10 Ace (%)
CIFAR-100 Acc (%)
GBROPS
Latency (hour)
DoReFa WAGEUBN
2.67e8
8/8
91.35
69.61
2.67c88
44.6
SBM
92.57
71.44
2.67e8
RosNet-74
Proposed CPT
8-8/8
93.23
72.35
1,8c88
35.04
Improv.
+0.66
+0.91
-37.1%
-1..%
DoReFa
6/6
90,94
69.01
1,5088
31.2
WAGEUBN
6/6
91.01
69.37
[. 1.5088
33.2
SBM
6/6
91.15
70.31
1.50c8
33.2
ResNet-74
Proposed CPT
3-6/6
92.4
70.83
1.0588
27.5
Improv.
+1.25
+0.52
-30,0%
-17.2%
DoReFa
8/8
91,40
70,90
6.04c8
101.9
WAGEUBN
92.5
71.86
6.04c8
SBM
8/8
93.63
72.53
6.04c8
101.9
Proposed CPT
3-8/8
93.83
72.9
3.8e8
80.5
Improv.
+0.20
+0.37
-37.1%
-21.0%
DoReHa
6/6
91.13
70.53
3.40c8
76.7
WAGEUBN
92.44
3.40e8
SHM
6/6
91.95
70.34
3.40c8
76.7
ResNet-164
Pruposed CPT
3/6/6
93.02
71.79
2.37e8
Improv.
+1.07
+0.29
-30.3%
-1..2%
DoReFa
8/8
91.03
70.17
1.49c8
26.2
WAGEUBN
8/8
92.32
71.45
1.49c8
26.2
SBM
8/8
93.57
75.
1.49c8
26.2
MobileNetV2
Proposed CPT
4-8/8
93.76
75.65
L.4c8
21.6
Improv.
+0.19
+0.37
.30.%%
-17.6%
DoReFa
90.25
68.4
8.39e7
18.4
WAGEUBN
6/6
91,00
71.05
8.39c7
18.4
SBM
72.31
8.39c7
MobileNetV2
Proposed CPT
4-6/6
91.81
73.18
6.63e7
15.7
Improv.
+0.25
+0.87
-21.0%
-1477
Benchmark over CLR on top of SBM: We further benchmark CPT with CLR (Loshchlovv & Hutter 2016) (inherit its besting setting CIFAR-100), with both being applied top SBM it achieves the best performance among SOTA quantizers as shown in Tab. l based on more DNN models and precision shown in Fig. We with a better accuracy and efficiency trade-off, and (2) CLR on top of SBM leads to negative effect on the accuracy, which conjecture is caused by both the instability of gradients and the sensitivity of gradients to the learning rate under low- www-precision training, as discussed in (Wang et al. 2018a), showing that CPT is more applicable low-precision training than CLR.
that (1) CPT still consistently outperforms all the baselines
Loss landscape visualization: To better understand the superior performance achieved by CPT, we visualize the loss landscape following the method in (Li et al.) 2018). shown in Fig. covering both non-compact and compact models and two low-precision settings (i.e., 6-bit and 8-bit) which are bottlenecks for low-precision DNN training. We can observe that, again, both standard and compact DNNs trained with CPT experience wider contours with less sharpness, indicating that CPT helps DNN training optimization converge to better local optima.
Benchmark on ImageNet. To verify the scalability of CPT on more complex tasks and larger models, benchmark CPT with the SOTA static precision training method SBM (Banner et al.) 2018) ResNet- 18/34/50 with ImageNet, under which it is challenging for low-precision training such as 4-bit work. As shown in Tab. 3 can observe that CPT still achieves reduced computational cost (up to -3.44% with comparable accuracy (-0.20% +0.06%). In particular, CPT works well
400
430
ResNet-110 Static
ResNet-110 BWW
MobileNetV2 Cyclic FWN4.6) BWS
Figure 6: Loss landscape visualization of ResNet-1 and MobileNetV2 trained on CIFAR-100.
7

---

Published as a conference paper at ICLR 2021
Table 3: The accuracy and computational cost of ResNet-118/3450 on ImageNet, trained with the proposed CPT and SBM (Banner et al. 2018).
Model
Method
Precision (FW/BW)
Acc (%)
GBitOPs
Precision
Acc (%)
GBitOPs
SBM
8/8
2.86c9
1.6199
Proposed CPT
4-8/8
69.64
1.9999
4-6/6
69.33
1.27e9
ResNet-18
Improv.
+0.04
-304%%
+0,03
-21.1%
SBM
8/8
73.32
5.77c9
6/6
72.82
3.24c9
ResNet-34 Proposed CPT
8-8/8
73.12
4.03e7
6/6
72.74
2.57c9
Improv.
-0.20
-30.2%
-0.08
-20.7%
SBM
8/8
76.29
6.47c9
6/6
75.72
3.63e9
ResNet-50 Proposed CPT
4 8/8
76.35
4.51e9
4
6/6
75.74
2,87e9
Improv.
+0.06
-30.3%
+0.02
-20.9%
on ResNet-50, leading to both slightly higher accuracy and a better training efficiency, indicating the scalability CPT with model complexity, and thus, its potential application large scale training, in addition to on-device training scenarios.
CPT for boosting accuracy: An important perspec- tive CPT its potential to improve training opti- mality in addition to efficiency. We illustrate CPT's advantage in improving the final accuracy through training ResNet-118/3 on ImageNet using CPT and static full precision. As shown in Tab. 4 CPT on ResNet- 18/34 achieves a 0.91 19%%/0.8444 higher accu- racy than their full precision counterparts on Ima- geNet, indicating that CPT can be adopted as gen- eral technique to improve the final accuracy in addition to efficient training.
Table 4: The test accuracy of ResNet- 18/34
ImageNet: CPT (8-32) vs. full precision.
Network
Method
Precision
Acc (%)
Precisiin
32
69.76
ResNet-18
Proposed CPT Improv.
8-32
70.67
+0.91
Full Precision
32
73.30
ResNet-34
Proposed CPT
8-32
74.14
Improv.
+0.84
Benchmark on WikiText- 103 and PTB. We also apply CPT on language modeling tasks (including
WikiText- 103 and PTB) (see Tab. 5) to show that CPT is also applicable to natural language processing models. Tab. shows that (1) CPT again consistently achieves a win-win in terms accuracy (i.e.,
perplexity the lower the better) and training efficiency, and (2) language modeling models/tasks are more sensitive to quantization, especially in LSTM models, as it always adapts to larger lower precision bound, which is consistent with SOTA observations (Hou et al.) 2019).
Table 5: The test accuracy and computational cost of (1) Transformer on WikiText- 103 and (2) 2-LSTM (two-layer LSTM) on PTB, trained with CPT and SBM (Banner et al. 2018).
Method
Precision (FW/W))
Perplexity
GBROPS
Precision
Perplexity
GBROPSs
SBM
8/8
31.77
144c6
6/8
32.41
9.7c5
Transformer
Proposed CPT
4-8/8
30.22
10c6
4-6/8
7.66e5
WilTText-103
Improv.
.1.55
-30.2%
.1.41
-22.4%
96.95
4.03e3
2.77e3
2.LSTM PTB
Proposed CPT
8-8/8 8/8
96.39
3.09c3
5-6/8
97.0
2.48e3
Improv.
-23.2%
4.3
ABLATION STUDIES OF CPT
CPT with different precision ranges. We also evaluate CPT under wide range of upper precision bounds, which
correspond different target efficiency, to see if CPT still works well. Fig. 7 plots a boxplot with each exper-
Test Accuracy $
iment being repeated ten times. We can see that (1) gardless of the adopted precision ranges, CPT consistently achieves a win-win (a +0.74% +2.03 +2.03% higher accuracy and -1.33 -48.0% reduction in computational cost), especially under lower precision scenarios, and (2) CPT shrinks the accuracy variance, which better aligns with the practical goal of efficient training.
FRISROTA
Log (BiHOPSs)
Figure 7: Training ResNet- 74 CIFAR- 100 with CPT and its static counterpart.
CPT with different adopted numbers of precision schedule cycles. To evaluate CPT's sensitivity to the total number of adopted cyclic precision schedule cycles, we apply CPT on ResNet-38/74 and CIFAR- 100 when using different numbers of schedule cycles. Fig. plots the mean of the achieved
8

---

Published as a conference paper at ICLR 2021
ResNet-38 on CIFAR-100
ResNet-74 on CIFAR-100
703
Test Accuracy (%)
Test Accuracy (%)
30.00
20.93
2PM
Cycles
Cycles
Figure 8: The achieved test accuracy of CPT under different adopted numbers of precision schedule cycles, as compared to the static precision baseline SBM, when training ResNet-38/74 on CIFAR- 100.
accuracy based on ten repeated experiments. We can see that (1) different choices for the total number of cycles lead to comparable accuracy (within 0.5% accuracy) CIFAR- 10/1000000 and (2) CPT with different number of precision schedule cycles consistently outperforms the static baseline SBM the achieved accuracy. Based on this experiment, set N 32 for simplicity.
CPT with different eyclic precision schedule patterns. We also evaluate CPT using other cyclic precision schedule patterns in addition the cosine one, including triangular schedule motivated by (Smith) (017)) and a cosine annealing schedule as the learning rate schedule in @Loshchilov & d Hutter (216)) with all adopting cyclic cycles for fairness. Experiments Tab, show that (1) CPT with different schedule patterns is consistently effecttvv,, and (2) CPT with the other two schedule patterns even surpasses the cosine one in some cases but underperforms on compact models. We leave how to determine the optimal cyclic pattern for a given model and task as a future work.
2011)r
Table CPT with different precision schedules into cyclic precision training for ResNet-74/ 74/164 and MobileNetV2 on CIFAR-100. Cosine (CPT) is the current schedule adopted by CPT.
ResNet-74
ResNet-164
MobileNeeV2
Schedulc
FW(38)) BWS
FW(366) BW6
FWG.8)BBS
FW(3.6) BW6
PW(4.8/BWW
BW(
Cosine CPT)
72.35
70.83
72.7
71.77
75.65
73.18
Triangular
72.94
Casine anneal
21.37
70.80
72.4
72.03
74.59
73.12
DISCUSSIONS ABOUT FUTURE WORK
The theoretical perspective of CPT. There has been a growing interest understanding and optimiz- ing DNN training. For example, (Li et al. 2019) shows that training DNNs with large initial learning rate helps the model to memorize more generalizable patterns faster and better. Recently, (Zhu et al. reduced-precision DNN
2020) showed that under convexity assumption the convergence bound of reduced- precision training is determined by a linear combination of the quantization noise and learning rate. These findings regarding DNN training seem to be consistent with the effectiveness of our CPT.
The hardware support for CPT. Recent progresses in mixed-precision DNN accelerators (Lee et al., 2019; Kim al. 2020) with dedicated modules for supporting dynamic precisions are promising to support our CPT. We leave the optimal accelerator design for CPT as our future works.
CONCLUSION
We hypothesize that DNNs' precision has a similar effect as the learning rate when comes to DNN training, i.c., low precision with large quantization noise helps DNN training exploration while high precision with more accurate updates aids model convergence, and thus advocate that dynamic precision schedules help DNN training optimization. We then propose the CPT framework which adopts periodic precision schedule for low-precision DNN training in order to boost the achievable Pareto frontier of task accuracy and training efficiency. Extensive experiments and ablation studies verify that CPT can reduce computational cost during training while achieving a comparable or even better accuracy. Our future work will strive to identify more theoretical grounds for such dynamic low-precision training.
ACKNOWLEDGEMENT
The work is supported by the National Science Foundation (NSF) through the Real- Machine Learning (RTMI.) program (Award number: 1937592) and the NSF Award 180185 801865.

---

Published as a conference paper at ICLR 2021
REFERENCES
Alexei Baevski and Michael Auli. Adaptive input representations for neural language modeling. CoRR, abs/1809 1085,, 2018. URL.bttp://arxiv. wwtp:/wwwwwwww.///////////////// 10853
Ron Banner, Itay Hubara, Elad Hoffer, and Daniel Soudry. Scalable methods for 8-bit training of neural networks. Advances in neural information processing systems, pp- 5145-5153, 2018.
Jeremy Bernstein, Yu- Yu-Xiang Wang, Kamyar Arizzadenesheli, and Anima Anandkumar, signsgd: Compressed optimisation for non-convex problems. arXiv preprint arXiv: 2018.
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 696-697, 2020.
Matthieu Courbariaux, Yoshua Bengio, and networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
David. Binaryconnect: Training deep neural
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran EI- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint arXiv: orXiv:/602.000 010.6600222000 2016.
Christopher De Sa, Matthew Feldman, Christopher Ré, and Kunle Olukotun. Understanding and optimizing asynchronous cow-precision stochastic gradient descent. In Proceedings of the 44th Annual International Symposium on Computer Architecture, pp. 561-574, 2017.
J. Deng. W. Dong. R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR09, 2009.
Ahmed Taha Elthakeh, Prannoy Pilligundla, Fatemeh Mireshghallah, Amir Yazdanbakhsh, and Hadi Esmaeilzadeh. Releq: A reinforcement learning approach for automatic deep quantization of neural networks. IEEE Micro, 2020.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. arXiv preprint arXiv: /000:.990..000 08133, 2019.
Yonggan Fu, Haoran You, Yang Zhao, Yue Wang. Chaojian Li, Kailash Gopalakrishnan. Zhangyang Wang, and Yingyan Lin. Fractrain: Fractionally squeezing bit savings both temporally and spatially for efficient dnn training. arXiv preprint arXiv:2012.13113 2020.
Yves Grandvalet, Stéphane Canu, and Stéphane Boucheron. Noise injection: Theoretical prospects. Neural Computation, 10)3109331100 1997.
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learning with limited numerical precision. In International Conference on Machine Learning, pp. 1737-1746. 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016. URL
fb. resnet. taaeeetttoree torch
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Lu Hou, Jinhua Zhu, James Kwok, Fei Gao, Tao Qin, and Tie-yan Liu. Normalization helps training of quantized lstm. In Advances in Neural Information Processing Systems, pp. 7346-7356, 2019.
Jacob, Skinmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-aribbmetic-olly inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
10

---

Published as a conference paper at ICLR 2021
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Soroosh Khoram and Jing Li. Adaptive quantization of neural networks. In International Conference on Learning Reprexentations, 2018.
Chang Hyeon Kim, Jin Mook Lee, Sang Hoon Kang, Sang Yeob Kim, Dong Seok Im, and Hoi Jun Yoo. 1b-16b variable bit precision dnn processor for emotional hri system in mobile devices. Journal of Integrated Circuits and Systems, 6(3), 2020.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Jinsu Lee, Juhyoung Lee, Donghyeon Han, Jinmook Lee, Gwangtae Park, and Hoi-Jun Yoo. 7.7 Inpu: A 25.3 tflops/w sparse deep-neural-netwoor learning processor with fine-grained mixed precision of 1p8-ffl6. In 2019 IEEE International Solid-State Circuits ConSerence-((SSCC), pp. 142-144.
IEEE, 2019.
Chaojian Li, Tianlong Chen, Haoran You, Zhangyang Wang, and Yingyan Lin. HALO: Hardware- aware learning optimize. In The 16th European Conference Computer Vision (ECCV 2020), 2020.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks. arXiv preprint arXiv:160..044 1605.0471. 2016.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems, pp. 6389-6399, 2018.
Yuanzhi Li, Colin Wei, and Tengyu Ma. Towards explaining the regularization effect of initial large learning rate in training neural networks. In Advances in Neural Information Processing Systems, pp. 11674-11685, 2019.
Sicong Liu, Yingyan Lin, Zimu Zhou, Kaiming Nan, Hui Liu, and Junzhao Du. On-demand deep model compression for mobile devices: A usage-driven model selection framework. MobiSys '18, pp. 389-400. Association for Computing Machinery, 2018. ISBN 978 9781450357203
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03000 2016
Mitchell Mareus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models. arXiv preprint 1609.0:600.078800 2016.
Stephen Merity, Nitish Shirish Keskar, and Richard Socher. Regularizing and optimizing Istm language models. arXiv preprint arXiv: 32X8:07708.02/8888 2017
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv: 2017a.
Paulius Micikevicius. Sharan Narang, Jonah Alben, Gregory F. Diamos, Erich Elsen, David García, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, and Hao Wu. Mixed precision training. CoRR, abs/1710.037400, 2017b. URL /tRl:/tttpp///rxi.orr/////
03740
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low-precision network accuracy. arXiv preprint arXiv:1711.05852 2017.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpu: wide reduced-precision networks. arXiv preprint arXiv:1709.000 134, 2017.

---

Published as a conference paper at ICLR 2021
Arvind Neelakantan, Luke Vilnis, Quoe V Le, Ilya Sutskever, Lukasz Kaiser, Karol Kurach, and James Martens. Adding gradient noise improves learning for very deep networks. arXiv preprint arXiv:1511.06807, 2015.
Eunhyeok Park and Sungjoo Yoo. Profit: A novel training method for sub-4-bit mobilenet models. arXiv preprint arXiv:2008.04000 04693, 2020.
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-entropy-bass quantization for deep neu- ral networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5456-5464, 2017.
Nasim Rahaman, Aristide Baratin, Devansh Arpit, Felix Draxler, Min Lin, Fred Hamprecht, Yoshua Bengio, and Aaron Courville. On the spectral bias of neural networks. In Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 5301-5310. PMLR, 09-15 Jun 2019.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Frank Seide, Hao Fu, Jasha Droppo, Gang Li, and Dong Yu. 1-bit stochastic gradient descent and its application to data-parallel distributed training of speech dnns. In Fifteenth Annual Conference of the International Speech Communication Association, 2014.
Jianghao Shen, Yue Wang. Pengfei Xu, Yonggan Fu, Zhangyang Wang, and Yingyan Lin. Fractional skipping: Towards finer-grained dynamic cnn inference. 2020.
Leslic N Smith. Cyclical learning rates for training neural networks. In 2017 IEEE Winter Conference Applications of Computer Vision (WACV), pp.
IEEE, 2017.
Xiao Sun, Jungwook Choi, Chia- Yu Chen, Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8-bit floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems, pp. 4900-4909, 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 6008, 2017.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware- aware automated quantization with mixed precision. In Proceedings of the IEEE conference on computer vision and pattem recognition, pp. 8612-8620, 2019a.
Naigang Wang, Jungwook Choi, Daniel Brand, Chia- Yu Chen, and Kailash Gopalakrishnan, Training deep neural networks with 8-bit floating point numbers. In Advances in neural information processing systems, pp. 7675-7684, 201 8a.
Xin Wang, Fisher Yu, Zi-Yi Dou, Trevor Darrell, and Joseph E Gonzalez. Skipnet: routing in convolutional networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 409-424, 2018b.
Learning dynamic
Yue Wang, Ziyu Jiang, Xiaohan Chen, Pengfei Xu, Yang Zhao, Yingyan Lin, and Zhangyang Wang. E2-Train: Training ofate-of-the-art enns with over 80% energy savings. In Advances in Neural Information Processing Systems, pp. 5139-5151, 2019b.
Wei Wen, Cong Xu, Feng Yan, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Temgrad: Ternary gradients to reduce communication in distributed deep learning. In Advances in neural information processing systems, pp. 1509-1519, 2017.
12

---

Published as a conference paper at ICLR 2021
Junru Wu, Yue Wang, Zhenyu Wu, Zhangyang Wang, Ashok Veeraraghavan, and Yingyan Lin. Deep k-means: Re-training and parameter sharing with harder cluster assignments for compressing deep convolutions. arXiv preprint 1ox.:::00000022288, 2018.
evalauuttiiiooo Xilinx. Xilinx zynq soc zc evaluation kit. https://ww/ xiiiixxnix..... com/ com/products: (Accessed on 09/30/2020).
Yuhui Xu, Shuai Zhang, Yingyong Qi, Jiaxian Guo, Weiyao Lin, and Hongkai Xiong. Dnq: Dynamic network quantization. arXiv preprint arXiv:/812.02235,, 2018.
Zhi-Qin John Xu, Yaoyu Zhang, Tao Luo, Yanyang Xiao, and Zheng Ma. Frequency principle: Fourier analysis sheds light on deep neural networks. arXiv preprint 191.v:190. 2019.
Linjie Yang and Qing Jin. Fracbits: Mixed precision quantization via fractional bit-widths. arXiv preprint ar wtxiiv0007.00017. 2020.
Yukuan Yang, Lei Deng. Shuang Wu, Tianyi Yan, Yuan Xie, and Guoqi Li. Training hegh-performance and large-scale deep neural networks with full 8-bit integers. Neural Networks, 125:70-82, 2020.
Haoran You, Xiaohan Chen, Yongan Zhang, Chaojian Li, Sicheng Li, Zihao Liu, Zhangyang Wang, and Yingyan Lin. Shiftaddnet: A mardware-inspiee deep network, 2020a.
Haoran You, Chaojian Li, Pengfei Xu, Yonggan Fu, Yue Wang, Xiaohan Chen, Yingyan Lin, Zhangyang Wang, and Richard G. Baraniuk. Drawing early-bird tickets: Toward more efficient training of deep networks. In International Conference Learning Representations, 2020b. URL
htttps://openrviite www.eeneeiewwweeeee.... oww.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv:1606.06111 2016.
Yiren Zhou, Seyed-Mohser Moosavi-Dezfooli, Mgai-Man Cheung, and Pascal Frossard. Adaptive quantization for deep neural network. arXiv preprint 1rXiv:1712.01048, 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint arXiv: 16.2.0//22.44 2016.
Feng Zhu, Ruihao Gong, Fengwei Yu, Xianglong Liu, Yanfei Wang, Zhelong Li, Xiuqi Yang, and Junjie Yan. Towards unified int8 training for convolutional neural network. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 1969-1979, 2020.
Bohan Zhuang, Chunhua Shen, Mingkui Tan, Lingqiao Liu, and Ian Reid. Towards effective low- bitwidth convolutional neural networks. In Proceedings of the IEEE conference computer vision and patlern recognition, pp. 7920-7928, 2018.
A VISUALIZATION OF THE PRECISION SCHEDULE CPT
Fig. 9 visualizes the precision schedule FW(3,8) with eight cycles for training on CIFAR-10/100.
Table 7: The test accuracy of ResNet-38/74 on CIFAR-10 trained with CPT enabled at different epochs. Note that the lowest training precision is applied before enabling CPT in all the experiments.
Network
Starting Epoch
60
80
100
120
160 (No CPT)
FW(3,8) BW8
70.24
69.60
68.37
68.04
67.65
62.3
ResNet-38
FW(3,6) / BW6
69.30
68.65
68.00
67.83
67.85
62.3
FW(388) BW8
72.35
70.55
69.44
69.19
68.96
64.2
74
FW(3,6) / BW6
70.83
70.32
69.25
68.72
68.56
64.2
13

---

Published as a conference paper at ICLR 2021
Precision
10000
20000
30000
60000
50000
60000
Iterations
Figure Visualization of the precision schedule FW(3,8) with eight cycles, where the red line the cosine-manner schedule and the blue line is the adopted precision after rounding.
B ABLATION STUDY: THE STARTING EPOCH FOR ENABLING CPT
We conduct another ablation study to explore whether CPT always necessary during the whole training process. We start from training with the lowest precision in the precision range of CPT, and then enable CPT at different epochs to see the influence on the final accuracy. Specifically, we train ResNet-38/74 on CIFAR-100 for 160 epochs, considering two CPT precision ranges with different starting epochs to enable CPT as shown Tab. 7 We consistently observe that (I) an early starting epoch of CPT leads better accuracy all the considered precision ranges and networks, and (2) even enabling CPT at a later training stage still leads to a notable better accuracy than training without CPT.
C ABLATION STUDY: CPT ON TOP OF GRADIENTS
We decide not apply CPT on top of gradients since (1) the resulting instability of low precision gradients during train- ing (Wang al. 2018a) (2) generally the required precision of gradients is higher than that of weights and activations, so that the benefit of applying CPT on top of gradients in terms of efficiency is limited.
Table 8: Training ResNet-74 CIFAR-100 with static precision and CPT top of gradient only.
harm the final convergence, and
FW
BW
Accuracy (%)
6
70.31
To validate this, show the results of applying CPT top of gradients with fixed precision for weights and activations in
70.51
6
6-8
69.45
Tab.8 As expected, CPT on top gradients can hardly benefit either accuracy efficiency compared with its static preci- sion baseline. Therefore, we decide adopt fixed precision for gradients in all other experiments.
6
6-10
70.32
6-12
69.59
6-16
70.24
14


Reviews:
[{"review_id": "y1ns1iksnAN", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "A simple yet apparently effective technique. The paper is clear and well written.\n\nThe authors demonstrate that cyclically changing the precision of weights and activations during training leads to better results (both accuracy and training cost) than static quantisation methods. The method is simple and easy to implement . The  authors provide a method to automatically select the method's hyperparameters.\n\n1. This work appears to be an another example of multiprecision training (with a precision-switching mechanism): https://arxiv.org/abs/2006.09049\n\n2. The abstract claims that the variance of accuracy decreases when using CPT. This is briefly mentioned around Figure 7, but I would also expect to see these figures available for comparison in Tables 1-3, and some more discussion.\n\n3. The original hypothesis was that cyclic precision would have a similar effect to cyclic learning rate, but there's little discussion of this in the paper. There's one paragraph in Section 4.2 which compares CPT against CLR.  Could both techniques be applied together?\n\nMinor\n\n* Table 1: Apologies if I am missing something obvious, it's unclear to me what \"GBitOPs\" and \"latency\" refer to. It appears that two separate models have been trained, one for CIFAR-10 and one for CIFAR-100, but only one GBitOPs/latency figure is reported?\n\n* I find it quite hard to parse paragraph 2 of section 4.2 (Benchmark over CLR on top of SBM....)\n\n* Section 4.2, paragraph 4, \"staic\" should be \"static\"\n\n* There are several capitalisation issues in the references \n\nEDIT: Thanks to the authors' for their clear responses, I'm happy to raise my score to a 7.", "extra_scores": {}, "extra_text": {}}, {"review_id": "uIzXrOumyN_", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose cyclic precision training (CPT), a method to train integer-quantized neural networks with high precision while saving bit operations. CPT alternates the numerical precision of the network during training between low (2-3 bits) and high (the final desired precision, e.g. 8 bits). A total of 32 cycles of low to high are used, and this is only applied to the weights and activations, the backward pass is always in high precision. CPT is able to achieve improved accuracy across a variety of models on CIFAR-10/100 as well as transformers on PTB and WikiText. On ImageNet, CPT achieves accuracy on part with regular quantized training but saves bit ops.\n\nThe authors include a large number of other interesting data to show that CPT is insensitive to hyperparameters. Regardless of what the low precision setting, CPT(a,b) achieves better accuracy than b-bit quantized training on CIFAR-100. Varying the number of cycles between 8 and 128  affects the results by at most 0.5%. These experiments and the loss visualizations show that CPT is robust on CIFAR.\n\nThe major thing holding the paper back for me is that the lack of a good theoretical explanation for the CPT phenomenon. The authors draw a parallel between cyclic learning rates and CPT, but the paper itself shows that cyclic LR is not effective with quantization. I believe that lower accuracy means higher noise which resembles a higher LR, but clearly the two are not equivalent. One other idea is that lower accuracy may resemble lower batch size (higher SGD noise), but not sure what that would mean.\n\nIn general, the paper presents overwhelming evidence that CPT is a real phenomenon which may help train quantized DNNs. Even though the mechanism is currently not well understood, the paper deserves to be seen by others.\n\nQuestions:\n - I assume the CNNs are all trained with SGD. Was the transformer trained with SGD or ADAM or another optimizer? \n - What if you just trained in low precision until say, epoch 80, then use CPT? Does cyclic precision have to be used throughout training or just near the end?\n\nEDIT: I managed to set this up on a small transformer and was able to replicate the paper results. The transformer was trained with ADAM. Definite accept from me.", "extra_scores": {}, "extra_text": {}}, {"review_id": "HjAMWbR7Mt1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "### Overview\nIn this paper, the authors proposed Cyclic Precision Training (CPT) for low precision training. CPT varies the precision between two boundary values cyclically during training. It improves the accuracy, converges to a wider minima, reduces training variance, and reduces the training cost.\n\n### Clarity\nThe paper is well written and clearly organized. The figures and tables are informative and clearly organized.\n\n### Pros\n1. The paper provides an in-depth analysis of the advantages of varying precision. It shows the connection between precision and learning rate, and dynamic precision helps convergence to a better minima, verified by the visualization.\n2. The proposed CPT is effective for improving training convergence and reducing training cost. As shown in Figure 5, the accuracy v.s. training cost trade-off is better compared to static low-precision training. CPT even slightly improves the final accuracy.\n3. The author proposed an automatic method to determine the boundaries for cyclic precision at negligible overhead.\n4. The experiments are solid. The advantage of CPT is consistent over different models and tasks. Ablation study shows the robustness of CPT w.r.t. different hyper-parameters.\n\n### Cons\n1. The results of CPT is impressive. However, the actual application scenario could be quite restricted at this time. Most of the hardware does not support flexible bit-precision like 3-8 in this paper. Supporting multiple bit-precision will require extra compute units, leading to higher cost and energy consumption. It is unclear if the trade-off is worthwhile. Using FPGA can support flexible bit like in this paper, but using ASIC supporting int8 training (like GPU) might be more efficient compared to the improvement from CPT (I think generally FPGA is not more efficient than GPU for NN training). \n2. The precision range test is not explained clearly. Is the threshold consistent for all experiments? In figure 4, the training accuracy for the switching point on two curves are not the same. Does it indicate we need to manually tune the threshold for each experiment?\n3. The authors claim that CPT improves convergence for quantization training. Since the hypothesis is that the precision may work similarly to the learning rate, a baseline would be training with fixed precision (just like in this paper) while using cyclic *learning rate*.  Can CPT also outperform this baseline?\n\nEDIT: The response is very helpful and it addresses my concerns. I raised my score to 7 after reading the response.", "extra_scores": {}, "extra_text": {}}, {"review_id": "254P0jrcF-t", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors proposed an interesting low-precision training method using a dynamic precision schedule. Their proposed Cyclic Precision Training (CPT) cyclically varies the precision during the training and the boundary of precision values is determined by a precision range test (PRT). As shown in their empirical results, CPT largely reduces time/energy costs during training while maintaining comparable accuracy. While CPT does show a promising empirical performance, the motivation and reason behind the proposed method require further explanations. \nSpecifically,\n\n1. Motivation hypothesis in section 3.1 lacks explanation and justification. Authors point out that dynamic learning rate schedules help to improve the training and DNNs learn to fit different patterns at different training stages, but there is no strong connection to why dynamic learning rate could relate to dynamic precision schedule. \n\n2. Visualization examples aiming for supporting the hypothesis are not strong enough. The progressive precision case only achieves slightly higher accuracy and such benefit may be diminished given a large number of repeated experiments. Therefore I don't fully agree authors' statement \"progressive precision schedule helps to converge to better local minima with wider contours, indicating a lower generalization error\".\n\n3. Authors do point out that low-precision with large quantization noise may have a similar effect as a high learning rate with large stochastic noise, which helps the exploration over loss surface during DNN training. This hypothesis is interesting and it would be better to support the hypothesis with empirical validation and theoretical justification.\n\nIn addition, I think the proposed precision range test (PRT) method is unfair and requires modification.  PRT starts from the lowest possible precision and gradually increases the precision while monitoring the difference in the training accuracy magnitude averaged over several consecutive iterations. However, such a difference could also be affected by the training stages at different iterations. A better test should be evaluating all precision settings from the same starting point (e.g. training from scratch), and then measuring the differences after reaching the convergence stage.  \n\nThe paper is mostly well-written. Section 4.4 seems redundant and should be put before the experimental section. The connections are also required to be explained in detail.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All of the reviewers are impressed by this paper's empirical results and they agree that this is a good paper and should be accepted. Some questions about the theoretical justification of the proposed method and its potential practical impact remain open, but the empirical results are impressive and can result in more research in understanding Cyclic Precision Training (CPT) and improving quantized training of neural nets. I suggest acceptance as a spotlight presentation.
Decision: Accept (Spotlight)