LEARNED STEP SIZE QUANTIZATION
Authors: Steven K. Esser
Jeffrey L. McKinstry
Deepika Bablani
Rathinakumar Appuswamy
Dharmendra S. Modha
Venue:  (2020)
Keywords: deep learning
low precision
classification
quantization

Abstract: Deep networks run with low precision operations at inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- or 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce a novel means to estimate and scale the task loss gradient at each weight and activation layer's quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for a given system and requires only a simple modification of existing training code.

Published as a conference paper at ICLR 2020
LEARNED STEP SIZE QUANTIZATION
Steven K. Esser ;Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, Dharmendra S. Modha
IBM Research
San Jose, California, USA
ABSTRACT
Deep networks run with low precision operations inference time offer power and space advantages over high precision alternatives, but need to overcome the challenge of maintaining high accuracy as precision decreases. Here, we present a method for training such networks, Learned Step Size Quantization, that achieves the highest accuracy to date on the ImageNet dataset when using models, from a variety of architectures, with weights and activations quantized to 2-, 3- 4-bits of precision, and that can train 3-bit models that reach full precision baseline accuracy. Our approach builds upon existing methods for learning weights in quantized networks by improving how the quantizer itself is configured. Specifically, we introduce novel means to estimate and scale the task loss gradient each weight and activation layer's 'ss quantizer step size, such that it can be learned in conjunction with other network parameters. This approach works using different levels of precision as needed for given system and requires only simple modification of existing training code.
INTRODUCTION
Deep networks are emerging as components of number of revolutionary technologies, including image recognition (Krizhevsky et al. 2012). speech recognition (Hinton al. [012). and driving assistance (Xu et al. 2017). Unlocking the full promise of such applications requires system perspective where task performance, throughput, energy-e mnergy-ffficienyy, and compactness are all critical considerations to be optimized through co-design of algorithms and deployment hardware. Current research seeks to develop methods for creating deep networks that maintain high accuracy while reducing the precision needed to represent their activations and weights, thereby reducing the computation and memory required for their implementation. The advantages of using such algorithms to create networks for low precision hardware has been demonstrated in several deployed systems (Esser et al. 2016 Jouppi et al. 2017 Qiu et al. 2016).
It has been shown that low precision networks can be trained with stochastic gradient descent by updating high precision weights that are quantized, along with activations, for the forward and backward pass (Courbariaux et al.) [2015] Esser et al.. 2016). This quantization is defined by a mapping of real numbers to the set of discrete values supported by given low precision representation (often integers with 8-bits less). We would like mapping for each quantized layer that maximizes task performance, but it remains an open question how to optimally achieve this.
To date, most approaches for training low precision networks have employed uniform quantizers, which can be configured by single step size parameter (the width of quantization bin), though more complex nonuniform mappings have been considered (Polino et al. 2018). Early work with low precision deep networks used a simple fixed configuration for the quantizer (Hubara et al. 2016 Esser al., 2016)) while starting with Rastegari et al. (2016), later work focused on fitting
quantizer
to the data, either based statistics of the data distribution (Li & Liu 2016 Zhou et al. 2016 Cai
et al. 2017 McKinstry et al., 2018) or seeking to minimize quantization error during training (Choi
et al. 2018c Zhang al. 2018). Most recently, work has focused using backpropagation with
ccorresponding author sesser@us.ibm.cc

---

Published as a conference paper at ICLR 2020
Table 1: Comparison of low precision networks ImageNet. Techniques compared are QIL. Jung et al. 2018), FAQ (McKinstry et al. 2018). LQ-Nets (Zhang et al. 2018), PACT (Choi et al. 2018b 8b
Regularization (Choi al. 201 and NICE (Baskin al. 2018).
Top-1
Accuracy @ Precision
Top-5 Accuracy @ Precision
Network
Method
2
3
4
8
2
3
ResNet-18
Full precision:
70.5
Full precision:
LSQ (Ours)
67.6
70.2
71.1
71.1
87.6
89.4
90.0
90.1
QIL
65.7
69.2
70.1
FAQ
69.8
70.0
89.1
89.3
LQ-Nets
64.9
68.2
69.3
85.9
87.9
88.8
PACT
64.4
68.1
69.2
85.6
88.2
89.0
NICE
67.7
69.8
87.9
89.21
Regularization
61.7
67.3
68.1
84.4
87.9
88.2
ResNet-34
Full precision; 74.1
Full precision: 91.8
LSQ (Ours)
71.6
73.4
74.1
74.1
90.3
91.4
91.7
91.8
QIL
70.6
73.1
73.7
LQ-Nets
69.8
71.9
89.1
90.2
NICE
71.7
73.5
90.8
91.4
FAQ
73.3
73.7
91.3
91.6
ResNet-50
Full precision: 76.9
Full precision: 93.4
LSQ (Ours)
73.7
75.8
76.7
76.8
91.5
92.7
93.2
93.4
PACT
72.2
75.3
76.5
90.5
92.6
93.2
NICE
75.1
76.5
92.3
FAQ
76.3
76.5
92.9
93.1
LQ-Nets
71.5
74.2
75.1
90.3
91.6
92.4
ResNet-101
Full precision: 78.2
Full precision: 94.
LSQ (Ours)
76.1
77.5
78.3
78.1
92.8
93.6
94.0
94.0
RcsNet-152
Full precision: 78.9
Full precision: 94.3
LSQ (Ours)
76.9
78.2
78.5
78.5
93.2
93.9
94.1
94.2
FAQ
78.4
78.5
94.1
94.1
VGG-16bn
Full precision; 73.4
Full precision: 91.5
LSQ (Ours)
71.4
73.4
74.0
73.5
90.4
91.5
92.0
91.6
FAQ
73.9
73.7
91.7
Squeeze
Full precision: 67.3
Full precision: 87.8
Next-23-2x
LSQ (Ours)
53.3
63.7
67.4
67.0
77.5
85.4
87.8
87.7
stochastic gradient descent to learn quantizer that minimizes task loss (Zhu al. 2016 Mishra &
Marr 2017 Choi et al. 2018b a Jung et al. 2018 Baskin et al. 2018 Polino et al. (018).
settings place guarantees on optimizing network performance, and quantization error minimization schemes might perfectly minimize quantization error and yet still be non optimal if different quantization mapping actually minimizes task error. Learning the quantization mapping by seeking to minimize task loss is appealing as directly seeks improve the metric of interest. However, as the quantizer itself is discontinuous, such an approach requires approximating its gradient, which existing methods have done in a relatively coarse manner that ignore the impact of transitions between quantized states CCoi et al. 2018b Jung et al. 2018).
While attractive for their simplicity, fixed mapping schemes based
Here, we introduce a new way to learn the quantization mapping for each layer in a deep network, Learned Step Size Quantization (LSQ), that improves prior efforts with two key contributions. First, we provide simple way to approximate the gradient to the quantizer step size that sensitive to quantized state transitions, arguably providing for finer grained optimization when learning the step size as a model parameter. Second, we propose a simple heuristic to bring the magnitude step size updates into better balance with weight updates, which approach is usable for quantizing both activations and weights, and works with existing methods for backpropagation and stochastic gradient descent. Using LSQ train several network architectures
show improves convergence. The overall
2

---

Published as a conference paper at ICLR 2020
the ImageNet dataset, demonstrate significantly better accuracy than prior quantization approaches (Table ( and, for the first time that we are aware of, demonstrate the milestone 3-bit quantized networks reaching full precision network accuracy (Table 4).
2
METHODS
We consider deep networks that operate inference time using low precision integer operations for computations in convolution and fully connected layers, requiring quantization of the weights and activations these layers operate Given data quantize quantizer step size the number of positive and negative quantization levels Qr and QN. respectively, we define quantizer that computes quantized and integer scaled representation of the data, and é, quantized representation of the data at the same scale as v:
[clip(%/>. QN, Qr)},
(2)
Here, clip(z, 72) returns a with values below set to and values above set to and [>] rounds the nearest integer. Given encoding with b bits, for unsigned data (activations) QN and Qr 2 and for signed data (weights) QN = 2 and Qr 2
For inference, to and values can be used input low precision integer matrix multiplication units underlying convolution or fully connected layers, and the output of such layers then rescaled by step size using a relatively low cost high precision scalar-tensor multiplication, step that can potentially be algebraically merged with other operations such as batch normalization (Figure
Quantizer
Multioler
Layer
Quantizer
Low Precision
Figure 1: Computation of a low precision convolution or fully connected layer, as envisioned here.
2.1 STEP SIZE GRADIENT
LSQ provides means to learn s based on the training loss by introducing the following gradient through the quantizer to the step size parameter:
%/6 1%/1
if if-QN < <1/3 Qr
06
QN
if S -QN
(3)
Øs
Qr
if Qr
This gradient is derived by using the straight through estimator (Bengio et 2013) to approximate the gradient through the round function as pass through operation (though leaving the round itself in place for the purposes of differentiating down stream operations), and differentiating all other operations in Equations I and normally.
This gradient differs from related approximations (Figure 2 which instead either learn a transforma- tion of the data that occurs completely prior to the discretization itself (Jung et al. 2018) or estimate the gradient by removing the round operation from the forward equation, algebraically canceling terms, and then differentiating such that as 06/0. 0 where - QN '/8- Qp (Choi et al. 201 a). In both such previous approaches, the relative proximity of v to the transition point between quantized states does not impact the gradient to the quantization parameters. However,
reason that the
3

---

Published as a conference paper at ICLR 2020
A)
8)
LSQ
QIL
PACT
3R
1
Tranwsson
Themation
2
Thanattonn
4
Point
1
Poin
PoRrE
3
V
Figure 2: Given 1, QN 0. Qp 3, A) quantizer output and B) gradients of the quantizer output with respect to step size, for LSQ, or related parameter controlling the width of the quantized domain (equal s(Qp p + QN)) for QIL (Jung et al. 2018) and PACT (Choi al., 2018b). The gradient employed by LSQ is sensitive to the distance between v and each transition point, whereas the gradient employed by QIL (Jung et al., 2018) is sensitive only to the distance from quantizer clip points, and the gradient employed by PACT (Choi et al. (2018)) 8b) is zero everywhere below the clip point. Here, we demonstrate that networks trained with the LSQ gradient reach higher accuracy than those trained with the QIL or PACT gradients in prior work.
closer a given v is to a quantization transition point, the more likely it is to change its quantization bin (7) as a result of learned update to (since smaller change in is required), thereby resulting in large jump in 0. Thus, we would expect 06/as to increase as the distance from v to transition point decreases, and indeed this gradient naturally falls out of our simple quantizer formulation and use of the straight through estimator for the round function.
observe this relationship in the LSQ gradient. It is appealing that
For this work, each layer of weights and each layer activations has a distinct step size, represented as fp32 value, initialized 2(Iv1)//QP. computed on either the initial weights values the first batch of activations, respectively.
2.2 STEP SIZE GRADIENT SCALE
has been shown that good convergence is achieved during training where the ratio of average update magnitude to average parameter magnitude is approximately the same for all weight layers in a network (You et al., 2017). Once learning rate has been properly set, this helps to ensure that all updates are neither large to lead to repeated overshooting of local minima, nor small to lead to unnecessarily long convergence time. Extending this reasoning, we consider that each step size should also have its update magnitude to parameter magnitude proportioned similarly to that of weights. Thus, for network trained on some loss function L, the ratio
V.L.
/ V..T |||||
should average be near 1, where ||^|| denotes the 42-norm of However, expect the step size parameter to be smaller as precision increases (because the data is quantized more finely), and step size updates to be larger as the number of quantized items increases (because across when computing its gradient). To correct for this, we multiply the step size loss by gradient scale, g. where for weight step size g = 3/VNw QF and for activation step size g 1/VNFQr. where Nw is the number of weights in layer and N, is the number of features in layer. In section 3.4 we demonstrate that this improves trained accuracy, and we provide reasoning behind the specific scales chosen in the Section A of the Appendix.
items summed
2.3 TRAINING
Model quantizers are trained with LSQ by making their step sizes learnable parameters with loss gradient computed using the quantizer gradient described above, while other model parameters be trained using existing techniques. Here, we employ a common means of training quantized networks (Courbariaux et al. 2015), where full precision weights stored and updated, quantized weights
4

---

Published as a conference paper at ICLR 2020
and activations used for forward and backward passes, the gradient through the quantizer round function is computed using the straight through estimator (Bengio et al. 2013) such that
{1
if QN < <// < QP
(5)
otherwise,
and stochastic gradient descent is used to update parameters.
For simplicity during training, we use ê as input to matrix multiplication layers, which is algebraically equivalent to the previously described inference operations. We set input activations and weights to either 2-, 3-, 4-, 8-bit for all matrix multiplication layers except the first and last, which always 8-bit, as making the first and last layers high precision has become standard practice for quantized networks and demonstrated represented using fp32. All quantized networks are initialized using weights from a trained full precision model with equivalent architecture before fine-tuning in the quantized space, which is known to improve performance (Sung et al. 2015 Zhou et al. 2016 Mishra & Marr 2017 McKinstry ct al. 2018).
provide large benefit to performance. All other parameters are
Networks were trained with momentum of 0.9, using softmax cross entropy loss function, and cosine learning rate decay without restarts (Loshchilov & Hutter 2016). Under the assumption that the optimal solution for 8- 8-bit networks is close to the full precision solution (McKinstry et al. 2018) 8-bit networks were trained for I epoch while all other networks were trained for 90 epochs. The initial learning rate set to 0.1 for full precision networks, 0.01 for 2-, 3-, and 4-bit networks and to 0.001 for 8-bit networks. All experiments were conducted on the ImageNet dataset (Russakovsky al. 2015), using pre-activation ResNet (He al.) 2016). VGG (Simonyan & Zisserman 2014 4) with batch norm, or SqueezeNext (Gholami et al. 2018). All full precision networks were trained from scratch, except for VGG-16bn, for which we used the pretrained version available the Py PyTorch model zoo, Images were resized to 256 x 256, then 224 x 224 crop was selected for training, with horizontal mirroring applied half the time. test time, 224 224 centered crop was chosen. We implemented and tested LSQ in PyTorch.
3 RESULTS
3.1 WEIGHT DECAY
We expect that reducing model precision will reduce model's tendency to overfit, and thus also reduce the regularization in the form of weight decay necessary to achieve good performance. To investigate this, we performed a hyperparameter sweep on weight decay for ResNet-18 (Table and indeed found that lower precision networks reached higher accuracy with less weight decay. Performance was improved by reducing weight decay by half for the 3-bit network, and reducing by quarter for the 2- network. We used these weight decay values for all further experiments.
Table 2: ResNet- 18 top-1 accuracy for various weight decay values.
Weight Decay
2-bit
3-bit
4-bit
8-bit
10-4
66.9
70.1
71.0
71.1
x
67.3
70.2
70.9
71.1
0.25 x 10-4
67.6
70.0
70.9
71.0
0.125 x 10
67.4
66.9
70.8
71.0
3.2 COMPARISON WITH OTHER APPROACHES
We trained several networks using LSQ and compare accuracy with other quantized networks and full precision baselines (Table J]. To facilitate comparison, we only consider published models that quantize all convolution and fully connected layer weights and input activations to the specified precision, except for the first and last layers which may higher precision (as for the LSQ models). In some cases, we report slightly higher accuracy on full precision networks than in their original publications, which we attribute to our use of cosine learning rate decay (Loshchilov & Hutter 2016).
5

---

Published as a conference paper at ICLR 2020
We found that LSQ achieved higher top-1 accuracy than all previous reported approaches for 2-, 3- and 4- bit networks with the architectures considered here. For nearly all cases, LSQ also achieved the best-to-date top-5 accuracy these networks. In most cases, we found no accuracy advantage to increasing precision from 4-bit to 8-bit. It is worth noting that the next best low precision method (Jung et al. 2018) used progressive fine tuning (sequentially training a full precision to 5-bit model, then the 5-bit model to a 4-bit model, and so on), significantly increasing training time and complexity over our approach which fine tunes directly from a full precision model to the precision of interest.
these networks, and best-to-date accuracy on 8-bit versions of
is interesting to note that when comparing full precision a 2-bit precision model, top-1 accuracy drops only 2.9 for ResNet-18, 18, but 14.0 for SqueezeNextt2333 2x-2xx One interpretation of this is that the SqueezeNext architecture was designed maximize performance using as few parameters as possible, which may have placed it at a design point extremely sensitive to reductions in precision.
3.3 ACCURACY VS. MODEL SIZE
For a model size limited application, it is important to choose the highest performing model that fits within available memory limitations. To facilitate this choice, we plot here network accuracy against corresponding model size (Figure 3).
We can consider the frontier of best performance for a given model size of the architectures considered here. On this metric, we can see that 2-bit ResNet-34 and ResNet-50 networks offer an absolute advantage over using smaller network, but with higher precision. We can also note that at all precisions, 66G exists below this frontier, which is not surprising as this network was developed prior to number of recent innovations in achieving higher performance with fewer parameters.
ResNet-152
ResNet-101
75
ResNet-S0
ResNet-34
VGG-16bn
Top-1 Accuracy
70
ResNet-18
SqueazeNext-22
65
Full Precisaa Modill Sizes (MB)
Precision
60
Reshet- Rechet: 83
Restect 101: 170
55
VGG 18bn: 525
1
2
Model Size (MB)
Figure 3: Accuracy model size for the networks considered here show some 2-bit networks provide the highest accuracy at given model size. Full precision model sizes are inset for reference.
3.4 STEP SIZE GRADIENT SCALE IMPACT
To demonstrate the impact of the step size gradient scale (Section 2.2 we measured It (see Equation averaged across 500 iterations in the middle of the first training epoch for ResNet-18, using different step size gradient scales (the network itself was trained with the scaling as described in the methods avoid convergence problems). With no scaling, we found that relative to parameter size, updates to step size were 2 to 3 orders of magnitude larger than updates to weights, and this imbalance increased with precision, with the 8-bit network showing almost an order of magnitude greater imbalance than the 2-bit network (Figure 4 left). Adjusting for the number of weights per layer (g = 3/vNww). ), the imbalance between step size and weights largely went away, through the imbalance across precision remained (Figure 4 center). Adjusting for the number of number of weights per layer and precision (g 1//Nw Qr), this precision dependent imbalance was largely removed as well (Figure 4 right).
We considered network accuracy after training a 2-bit ResNet-18 using different step size gradient scales (Table 3). Using the network with the full gradient scale (g 1/VNwQ and g 1//NrQF

---

Published as a conference paper at ICLR 2020
10a
2-bit
3-blt
10a
4-bit
10
8-bit
10e
Step size grad scale
Figure 4: Relative parameter update magnitudes given different step size gradient scales. A gradient scale of 1/Nw Qr better balances relative step size and weight gradient magnitudes (right vs, left).
for weight and activation step size respectively) as baseline, we found that adjusting only for weight and feature count led to a 0.3 decrease in top-1 accuracy, and when no gradient scale was applied the network did not converge unless we dropped the initial learning rate. Dropping the initial learning achieved using gradient scale was 3.4 below
rate in multiples of ten, the best top-1 accuracy baseline, using an initial learning rate of 0.0001. Finally, we found that using the full gradient scaling with additional ten-fold increase decrease also reduced top-1 accuracy. Overall, this suggests benefit to our chosen heuristic for scaling the step size loss gradient.
Table 3: Top-1 accuracy for various gradient scale values and learning rates for 2-bit ResNet-
Gradient scale
Learning Rate
Accuracy
1/NQpp
0.01
67.6
1/V8
0.01
67.3
0.01
Did not converge
1
0,0001
64.2
10//NQF
0.01
67.4
1/1/NQF
0.01
67.3
3.5 COSINE LEARNING RATE DECAY IMPACT
We chose cosine learning rate decay in experiments as it removes the need to select learning rate schedule hyperparameters, is available in most training frameworks, and does not increase training time. To facilitate comparison with results in other publications that step-based learning rate decay, we trained 2-bit ResNet-18 8 model with LSQ for 90 epochs, using an initial learning rate of 0.01, which was multiplied by 0.1 every 20 epochs. This model reached top-1 accuracy of 67. 22, a reduction of 0.4 from the equivalent model trained with cosine learning rate decay, but still marking an improvement of 1.5 over the next best training method (see Table T).
3.6 QUANTIZATION ERROR
We next sought to understand whether LSQ learns a solution that minimizes quantization error (the distance between i and on some metric), despite such an objective not being explicitly encouraged. For this purpose, for given layer we define the final step size learned by LSQ & and let $ be the set of discrete values {0.013, 0.025, 20.003). For each layer, on a single batch of test data we computed the value of s E S that minimizes mean absolute error, <I(6(s) u)l}, mean square error, ((6(s) 2) and Kullback-Leille divergence, p(v) log p(v) - p(v) log ((((s) where p and q are probability distributions. For purposes of relative comparison, ignore the first term of Kullback- Leibler divergence, as it does not depend on 6, and approximate the second term as -E(log(g(((s))) where the expectation is over the sample distribution.
7

---

Published as a conference paper at ICLR 2020
For 2-bit 18 model found & = 0.949 = 0.206 for activations and 0.025 = 0.019 for weights (mean + standard deviation). The percent absolute difference between & and the value of s that minimizes quantization error, averaged across activation layers was 50% for mean absolute error, 63% for mean square error, and 64% for Kullback-Leiin layers, was 47% for mean absolute error, 28% for mean square error, and 46% for Kullback-L.eibler divergence. This indicates that LSQ learns solution that does not in fact minimize quantization error. As LSQ achieves better accuracy than approaches that directly seek minimize quantization error, this suggests that simply fitting quantizer to its corresponding data distribution may not be optimal for task performance.
divergence, and averaged across weight
3.7 IMPROVEMENT WITH KNOWLEDGE ANOWLEDGGE-DSTIILLAIIO DISTILLATION
To better understand how well low precision networks can reproduce full precision accuracy, we combined LSQ with same-architecturg knowledge distillation, which has been shown to improve low precision network training (Mishra Marr 2017). Specifically, we used the distillation loss function of Hinton et al. (2015) with temperature of 1 and equal weight given to the standard loss and the distillation loss (we found this gave comparable results to weighting the the distillation loss two trained
times more less than the standard loss 2-bit ResNet-18), The teacher network full precision model with frozen weights and of the same architecture as the low precision network trained. As shown in Table 4 this improved performance, with top-1 accuracy increasing by up to 1.1 (3-bit ResNet-50), and with 3-bit networks reaching the score of the full precision baseline (see Table I for comparison). As a control, also used this approach distill from the full precision teacher to full precision (initially untrained) student with the same architecture, which did not lead an improvement the student network accuracy beyond training the student alone. These results reinforce previous work showing that beoooedgeedistillateee can help low precision networks catch up to full precision performance (Mishra & Marr 2017)
Table 4: Accuracy for low precision networks trained with LSQ and knowledge distillation, which is improved over using LSQ alone, with 3-bit networks reaching the accuracy of full precision (32-bit) baselines (shown for comparison).
Top-1 Accuracy @ Precision
Top-5 Accuracy @ Precision
Network
2
3
4
8
32
2
3
8
32
R8
67.9
70.6
71.2
71.1
70.5
88.1
89.7
90.1
90.1
89.6
ResNet-34
72.4
74.3
74.8
74.1
74.1
90.8
91.8
92.1
91.7
9/.8
ResNet-50
74.6
76.9
77.6
76.8
76.9
92.1
93.4
93.7
93.3
93.4
4
CONCLUSIONS
The results presented here demonstrate that the ImageNet dataset across several network archi- tectures. LSQ exceeds the performance of all prior approaches for creating quantized networks. We found best performance when rescaling the quantizer step size loss gradient based on layer size and precision. Interestingly, LSQ does not appear to minimize quantization еитоr, whether measured using mean square error, mean absolute error, or kullback-Leible divergence. The approach itself simple, requiring only single additional parameter per weight or activation layer.
Although our goal to train low precision networks to achieve accuracy equal to their full precision counterparts, it is not yet clear whether this goal is achievable for 2-bit networks, which here reached accuracy several percent below their full precision counterparts. However, we found that such 2-bit solutions for state-of-the- aft networks model size, for example, with an 8MB model size limit, 2-bit ResNet-50 was better than a 4-bit ResNet-34 (Figure
useful in that they can give the best accuracy for the given
This work is a continuation of trend towards steadily reducing the number bits of precision necessary to achieve good performance across range of network architectures ImageNet. While unclear how far it can be taken, it is noteworthy that the trend towards higher performance at lower precision strengthens the analogy between artificial neural networks and biological neural networks,
8

---

Published as a conference paper at ICLR 2020
which themselves employ synapses represented by perhaps few bits of information (Bartol Jr et al. 2013) and single bit spikes that may be employed in small spatial and/or temporal ensembles to provide low bit width data representation. Analogies aside, reducing network precision while maintaining high accuracy is promising means of reducing model size and increasing throughput to provide performance advantages in real world deployed deep networks.
REFERENCES
Thomas M Bartol Jr, Cailey Bromer, Justin Kinney, Michael Chirillo, Jennifer N Bourne, Kristen M Harris, and Terrence J Sejnowski. Nanoconnectomic upper bound on the variability of synaptic plasticity, Elife. 1:770 2015.
Chaim Baskin, Natan Liss, Yoav Chai, Evgenii Zheltonozhskii, Eli Schwartz, Raja Girayes, Avi Mendelson, and Alexander M Bronstein. Nice: Noise injection and clamping estimation for neural network quantization. arXiv preprint arXiv:1810.0011 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 1300v:1308.30 3432, 2013.
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half-wave gaussian quantization. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition, pp. 1916-592,, 2017.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint htf.::v80700666 2018a.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint 1805i 2018b.
Yoojin Choi, Mostafa El-Khamy, and Jungwon Lee. Learning low precision deep neural networks through regularization. arXiv preprint arXiv:1809.0000 2018c.
Matthieu Courbariaux, Yoshua Bengio, and Sean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123-3131, 2015.
Steven K. Esser, Paul A. Merolla, John V. Arthur, Andrew S. Cassidy, Rathinakumar Appuswamy, Alexander Andreopoulos, David J. Berg, Jeffrey L. McKinstry, Timothy Melano, Davis R. Barch, Carmelo di Nolfo, Pallab Datta, Arnon Amir, Brian Taba, Myron D. Flickner, and Dharmendra S. Modha. Convolutional networks for fast, energy-efficient neuromorphic computing. Proceedings of the National Academy of Sciences, (1441):1144111111 2016.
Amir Gholami, Kiseok Kwon, Bichen Wu, Zizheng Tai, Xiangyu Yue, Peter Jin, Sicheng Zhao, and Kurt Keutzer. Squeezenext: Hardware-aware neural network design. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pp. 1638-1647, 2018.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Li Deng. Dong Yu, George E Dahl, Abdel-rahman Mohamed, Navdeep Jaitly, Andrew Senior, Vincent Vanhoucke, Patrick Nguyen, Tara N Sainath, et al. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups. IEEE Signal processing magazine, 29(6):82-97, 2012
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint 150.://53330331 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks. In Advances in neural information processing systems, pp. 4107-4115, 2016.

---

Published as a conference paper at ICLR 2020
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXivv15022.3316, 2015.
Norman P Jouppi. Cliff Young, Nishant Patil, David Patterson, Gaurav Agrawal, Raminder Bajwa, Sarah Bates, Suresh Bhatia, Nan Boden, Al Borchers, et al. In-datacenter performance analysis of tensor processing unit. In Computer Architecture (ISCA), 2017 ACM/IEEE 44th Annual International Symposium on, pp. 1-12. IEEE, 2017.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Youngjun Kwak, Jae-Joon Han, and Changkyu Choi. Joint training of low-precision neural network with quantization interval parame- ters. arXiv preprint arXiv:1808.05779 2018.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances neural information processing systems, pp. 1097-1105, 2012.
Li and B Liu. Ternryy weight networks.(2016), arXiv preprint 16Xv:1605..0771,, 2016.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 0608v:/608.0399 2016.
Jeffrey L McKinstry, Steven K Esser, Rathinakumar Appuswamy, Deepika Bablani, John V Arthur, Izzet B Yildiz, and Dharmendra Modha. Discovering low-precision networks close to full- precision networks for efficient embedded inference. arXiv preprini arXiv: (10.:0v88000449990 2018
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve
low-precision network accuracy. arXiv preprint arXiv:1711.05833 2017.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quantiza- tion. arXiv preprint arXiv: arXiv: 1802.0568, 2018.
Jiantao Qiu, Jie Wang. Song Yao. Kaiyuan Guo, Boxun Li, Erjin Zhou, Jincheng Yu, Tianqi Tang, Ningyi Xu, Sen Song, al. Going deeper with embedded fpga platform for convolutional neural network. In Proceedings of the 2016 ACM/SIGIA International Symposium on Field- Programmable Gate Arrays, pp. 26-35. ACM, 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115(3)22112255 2015.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Wonyong Sung, Sungho Shin, and Kyuyeon Hwang. Resiliency of deep neural networks under quantization. arXiv preprint arXiv: 15/ 06488, 2015.
Huazhe Xu, Yang Gao, Fisher Yu, and Trevor Darrell. End-to-end learning of driving models from large-scale video datasets. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2174-21111 2017.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 1708.
2017.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 166.06606606160, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally. Trained ternary quantization. arXiv preprint 11110.16/..01064, 2016.
10

---

Published as a conference paper at ICLR 2020
A STEP SIZE GRADIENT SCALE DERIVATION
We compute our gradient scale value by first estimating R (Equation 4 starting with the simple heuristic that for layer with Nw weights
/w/ll/ NwQr-
(6)
To develop this approximation, we first note that the expected value of an 12-norm should grow with the square root of the number of elements normalized. Next, we assume that where Qp = 1, step size should be approximately equal to average weight magnitude as split the weight distribution into zero and non zero values in a roughly balanced fashion. Finally, we assume that for larger Qp, step size should be roughly proportional to vT/Qp, so that as the number of available quantized states increases, data between the clip points will be quantized more precisely, and the clip points themselves (equal to sQN and sQp) will move further out to better encode outliers.
We also note that, in the expectation, LI and V,L are of approximately the same order. This can be shown by starting from the chain rule
Nw
aL 0w:
V.L=
Σ
Ow. Os
then assuming /0s is reasonably close to (see for example Figure 2 uncorrelated zero-centered random variables, compute the following expectation across weights:
and treating all 0L/aa; as
aL iu
[ LL R Nw x E
(8)
By assuming 06 06/oww 1 for most weights, we similarly approximate
V [| Nw x
E
(9)
Bringing all of this together, then estimate
NwQP.
(10)
Knowing this expected imbalance, compute our gradient scale factor for weights by simply taking the inverse R, so that is set to 1/VNw Qr.
As most activation layers are preceded by batch normalization (loffe & Szegedy 2015). and assuming updates to the learned batch normalization scaling parameter is the primary driver of changes to rre-quantization activations, imbalance between step size updates and update driven changes to activations that grows with the number of features in layer, Nr well Qp. Thus, for activation step size set g to 1//NeQP.
can similar approach to the above show that there is an
B IMPLEMENTATION
In this section we provide pseudocode to facilitate the implementation of L.Q.. We assume the use of automatic differentiation, as supported by a number of popular deep learning frameworks, where the desired operations for the training forward pass are coded, and the automatic differentiation engine computes the gradient through those operations in the backward pass.
Our approach requires two functions with non standard gradients, gradscale (Function I and roundpass (Function 2) We implement the custom gradients by assuming a function called detach that returns its input (unmodified) during the forward pass, and whose gradient during the backward pass is zero (thus detaching itself from the backward graph). This function is used in the form:
y = detach(1 - x2) x2.
(11)
so that in the forward pass, y Dy (as the Ty terms cancel out), while in the backward pass 01/02z, = (as detach blocks gradient propagation to xy and 0L/az, 0L/ay- We also assume a function n. feauures that given activation tensor, returns the number of features in that tensor, and
11

---

Published as a conference paper at ICLR 2020
nweights that given weight tensor, returns the number of weights in that tensor. Finally, the above are used to implement a function called quantize, which quantizes weights and activations prior to their use in each convolution fully connected layer.
The pscudocode provided here chosen for simplicity implementation and broad applicability to many training frameworks, though more compute and memory efficient approaches are possible. This example code assumes activations are unsigned, but could be modified to quantize signed activations.
Function gradscale(x, scale):
x: Input tensor
# scale: Scale gradient by this
yOut=
yGrad scale
y= detachhyOut yGrad) + yGrad # Return yOut in forward, pass gradient to yGrad in backward
return
Function roundpass(x):
x: Input tensor
yOut round(xx Round to nearest
yGra =x
= detach(yOut yGrrd) + yGrad # Return yOut in forward, pass gradient to yGrad in backward
return
Function quantize(y, s, isActivation):
Input tensor
s: Step size, a learnable parameter specific to weight or activation layer being quantized Quantization bits of precision
p:
isActivation: True v activation tensor,
False if weight tensor
Compute configuration values if isAtiivation:
0
Qp=2pp11
gradScaleFFattor sqr((nfeatures(() x Qp)
else: # is weights
Qn 1p-1-
Qp (2-11)
gradScaleFactor 1 sgr((nweighrs(v) Qp)
Quantize
gradscale(s, gradScaleFaator)
= v/s
clip(v. Qn, Qp)
vbar = roundpass(v)
vhat vbar s return vhat
12


Reviews:
[{"review_id": "rJx7-yzYqB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation of the paper is to be able to train low precision networks to a high-accuracy. Quantization is a useful tool in model compression, and doing it well for very low-precision models (2-3 bit precision specifically), is challenging.\n\nThe main contribution of the paper comes from:\na) Step Size Gradient: They propose a gradient which is sensitive to the distance between the value and the transition point. This is different from other methods which have gradients dependent only on the clip point.\nb) Step Size Gradient Scale: This is an interesting contribution, where they try to match the ratio of average update of the step size ‘s’ and average magnitude of ‘s’, with that of the network weights. This leads them to scale the gradient according to the precision and number of parameters. They demonstrate that this scaling actually helps improve the accuracy.\n\nThe results for 8-bit precision are not new. Several results (Quantization and Training of Neural Networks for Efficient\nInteger-Arithmetic-Only Inference, Jacob et al., Quantizing deep convolutional networks for\nefficient inference: A whitepaper, Krishnamoorthi et al.), show 8-bit quantization results where the accuracy matches floating point accuracy, and in some case exceeds it (low precision quantization acting as a regularizer). However, the results for lower precision are impressive.\n\nThere are a few questions:\n1. In sec 2.1, you mention that ‘each layer of weights and activations has a distinct step size, represented as an fp32 value, initialized to …’. Can you explain the intuition behind the initial value of the step size, and how is it a function of v?\n2. ‘Model Compression via Distillation and Quantization’ (Polino et al.) shows distillation actually helps significantly improve accuracy. I wonder if the authors have tried different weight combinations for the distillation loss, and using bigger models as teacher models.\n3. I would like to get more details of the inference setup, specifically the size and inference latency improvements over full-precision networks. The practical applicability of low-precision networks, specifically 2-bit and 3-bit networks, equally depends on the inference infrastructure, as it does on the training improvements.\n4. Have you evaluated your method for a non-Vision usecase?\n\nOverall this is a good work, I would tend towards accepting this.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "BygMO0Lhtr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper is concerned with neural networks using low-precision operations. This is an important research are both for applications (e.g., deploying neural networks on embedded systems or other constrained hardware, or increasing throughput on a production system) and for theoretical reasons (e.g., potentially modelling the behavior of biological neural networks).\n\nThe paper makes two primary contributions: (1) It proposes a new approximation for the gradient of the quantized data with respect to the step size parameter (equation 3) based on the well-known straight through estimator. (2) It proposes a heuristic for scaling gradient updates based on the number of weights in a given layer and the quantization levels (clipping thresholds). Experimental results showing higher accuracy than previously reported approaches in a number of settings and for common metrics, and best-to-date results for other settings.\n\nI recommend the paper for publication.\n\nIn terms of improving the paper further, the authors could expand their experiment section. E.g., how does their method compare to current SOTA knowledge-distillation methods; how does it deal with other architectures (recurrent models, attention, etc).\n\nMinor notes on the manuscript:\n\nIn section 3.4, both the term “population size” as well as the symbol N_W come without any definition. While closer reading makes it clear what is meant is the number of weights per layer, a definition closer to their first usage would be good. This goes in particular for N_W which comes in several versions (N_W, N, nweights).\n\nIn section 3.7, the sentence starting with “We found that using this approach to distill” does not seem to fit within its context for this reader, as it describes “full-to-full” distillation. Perhaps it is meant to read “low precision student network”?", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bke1XVDW9S", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper trains low-precision network with quantized weights and quantized activation. The main idea is to split the scale and quantized values. Both scales and weights are updated with backprop and SGD. The paper presents excellent experimental results on ImageNet. \n\nThe paper is generally well written and easy to follow. However, there does exist quite some grammar errors, especially in abstract, which could be improved. \n\nMoreover, I would like the authors to clarify some technical details. Are the scales s, so called step size in the paper, for every weight, every convolutional kernel, or very layer? How do you deal with BatchNorm?\n\nWhat is the main benefits of the proposed quantization method in general? Is it for fast inference, fast training, or just memory compression? Do the authors see the real benefits in practice besides claiming the accuracy does not drop?\n\nI would suggest the authors discuss and compare with XNOR network in detail. The proposed method looks similar. \n\nI am wondering how the baseline methods are tuned. There are quite a few “tricks” like learning rate scheduler and weight decay, which I do consider them as contributions of the paper. But would baseline methods also benefit from more hyper-parameter tuning?\n\nMinor issue, I donot get the explanation of eq (4), and it looks rather unnecessary. It sounds to me starting from a trained network and then train 90 epochs is a rather long time. Could the authors convince me this is a standard setting by providing some reference?\n\n\n================ after rebuttal=========================\nThank the authors for reply. My rating does not change. The proposed does look similar to XNOR, and the only difference seems to be how the scales are updated. Since there is only one scale per layer, I will be quite surprised if the proposed method can be much better than XNOR. Moreover, since BatchNorm is not quantized and it is everywhere in a ResNet-like architecture, it surprises me how much the scale helps. Finally, I am worried about practical benefits towards the authors' claim because the networks are not fully quantized. \n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Main content: Paper is about training low precision networks to a high-accuracy.

Discussion:
reviewer 2: impressive results, main questions are around some clarity in the experiments tried, but sounds like authors addressed most of this in rebuttal.
reviewer 1: well written paper, but authors think some technical details could be clarified. 
reviewer 3:  well written but experimental section could be improved.
Recommendation: all reviewers are in consensus, well written paper but some experiments/technical details could be improved. i vote poster.
Decision: Accept (Poster)