Rethinking Neural Network Quantization
Authors: Qing Jin
Linjie Yang
Zhenyu Liao
Venue:  (2020)
Keywords: Deep Learning
Convolutional Network
Network Quantization
Efficient Learning

Abstract: Quantization reduces computation costs of neural networks but suffers from performance degeneration. Is this accuracy drop due to the reduced capacity, or inefficient training during the quantization procedure? After looking into the gradient propagation process of neural networks by viewing the weights and intermediate activations as random variables, we discover two critical rules for efficient training. Recent quantization approaches violates the two rules and results in degenerated convergence. To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training. We also analyze the quantization error introduced in calculating the gradient in the popular parameterized clipping activation (PACT) technique. Through SAT together with gradient-calibrated PACT, quantized models obtain comparable or even better performance than their full-precision counterparts, achieving state-of-the-art accuracy with consistent improvement over previous quantization methods on a wide spectrum of models including MobileNet-V1/V2 and PreResNet-50.

Under review as a conference paper at ICLR 2020
RETHINKING NEURAL NETWORK QUANTIZATION
Anonymous authors
Paper under double-bling blind review
ABSTRACT
Quantization reduces computation costs of neural networks but suffers from per- formance degeneration. Is this accuracy drop due to the reduced capacity, or in- efficient training during the quantization procedure? After looking into the gra- dient propagation process of neural networks by viewing the weights and inter- mediate activations random variables, we discover two critical rules for effi- cient training. Recent quantization approaches violates the two rules and results in degenerated convergence. To deal with this problem, we propose a simple yet effective technique, named scale-adjusted training (SAT), to comply with the discovered rules and facilitates efficient training. We also analyze the quantiza- tion enror introduced in calculating the gradient in the popular parameterized clip- ping activation (PACT) technique. Through SAT together with gradient-calibrate PACT, quantized models obtain comparable even better performance than their full-precision counterparts, achieving state-of-the-art accuracy with consistent im- provement over previous quantization methods on wide spectrum of models in- cluding MobileNet-V1//2 and PreResNet-50.
INTRODUCTION
Deep neural networks have gained rapid progress tasks including computer vision, natural lan- guage processing and speech recognition (Voulodimos et al. (2088) Young al. (0088 Nassif al.
(2019 J. which have been applied to real world systems such as robotics and self-driving cars Li-
licrap et al. (2015): Janai et al. [017]] Recently, becomes a significant challenge deploy the platforms such as mobile phones and wearable devices. To make deep neural networks more efficient on model size, latency and energy, people have devel- oped several approaches such weight prunning (Han et al. (2015),, 1, model slimming (Liu et al. YYu (2018). and quantization [Courbariaux et al. 2015][2016 As one of the promis- ing methods, quantization provides the opportunity to embed bulky and computationnintensiie mod- els onto platforms with limited resources. By sacrificing the precision weights [Courbariaux et al.) Han et al. (2015): Li al. [2016] Zhu et al. (2016 Leng et al. (2018) and even fea- tures Courbariaux et al. [aste)aaasteaar et al. (2016); Zhou et al. (2016)
heavy deep models to resource-constraine
2015:
Park ct all (2017): al. 2018 Choi al.
Mellempudi et al. 22177 Mishra & Marr (2017p Mishra al.
2017
2018): Jacob et al. (2018) ZZang et al. 2018): : Zhuang et al.
2019 Gong et (211)). model
sizes can be shrunk large extent, and full-precision multiplication is replaced by low-precision
fixed-point multiplication, addition or even bitwise operations, requiring much reduced latency and energy consumption during inference.
The quantized models are either directly derived from full-precision counterparts Sheng et al. 2018 Krishnamoorthi (0188) 1. or through regularized training [Rastegari et al. (2016) Choi et al. 2018 Bai et al. (2018) Training-basee methods better exploit the domain information and nor- mally achieve better performance. However, the quantized models still suffer from significant ac- curacy reduction. To alleviate this problem, instead of manually designing the number of bits in different layers, recent works resort to automated search methods [Elhhaeeb et al, (0188) : Wu et al. 2018 Wang et al. (2019): Uhlich et al. (2019): Lou et al. [201 Despite all the effort im- prove quantized models, we notice that a fundamental problem of network quantization remains untouched, that is. whether accuracy reduction is due the reduced capacity of quantized mod- els, or due to the improper quantization procedure. the performance is indeed affected by the quantization procedure, is there way correct it and boost the performance?

---

Under review as a conference paper at ICLR 2020
ResNet-50 ImageNet (Both Quantization)
ResNet-50 on ImageNet (Weight-Only Quantization)
Top1 Valld Accuracy (8)
75
Top1 Valid Accuracy
75
74
73-
74-
PAST LOWet Destiiation
73
DeepCompresston
72-
Autoqu
LSO
70
HAQ
LON
SAT
69
SAT
2
3
Bltwidth
B Itwidth
Figure 1: Comparison of approaches with ResNet50 on ImageNet dataset under different quantiza- tion levels. Left: Quantization on both weight and activation. Right: Weight-only quantization. The bit-width represents equivalent computation cost for preccssoiiiionn methods (AutoQB and HAQ).
In this paper, we first investigate the condition of efficient training of a generic deep neural network by analyzing the variance of the gradient with respect to the effective weights involved in convo- lution or fully-connected operations. Here, effective weights denote the weights directly involved in convolution and other linear operations, which can be transformed weights in either quantized or unquantized training. With some semi-quantitative analysis (together with empirical verfication) of the scale of logit values and gradient flow in the neural network during training, we discover that proper convergence of the network should follow two specific rules of efficient training. It is then demonstrated that deviation from these rules leads to improper training and accuracy reduction, regardless of the quantization level of weights or activations.
To deal with this problem, simple yet effective technique named scale-adjusted training (SAT) is proposed, with which the rules of efficient training are maintained so that performance can be boosted. It is general approach which can be integrated with most existing quantization-awa training algorithms. We showcase the effectiveness of the SAT approach approach PACT [Choi et al. (011)). On the other hand, we also discover the quantization error intro- duced in calculating the gradient with respect to the trainable clipping level in the PACT technique, which also degenerates accuracy, especially for low precision models. schedule with calibrated gradient for PACT, namely CG-PACT, is proposed to correct the inaccurate gradient. With the pro- posed SAT and CG-PACT, we achieve haae-oof-the-an performance of quantized neural networks on wide range of models including MobileNet- 1/V2 and PreResNet-50 on ImageNet dataset, with consistent improvement over previous methods [an et al. (2015); Choi et al. (2018); Zhang et al.
recent quantization
2018: Zhuang et al. 2019: Wang et al. (2019 Lou al. (011)) Esser et al. [2019] under the same effciency constraints. As an example, Fig. I compares our method with some recent quan- tization techniques applied ResNet-50 on ImageNet classification task. We believe our analysis and approaches will shed light on some existing issues of network quantization, and facilitate more
advanced algorithms and applications.
2 RELATED WORK
Uniform-preeision quantization. Quantization of deep models has long been discussed since the
early work of weight binarization [Courbariaux et al. [2015] 2016)] and model compression [Han
al. J.15)J.The majority of previous methods enforce the same precision for weights/activatioon
in different layers during quantization. Early approaches focus on minimizing the difference in
values |Rastegari et al! [2016]] or distributions [Han al. (2015): Zhang et al. (2018) between quantized weights/activations and full-precision ones. Recently, /Zhang et al. (2018) proposes learning-based quantization method, where the quantizer is trained from data. Regularizer for quan-
tization is also proposed to implement binarized weights [Bai et al. (2018). Ensemble of multiple models with low precision has also been studied (Zhu et al. (2019) demonstrating improved per- formance than single model under the same computation budget. [Esser et al. [2019] proposes quantizer with trainable step size, and improves training convergence by balancing the magnitude of step size updates with weight updates, based on some heuristic analysis. However, this method
2

---

Under review as a conference paper at ICLR 2020
focuses training the step size, and scales the gradients, instead of analyzing the impact of model weights themselves on the training dynamics. Previous work have paid little attention to the impact of quantization procedure training dynamics, and it remains unclear if previous algorithms are efficient in training during quantization procedure.
Mixed-precision quantization. Recent work attempts to mixed-preed precision in one model, and weights and activations in different layers are assigned different bit-widths, resulting in better trade- off's between efficiency and accuracy of neural networks. Towards this end, automated algorithms are adopted to determine the most appropriate bit-width for each layer. Reinforcement learn- ing are adopted to search bit-width configurations with guidance from memory and computation cost [Elthakeb et al. (2018): : et al. [2019]] or latency and energy produced by hardware simula- tors Wang et al. (2019 ]. Wu et al. (2018) and [Uhlich et al. (019) apply differentiable neural architecture search method to efficiently explore the searching space. Although these methods result in more flexible quantization architectures, the training procedure of quantized models still follows
paradigms of mmiform-precisiio quantization.
Efficient Training. Training dynamics of neural networks has been extensively studied, both on the impact of initialization schemes [He et al. (013 and on the convergence condition with mean field theory [Poole et al. I. Early work mainly focuses on networks with simple structure, such as
with only fully-connected layers [Schoenholz ct al. 6 More advanced topics including resid-
ual networks and batch normalization are discussed with more powerful tools (ang & Schoenholz
2017 Yang et al. [2019]] facilitating efficient training of CNN with 10k layers through orthogonal
initialization of kernels Xiaoo et al. (2018)] From a different angle of view, [Zhang et al. (2019)
proposes properly rescaled initialization scheme to replace normalization layers, which also en- ables training of residual network with a huge number of layers. These approaches mainly focus on weight initialization methods to facilitate efficient training. We extend this analysis to the whole training process, and discovered some critical rules for efficient training. These rules guide our design of an improved algorithm for network quantization.
3 EFFICIENT TRAINING OF NETWORK QUANTIZATION
3.1 BASIC QUANTIZATION STRATEGY
Historically, quantization of neural networks follows different conventions and settings Krish- namoorthi [. Here we describe the convention adopted in this paper to avoid unnecessary ambiguity. We first train the full-precision model, which is used as a baseline for the final compar- ison. For quantized models, we use the pretrained full-precision model as initialization, and apply the same training hyperparameters and settings as full-precision model (including initial learning rate, learning rate scheduler, weight decay, the number of epochs, optimizer, batch size, etc.) fine- tune the quantized model. For the input image to the model, use unsigned Sbit integer (uint8), which means we do not apply standardization (neither demeaning nor normalization). Previous work sometimes avoid quantizing the first and last layers due accuracy drop (Zhou et al. (2016) Wu al. ). J. We follow a more practical setting to quantize weights in both layers with a minimum precision of 8bit [Choi al. [2018) in our main results. To investigate the effect of quantization level in these two layers, some additional results are shown in Appendix [i] The input to the last layer quantized with the same precision as other layers. As a widely adopted convention [Choi et al. 2018); Wang et al. (211)), J. bias in the last fully-connected layer(s) and the batch normalization (BN) layers (including weight, bias and the running statistics) are not quantized, but as shown in Appendix G batch normalization layers of most networks can be eliminated except for few cases. Note that no bias term is used in convolution layers.
3.2 RULES OF EFFICIENT TRAINING
Before detailed analysis, we first formulate the problem we will investigate. Following [He et al. 2013): Schoenholz et al. (016)), ], we will discuss a simplified network composed of repeated blocks, each consisting of a linear layer (either convolution
cully-connected) without bias, BN layer, an activation function and a pooling layer. The last layer is a single fully-con fully-connected layer without bias, which produces the logit values for the loss function.
3

---

Under review as a conference paper at ICLR 2020
Suppose for the I-th block, the input is z0), and the effective weight of the fully-connected layer is =(), which is called the effective weight because it is the de facto weight involved in forward and backward propagation of the network, In other words, we have
L 3
(la)
Vi
(1)
74
(1)
4
(lb)
a((+1)
h(y (1)
(1c)
((+1)
((+1)
(Id)
(2
PEP
where I = 1..... L, L is the depth of the network, and l stand for running statistics, and y and denote the trainable weights and bias in the BN layer, respectively. h(-) is the activation function, which is assumed to be quasi-linear, such as the ReLU. kppool is the kemel size of the pooling layer, which can be set for the usual case with no pooling, and P is the set of pooling indices. Note that for the last layer I = I,, only Eq. la applies, and 2L) are the logits for calculating the cross entropy loss.
Optimization of the cross entropy loss is highly related the scale of weights in the last fully- connected layer. Suppose the input to this layer is normalized with BN layer before pooling. To avoid saturation issue for calculating gradient, have the following rule of efficient optimization for the cross entropy loss.
Efficient Training Rule (ETR)) I) To prevent the logits from entering the saturation region of the cross entropy loss, the scale (measured by the variance) of the effective weight E(L) in the last fully-connected layer should be sufficiently small. Specifically, we have
Popooly (A:
V VARE
nL
where n is the length of input features of the fully-comnected layer, and kppoo is the kernel size of the preceding pooling layer (k-p..a = if the preceding layer not pooling). Empirically, we find ratio less than 0.1 is adequate.
Detailed derivation of this rule is given in Appendix A Besides optimization of the cross entropy operation, training dynamics is also critical for efficient training of deep neural network. To analyze the training dynamics, we derive the following phenomenological law for gradient flowing during training.
The Gradient Flowing Law For the aforementioned network, the variances of gradients of loss with respect to the effective weights in two adjacent layers related by
7o VAR (1+1)
VAIR
xi)
VAR(O_a-1)
(3)
(2pooly2
ny WARR
On the other hand, if there is no batch normalization layer, we have
VARIO - (0 1
(xpooly4 - VALIVARE "VAR 2 (
(4)
Here, both (1( and x2 are empirical parameters of order O(1) with respect to ny. L is the loss function, and ny a (kgonv)2 and ñu C++1 (Kponv)2 represent the number of input and output neurons of the (-th layer, respectively. q and c++1 denote the number of input and output channels, and kenny is the kernel size of this layer (which can be set to for fully-connected layer).
Note that similar results have been studied for network initialization in He et al. [2015) and exten- sively studied with mean field theory [Schoenholz et al. (016); Yang & Schocnholz (2017) Xiao cet al. ]. Here, we generalize the randomness assumption and empirically verify that the con- clusion holds along the whole training procedure (see Appendix A). Note that viewing weights and
4

---

Under review as a conference paper at ICLR 2020
imputs random variables along the whole training procedure is also adopted in previous literature of statistical mechanics approaches to analyze training dynamics of neural networks [Seung et al.
[9922]
Watkin et al. [9933;; Martin & Mahoney (2017
order when propagating from one layer to another. Suppose k.pool 1, that is, there is no pooling operation (since the number of pooling layers is much smaller than the number of linear layers, their effect can be ignored for simplicity). If we have BN after linear layer. then as long as the variance To avoid gradient exploding/vaniiee vanishing problems, the magnitudes of gradients should be on the same
of weights in different layers are kept on the same order, the scaling factor in Eq. ( will be (1). Note that we have implicitly assumed that ny and Ø1+1 do not differ in order. On the other hand, if
there is no BN layer directly following the linear layer, the scale of weights should be on the order of the reciprocal of the number of neurons to keep the scaling factor in Eq. (4 on the order of (1). Thus, we arrive at the basic rule for convergent training.
Efficient Training Rule II (ETR II) To keep the gradient of weights in the same scale across the whole network, either BN layers should be used after linear layers such as convolution and fully- connected layers, or the variance of the effective weights should be on the order of the reciprocal of the number of neurons of the linear layer (n4 or it).
Detailed derivation of the gradient flowing law and ETR II is given in Appendix A
With these rules, we are now ready to examine if training of quantized models is efficient. Following previous work PACT [Choi et al. 2018)) J, quantization, and the PACT technique for activation quantization. These methods are popular and typical for model quantization, and it is noteworthy that they are adopted only showcase the ef- fectiveness of our approach, instead of limiting our analysis. Violations of training rules also exists in other guantization approaches such BinaryNet [Courbariaux et [2015], XNORNet [Raste-] gari et al. (, Ternary Quantization [Zhu et al. (2016). HWGQ Caa et [217)]] as well as differentiable quantization approach Darts Quant [Uhlich et al. 2019], as weights distributions are
the DoReFa scheme [Zhou et al. 6 for weight
changed during quantiatiio//inarizaiion procedure which results in potential violation of Efficient
Training Rules. Thus we believe our approach general and can be used to boost the performance
of such quantization algorithms as well.
3.3 WEIGHT QUANTIZATION WITH SAT
The DoReFa scheme Zhou et al. [016 involves two steps, clamping and quantization. Clamping transforms the weights to values between 0 and 1, while quantization rounds the weights to integers. We here analyze the impact of both steps on training dynamics.
3.3.1 IMPACT OF CLAMPING
Before quantization, the weights are first clamped to the interval between 0 and 1. For weight matrix W, we first clamp it to
tanh(W (Wy)
WiJ = 2 max [tanh(W,.)) +1)
(5)
r.a
which is between 0 and 1. This transformation generally contracts the scale of large weights, and enlarges the difference of small scale elements. Thus, this clamping operation makes variables distributed more uniform in the interval [0, 1], which is beneficial for reducing quantization enror. However, the variance of the clamped weights will be different from the original ones, potentially violating the efficient training rules. Specifically, clamping leads to violation of ETR I in the last linear layer, and voilation of ETR II in linear layers without BN layers followed. For example, clamping on the full pre-activation ResNet [He et al. (016)) and VGGNet (Simonyan & Zisserman 2014 will both violate ETR II since they have linear layers without BN followed. In cases that convolution layers
followed by BN layers, the clamped weights are commensurate with each other and ETR II is preserved.
To understand the effect of clamping on ETR we first analyze model using clamped weights without quantization, following the DoReFa scheme
WB, 2W,
5

---

Under review as a conference paper at ICLR 2020
MobileNet-V2 ImageNet
(W( (W)
50
Vanilia (trait)
Vanilla (valld)
0.40
Clamp Only (train)
Clamp Only (valid)
Clamp and Ressale (train)
Clamp Rescale (vaild)
30
Error
0.35
10
0
Number of Neurons
Epach
(a)
(b)
Figure 2: Effect of weight clamping. (a) The ratio of variances with respect to the number of neurons. Note that the plot only a sampling result and different samples can give different results, but the order of magnitude remains meaningful. (b) Learning curves with different settings.
this case. Fig. 2a gives the ratio between variances of the clamped and the original weights with respect the number of neurons. As common practice (He ct al. 2013 }. the original weights W are sampled from a Gaussian distribution of zero mean and variance proportional to the reciprocal of the number of neurons. We find that for large neuron num- bers, the variance weights can be enlarged to tens of their original values, potentially increasing the opportunity of violating ETR I. Since the number of output neurons of the last linear layer is determined by the number of classes of data, we expect large dataset such as ImageNet (Deng et al. 2209 to be more vulnerable to this problem than small dataset such as CIFARIO (see Appendix for more details).
We have the effective weight E
To verify our analysis, we train the MobileNet-Va V2 ImageNet using traditional method well as using clamped weights, and compare their learning curves. As shown in Fig, 2b clamping impairs the training procedure significantly. reducing the final accuracy by as much as 1%. Also, we notice that clamping makes the model more prone to the over-fitting issue, which is consistent with previ- ous literature which claims that increasing weight variance neural networks might worsen their
generalization property [Seung et al. (1992 Watkin et al. (1993); Martin & Mahoney [017]]
To deal with this problem, we propose method named scale-adjusted training (SAT) to restore the variance of effective weights. We directly multiplies the normalized weight with the square root of the reciprocal of the number of neurons in the linear layer in Eq. 77) Here VAR (Wr]) is the sample variance of elements in the weight matrix, calculated by averaging the square of elements in the weight matrix. In back pook-prop,g,tion, VAR is viewed constant and receives gradient. This simple strategy is named constant rescaling and works well empirically across all of the ex- periments. Note that here we have ignored the difference between weight variances across channels and just use variance the weights in the whole layer for simplicity.
W
AVAR
W
Fig. 2b compares the learning curves of vanilla method, and weight clamping with and without constant rescaling. It shows that SAT recovers efficient training and improves validation accuracy significantly after weight clamping. We also experiment with an alternative rescaling approach in Appendix notice similar performance. In the following experiments we will always use con- stant rescaling. For MobileNet-V2, we only need to apply SAT to the last fully-connected layer. For other models where convolution is not directly followed by BN such as full pre-activation ResNet [He et al. [, we need to apply SAT to all such convolution layers (see Appendix B for more details). Note that the proposed method does not mean to be exclusive, and there are other methods to preserve the rules of efficient training. For example, different learning rates can be used for the layers with BN layer followed. Before further discussion, want emphasize that

---

Under review as a conference paper at ICLR 2020
the clamping is only preprocessing step for quantization and there is quantization operation involved up to now.
3.3.2 IMPACT OF WEIGHT QUANTIZATION
With weights clamped to [o, 1]. the DoReFa scheme with the following function
Zhou et al. [016]] further quantizes weights
(z)
ax
(8)
where b is the number of Here, L-1 indicates rounding the nearest integer, and equals 2 quantization bits. Quantized weights are given by
Q₁ 2gk (Wi))
In this case, have the effective weight given E Q. To the impact of quantization on the training dynamics, we compare the variance of the quantized weight Qi with the variance of the full-precision clamped weight Wey in Appendix B We find that for precision higher than 3 bits, quantized weights have nearly the same variance as the full-precision weights, indicating quantiza- tion itself introduces little impact on training. However, the discrepancy increases significantly for low precision such as or 2 bits, thus we should use the variance of quantized weights Qij for stan- dardization, rather than that of the clamped weight Wyy- For simplicity, we apply constant scaling to the quantized weights of linear layers without BN by
Qi
V Nout VAR[Qrs] Qi
(10)
For typical models such as MobileNets and ResNets, only the last fully-connected layer needs to be rescaled, and such rescaling is only necessary during training for better convergence. For in- ference, the scaling factor (which is positive) can be discarded, with the bias term being modified accordingly, introducing no additional operations. For models with several fully-connected layers such VGGNet [Simonyan & Zisserman (2014)], or with convolution layers not followed by BN layers, such as fully pre-activation ResNet, the scaling factors for these layers can be applied after computation inteiisve convolutions matrix multiplications, adding marginal computation cost.
3.4 ACTIVATION QUANTIZATION WITH CG-PACT
For activation quantization. [Choi et al. 2018) ] proposes method called parameterized clipping
activation (PACT), where trainable parameter a is introduced. An activation value x first clipped to the interval [0, a] with the hard tanh function, then scaled, quantized and rescaled to produce the quantized value q as
T x
|r al + a
(11a)
9 = aqi
(11b)
Quantized value obtained this way is called dynamic fixed-point number in Mellempudi et al. 2017 because a is floating number.
Since is trainable, need calculate gradient with respect it besides the activation itself. For this purpose, the straight through estimation (STE) method [Bengio et al. (2013) is usually adopted for backward propagation, which gives
(x) =
(12)
As (z(2), the domain of definition for (x) is also restricted to [0, 1]. Chain rule gives (see Ap- pendix D for more details)
0g
[k
x
a
(13)
aa 1
x > a
7

---

Under review as a conference paper at ICLR 2020
MobileNet-VV V2 (2bit) on ImageNet
MobileNet V2 (4bit) on ImageNet.
Gradient Caltralam (trai))
Gradient Colbrratinn bolld
0.35
Gredient Cellbration thrin)
Gradilent Calbrration (valid)
0.60
Cateration (val))
Gradiant (vaild)
0.53
0.45
20
100
120
Epoch
Figure 3: Effect of quantization error in caleulting gradient for PACT. Left: 2bit MobileNet- V2 ImageNet. Right: 4bit MobileNet-V2 on ImageNet.
Table 1: Comparison of quantization techniques with both weights and activation quantized.
MobileNet-V VI
MobileNet-Va V2
Quant. Method
Bit-widths
Acc.-1
Acc.-5
Acc.-1
Acc.-5
PACT
4bits
70.3
89.2
70.4
89.4
HAQ
flexible
67.40
87.90
66.99
87.33
SAT (Ours)
4bits
71.3
89.9
71.1
90.0
PACT
5bits
71.1
89.6
71.2
89.8
HAQ
flexible
70.58
89.77
70.90
89.91
SAT (Ours)
5bits
71.9
90.3
72.0
90.4
PACT
6bits
71.2
89.2
71.5
90.0
HAQ
flexible
71.20
90.19
71.89
90.36
SAT (Ours)
6bits
72.3
90.4
72.3
90.6
PACT
8bits
71.3
89.7
71.7
89.9
HAQ
flexible
70.82
89.85
71.81
90.25
SAT (Ours)
8bits
72.6
90.7
72.5
90.7
PACT
FP
72.1
90.2
72.
90.5
SAT (Ours)
FP
71.7
90.2
71.8
90.2
Note that give different result for the case of x in Eq. compared to the original PACT Choi et al. (2018), where quantization error is ignored and the derivative of is approxi- mated 0, i.e., qk
Such quantization eтrоr will introduce sigmificant difference for high precisions, but the enror introduced might be harmful for low precisions such as 4bit lower. Notably, a concurrent work [Jain et al. (2019)] presents similar results.
To see the effect of this quantization error, we train MobileNet-V V2 on ImageNet with low-precision of 2 and 4 bits, with the gradient calibrated as in Eq. 13 or not as original PACT CChoi et al. 2018 J. The learning curves are illustrated in Fig. 3 which show that calibration of PACT gradient facilitates efficient training for low precision models. This gradient calibration technique is named CG-PACT for brevity.
4 EXPERIMENTS
Based on previous analysis, we apply the SAT and CG-PACT technqiues to popular models includ- ing MobileNet-Vi V1, MobileNet-V V2, PreResNet-50 on ImageNet. For all experiments, we use cosine learing rate scheduler [Loshchilov & Hutter [2016]] without restart. Learning rate is initially set to 0.05 and updated every iteration for totally 150 epochs. We use SGD optimizer, Nesterov momen- tum with momentum weight of 0.9 without damping, and weight decay of 4 10 The batch size
8

---

Under review as a conference paper at ICLR 2020
Table Comparison of quantization techniques with only weights quantized.
MobileNet-V V1
MobileNet-We V2
Quant. Method
Weights
Acc.-1
Acc.-5
Acc.-1
Acc.-5
Deep Compression
2bits
37.62
64.31
58.07
81.24
HAQ
flexible
57.14
81.87
66.75
87.32
SAT (Ours)
2bits
66.3
86.8
66.8
87.2
Deep Compression
3bits
65.93
86.85
68.00
87.96
HAQ
flexible
67.66
88.21
70.90
89.76
SAT (Ours)
3bits
70.7
89.5
71.1
89.9
Deep Compression
4bits
71.14
89.84
71.24
89.93
HAQ
flexible
71.74
90.36
71.47
90.23
SAT (Ours)
4hits
72.1
90.2
72.1
90.6
Deep Compression
FP
70.90
89.90
71.87
90.32
HAQ
FP 70.90
89.90
71.87
90.32
SAT (Ours)
FP
71.7
90.2
71.8
90.2
Table 3: Comparison of quantization techniques on ResNet-50..
Both Quantization
Weight-Only Quantization
Quant. Method
Bit-widths
Acc.-1
Acc.-5
Quant. Method
Weights
Acc.-I
Acc.-5
PACT
2bits
72.2
90.5
DecpCompression
2bits
68.95
88.68
LQNet
2bits
71.5
90.3
LQNet
2bits
75.1
92.3
LSQ
2bits
73.7
91.5
HAQ
flexible
70.63
89.93
SAT (Ours)
2bits
74.1
91.7
SAT (Ours)
2bits
75.3
92.4
PACT
3bits
75.3
92.6
DeepCompression
3bits
75.10
92.33
LQNet
3bits
74.2
91.6
LQNet
3bits
NA
NA
LSQ
3bits
75.8
92.7
HAQ
flexible
75.30
92.45
SAT (Ours)
3bits
76.6
93.1
SAT (Ours)
3bils
76.3
93.0
PACT
4bits
76.5
93.2
DeepCompression
4bits
76.15
92.88
LQNet
4bits
75.1
92.4
LQNet
4bits
76.4
93.1
LSQ
4bits
76.7
93.2
HAQ
flexible
76.14
92.89
SAT (Ours)
4bits
76.9
93.3
SAT (Ours)
4bits
76.4
93.0
PACT
FP
76.9
93.1
DeepCompression
FP
76.15
92.86
LQNet
FP
76.4
93.2
LQNet
FP
76.4
93.2
LSQ
FP
76.9
93.4
HAQ
FP
76.15
92.86
SAT (Ours)
FP 75.9
92.5
SAT (Ours)
FP
75.9
92.5
PACT and SAT use full pre-activation ResNet, LSQ and HAQ use vanilla ResNet, and LQNet uses vanilla
ResNet without convolution operation shortcut (type-A shortcut).
PACT and LQNett use full- full-precision for the first and last layers, LSQ and SAT use Shit for both layers, and HAQ uses Shit for the first layer.
is set to 2048 for MobileNer- V1/V2 and 1024 for PreResNet-50, and we adopt the warmup strategy suggested in (Goyal et al. (017) by linearly increasing the learning rate every iteration larger value (batch size/256 256 x 0.05) for the first five epochs before using the cosine annealing scheduler. The input image is randomly cropped to 224 x 224 and randomly flipped horizontally, and kept as 8 bit unsigned integer with standardization applied. Note that clamped weight as initial points to finetune quantized models.
full-precision models with
We compare method with techniques in recent literature, including uniform- prrcciioprecision quan- tization algorithms such as DeepCompression [Han et al. [2015]] PACT [Chol et al. (2018). J, I. L.Net (Zhang et al. (2018). LSQ (Esser al. (2019 and mixed-precision approaches. such

---

Under review as a conference paper at ICLR 2020
HAQ Wang et al 2019 and AutoQB Lou ct 2019 Validation accuracy with respect to quantization levels for ResNet-50 is plotted in Fig. I It is obvious that our method gives significant and consistent improvement
previous methods under the same resource constraint. More thor- ough comparisons for quantization on MobileNets with or without quantized activation are given in Table I and 2 respectively. Table compares different quantization techniques on ResNet-50. Surprisingly, ly, the quantized models with our approach not only outperform all previous methods, in- cluding mixed-precision algorithms, but even outperform full-precision ones when the quantization is moderate ( 2 bits for both quantization and 4 bits for weight-only quantization on MobileNet- V1/V2, 23 bits for either both and weight-only quantization ResNet-50).
Our method reveals that previous quantization techniques indeed suffer from inefficient training. With proper adjustment abide by the efficient training rules, the quantized models achieve com- parable or even better performance than their full-precision counterparts. In this case, we have to rethink about the doctrine the model quantization literature that quantization itself hampers the capacity of the model. It seems with mild quantization, the generated models do not sacrifice in ca- pacity, but benefit from the quantization procedure. The claimping and rescaling technique does not contribute the gain in quantized models since they already used in full- precisiecinnn training. One potential reason is that quantization acts as a favorable regularization during training and help the model generalize better. Note that quantizing both weight and activation gives better results than weight-only quantization in some cases (3 and 4 bits on ResNet-50), which also indicates that the quantization the activations acts proper regularization and improves generalization capability of the network. The underlying mechanism is not clear yet. We left in-depth exploration as future work.
A Ablation Study To see the different impacts of the two methods we proposed (SAT and CG-PACT), we experiment MobileNet-V VI under two different quantization levels 4bit and Sbit with combi- nations of the two approaches. As shown in Table both techniques contribute to the performance, and SAT is more critical than CG-PACT both settings. In mild quantization such as bit, SAT itself is sufficient to achieve high performance, which matches our previous analysis.
Table 4: Ablation study of the SAT and CG-PACT on quantized MobileNet-Vi V1. Here CG denotes CG-PACT method.
4bits
8bits
Quant. Method
Acc.-1
Acc.-5
Acc.-1
Acc.-5
PACT
70.3
89.2
71.3
89.7
PACT+SAT
70.9
89.5
72.6
90.6
PACT+CG
70.4
89.3
71.0
89.5
PACT+SAT+CG
71.3
89.9
72.6
90.7
5 CONCLUSION
This paper studies efficient training quantized neural networks. By analyzing the optimization of cross entropy loss and training dynamics in neural network, we present law describing the gradient flow during training with qunitquantttative analysis and empirical verification. Based on this, we suggest two rules of efficient training, and propose scale-adjusted training technique to facilitate efficient training of quantized neural network. Our method yields state-of-the-ar perfor- mance quantized neural network. The efficient training rules not only applies to the quantization scenario, but can also be used as inspection tool in other deep learning tasks.
REFERENCES
Yu Bai, Yu-Xiang Wang, and Edo Liberty. Proxquant: Quantized neural networks via proximal operators. arXiv preprint arXiv:/810.00861. 2018.
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: /
308.340833333 3432, 2013.
10

---

Under review as a conference paper at ICLR 2020
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by hale-wave gaussian quantization. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5918-5998869 2017.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pieree I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv: 1805.06085, 2018.
Matthieu Courbariaux, Yoshua Bengio, and lean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances neural information processing systems, pp. 3123-3131, 2015.
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El-Yanix and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+ or-1. arXiv preprint drXiv:1602.02230, 2016.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 255. leee, 2009.
Ahmed T. Elthakeb, Prannoy Pilligundla, FatemehSadat Mireshghallah, Amir Yazdanbakhsh, Sicun Gao, and Hadi Esmaeilzadeh. Releq: an automatic reinforcement learning approach for deep quantization of neural networks. arXiv preprint 1rXiv:1811.01744 2018.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S. Modha. Learned step size quantization. arXiv preprint 1rXiv:1902.08/55 2019.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- precisseoo and low-bit neural networks. arXiv preprint arXiv: 190800330, 2019.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, An- drew Tulloch, Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet in hour. arXiv preprint arXiv: 1xXiv:1706.0200 1706.02677 2017.
Song Han, Huizi Mao, and William Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.00149, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient arigger-rittmmtic-onf. inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 77704-2713. 2018.
Sambhav R. Jain, Albert Gural, Michael Wu, and Chris Dick. Trained uniform quantization for accurate and efficient neural network inference on fixed-point hardware. arXiv preprint 155i::1903.00000 2019.
Joel Janai, Fatma Güney, Aseem Behl, and Andreas Geiger. Computer vision for autonomous vehi- cles: Problems, datasets and state-of-the-art. arXiv preprint 1rXiv:1704.05512 2017.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 180.0028006. 2018
Cong Leng, Zesheng Dou, Hao Li, Shenghuo Zhu, and Rong Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In Thirry-Sccond AAAI Conference on Artificial Intelligence, 2018.
11

---

Under review as a conference paper at ICLR 2020
Fengfu Li, Bo Zhang, and Bin Liu. Temary weight networks. arXiv preprint arXiv:1605.04711 2016.
Timothy Lillicrap, Jonathan Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. arXiv preprint arXiv:/509.0291 2015.
Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learn- ing efficient convolutional networks through network slimming. Proceedings of the IEEE International Conference on Computer Vision, 2017.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv: 030800383,, 2016.
Qian Lou, Lantao Liu, Minje Kim, and Lei Jiang. Autogb: Automl for network quantization and binarization mobile devices. arXiv preprint arXiv: 1902.05690, 2019.
Charles H Martin and Michael W Mahoney. Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior. arXiv preprint arXiv:1710.0955: 2017.
Naveen Mellempudi, Abhisek Kundu, Dheevatsa Mudigere, Dipankar Das, Bharat Kaul, and Pradeep Dubey. Ternary neural networks with fine-grained quantization. arXiv preprint arXiv: 170500170, 2017.
Asit Mishra and Debbie Marr. Apprentice: Using knowledge distillation techniques to improve low www-precision network accuracy. arXiv preprint arXiv:1711.0588 2017.
Asit Mishra, Eriko Nurvitadhi, Jeffrey Cook, and Debbie Marr. Wrpn: wide reduccnnnrecision networks. arXiv preprint UrXiv:1709.011 2017
Ali Bou Nassif, Ismail Shahin, Imtinan Attili, Mohammad Azzeh, and Khaled Shaalan. Speech recognition using deep neural networks: systematic review. IEEE Access, 7:19143-19165, 2019.
Eunhyeok Park, Junwhan Ahn, and Sungjoo Yoo. Weighted-entropy-bbere quantization for deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition, pp. 5456-5464, 2017.
Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponen- tial expressivity in deep neural networks through transient chaos. In Advances in neural informa- tion processing systems. pp. 3360-3368, 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European Conference on Computer Vision, pp. 525-542. Springer, 2016.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. arXiv preprint arXiv:1611.01233 2016.
Hyunjune Sebastian Seung, Haim Sompolinsky, and Naftali Tishby. Statistical mechanics of learn- ing from examples. Physical review A, 45(8):6056, 1992.
Tao Sheng, Chen Feng, Shaojie Zhuo, Xiaopeng Zhang, Liang Shen, and Mickey Aleksic. A ceattizzatio-frienniie separable convolution for mobilenets. In 2018 1st Workshop on Energy Efficient Machine Learning and Cognitive Computing for Embedded Applications (EMC2), pp. 14-18. IEEE, 2018
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.153: 2014.
Stefan Uhlich, Lukas Mauch, Kazuki Yoshiyama, Fabien Cardinaux, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Differentiable quantization deep neural networks. arXiv preprint arXiv: /905. 1452, 2019.
12

---

Under review as a conference paper at ICLR 2020
Nicolaas Godfried Van Kampen. Stochastic processes in physics and chemistry. Elsevier, 1992.
Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, and Eftychios Protopapadakis. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience, 2018.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8612-8620, 2019.
Timothy LH Watkin, Albrecht Rau, and Michael Bichl. The statistical mechanics of learning a rule. Reviews of Modern Physics, 65(2))499, 1993.
Bichen Wu, Yanghan Wang, Peizhao Zhang, Yuandong Tian, Peter Vajda, and Kurt Keutzer. Mixed precision quantization of convnets via differentiable neural architecture search. arXiv preprint arXiv: 2018.
Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel S Schoenholz, and Jeffrey Penning- ton. Dynamical isometry and mean field theory of cnns: How to train 10,000-layer vanilla convolutional neural networks. arXiv preprint (rxxiv:806.055300 05393, 2018.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao. Zhirong Wang, and Hong- bin Zha. Alternating multi-bit quantization for recurrent neural networks. arXiv preprint arXiv: 1802.00/50, 2018.
Ge Yang and Samuel Schoenholz. Mean field residual networks: On the edge of chaos. In Advances in neural information processing systems, pp. 7103-7114, 2017.
Greg Yang. Jeffrey Pennington, Vinay Rao, Jascha Sohl- Docl-Dickstten, and Samuel S Schoenholz. A mean field theory of batch normalization. arXiv preprint arXiv:/902.08/22 2019.
Tom Young. Devamanyu Hazarika, Soujanya Poria, and Erik Cambria. Recent trends in deep learn- ing based natural language processing [review article]. IEEE Computational Intelligence Maga- zine, 13:55-75, 08 2018. doi:
8.2840738.
Jiahui Yu, Linjie Yang. Ning Xu, Jianchao Yang, and Thomas Huang. Slimmable neural networks. arXiv preprint arXiv: arXiv:1812.0892. 2018
Dongging Zhang, Jiaolong Yang, Dongqiangzi Ye, and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 1901::9901.093300 2019.
Shu-Chang Zhou, Yu-Zhi Wang, He Wen, Qin-Yao He, and Yu-Heng Zou. Balanced quantization: An effective and efficient approach to quantized neural networks. Journal of Computer Science and Technology, 32(4):667-682, 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint a6Xiv:1606.0 06160, 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.
Shilin Zhu, Xin Dong, and Hao Su. Binary ensemble neural network: More bits per network or more networks per bit? In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4923-4933 2019.
Bohan Zhuang, Jing Liu, Mingkui Tan, Lingqiao Liu, Ian Reid, and Chunhua Shen. Effective train- ing of convolutional neural networks with low-bitwidth weights and activations. arXiv preprint arXiv:1908..4460 04680, 2019.
13

---

Under review as a conference paper at ICLR 2020
A SEMI-QUANTIAAIVE ANALYSIS AND EMPIRICAL VERIFICATION OF LOGIT SCALE AND GRADIENT FLOW
In this section, we first make some semi-quantitative analysis of the scale of logit values and the gradient flow to derive the laws and rules introduced in Section 3.2 After that, we present some experimental results verify the laws, together with some comments want to emphasize that our analysis is only semi-quantitative to give insights for practice, and is not guaranteed be rigorous. Complete and detailed analysis is far beyond the scope of this paper. In contrast, we will focus on order analysis of quantities, instead of accurate values, so we assume all variables are random unless specifically mentioned, and variables in the same tensor (weight matrix or activation matrix in layer) are distributed identically. We will also widely adopt the assumption of independence between different random variables (with different names, layer indices or component indices), as claimed explicitly in the Axiom 3.2 of [Yang & Schoenholz (2017)] This assumption is also adopted in other literature, such as [He et al. 22015]]Schochoo22 ct al. (016)]. It
our basic assumption. We
is not rigorously correct, but it leads to useful conclusion in practice [Xiao et al. (2018) In spite of this, we will give some comments on this assumption in the end of this section.
A.1 SEMI-QUANTITATIVE ANALYSIS
As mentioned in Section 3.2, we discuss simplified network composed of repeated blocks, each consisting of a linear (either convolution or fully-connected) layer without bias, BN layer, the activation function and pooling layer. The final stage is a single fully-connected layer without bias, which produces the logit values for the loss function. The pooling operation can be either average pooling or max pooling, but we only analyze the case of average pooling, and leave the analysis of max pooling as future work. We will first analyze the scale of logit values produced by the last fully-comnected layer, and then discuss the training dynamics of the network.
A.1.1 ANALYSIS OF THE LOGIT SCALE
The logit values produced by the final fully- layer is
= >=
(14)
J
where L is the depth of the network, ((.)) is the effective weight of this layer, and (x.)) is its input given by the previous block. The BN, nonlinear activation and pooling operations can be represented by
(L-1)
(1-1)
y1 %
(L-1)
(1-1) 2
(1-1)
14
+ 8 (L-1)
(15a)
0,
x1(4)
h(v! (11)
(15b)
5 (kpoo)) )2
(1)
L))
(15c)
PEP
From this, we have
VARI2 (1) 1
V (1
nz 1n
(16a)
=
E ((
(16b)
Nopool
P
=
E((E(()) )
(16c)
ponl
n4.
VAR J/ (1-1) 13
(16d)
pool
nz [
(kpoon) )2 (75 -11
(16e)
14

---

Under review as a conference paper at ICLR 2020
where Ry is the length of the imput features of this last fully- connected layer. For wide range of models including ResNets and MobileNets, Ry equals to the number of output channels of the last convolution layer. Note that have extensively used the assumption of independence, and rely the fact that the activation function h(.)) is quasi-linear.
Typically, the weight of the batch normalization layer y is the order of O(1), and thus have
VAR[2 1 m VARE
(17)
1.Pool
L-
To avoid output saturation, the softmax function for calculating the cross entropy loss has preferred range of the logit values for efficient optimization. To get more intuitive understanding of such preference, we plot the sigmoid function and its derivative in Fig. 4 as a simplification of softmax function. It is clear that for efficient optimization, the scale of the input æ should be well below 1, from which we can derive that
(A-p)))
VARE!
(18)
which is the Efficient Training Rule I.
For more intuitive understanding of the ETR I, here we analyze the impact of clamping operation for some typical models, based on the variance ratio given in Fig. 2a The results are illustrated in Table where the variance of the original weight (WR(W sj
is given by
VAR W
(19)
and we need to determine the value of Ky defined as
K0
nz V
(20)
sufficiently small in comparison with Note that W for vanilla models and E W for models with clamping. From the results we can see that models with clamping have much larger Ko, which results violation of the ETR I. This explains the degeneration of the learning curve for clamping in Fig. 2b
Table Impact of clamping on ResNet- 18/50 and MobileNet V1/V2 with ImageNet.
model
ResNet- 18
ResNer-50
MobileNet-Vi
MobileNet-V2
nL
512
512x4
1024
1280
7r n1+1
1000
1000
1000
1000
7
7
7
7
Kg
yanilla
0.01
0.04
0.02
0.026
.elmpp no
0.5
2
1.3
15

---

Under review as a conference paper at ICLR 2020
600
0.8
0.4
0.6
0.3
0.2
0.0
-2
(a)
(b)
Figure 4: Plot of the sigmoid function (a) and its derivative (b).
A.1.2 TRAINING DYNAMICS
Now we analyze the training dynamics of the network. Suppose for the I-th block, the input is x((), and the effective weight the linear layer is E(), In other words, we have
2
(21a)
(1)
(
(1)
2
8(0)
7
(1)
(21b)
((+1)
h(v? (1)
(21c)
x
((+1)
(xpoolyy PEP
Σ
(1+1)
p
(21d)
where = I,...,
Since the parameters (1) and 800 constants (although trainable), and the activation function is quasi-linear, we have
VAR (1) ())
(22a)
EI 1)) VAIR (1 ~ 132
(22b)
700) 2 R.pool
B[
((((())))0]11
(2 B[(E!! ((((((+)))
(22c)
(0(1)2 WAR (
(22d)
TiVAR [1 (0)))
(22c)
7 (1-1)
ni VARE A.Pool
(22f)
where ny is the number of input neurons in the (-th layer, i.c. ny = q.- - (kponv)?, with o and RÇORy denoting the number of input channels and kernel size of the convolution layer, respectively. For fully-connnected layer, we can set Aconv 1. Here, we have omitted the indices of trainable parameters in the batch normalization layer by simply assuming that they of the similar order. The second equality holds because the activation function is quasi-linear, and in the third equality we have used the assumption of independence. Meanwhile, in the last three equalities, we have used the definition of o and also the assumption that WO) and (1) are all independent with each other.
16

---

Under review as a conference paper at ICLR 2020
During training, both 14; (1) and o
(1)
functions of 2/
(1)
and we have
0p4 (1) 1
(23a)
02, (1) mB
0o (1)
(1)
2
p4
(23b)
02 (1)
mB
0
therefore,
O (1)
1 0p
(1)
(1)
(0)
-1 do (1)
14
0=())
(1)
(1)
Oz, (1)
(0 )2 0z, (9)
(2 14 (1)
(0
1
1
2
(1)
mB
0
(1)
mB
(o)))
70
(mn >
1)
(24)
(1)
Here, m. is the batch size.
From the above results, derive the relationship between the gradients with respect to the weights and activations in two adjacent layers. Actually, from Eq. 21 and Eq. 24 we have
2=m L ~ x
7
11 [y
(1)
(25a)
0 10 L = Σ (1)
h' N'(,, 12
(25b)
2,
0.(41)
2 (+1) C
(25c)
(1. (kpooly3 32
The variances of the gradients are thus given by
7000)
2
VAR mm C]
B[
(1)
(26a)
(1)
0,
((()
VAR m C]
VARE JE (E ( 2
(26b)
23
(
V VAR (-1)
C]
V VAR e+ C]
(26c)
(kpooly y4
where îu = C++1 (kFonv)2 and C+11 the number of output channels. We use the assumption of independence for different variables here again.
From this we can get
7.01)
2 [ (1+1) y2]
VAR 1 C
VARE
(1-1)
VAR (27)
[x
(K) (kponly4
Note that the subscripts of variables inside VAR can be eliminated variables in the same tensor are distributed identically by assumption.
If have batch normalization layer after the linear layer, based on Eq. 22c and Eq. have
22e
To VARE (1+1)
VAR (0_ 10())
VAR
L
(28)
(kpoolyz
M (1)
(1) Here, [1 is some empirical parameter of order (1)) with respect n..
17

---

Under review as a conference paper at ICLR 2020
10
14
0.10
11
15
0.08
0.08
10a
0.02
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Iteration
x10*
Iteration
x107
(a)
(b)
Figure 5: Empirical verification of Eq. and Eq.
28
(a) Ro calculated with Eq.
30ap. (b)
K1
given by Eq. 30bb for different layers.
On the other hand, if there no batch normalization layers, (0) and o will be removed from Eq. 27 and VAR C] can be formulated as =3
VARIO_0 [] m22)
(kpoolya 1 WVAR C]
(29)
For typical case, K2 (1)
is another empirical parameter order ((1) with respect ny.
Both Eq.
28 and Eq. are referred to as the gradient flowing law.
A.2
EMPIRICAL VERIFICATION
We are now ready verify Eq. and Eq. 28 with experiments. For this purpose, we train vanilla ResNet18 8 with Kaiming initialization et 2015) and batch normalization to guarantee
convergent training. In this case, we have E W. We calculate the two parameters Ko and Kj given by
VAR (
No = (k:pool)z
(30a)
Ty VAI
VAR 2 w
C]
( (kpool)2.
(30b)
Foz WAR
VAR 21)
Here Kg is calculated using statistics from the last fully-connected layer and Kn is calculated for all convolution layers along the whole training procedure. We plot the values of Kn and кι in Fig. 5 As we can see, Ku < during the whole training process as expected. x] in different layers are all of order (1) during the training process, except for the first layer, which involves its analysis is out of our scope (we set x.pool 1 in the computation for this layer). Nevertheless, we notice that the discrepancy is not large.
pooling and
A.3 COMMENTS ON THE INDEPENDENCE ASSUMPTION
Here we want to present some further discussion on the independence assumption, which is ex- tensively used in analysis. Our discussion is to facilitate more intuitive understanding of such assumption, but not in pursue rigorous proof.
The first point want to emphasize is related to the central limit theorem (CLT). Independence is one important condition for the CLT. but sufficiently weak dependence does not harm [Van Kam- pen [1992]] Typical examples where independence does not hold include the momentum of molecules in an ideal gas described by the micro-canonicaa ensemble, or random walk with per- sistence. Even with independence assumption violated, the sum of random variables is able to
18

---

Under review as a conference paper at ICLR 2020
converge to Gaussian distribution as the number of random variables increases indefinitely. Thus, with sufficient number neurons, linear operation can result in normally distributed outputs, which further leads to the normal distribution of gradients. Now since the weights initialized with normal distribution, updating with normally distributed gradients will result in some new Gaussian random variables, and thus Experiment shows that the trained parameters still resemble Gaussian distribution (sometimes with some small spurs), especially for the last several layers.
focus the variance to characterize their statistical properties.
Second, we investigate a simple example to show how the independence property can be derived with zeroth-order approximation. For simplicity. we analyze the product of two random variables U and V. both of which are some function of another random variable X, that is
= ((X)
(31a) (31b)
V = (X)
We can see that and are generally not independent.
The random variables be viewed quantity with some true value given by the mean, but perturbed by some small random noise, that is
U U + 6U
(32a)
V E[ + 8V
(32b)
and we have
EUV] [[BB[]]+
(33a)
+ E[öUSV]
(33b)
[
(33c)
For zeroth-order approximation, we can ignore the second term, which is second-order quantity, and get
BUV] = EU
(34)
For variance of UV, EUU] = 0, we have
VAR = 13 (E(UV))
(35a)
= 2
(35b)
= EVV?
(35c)
VAR[U]E[V2]
(35d)
which an approximated independence property widely used in our derivation. Note that such zeroth-order approximation is broadly adopted in statistical mechanics and related areas for analyz- ing macroscopic law of noisy physical systems, where nonlinear mechanisms exist [Van Kampen 1992 More accurate results could be derived by expansion of the master equation, which is beyond the scope of this paper. Such zeroth-order approximation is also similar to the annealed approximation method in condensed matter physics [Seung et al. [.
B IMPACT OF QUANTIZATION ON WEIGHT VARIANCE
In this section, we examine the impact of quantization on weight variance. Fig. 6 shows the ratio between the standard deviations of them with respect to the number of bits for different channel numbers, which determines the variances of the original non-clamped weights W. We can find that for precision higher than 3 bits, quantized weights have nearly the same variance as the full- preliireoiioon weights. indicating quantization itself introduces little impact training. However, the discrepancy increases significantly for low precision such as or 2 bits. Also, different channel numbers give similar results.
19

---

Under review as a conference paper at ICLR 2020
etQ)
32 channels
64 channels
a(
256 charnels
charnels
2049 chamels
3-
Num ber of Bits
Figure 6: Impact of weight quantization the variance of effeettve weight under different channel numbers.
MobileNet-V V2 on ImageNet
0.40
Constant (train)
Constant (valid)
Standard Deviation (train)
Standard Deviation (valid)
0.35
Error
0.30
0.25
20
40
60
80
100
120
140
Epoch
Figure 7: Comparison of constant rescaling and rescaling with standard deviation.
C COMPARISON OF RESCALING METHOD
this section, we will compare the constant rescaling methods proposed in and another method named rescaling with standard deviation. It standardizes the effective weights and then rescales them with the standard deviation of the original weights as in Eq. 36 where VAR[[- the sample variance of elements in the weight matrix, calculated by averaging the square of elements in the weight matrix.
VAR VAR W,
W
(36)
We train MobileNet- V2 ImageNet with weight clamping and rescale the last colne-connected layer using the two methods, and plot their learning curves as shown in Fig, 7 We find that the two methods give similar results.
20

---

Under review as a conference paper at ICLR 2020
D GRADIENT CALCULATION IN PACT
Here we derive the result in Eq. 13 The activation value x is clamped into the interval [0, a] and then quantized follows
1 3
al
+
(37a)
0 x <<< 0
= r 0<z<a
(37b)
Σ>a
q aqu
(37c)
Based on the STE rule, we have Taking derivative with respect
1/ (x)
(38)
in Eq.
37
have
0q
0
da = qu
Qk
(39a)
da
0
qh
+
(39b)
da
di
+ a
at
(39c)
a da
a
+ a 0 + 07
1 <<<
(39d)
qk
+ a
1 + 0.
a
8 >
qh
I << a
(39e)
1 1 Q
(-23)
oz
>>0
q
x << a
(39f)
W>0 a
which is the result in Eq.
13
E BITWIDTHS OF THE FIRST AND LAST LAYERS
Here we study the impact of quantization levels of the first and the last layers. Using MobileNet- VI, compare the two settings of quantizing these two layers to fixed 8 bits to the same bit-width as other layers. As shown in Table 6 we find that the accuracy reduction is negligible for quantization levels higher than 4bits.
Table 6: Impact of precisions of the first and the last layers on MobileNet-VI.
8bits Both Layers
Uniform Quantization
Bitwidths of Internal Layers
Acc.-1
Acc.-5
Acc.-1
Acc.-5
4bits
71.3
89.9
71.2
89.8
5bits
71.9
90.3
72.1
90.3
6bits
72.3
90.4
72.5
90.5
F RESCALING CONVOLUTION LAYERS WITHOUT BATCH NORMALIZATION FOLLOWED
In this section, we study the impact of rescaling to convolution layers without BN layers followed. As shown in Fig. & plot the learning curves of Pre-Res ResNet50 with bits weight-only quantization
21

---

Under review as a conference paper at ICLR 2020
PreResNet- 50 (2bit Weight-Onlyl on ImageNet
0.7
Only L-yLayer (train)
Only FCyayer (vali))
All Layers (trainl....
All Layers (valid)
Error
0.3
20
80
100
120
140
Epach
Figure 8: Learning curves for PreResNet-50 with 2bit weight-only quantization on ImageNet. For red lines, rescaling is only applied to the last fully-connected layer, while for blue lines, all layers are rescaled.
on ImageNet, with constant rescaling applied to only the fully-connected layer or all layers. We can see that applying rescaling to only the last fully-connected layer is not sufficient for efficient training, and rescaling on all convolution layers successfully achieves efficient training.
G ELIMINATION OF BATCH NORMALIZATION LAYERS
As mentioned in Section 3.1 batch normalization layers are not quantized in our experiments, and full-precision multiplication involved in BN is a considerable amount of computation during infer- ence. Here, we want to discuss the possibility to eliminate such layers, especially for light-weight models such as MoDi
As shown in Eq. (8) quantization of activation function already involves full- precision multiplica- tion, so we want to examine if it is possible to absorb the batch normalization operations into this clipping function. The quantized input to a linear layer can be written as
78
(40)
a
where n is the integer output from the rounding function in quantizing the activation of previous layer, a 2 determined by the bitwidth of the previous layer, and a is the clipping level of the activation function in the previous layer. The quantized output of the current layer is given by
R/
P3
clipping ((())): + +3,00,
(41a)
21| P1
21 R1
22
& 0,
24
7v- clipping (Qx); +
(41b)
Pt
3 0, ā
7 -clipping
(n))); +
(41c)
vi
22| R/
a <=>
clipping (Qn); + 0, 7
3 a
à a
(41d)
76
where ā and ä are corresponding parameters for the current layer, clipping(.) is the clipping func- tion given by Eq. [Ta), and we have simplified the batch normalization by absorbing the running statistics into the parameters. Here, we have focused on one channel of the batch normalization layer, but the generalization is straightforward. From this, we can see that we only need to use different biasing and clipping levels for different channels, which can be pre-computed. The fuel-prrcision multiplications in BN layers are eliminated. Meanwhile, since Q and n are both quantized, lower precision can be adopted for computing the clipping levels.
22

---

Under review as a conference paper at ICLR 2020
Note that the above discussion is only applicable models without skip-connection, such as MobileNet-VLL V1, or models where convolution is not involved in coip-connection, such as MobileNet- V2. For ResNet and PreResNet, the above method is not applicable and more sophisticated methods might be necessary to eliminate BN layers.
23


Reviews:
[{"review_id": "rklidewTFr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents network quantization techniques for weight quantization (SAT) and activation quantization (CG-PACT). The authors first formulate efficient training rules (ETR I and ETR II) and propose a normalization scheme for weight quantization based on the rules. Independently, the authors also propose an activation quantization rule by removing the approximated term in PACT. The authors present the effectiveness of their algorithm in MobileNet V1, V2, and PreResNet-50.\n\nOverall, I think the proposed algorithms lack novelty due to the following reasons.\n\n- SAT suggests a normalization scheme under some assumptions (ETR I and ETR 2), however, I am not sure this is a significant contribution compared to DoReFa. In particular, I have some doubt in assumptions which is listed in ‘Other comments’. \n\n- CG-PACT seems to be a simple variant of PACT. In methodological contribution, I believe that it could only be a marginal improvement with respect to PACT. In terms of empirical contribution, it was hard to see the improvement made by CG-PACT as PACT and PACT+CG exhibit similar performance in Table 4.\n\nOther comments\n- In the derivation of ETR 1, some approximations are not clear for me. For example, the authors drop \\gamma in (17). In addition, approximation in (16d) depends on the activation function and BN parameters but they are also ignored. I believe that incorporating all these would not introduce much computational overhead.\n\n- In (3) and (4), I think that equality only holds under some assumptions (e.g., an infinite number of parameters, i.i.d. weights, etc.) but they are not clearly mentioned. \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "HkxtvrkCKr", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes two rules for efficient training of quantized networks by investigating the scale of the logit values and gradient flow. The authors claim that accuracy degradation of recent quantization methods results from the violation of these two rules.\n\nOne of my main concerns is that the analysis of the rules for weight and activation quantization are separated. E.g., the analysis of weight quantization in Section 3.2 is based on eq (1a)-(1d) where no activation quantization is considered. In this case, does the analysis still hold when applying weight and activation are quantized simultaneously?\n\nMoreover, the analysis is only suited for a limited range of quantization methods. In the proposed SAT, the authors propose to multiplies the normalized weight with the square root of the reciprocal of the number of neurons in the linear layer, to make up for the variance difference caused by quantization. However, this increase indeed depends on the initialization of the weights. If the weights are not sampled from \"a Gaussian distribution of zero mean and variance proportional to the reciprocal of the number of neurons\" as at the end of page 5, then this recipe may not work any longer. Moreover, the proposed SAT seems to be only suited for the specific quantization function for Dorefa-Net in (5), what about many other recent quantization functions that do not need this kind of clamping?\n\nOthers:\n1. The citation format is wrong.\n2. In the abstract, \"Recent quantization approaches violates ... and results ...\" => \"Recent quantization approaches violate ... and result ...\"\n3. What is the \"scaling factor in Eq. (3)\" before the subsection \"Efficient Training Rule II (ETR II)\"?\n4. Keep the same number of decimal places in the tables.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HkgNx-OiqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Quick Summary: Based on semi-quantitative analysis the paper first proposes two rules for quantization of DNNs, then extends previous methods based on these rules to propose specific technique for quantizing activations and weights. Experimental results are provided on MobileNet V1/VB2 and ResNet50 and compare favourably against baselines (which are old and unfortunately do not represent recent developments in the literature). Overall I found novelty low and also there are several recent papers already published before this submission which provide results which are at least as good or even better\n\nDetails\nSpecifically the rules are: \n1. To prevent logits from entering saturation region of the cross entropy loss, the effective weight in the last fully connected layer should be small.  \n2.To keep the gradient of weights in the same scale across the whole network, either BN layers should be used after linear layers such as convolution and fully- connected layers, or the variance of the effective weights should be on the order of the reciprocal of the number of neurons of the linear layer (n_l).\nThey use DoReFa for weight quantization and PACT for activation quantization based on the above observations/rules. However the contributions were not really very novel in my opinion. See especially the paper \"LEARNED STEP SIZE QUANTIZATION\" I linked below when explaining novelty\n\nThis reviewer feels that the authors were perhaps unaware of several important contributions to the literature this year which are substantially better than the baselines compared against in the paper. I list a few below for the authors to compare against, and to explain their novelty in contributions in the rebuttal phase. Unfortunately this makes me feel the paper should be a clear reject in its current state, unless the authors can convince us otherwise. \n\nhttps://arxiv.org/pdf/1902.08153.pdf\nhttps://arxiv.org/pdf/1903.08066.pdf\nhttps://arxiv.org/pdf/1905.11452.pdf\nhttps://arxiv.org/pdf/1905.13082.pdf\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission proposes methodology for quantizing neural networks.  The reviewers were unanimous in their opinion that the paper is not suitable for publication at ICLR.  Concerns included novelty over previous works, comparatively weak baseline comparisons, and overly restrictive assumptions.
Decision: Reject