Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks
Authors: Yuhang Li
Xin Dong
Wei Wang
Venue:  (2020)
Keywords: Quantization
Efficient Inference
Neural Networks

Abstract: We propose Additive Powers-of-Two~(APoT) quantization, an efficient non-uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization levels as the sum of Powers-of-Two terms, APoT quantization enjoys high computational efficiency and a good match with the distribution of weights. A simple reparameterization of the clipping function is applied to generate a better-defined gradient for learning the clipping threshold. Moreover, weight normalization is presented to refine the distribution of weights to make the training more stable and consistent. Experimental results show that our proposed method outperforms state-of-the-art methods, and is even competitive with the full-precision models, demonstrating the effectiveness of our proposed APoT quantization. For example, our 4-bit quantized ResNet-50 on ImageNet achieves 76.6% top-1 accuracy without bells and whistles; meanwhile, our model reduces 22% computational cost compared with the uniformly quantized counterpart.

Published as a conference paper at ICLR 2020
ADDITIVE POWERS-OF-TWO QUANTIZATION: AN EFFICIENT NON-UNIFORM DISCRETIZATION FOR NEURAL NETWORKS
Yuhang Li i Xin D Wei Wang
NNational University of Singapore, HHarvard University loa @ com, x xindongêg. ha rvard.ede wangwei wangweiecomp. nus .eds.eduu... .gg
ABSTRACT
We propose Additive Powers-of-Two (APoT) quantization, an efficient non- uniform quantization scheme for the bell-shaped and long-tailed distribution of weights and activations in neural networks. By constraining all quantization lev- els as the sum of Powers-of-Tie tational efficiency and a good match with the distribution of weights. A simple reparameterization of the clipping function is applied to generate better-defined gradient for learning the clipping threshold. Moreover, weight normalization is presented to refine the distribution of weights to make the training more stable and consistent. Experimental results show that our proposed method outperforms state-of- ofc--o----errrt art methods, and is even competitive with the full full-precision models, demonstrating the effectiveness of our proposed APoT quantization. For exam- ple, our 4-bit quantized ResNet-50 on ImageNet achieves 76.6% top-1 accuracy without bells and whistles: meanwhile, our model is capable to decrease 22% computational cost compared with the uniformly quantized counterpart.
terms, APoT quantization enjoys high compu-
INTRODUCTION
Deep Neural Networks (DNNs) have made a significant improvement for various real-world appli- cations. However, the huge memory and computational cost impede the mass deployment of DNNs, e.g., on resource-constrainee devices. To reduce memory footprint and computational burden, sev- eral model compression methods such as quantization (Zhou et al., 2016), pruning (Han et al., 2015) and low-rank decomposition (Denil et al., 2013) have been widely explored.
In this paper, focus on the neural network quantization for efficient inference. Two operations are involved in the quantization process, namely clipping and projection. The clipping operation sets full precision number to the range boundary if it is outside of the range; the projection operation maps each number (after clipping) into a predefined guantization level (a fixed number). We can that both operations incur information loss. A good quantization method should resolve the two following questions/challenges, which correspond to two contradictions respectively.
How to determine the optimal clipping threshold balance clipping range and projection resolu- tion? The resolution indicates the interval between two quantization levels; the smaller the interval, the higher the resolution. The first contradiction is that given fixed number of bits represent weights, the range and resolution are inversely proportional. For example, a larger range can clip fewer weights; however, the resolution becomes lower and thus damage the projection. Note that slipshod clipping of outliers can jeopardize the network a lot (Zhao et al., 2019) although they may only take 1-2% of all weights in one layer. Previous works have tried either pre-defined (Cai al., 2017; Zhou et al., 2016) or trainable (Choi et al., 201 8b) clipping thresholds, but how to find the optimal threshold during training automatically is still not resolved.
"Equal Contribution. Y. completed this work during his internship at NUS.
CCode is available /ettss:/giitiie com/ 20hh// //PoT Tepantttuuaiiitte ion

---

Published as a conference paper at ICLR 2020
How to design quantization levels with consideration for both the computational efficiency and the distribution of weights? Most of the existing quantization approaches (Cai et al., 2017; Gong
et al., 2019) use uniform quantization although non-uniform quantization usually achieve better accuracy (Zhu et al., 2016). The reason is that projec- tion against uniform quantization levels much more hardware-friendly (Zhou et al., 2016). However, empirical study (Han al., 2015) has shown that weights in a layer of DNN follow a bell-shaped and long-tailed distribution instead of uniform distribution (as shown in the right figure). In other words, fair percentage of weights con- centrate around the mean (peak area); and few weights are of relatively high magnitude and out of the quanti- zation range (called outliers). Such distribution also ex- ists in activations (Miyashita et al., 2016). The second contradiction is: considering the bell-shaped distribution of weight, it is well-motivated to assign higher resolution (i.e. smaller quantization interval) around the mean; how- ever, such non-uniform quantization levels will introduce high computational overhead. Powers-off Two quantization levels (Miyashita et al., 2016; Zhou al., 2017) are then proposed because of its cheap multiplication implemented by shift operations on hardware, and super high resolution around the mean. However, the vanilla powers-of-two quan- tization method only increases the resolution near the mean and ignores other regions at all when the bit-width is increased. Consequently, it assigns inordinate quantization levels for tiny range around the mean. To this end, we propose additive Powers-of-Two (APoT) quantization to resolve these two contradictions,
More weights in peak area
0.5
Density
0.3
0.2
Less weights in edge area
0.0
-2
2
Figure 1: Density of weights ResNet-18
contribution can be listed as follows:
1. We introduce the APoT quantization scheme for the weights and activations of DNNs. APoT is non-uniform quantization scheme, in which the quantization levels is sum of several PoT terms and can adapt well to the bell-shaped distribution of weights. APoT quantization enjoys approximate 2x multiplication speed-up up compared with uniform quantization on both generic and specific hardware.
2. We propose a Reparameterized Clipping Function (RCF) that can compute a more accurate gradient for the clipping threshold and thus facilitate the optimization of the clipping thresh- old. We also introduce weight normalization for neural network quantization. Normalized weights in the forward pass are more stable and consistent for clipping and projection.
3. Experimental results show that our proposed method outperforms state-of-the- art methods, and is even competitive with the full-precision implementation with higher computational efficiency. Specifically, our 4-bit quantized ResNet-50 on ImageNet achieve 76.6% Top-1 and 93.1 1.11% 1% Top-5 accuracy. Compared with uniform quantization, our method can decrease 22% computational cost, demonstrating the proposed algorithm is bardware-friendll.
METHODOLOGY
2.1 PRELIMINARIES
Suppose kernels in a convolutional layer are represented by a 4D tensor W E XCo where Cout and Cin the number of output and input channels respectively, and K is the kernel size. We denote the quantization of the weights as
(1) where a is the clipping threshold and the clipping function a' clips weights into [-a, a.. After clipping, each element of W is projected by II(-) onto the quantization levels. We denote Q(a, b) for a set of quantization levels, where is the bit-width. For uniform quantization, the quantization levels defined as
W = II (W, [W,a],
(" = a x {0, #1
22
+3
(2)
For every floating soating-ppint number. uniform quantization maps it to a b-bit fixed-point representation (quantization levels) in (" b). Note that is stored separately as full-precision floating-poin
2

---

Published as a conference paper at ICLR 2020
1.5
8.8
Quanie
04
Quanttcal
14.6
Qutnited
pcak
Morellevels
0.2
0.2
6.2
3-bit
3-bit
3-bit
1.8
10
Rigid Reslution
edge area
Quartto
Quariied
6:90
8.2
Quantted
0.4
9.4
6.2
4-bil
4-bit
6.
0.2
0.4
0.6
6.5
1.8
0.2
0.8
Float
Flaas
(a) Uniform Quantization
(b) Power-off.ww Quantization
(c) Additive Power-of. Quantization
Figure 2: Quantization unsigned data to 3-bit 4-bit (a 1.0) using three different quantization levels. APoT quantization has a more reasonable resolution assignment and does not suffer from the rigid resolution.
number for each whole W. Convolution is done against the quantization levels first and the re- sults are then multiplied by a. Arithmetical computation, e.g., convolution, can be implemented using low-precision fixed point operations on hardware, which are substantially cheaper than their Hoating-poin contradictory (Goldberg, 1991). Nevertheless, uniform quantization does not match the distribution of weights (and activations), which is typically bell-shaped (Han et al., 2015). A straightforward solution
assign more quantization levels (higher resolution) for the peak of the distribution and fewer levels (lower resolution) for the tails. However, it is difficult to implement the arithmetical operations for the non-uniform quantization levels efficiently.
2.2 ADDITIVE POWERS-OF-TWE QUANTIZATION
To solve the contradiction between non-uniform resolution and hardware efficiency, Powers-of- Two (PoT) quantization (Miyashita et al., 2016; Zhou et al., 2017) proposed by constraining quantization levels be powers-of-two values zero, i.e.
(a,
{0, 20.22-2
-2
12-1,
(3)
Apparently, as non-uniform quantizer, PoT has higher resolution for the value range with denser weights because of its exponential property. Furthermore, multiplication between Powers-of-wwe number 2" and the other operand r can be implemented by bit-wise shift instead bulky digital multipliers, i.e.,
= =
2"r
if > 0, <>>
r<<I
(4)
if <<
where >>> denotes the right shift operation and is computationally cheap, which only takes clock cycle in modern CPU architectures.
However, we find that PoT quantization does not benefit from more bits. Assume a is 1, as shown in Equation (3), when we increase the bit-width from b to b 1, the interval [-2 1+1 2" 111 will be split into 2b-1 1 sub-intervals, whereas all other intervals remain unchanged. In other words, by increasing the bit-width, the resolution will increase only for [-2 ++1 2-1--++1 We refer this phenomenon as the rigid resolution of PoT quantization. Take Q" 5) as an example, the two smallest positive levels are 2-15 and 2 2-14, which is excessively fine-grained. In contrast, the two largest levels are 2-1 and 20, whose interval is large enough to incur high projection error for weights between 12-1,20), e.g., 0.75. The rigid resolution is demonstrated in Figure 2(b). When change from from 3-bit to 4-bit, all new quantization levels concentrate around 0 and thus cannot increase the model's expressiveness effectively.
3

---

Published as a conference paper at ICLR 2020
Weights Buller
Unitorm (k=1)
Sign
Shitt bit
x
APoT
(k=2)
...
+
Oulput
Accumulator
PoT
(k=b)
Activations Buffer
Figure 3: Hardware accelerator with different quantization schemes. When increase, weights usually has less PoTT terms, thus accelerates the computation.
To tackle the rigid resolution problem, we propose Additive Powers-of-Two (APoT) quantization. Without loss of generality, in this section, we only consider unsigned numbers for simplicity?. In APoT quantization, each level is the sum of PoT terms shown below,
1 Σ Pi where E {0, 2 20th 21+ 2)m ==
=
(5)
where y a scaling coefficient to make sure the maximum level in Qa a. k is called the base bit-width, which is the bit-width for each additive term, and n is the number of additive terms. When the bit-width and the base bit-width & is set, n can be calculated by = -. There are 2 = 2 levels in total. The number of additive terms in APoT quantization can increase with bit-width b, which provides a higher resolution for the non-uniform levels.
We use b = 4 and k 2 as an example to illustrate how APoT resolves the rigid resolution problem. For this example, we have pa E {0, 20, 2 2,2,2-4 }, p1 E {0, 20,2-1,2 1,2 3,2 3- = 2a/3, 16) combinations of Po and p1. First, we can see the smallest positive quantization level in (1, 4) 2-3/3. Compared with the original PoT levels, APoT allocates quantization levels prudently for the central area. Second, APoT generates 3 new quantization levels between 2D and 2-1, to properly increase the resolution. In Figure 2, the second row compares the 3 quantization methods using 4 bits for range [0, IJ. APoT quantization has reasonable distribution of quantization levels, with more levels in the peak area (near 0) and relatively higher resolution than the vanilla PoT quantization the tail (near 1).
and Q" (a, kn)
{y x (po + pı)} for all (2
Relation to other quantization schemes. On the one hand, the fixed- fixed-point point number representations used in the uniform guantization quantization levels is sum of b PoT terms or O. In the fixed-point representations, each bit indicates one specific choice of the additive terms. On the other hand, when and (o, b) becomes Q" b). i.e., PoT quantization. We can conclude that when & decreases, APoT levels are decomposed into more PoT terms, and the distribution of levels becomes more uniform. Our experiments use k 2, which is intermediate choice between the uniform (k 1) and the vanilla PoT case (I: b).
special case of APoT. When k in Equation (5), the
there is only one PoT term
Computation. Multiplication for fixed-point numbers can be implemented by shifting the mul- tiplicand (i.e., the activations) and adding the partial product. The n. in Equation (5) denotes the number of additive PoT terms in the multiplier (weights), and control the speed of computation. Since n = h either decreasing or increasing can accelerate the multiplication. Compared with uniform quantization (k: 1), our method (k 2) is approximately 2x faster in multiplication. As for the full precision a, it is coefficient for all weights in a layer and can be multiplied only once after the multiply aactuply-accumullte operation finished. Figure shows the hardware accelerator, the weights buffer takes b-b as PoT term and shift-adds the activations.
Generalizing 2n + bits. When 2, APoT quantization can only leverages 2n-bit width for quantization. To deal with 2n + 1-bit quantization, we choose to add n + PoT terms, one of which
"To extend the solution for the signed numher, we only need add more bit for the sign.
4

---

Published as a conference paper at ICLR 2020
only contains 2 levels. The formulation is given by
1 1
1
(a 2n + 1) - 7 x
Σ p P where ps E {0, 2i 21+n' 22:22+1
J, Pe E {0, 2i+2n
(6)
Take 3-bit APoT quantization as an example, every level is a sum of one Po and one p, where Po {0,2-* 1-22-22-2 2.2 +} and p 20,2-3) 33. The forward function is plotted in Figure 2(c).
2.3 REPARAMETERIZED CLIPPING FUNCTION
Besides the projection operation, the clipping operation [W,a a is also important for quantization. is a threshold that determines the value range weights quantized layer. Tuning the clipping threshold a is a key challenge because of the long-tail distribution of the weights. Particularly, if a too large (e.g.. the maximum absolute value of W). Q(a, b) would have wide range and then the projection will lead to large error as result of insufficient resolution for the weights in the central area; if a too small, more outliers will be clipped slipshodly. Considering the distribution of weights can be complex and differs across layers and training steps, static clipping threshold a for all layers is not optimal.
To jointly optimize the clipping threshold and weights via SGD during training, Choi et al. (2018b) Estimator (STE) (Bengio et al., 2013) to do the backward propagation
apply the Straight-Thronl for the projection operation. According to STE, the gradient to a is computed by aw 0 W.Wa 0cx
8e ~
sign(W) when [W] > a otherwise O, where the weights outside of the range cannot contribute to the gradients, which results in inaccurate gradient approximation. To provide refined gradient for the clipping threshold, we design a Reparameterized Clipping Function (RCF) as
W = all o(1
Instead of directly clipping them to [-a, a)., RCF outputs constant clipping range and re-scales weights back after the projection, which mathematically equivalent to Equation (1) during for- ward. In backpropagation, STE is adopted for the projection operation and the gradients of calculated by
sign(W)
if M
Ow da T a
W W
if i Sa
The detail derivation of the gradients is shown Appendix A. Compared with the normal clipping function, RCF provides more accurate gradient signals for the optimization because both weights inside (WW)= S a) and out of (WI > a') the range can contribute to the gradient for the clipping threshold. Particularly, the outliers are responsible for the clipping, and the weights in (-a, a] are for projection. Therefore, the update of considers both clipping and projection, and tries to find balance between them. In experiments, universally smaller when the bit-width is reduced to guarantee sufficient resolution, which further validates the efficaciousness of the gradient in RCF.
observe that the clipping threshold will become
2.4 WEIGHT NORMALIZATION
In practice, we find that learning a for weights is quite arduous because the distribution of weights pretty steep and changes frequently during training. As result, jointly training the clipping threshold and weights parameters is hard to converge. Inspired by the crucial role of batch nor- malization (BN) (Ioffe & Szegedy, 2015) in activation quantization (Cai et al., 2017), we propose weight normalization (WN) to refine the distribution of weights with zero mean and unit variance,
W W-P P W
where μ Σ W. o
Σ (w - m)²
a +
where e is a small number (typically 10-5) for numerical stability, and I denotes the number of weights in one layer. Note that quantization of weights is applied right after this normalization.
5

---

Published as a conference paper at ICLR 2020
1.
Layer Laper 2
Layar
Layer
Laper
Loper
Clipping Ruite
11.85
weo normalization
wi normaltcution
0.6
wio normalization
wi normalization
Same threelald
Incersitrat cityping rade
0.75
Layar Laye Layar
Layer Layer
combitet clappingeatte
0.79
Layer
0.0
1.8
0.65
159
200
59
100
150
value of
value of
Epoch
Epech
(a) Evolution of clipping ratio with fixed weights
(h) Evolution of clipping ratio with fixed threshold
Figure 4: The evolution of clipping ratio of the first three layers in ResNet-20. (a) demonstrates clipping ratio too sensitive threshold hurt optimization without weights normalization. (b) shows that weights distribution after normalization is relatively more stable during training.
Algorithm Forward and backward procedure for an APoT quantized convolutional layer
Input: input activations Xin, the full precision weight tensor W, the clipping threshold for weights and activations ow, ax2, the bit-width b of quantized tensor.
Output: the output activations Nout
1: Normalize weights W to W
2: Apply RCF and APoT quantization the normalized weights W = a
Low 1]
W
3: Apply RCF and APoT quantization to the activations X.,, = ax axIIlo=(..)) Xin
4: Compute the output activations Xout = Conv(MV,
ac 5: Compute the loss L and the gradients ax..nt
6: Compute the gradients of convolution axi.. aw _c 8C based on Equation (8)
8c
7: Compute the gradients for clipping threshold
Jaw' Bax
Oc aw aw 8: Compute the gradients to the full precision weights aW aw aw 8W
9: Update W, oww, ox with learning rate J/V, Vlos fox
Normalization is important to provide a relatively consistent and stable input distribution to both clipping and projection functions for smoother optimization of over different layers and itera- tions during training. Besides, making the mean of weights to be zero can reap the benefits of the symmetric design of the quantization levels.
Here, we conduct a case study of ResNet-20 on CIFARIO to illustrate how normalization for weights can help quantization. For certain layer (at a certain training step) in ResNet-18, We firstly fix the weights, and let a go from 0 to max [W] to plot the curve of the clipping ratio (i.e. the proportion of clipped weights). As shown in Figure 4a, the change of clipping ratio much smoother after quan- tization. As a result, the optimization of will be significantly smoother. In addition, normalization also makes the distribution of weights quite more consistent over training iterations. We fix the value of a. and visualize clipping ratio over training iterations in Figure 4b. After normalization, the same will result in almost the same clipping ratio, which improves the consistency of optimization goal for More experimental analysis demonstrating the effectiveness of the normalization on weights can be found in Appendix B.
2.5 TRAINING AND DEPLOYING
We adopt APoT quantization for both weights and activations. Notwithstanding the effect in activa- tions not conspicuous, we adopt APoT quantization for consistency. During backpropagation, we 1. The detailed training procedure is aw
STE when computing the gradients of weights, iec. ay shown in Algorithm 1.
To save memory cost during inference, we discard the full precision weights W and only store the quantized weights W. Compared with other uniform quantization methods, APoT quantization is more efficient and effective during inference.

---

Published as a conference paper at ICLR 2020
3 RELATED WORKS
Non- Mon-Uniform Quantization. Several methods are proposed for the non non-uniform distribution of weights. LQ-Nets(Zhang al., 2018) learns quantization levels based on the quantization error minimization (QEM) algorithm. Distillation (Polino et al., 2018) optimizes the quantization lev- els directly to minimize the task loss which reflects the behavior of their teacher network. These methods use finite floating-point numbers to quantize weights (and activations), bringing extra com- putation overhead compared with linear quantization. Logarithmic quantizers (Zhou et al., 2017; Miyashita et al.. 2016) leverage powers-of-2 values to accelerate the computation by shift opera- tions; however, they suffer from the rigid resolution problem.
Jointly Training. Many works have explored to optimize the quantization parameters (e.g., o) and the weights parameters simultaneously. Zhu et al. (2016) learns positive and negative scaling coefficients respectively. LQ-Nets jointly train these parameters to minimize the quantization error. QIL (Jung et al., 2019) introduces a learnable transformer to change the quantization intervals and optimize them based on the task loss. PACT (Choi et al., 2018b) parameterizes the clipping threshold in activations and optimize it through gradient descent. However, in PACT, the gradient of a is not accurate, which only includes the contribution from outliers and ignores the contribution from other weights.
Weight Normalization. Previous works on weight normalization mainly focus on addressing the limitations of BatchNorm (Ioffe & Szegedy, 2015). Salimans & Kingma (2016); Hoffer et al. (2018) decouple direction from magnitude accelerate the training procedure. Weight Standardiza- tion (Qiao et al., 2019) normalizes weights to zero mean and unit variance during the forward pass. However, there is limited literature that studies the normalization of weights for neural network quantization. (Zhu et al., 2016) uses feature-scaling to normalize weights by dividing the maximum absolute value. Weight Normalization based Quantization (Cai & Li, 2019) also uses this feature scaling and derive the gradient to eliminate the outliers in the weights tensor.
4 EXPERIMENT
In this section, we validate our proposed method on ImageNet-ILSVRR2002 (Russakovsky et al., 2015) and CIFARIO (Krizhevsky al., 2009). We also conduct ablation study for each component of our algorithm.
4.1 EVALUATION ON IMAGENET
We compare our methods with several strong baselines on ResNet architectures (He et al., 2016), including ABC-Net (Lin et al., 2017), DoReFa-Net (Zhou et al., 2016), PACT (Choi et al., 2018b), LQ-Net (Zhang et al., 2018), DSQ (Gong et al., 2019), QIL (Jung et al., 2019). Both weights and activations of the networks are quantized for comparisons. All the state-of-the-art methods use full precision (32 bits) for the first and the last layer, which incur more memory cost. In our imple- mentation, we employ 8-bit quantization for them to balance the accuracy drop and the hardware overhead.
For our proposed APoT quantization algorithm, four configurations of the bit-width, i.c., 2,3,4, and 5 (k: = 2 and n = or 2 in Equation (5) and (6) are tested, where one bit is used for the sign of the weights but not for activations. Note that for the 2-bit symmetric weight quantization method, ((a, 2) can only be (=ta00) 0). therefore only RCF and WN are used in this setting. To obtain a reasonable initialization, we follow Lin et al. (2017); Jung et al. (2019) to initialize our model. Specifically, the 5-bit quantized model is initialized from the pre-trained full precision one", while the bit network is initialized from the trained 5-bit model. We compare the accuracy, memory cost, and the fixed point operations under different bit-width. To compare the operations with dif- ferent bit- we use the oit-op computation scheme introduced in Zhou et al. (2016) where the multiplication between m-bit and a i-bit uniform quantized number costs ml binary operation. We define one FixOP as one operation between an 8-bit weight and an 8-bit activation which takes 64 binary operations if uniform quantization scheme is applied. In APoT scheme, the multiplication
docs/ stab' e/ modul eestttrcrrrss 8 on /model areenet.
html
7

---

Published as a conference paper at ICLR 2020
Table 1: Comparison of accuracy performance well hardware performance of ResNets (He al., 2016) on ImageNet with existing methods.
METRON
PRECISION
ACCURACY($)
MODEL
FiXOPS
PRECISION (W/A)
ACCURACY(
MODEL
FixOPS
(W/A)) A)
ToP-1
ToP-5
TOP-1
TOP-5
F..RES18)
32/32
70.2
89.4
46.8 MB
182G
ABC-NETS
5/5
65.0
85.9
8.72 MB
781M
DORETA NET
5/5
68.4
88.3
8.72 MB
781M
4/4
68.1
88.1
7.39 MB
542M
PACT
5/5
69.8
89.3
8.72 MB
781M
474
69.2
89.0
7.39 MR
542M
LQ-NET
4/4
69.3
88.8
7.39 MB
542M
DSQ
4/4
69.6
7.39 MB
542M
QIL
5/5
70.4
8.72 MB
781M
4/4
70.1
MB
542M
APOT (OURS)
575
70.9
89.7
7.22 MB
616M
4/4
70.7
89.6
5.89 MB
437M
ABC-NETS
3/3
61.0
83.2
6.06 MB
357M
DOREFA-NET
373
67.5
87.6
6.06 MB
357M
2/2
62.6
84.6
4.73 MB
225M
88.2
2/2
LQ-NET
3/3
68.2
87.9
6.06 MB
357M
2/2
64.9
85.9
4.73 MB
225M
DSQ
3/3
68.7
6.06 MB
357M
2/2
65.2
4.73 MB
225M
QIL
3/3
69.2
6.06 MB
357M
2/2
65.7
4.73 MB
225M
SACT
2/2
67.0
5.36 MB
243M
APOT (OURS)
3/3
69.9
89.2
4.56 MB
298M
2/2
67.3
87.5
3.23 MB
198M
RE(3E)34)
32/32
73.7
91.3
ABC-NETS
575
68.4
88.2
14.8 MB
1.50G
DSQ
4/4
72.8
12.3 MB
1.00G
QIL
5/5
73.7
14.8 MB
1.50G
4/4
73.7
12.3 MB
1,00G
APOT (OURS)
575
73.9
91.6
13.3 MB
1.15G
4/4
73.8
91.6
10.8 MB
784M
NETS
3/3
66.4
87.4
9.73 MB
618M
LQ-NET
3/3
71.9
90.2
9.73 MB
618M
2/2
69.8
89.1
7.20 MB
340M
DSQ
3/3
72.5
9.73 MB
618M
2/2
70.0
7.20 MB
340M
QUI
3/3
73.1
9.73 MR
618M
2/2
70.6
7.20 MB
340M
APOT (OURS)
3/3
73.4
91.1
8.23 MB
493M
2/2
70.9
89.7
5.70 MB
285M
93.1
ABC-NETS
5/5
70.
89.7
22.2 MB 22.2 MB
1.67G
DOREFA-NET
515
71.4
93.3
1.67G
4/4
71.4
89.8
19.4 MB
1.11G
LQ-NET
4/4
75.1
92.4
19.4 MB
IJL1G
PACT
5/5
76.7
93.3
22.2 MB
1.67G
414
76.5
93.3
19.4 MB
IJ1GG
5/5
866M
DOREFA-NET
3/3
69.9
89.2
16.6 MB
680M
2/2
67.1
87.3
MB
PACT
373
75.3
92.6
16.6 MB
680M
2/2
72.2
90.5
13.8 MB
370M
LQ NET
3/3
74.2
91.6
16.6 MB
680M
2/2
71.5
90.3
13.8 MB
370M
PACT
2/2
74.2
23.7 MB
707M
APOT (OURS)
3/3
75.8
92.7
10.8 MB
540M
2/2
73.4
91.4
7,96 MB
308M
between m-bit activation and kn-bit weight only needs mn shift-adds operations, i... FixOPs. More details of the implementation are in the Appendix C.2.
2Xm
64
Overall results are shown in Table 1. The results of DoReFa-Net are taken from Choi al. (2018b), and the other results are quoted from the original papers. It can be observed that our 5-bit quantized network achieves even higher accuracy than the full precision baselines (0.7% Top-1 improvement on ResNet-15 18 and 0.2% Top-1 improvement on ResNet-34 and ResNet-50), which means quanti- zation may serve the purpose regularization. Along with the accuracy performance, our APoT quantization can achieve better hardware performance on model size and inference speed. For full precision models, the number the column of FixOPs indicates FLOPs.
4-bit and 3-bit quantized networks are also preserving (or approaching) the full-precision accuracy except for the 3-bit quantized ResNet-18 and ResNer-34, which only drops 0.5% and 0.3% aceu- racy respectively. When is further reduced to 2, our model still outperforms the baselines, which demonstrates the effectiveness RCF and WN. Note that Choi et al. (2018a) use a full precision shorteut in the model, reaching higher accuracy ResNet-50 however suffering from the hardware performance. In specific, the different precision between the main path and the residual path may result in greater latency in pipelined implementation.
4.2 EVALUATION ON CIFAR10
We quantize ResNet-20 and ResNet-56 (He al., 2016) CIFARIO for evaluation. We adopt progressive initialization and choose the quantization bit as 2, and 4. More implementations can be found in the Appendix C.2.
8

---

Published as a conference paper at ICLR 2020
Table 2: Accuracy comparison of ResNet architectures CIFARIO
ACCURACY(
MODELS
METHODS
BITS
BITS
BITS
DOREFA-NE (ZHOU ET AL., 2016)
88.2
89.9
90.5
PACT (CHOI ET AL., 2018B)
89.7
91.1
91.7
RESNET-20 (FP: 91.6)
LQ-NET (ZHANG ET AL., 2018)
90.2
91.6
PACT+SAWB+FFE (CHOI ET Al.,, 2018A)
90.5
APoT QUANTIZATION (OURS)
91.0
92.2
92.3
RESNET-56
PACT+SAWB+FSSC (CHOI ET AI.,, 2018A)
92.5
(FP: 93.2)
APoT QUANTIZATION (OURS)
92.9
93.9
94.0
Table. 2 summarizes the accuracy of our APoT in comparison with baselines. For 3-bit and 4-bit models, APoT quantization reached comparable results with the full precision baselines. It worthwhile to note that all ttae-of-thhe-rte methods in the table use 4 levels to quantize weights into 2-bit. Our model only employs ternary weights for 2-bit representation and still outstrips existing quantization methods.
4.3
ABLATION STUDY
Table 3: Comparison of quantizer, weight normalization and RCF of ResNet-18 ImageNet.
METHOD
PRECISION
WN
RCF
Acc.-1
RCF
Acc.-1
MODEL SIZE
FIXOPS
FULL PREC.
32 /2/32
70.2
70.2
46.8 MB
1,82G
APOT
5/5
70.9
70.0
7.22
616M
POT
5/5
/
70.3
68.9
7.22 MB
582M
UNIFORM
5/5
/
70.7
X
69.4
7.22 MB
781M
LLOYD
5/5
/
V
70.9
70.2
7.22 MB
1.81G
APOT
3/3
/
69.9
X
68.5
4.56 MB
298M
UNIFORM
3/3
69.4
67.8
4.56 MB
357M
LLOYD
3/3
70.0
69.0
4.56
1.81G
APOT
3/3
2.0
68.5
4.56 MB
198M
The proposed algorithm consists of three techniques, APoT quantization levels to fit the bell-shaped distribution, RCF to learn the clipping threshold and WN to avoid the perturbation of the distribution of weights during training. In this section, we conduct an ablation study for these three techniques. We compare the APoT quantizer, the vanilla PoT quantizer, uniform quantizer and non-uniform quantizer using Lloyd algorithm (Cai et al., 2017) to quantize the weights. And we either apply RCF to learn the optimal clipping range do not clip any weights (i.e. Normalization is also adopted or discarded to justify the effectiveness of these techniques.
max [W). Weight
Table summarizes the results of ResNet-18 using different techniques. Quantizer using Lloyd achieves the highest accuracy, however, the irregular non-uniform quantized weights cannot utilize the fixed point arithmetic to accelerate the inference time. APoT quantization attends the dis- tribution of weights, which reaches the same accuracy in 5-bit and only decreases 0.2% accuracy in 3-bit quantization compared with Lloyd, and shares a better tradeoff between task performance and hardware performance. We also observe that the vanilla PoT quantization suffers from the rigid resolution, and has the lowest accuracy in the 5-bit model.
Clipping range also matters in quantization, the comparison in Table 3 shows that a proper clipping range can help improve the robustness of the network. Especially when the network quantized to 3-bit, the accuracy will drop significantly because of the quantization interval increases. Applying RCF to learn the optimal clipping range could improve at most 1.6% accuracy. As we mentioned before, normalization of weight is important learn the clipping range, and the network diverges if RCF applied without WN. We refer to the Appendix B for more details weight normalization during training.

---

Published as a conference paper at ICLR 2020
5 CONCLUSION
In this paper, we have introduced the additive powers-of-two (APoT) quantization algorithm for quantizing weights and activations in neural networks, which typically exhibit a bell bell-shaped and long-tailed distribution. Each quantization level of APoT is the sum of set of powers-of-two terms, bringing roughly 2x speed-up in multiplication compared with uniform quantization. The distribution of the quantization levels matches that of the weights and activations better than existing quantization schemes. In addition, we propose to reparameterize the clipping function and normalize the weights to get more stable and better-defined gradient for optimizing the clipping threshold. We reach state-of-thee accuracy on ImageNet and CIFAR 10 dataset compared to uniform or PoT quantization.
ACKNOWLEDGEMENT
This work is supported by National University of Singapore FY2017 SUG Grant, and Singa- pore Ministry of Education Academic Research Fund Tier 3 under MOEs official grant number MOF2017-T3-1-0077
REFERENCES
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating propagating gradients through stochastic neurons for conditional computation. arXiv preprint 0rXiv:1308.33. 3432,, 2013.
Wen-Pu Cai and Wu-Jun Li. Weight normalization based quantization for deep neural network 2019.
compression. arXiv preprint
Zhaowei Cai, Xiaodong He, Jian Sun, and Nuno Vasconcelos. Deep learning with low precision by half wave gaussian quantization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 5918-5926, 2017.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang. Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). arXiv preprint 0ttt::1807.669900 2018a.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXiv:1805.0688; 2018b.
Misha Denil, Babak Shakibi, Laurent Dinh, Marc' Aurelio Ranzato, and Nando De Freitas. Pre- dicting parameters in deep learning. In Advances in neural information processing systems, pp- 2148-2156, 2013.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. URL https: wpw..eeee.eeeeeeeeee/ee/e. net. /o
David Goldberg. What every computer scientist should know about Moating-point arithmetic. ACM Computing Surveys (CSUR), 23(1):5-48 1991.
Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full-precision and low-bit neural networks. arXiv preprint arXiv: (ww.//98800000 2019.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv: 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
10

---

Published as a conference paper at ICLR 2020
Elad Hoffer, Ron Banner, Itay Golan, and Daniel Soudry. Norm matters: efficient and accurate normalization schemes in deep networks. Advances in Neural Information Processing Systems, pp. 2160-2170, 2018.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167, 2015.
Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 4350-4359, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
Xiaofan Lin, Cong Zhao, and Wei Pan. Towards accurate binary convolutional neural network. In Advances in Neural Information Processing Systems, pp. 345-353, 2017.
Daisuke Miyashita, Edward H Lee, and Boris Murmann. Convolutional neural networks using logarithmic data representation. arXiv preprint 1603: 2016.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. arXiv preprint arXiv:/802 05668, 2018
Siyuan Qiao, Huiyu Wang, Chenxi Liu, Wei Shen, and Alan Yuille. Weight standardization. arXiv preprint arXiv:1903. 1119000..200, 2019.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenct large scale visual recognition challenge. International journal of computer vision, 115())211-255, 2015.
Tim Salimans and Durk P Kingma. Weight normalization: simple reparameterizztioo accelerate training of deep neural networks. In Advances in Neural Information Processing Systems, pp- 901-909, 2016.
Dongqing Zhang, Jiaolong Yang, Dongqiangzi Yc. and Gang Hua. Lq-nets: Learned quantization for highly accurate and compact deep neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 365-382, 2018.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International Conference Machine Learning, pp. 7543-7552, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantiza- tion: Towards lossless cnns with low-precision weights. arXiv preprint arXiv: (et:::02200444 2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606.066660616 2016.
Chenzhuo Zhu, Song Han, Huizi Mao, and William Dally. Trained ternary quantization. arXiv preprint arXiv: 2016.

---

Published as a conference paper at ICLR 2020
APPENDICES
A GRADIENT DERIVATION
In this section, we derive the gradient estimation of PACT (Choi et al., 201 along with our pro- posed Reparameterized Clipping Function and show the distinction of these two estimation.
A.1 PACT
Equation (1) shows the forward of PACT. In backpropagation, PACT applies the Straight-Throus Estimator for the projection operation. In particular, the STE assumes that
Al ax
1,
(10)
which means the variable before and after projection are treated the same in backpropagation. There- fore, the gradients of a in PACT is computed by:
@[W,a] aw LT ((, [W alW.a]
(W))(W)
if W > a if W S a
da =
(11)
da
0
where the first term is computed by STE and the second term is because the clip operation |. returns sign(-)a when . >>> a. In this gradient estimation, the effect of a in the levels set Q( b)
ignored by the STE, leading inaccurate approximation.
A.2 RCF
To avoid the elimination of STE, we reparameterize the clipping function so that the output clipping range before projection is settled and the range is re-scaled after the projection. We can define general formation RCF by
a W - - c
! Sw.cl, cl, a
(12)
where e is a constant. This function clips the weights to [-c, c] before projection and re-scaled to [-a, a] after projection. Thus, the backpropagation is given by:
ow
da 1 -o
- w
@l La W. WLWW,c
a
x
(13a)
da
da
WL.Wcel
da
c
sign( = W) x ct a x0
if W| a
(13b)
11 W c
(- W
if W| s
a
a W) sign
if >0
(13c)
II Mo W "w-1 w if
Since the levels set Q not parameterized by a, the gradients will flow two parts in RCF: the re-scale coefficient and the scale in clipping function. The constant here do not impact the gradient estimation, therefore we choose for simplicity in the implementation. Note that in uniform quantization scheme, this function equivalent to the Learned Step Size Quantization (Esser al., 2020), where the setp size the same for all levels while RCF provides a more general formation for any levels set Q.
B How DOES NORMALIZATION HELP QUANTIZATION
In this section, we show some experimental results to illustrate the effect our weights normaliza- tion in quantization neural networks.
12

---

Published as a conference paper at ICLR 2020
Epoch
Epoch
Epoch
Density
-0.50
-0.25
0.00
0.25
0.50
-0.50
-0.25
0.00
0.25
0.50
-0.50
-0.25
0.00
0.25
0.50
(a) Density distribution of unnormalized weights W during training
0.6
Epoch 0
Epoch 10
Epoch 90
0.5
0.4
Density
0.3
0.2
0.1
0.0
-2
(b) Density distribution normalized weights W during training
Figure 5: When weights are normalized the distribution of weights are more stable. The dashed line shows the value of weights.
B.1 WRIGHTS DISTRIBUTION
We visualize the density distribution weights before normalization W and after normalization W during training to demonstrate its effectiivness.
Figure 5a demonstrates the density distribution of the fifth layer of the 5-bit quantized ResNet-18. from which we can see that the density of the unnormalized weights could be extensive high (> 8) the centered area. Such distribution indicates that even tiny change of clipping threshold would bring significant effect on clipping when a is small, as shown in Figure 4a. which means a small learning rate for a is needed. However, the learning rate too small, the change of a cannot follow the change of weights distribution because weights are also updated according to Figure 5a. Thus it is unfavorable to train the clipping threshold for unnormalized weights, while Figure 5b shows that the normalized weights can have stable distribution. Furthermore, the dashed line in the figure indicates W usually do not have zero mean, which may not utilize the symmetric design of quantization levels.
B.2 TRAINING BEHAVIOR
The above experiments use normalization during training to compare the distribution of weights. In this section, we compare the training of quantization neural networks with and without normalization investigate the real effect WN. Here, we train a 3-bit quantized (full precision for activations) ResNet-20 from scratch, and compare the results under different learning rate for a. The results are shown in Table 4, from which we can find that if weights are normalized during training, the net- work can converge to descent performances and is robust to the learning rate of clipping threshold. However, if the weights are not normalized, the network would diverge if the learning rate for a is too high. Even if the learning rate is set to lower value, the network does not outperform the normalized one. Based on the training behaviors, the learning rate for clipping threshold without WN in QNNs need careful choice.
13

---

Published as a conference paper at ICLR 2020
Table Accuracy comparison of 3-bit quantized ResNet-20 CIFARIO.
LEARNING RATE
0.1
0.01
0.001
0.0001
W/ NORMALIZATION
91.6
91.7
91.6
91.8
W/O NORMALIZATION
0.2
0.2
62.8
84.7
0.4
Prg
Rn RCT
Clpy
Pny for QUM
0.3
Thig fir QEM
MSE
0.2
Cli En QEM
0.00
Layer
0.0
Laycrs
(a) 5-bit quantized ResNet-18
(b) 3-bit quantized ResNet-18
Figure summary of projection error and clipping error different layers.
C EXPERIMENTAL DETAILS
C.I REVISITING QUANTIZATION ERROR
Typically, quantization error (A) is defined as the mean squared error between weights W and W before and after quantization respectively, defined A E[W - -W2 This quantization error is composed of two errors, the clipping error Adip produced by [t, a and the projection error Aproj produced by Ne. L.e.
Dolip Aproj =
Σ (niil 1/
+
Σ (Mi ( |W
(14)
Previous methods (Zhang et al., 2018; Cai et al., 2017) seek to minimize the quantization error to obtain the optimal clipping threshold (i.e. a arg min, (Aetip + Aproo))) while RCF directly optimized by the final training loss to balance projection error and clipping error. We compare the Quantization Error Minimization (QEM) method with our RCF on the quantized ResNet- 18 model. Figure 6 gives an overview of the clipping error and projection error using RCF or QEM.
For the 5-bit quamtized model. RCF has much higher quantization error. The projection error ob- tained by RCF is lower than QEM and QEM significantly reduces the clipping error. Therefore, we can infer that projection error has a higher priority RCF When quantizing to 3-bit, the clipping error in RCF still exceeds QEM except for the first quantized layer. This means RCF can identify whether the projection is more important than the clipping over different layers and bit-width. Gen- erally, the insight behind is that simply minimizing the quantization error may not be the best choice and it more direct to optimize threshold with respect to training loss.
C.2 IMPLEMENTATION: DETAILS
The ImageNet dataset consists of 1.2M training and 50K validation images. We use a standard data preprocess in the original paper (He et al., 2016). For training images, they are randomly cropped and resized to 224 224. Validation images are center-cropped to the same size. We use the Pytorch official code to construct ResNets, and they are initialized from the released pre-trained model. We use stochastic gradient descent (SGD) with the momentum of 0.9 to optimize both weight parameters and the clipping threshold simultaneously. Batch size is set to 1024 and the learning rate starts from 0.1 with a decay factor of 0.1 at epoch 30,60,80, 10,,80,100,, 100. The network is trained up to 120 epochs and weight decay is set to 10-4 for 3-bit quantized models or higher and 2 x 10 for 2-bit model.
The CIFARIO dataset contains 50K training and 10K test images with 32x32 pixels. The ResNet architectures for CIFARIO (He et al., 2016) contains convolutional layer followed by 3 residual
'https://9 htte:://ggthbu...oeeee whwb..com/pttorree /vi of on htm/b//l///msttee/rrr///// i on on/models/ resnet.py
14

---

Published as a conference paper at ICLR 2020
blocks and final FC layer. We train full precision ResNet-20 and ResNet-56 firstly and use them as initialization for quantized models. All networks were trained for 200 epochs with a mini-batch size of 128. SGD with momentum of 0.9 was adopted to optimize the parameters. Learning rate started at 0.04 and was scaled by 0.1 at epoch 80,120. Weight decay was set to 10-1
For clipping threshold a, set 8.0 for activations and 3.0 for weights initial value when training a 5-bit quantized model. The learning rate of a is set to 0.01 and 0.03 for weights and activations, respectively. During practice, we found that the learning rate of merely does not influence network performance. Different from PACT (Choi et al., 2018b), the update of a in our works already consider the projection error, so do not require relatively large 12-regularization. In practice, the network works fine when the weight decay for a is set to 10 and may increase to 10 when bit-widthh is reduced.
15


Reviews:
[{"review_id": "rkxQ9BYvKr", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThe authors propose to compress Neural Networks (NNs) by quantizing their weights to sums of powers-of-two. This allows both to take the non-uniform distribution of the weights into account and to perform fast inference on dedicated hardware. \n\nStrengths of the paper:\n- The problem is clearly stated (in particular the two questions about the clipping operation and the quantization levels are clearly presented in the Introduction). Similarly, the authors alleviate what they call the \"rigid resolution\" problem by using sums of powers-of-two and clearly explain their choice. \n- The paper addresses the inference time, which is an important metric. Indeed, researchers tend to focus on the size of the compressed weights as this is easily measurable and not questionable. Inference time however depends on the hardware but is crucial as compressed models often that run on embedded devices are often required to run in real-time (say image detection models). The paragraph \"Computation\" in Section 2.1 is therefore useful for the reader.\n\nWeaknesses of the paper:\n- Both the reparameterization of the clipping function and the weight normalization before quantization approaches seem not novel to me, see for instance: \"Weight Normalization based Quantization for Deep Neural Network Compression\", Cai et al. \n- The experiments are lacking the widely used ResNet-50 baseline (and Table 4 should be in the main paper instead of the appendix). Other tasks such as Image Detection (Mask R-CNN) could also strengthen the impact of the paper. \n- Moreover, unless I am mistaken, the authors \"do not quantize the first and last layers\". While not quantizing the first layer has no impact on the compressed size (weights of size 7x7x3x64 = 36 KB for a ResNet-18), not quantizing the last layer (ie the classifier, weights of size 512x1000 = 1.95 MB) seems problematic. It is indeed challenging to quantize the classifier, however this adds an overhead of 1.95 MB. Assuming the ResNet18 is compressed with 2 bits/weight, the compressed part has a size of  44.6 MB/16 = 2.8 MB. Thus, with the classifier, its size is 2.8+1.95 = 4.75 MB, which is roughly a x10 compression factor.  \n\nJustification of rating:\nThe paper presents an interesting idea that accounts both for the inference time and the bell-shaped distribution of weights in NNs. However, the results must be pushed further (see \"weaknesses\"). \n\n\n=== After the rebuttal ===\n\nI thank the authors for their rebuttal. They answer to all the raised concerns with clarity and concision and performed additional experiments. \n\nIn particular, the authors:\n- perform additional ResNet50 experiments\n- quantize the input layer and the classifier to 8 bits\n- clarifying the positioning and in particular with respect to weight normalization\n\nThe paper is substantially better in its updated version. Therefore, I am happy to update my rating to \"weak accept\".\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "SJgDOp8L5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors proposed a novel quantization method, additive powers-of-two quantization and show both the uniform quantization and powers-of-two quantization are the special case of this method. To train such a quantized network, new clipping function, and weight normalization are proposed. The numerical results show that the proposed method only introduces a small accuracy loss and sometimes even improve accuracy. Compared to the other methods, it also shows better performance. Overall, I think this work is valuable and may be considered for publication. But the reviewer is completely out of this neural network quantization area, and thus not very familiar with the related works.\n\nThe following are some more detailed comments:\n\n1. On page 3, \"when we increase the bit-width from b to b+1 ... be split into 2^b subintervals.\" Based on Equation 3, I think it might be 2^(b-1) +1 subintervals.\n\n2. From Figure 1 (c), the introduced quantization method introduces non-monotonic interval steps, which is a little unintuitive. Can the authors explain if this can be further improved?\n\n3. A very important theme of this paper is on hardware, however, I feel the paper doesn't have a satisfying discussion on the hardware implementation. In order to make the argument more solid, the authors may want to provide more discussion on the hardware implementation trade-offs. Also, a detailed comparison of the quantization size comparison should also be provided.\n\n4. The authors claim the quantized network can sometimes achieve better performance. This statement needs to be further checked. Since the performance provided in the baseline Resnet models can be slightly improved with further training and training schedule tuning. Without a thorough optimization, such a claim might be misleading.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "SJe9wpdScS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThe paper presents an approach based on power-two quantization to compress the weights of neural networks. The authors elaborate on a coding scheme that adds several quantized values with different log scales (typically 2^(-2k) and 2^(-2k+1) for varying k, for the sum of two log scales). More importantly, they emphasize on the importance of weight scaling and learning the clipping threshold. experiments are carried out with several versions of ResNet on CIFAR-10 and ImageNet, and the authors show better performances than competing baseline for a fixed quantization budget per float.\n\nThe results are better than the non-compressed baseline (CIFAR with 3 and 5 bit/flotat, ImageNet with 5bit/float.) This seems surprising. Is there any clear reason why? Shouldn't the performance decrease as a result of the compression?\n\nWhile the results are good, a large part of the paper is dedicated to the idea of summing powers-of-two quantizers. The motivation comes from the sub optimality of uniform quantizers. Some ablation studies demonstrate that the scheme power-of-two is better than uniform. Nonetheless, the results in Table 2 indicate that uniform already achieves very good performances (already better than the non-compressed network for top-1 accuracy), so that most of the gain seems to come from weight normalization and learning the clipping threshold.  \n\nThus, the advantage of the exact scheme that is proposed compared to other methods is unclear to me. There already are well-known algorithms for non-uniform quantization (e.g., Lloyd). Lin et al. (\"Towards Accurate Binary Convolutional Neural Network\", cited in the paper) already addressed the non-uniform quantization by learning different binary bases. While the results of the paper are better than a number of previous results, it seems to me that most of the gain does not come from the quantizer but rather from the other elements of the method. \n\nThe contribution then seems a bit incremental, as it is mostly weight normalization +  straight through estimator to learn the clipping constant by minimizing directly the loss function. One way to make the contribution stronger is to isolate this part to see its effect more generally on several non-uniform quantization schemes.\n\nother comments:\n- is there any (possibly intuitive) justification for the fact that the results of the compressed networks are better than the non-compressed ones?\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a quantization scheme with the advantage of high computational efﬁciency. The experimental results show that the proposed scheme outperforms SOTA methods and is competitive with the full-precision models. The reviewers initially raised some concerns including baseline ResNet performance,  detailed comparison of the quantization size, and comparison with ResNet50. Authors addressed these concerns in the rebuttal and revised the draft to accommodate the requested items. The reviewers appreciated the revision and find it highly improved. Their overall recommendation is toward accept, which I also support.
Decision: Accept (Poster)