Towards Effective 2-bit Quantization: Pareto-optimal Bit Allocation for Deep CNNs Compression
Authors: Zhe Wang
Jie Lin
Mohamed M. Sabry Aly
Sean I Young
Vijay Chandrasekhar
Bernd Girod
Venue:  (2020)

Abstract: State-of-the-art quantization methods can compress deep neural networks down to 4 bits without losing accuracy. However, when it comes to 2 bits, the performance drop is still noticeable. One problem in these methods is that they assign equal bit rate to quantize weights and activations in all layers, which is not reasonable in the case of high rate compression (such as 2-bit quantization), as some of layers in deep neural networks are sensitive to quantization and performing coarse quantization on these layers can hurt the accuracy. In this paper, we address an important problem of how to optimize the bit allocation of weights and activations for deep CNNs compression. We first explore the additivity of output error caused by quantization and find that additivity property holds for deep neural networks which are continuously differentiable in the layers. Based on this observation, we formulate the optimal bit allocation problem of weights and activations in a joint framework and propose a very efficient method to solve the optimization problem via Lagrangian Formulation. Our method obtains excellent results on deep neural networks. It can compress deep CNN ResNet-50 down to 2 bits with only 0.7% accuracy loss. To the best our knowledge, this is the first paper that reports 2-bit results on deep CNNs without hurting the accuracy.

Under review as a conference paper at ICLR 2020
TOWARDS EFFECTIVE 2-BIT QUANTIZATION: PARETO-OPTIMAL BIT ALLOCATION FOR DEEP CNNS COMPRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
State-of-the-arr quantization methods can compress deep neural networks down 4 bits without losing accuracy. However, when it comes to 2 bits, the performance drop is still noticeable. One problem in these methods is that they assign equal bit rate to quantize weights and activations in all layers, which is not reasonable in the case of high rate compression (such as 2-bit quantization), as some of lay- ers in deep neural networks are sensitive to quantization and performing coarse quantization on these layers can hurt the accuracy. In this paper, we address an important problem of how to optimize the bit allocation of weights and activations for deep CNNs compression. We first explore the additivity of output error caused by quantization and find that additivity property holds for deep neural networks which are continuously differentiable in the layers. Based on this observation, we formulate the optimal bit allocation problem of weights and activations in joint framework and propose a very efficient method to solve the optimization problem via Lagrangian Formulation. Our method obtains excellent results deep neural networks. It can compress deep CNN ResNet-50 down to 2 bits with only 0.7% accuracy loss. To the best of our knowledge, this the first paper that reports 2-bit results on deep CNNs without hurting the accuracy.
INTRODUCTION
Deep Convolutional Neural Networks (CNNs) Krizhevsky et al.) (2012); Simonyan & Zisserman (2055;; Szegedy et al. 2015); He et al. 2016 have dominated the applications in computer vision and related fields. However, one challenging issue is that deep CNNs usually involve large number of parameters and are computationally demanding, making it very difficult to deploy deep CNNs on essource-limites devices. Quantization for weights and activations of deep CNNs is an effective way to mitigate this issue. Cooperated with advanced retraining (or fine-tuning) strategies, state-of-the- art methods Zhuang al. (2018); /hou al. (2016); Zhhng al. 2018) compress deep CNNs down to 4 bits without losing accuracy.
State-of-the-ee quantization methods Zhuang al. (0018) Zhou et (2016); Zhang al. (2018) typically assign same bit rate to all layers for compression. However, when deep neural networks compressed very low precision, e.g., 2 bits, such equal bit allocation scheme is not reasonable. As some of layers in deep neural networks are sensitive to quantization, performing coarse quan- tization on these layers may significantly hurt the accuracy. better strategy is to adopt unequal bit allocation scheme to quantize weights and activations across layers.
In this paper, we address an important problem of how to optimize the bit allocation of weights and activations of deep CNNs. Finding the optimal bit allocation for deep CNNs is difficult, as traversing the whole searching space requires exponential computation complexity. Assume is the number of quantization levels and L is the number of layers. A brute-force search has O(M4) computation complexity, which is unaffordable if L is large in the case of deep neural networks. In work, first analyze how quantization affects the output of deep CNNs, and find that the mean squared output eтror caused by quantization has additivity property. Based on this observation, formulate the optimal bit allocation problem of weights and activations in joint framework

---

Under review as a conference paper at ICLR 2020
and adopt Lagrangian formulation Shoham & Gersho (9888) for optimization. We propose very efficient method to find the solution with polynomial computation complexity.
Prior adaptive quantization schemes proposed by Khoram & Li (2018) and Zhou et al. (2018) only consider the compression of weights. Since the volume of activations is considerable, the size of activations should be also reduced much as possible, which might otherwise slow down process- ing. ZZuu et al. (2018) proposed layerwise adaptive quantization for weights and activations, but their method decides each layer's quantization level individually and does not optimize the bit allocation across layers when performing quantization on multiple layers. Different with prior literature, our work formulates the optimal bit allocation problem for both weights and activations and proposes an efficient method to solve the challenging problem. The new contributions of this paper are sum- marized as following:
Exploring Additivity Mean Squared Output Error: In this paper, we adopt mean squared output error to measure quantization impact. We observe that mean squared out- put error has additivity property, i.e., the mean squared output error caused by quantizing multiple layers equals to the sum of mean squared output error due to the quantization of each individual layer. We show empirical results on two deep CNNs AlexNet and VGG-16 and also provide mathematical derivation for the additivity property.
Pareto Condition for Optimization: Based on the additivity of mean squared output error, we propose an optimal bit allocation framework for both weights and activations. Moreover, we propose an efficient method to solve the optimization problem by using La- grangian Formulation. We show that the optimization problem can be solved in polynomial computational complexity under Pareto condition, i.e.. the slopes of mean squared output error vs bit rate functions of each layer have to be equal.
The pattern of Pareto-optimal bit allocation of weights deep CNNs has very positive impacts on inference rate. It tends to allocate fewer bits per weight for layers that have a lot of weights, which helps to reduce the corresponding memory-access time which in turn reduces compute idle time and improves the overall inference rate. Simulation experiments on Google TPU hardware accelerator shows that Pareto-optimal bit allocation can further improve the inference rate on deep CNNs by 1.5X compared to its equal bit allocation counterpart.
Impact on Improving Inference Rate:
Results on ImageNet Deng et al. (2009) show that our method can compress deep CNNs (c.g.. ResNet-50 He et al. (016)) down to 2 bits and the accuracy loss is less than 0.7%. To the best of knowledge, this the first paper that reports 2-bit results on deep CNNs without hurting the accuracy. The remainder of this paper is structured as follows. Section 2 discusses related works. Section 3 analyzes the impact of quantization. Section 4 develops our bit allocation framework. Section 5 reports experimental results. Section 6 discusses the impact of inference rate.
2 RELATED WORKS
Some latest works Zhuang et al. 2018); Park et al. (0018); Choi et al. (2018bla) already reported 2-bit results on deep CNNs. Among them, the highest result is reported by Park et al. (2018). which loses 2.45% accuracy at 2 bits on ResNet-50 He et al. (016)) Although promising results are obtained, there is still considerable accuracy drop at bits. Note that all these works assign same bit rate to quantize weights and activations in different layers. Different with them, our method adopts a better strategy for bit allocation. The optimal bit allocation strategy critical for maintaining high accuracy when neural networks are compressed to very low precision like 2 bits.
The most relevant works related to our unequal bit allocation scheme are proposed by Khoram & (018) and Zhou et al. (2018). Khoram & Li (2018) suggest monitoring the loss function during training to allocate the bit rates of weights in different layers. Zhou et al. (2018) suggest applying adversarial noise Fawzi et al. (201)) as indicator to measure the quantization impact for bit allocation. Both of these two works only study the quantization problem of weights.
Zhou et al. (2018) empirically finds that the adversarial noise Fawzi et al. (2016) has additivity property. While, adversarial noise a measurement of top-1 accuracy in object classification task, which is not applicable to other tasks like object detection. Our work studies the additivity of mean
2

---

Under review as a conference paper at ICLR 2020
A
A
A
A + 4 i=
4 i= 2
33
i
2
i -=-1
0
Figure I: An example of uniform quantizer with dead zone.
squared output error. We do not make assumptions about the application scenarios of our method We provide both empirical results and a mathematical derivation for the additivity property.
Banner et al. (2019) proposed per-channel bit allocation for weights and activations. They formulate the bit allocation problem minimization problem of Mean Squared Error (MSE) of quantized values intermedia layers, and provide a solution with analytic expression in the case Laplacian distribution. Finally. the solution of bit allocation with analytic expression is rounded to integer values. Mean Squared Error (MSE) in intermedia layers can not reflect network networks' final output accuracy. Our method adopts mean squared output error measure the impact last layer's output accuracy. It finds the optimal bit allocation directly from the discrete search space and does not make any assumptions for the distributions.
Recently, Zhe et al. (219)) adopted the rate-diste ditedistortion Lagrangian formulation to allocate bitrates for weights and activations across layers. Our method builds top of the Lagrangian formulation approach. Beyond that, we provide the analysis of the mathematical derivation of additivity property, which is the precondition of Lagrangian formulation. The proposed quantization framework differs from Zhe et al. (219) in two-fold. First, we adopt dead zone to the quantization function of weights. Second, we apply the straight-through estimator (STE) Bengio et al. (2013) to perform back-propagation
the retraining stage for both quantized weights and activations. As illustrated
in the experiment section, dead zone and STE retraining are critical for improving the accuracy. In addition, we show that the bit allocation solved by our approach has positive impacts at inference stage. It can largely speed up the inference rate compared its equal bit allocation counterpart. We verify this point by performing hardware simulation experiments Google TPU v1 platform.
ANALYSIS OF QUANTIZATION IMPACT
We adopt uniform scalar quantization in compression framework. Each layer's weights and activations are quantized separately, followed by entropy coding the quantization index. The quantizer stepsize is individually chosen for each layer's weights and activations.
We incorporate dead zone into the quantizers for weights. Let A denote the stepsize of a quantizer. The width of the dead zone is A + B. Let 2k denote the total number of representative levels of the quantizer. Given scalar input x, if a is in dead zone (wwl A+*)) it is quantized to zero. Otherwise, # is quantized to
((x) sign(x) ( + 1i1 -1)
where integer i is the quantization index, and < | S k. Both 3 and are hyper parameters. We note that adding a dead zone improve compression. This will be discussed in the following sections.
We do not apply dead zone to the quantizers of activations. We assume activation function after Convolution layer is ReLU, which common practice in modern deep CNNs Krizhevsky et al.) 2012); Simonyan & Zisserman (2015): He et al. (2016). The quantizer for activations is define as
g(x) =i-
where quantization index 0 11
3.1 MEAN SQUARED OUTPUT ERROR CAUSED BY QUANTIZATION
Let W; denote weights in layer i and A; denote activations in layer for all 1 < i < L where L is total number of layers. Given neural network JF and input image I, we have output vector
V F(T). If we quantize the weights W, obtain modified output vector V. The mean sqaured
3

---

Under review as a conference paper at ICLR 2020
AlexNet
VGG-16
true output eiror Dr
8
oultput error
0.5
1.5 W1
2
0.5
1
1.5
2
estimated output error D + Dr
estimated output error ED
Wt
D2
Demonstration of the additivity of mean squared output error on AlexNet and VGG-16. For each time, we randomly select some of layers and perform quantization on the selected layers, which corresponds a point in the figures.
Figure 2:
output error caused by quantizing W; is defined as the expectation of the squared Euclidean distance between original output V and modified output V divided by the dimensionality of V
D E(d(V. V)) dim(V)
(3)
where F(.) denotes the expectation operator and d(X, Y) the squared Euclidean distance between vectors X and Y. The expectation is over the distribution of all random input images 7. Similarly, DA denotes mean squared output error caused by quantization of activations A, for all 1 S < L. In the following, squared output error" and "output error" will both be used interchangeably.
3.2 ADDITIVITY OF MEAN SQUARED OUTPUT ERROR
Consider mean squared output error when simultaneously quantizing all layers. Let D, denote mean squared output error caused by quantizing all layer's weights and activations, the following additivity property holds.
Proposition 1 The output error Dy. caused by quantizing all layers' weights and activations, equals the sum of all output error due to the quantization of individual layer's 's weights and acti- vations
Dy Dy
D + DJ
(4)
the neural network is continuously differentiable in every layer and quantization errors in different layers are independently distributed with zero mean.
In practice, additivity holds approximately in deep neural networks. We evaluate additivity on AlexNet and VGG-16. Figure [ shows the results. Vertical axis in Figure 2 represents the out- put error when quantizing all layers simultaneously. Horizontal axis represents the sum of output that data points in Figure 2 are all
caused by quantizing each layer individually. We very close to the diagonal, meaning that additivity property holds. This property can be shown mathematically by linearizing the quantization error of weights and activations using Taylor series expansion as the neural networks is continuously differentiable and the quantization error can be considered small deviations. The supplementary material provides a mathematical derivation.
UNEQUAL BIT ALLOCATION FRAMEWORK
Let RW; and R.A: denote the total bitrates of weights and activations in i-th layer, respectively. RW, (or R.A.) is determined by the number of representative levels f, the dead zone parameter /8 and
4

---

Under review as a conference paper at ICLR 2020
1=-10
7=-1
=-0.3
output error
selected
output error
output error
selected
selected
bit rate
bit rate
rate
curves of output error bit rate functions
slopes
selected points
Figure 3: Example of finding optimal bit allocation under Pareto condition (best viewed in color).
the number of weights (or activations) in layer i. According to the output error of quantizing all layers' weights and activations the sum of the output error caused by quantizing each layer's weights and activations individually. The bit allocation framework is thus formulated as
arg min RA,
D +D + DW +DA
(5)
s.l. R RA +...+ R R RTotal
where Rotal is the total number of bits needed to represent the weights and activations of all layers. Our goal is to find the optimum bit allocation { [RW:) and } to minimize network output error within constrained total bit budget for weights and activations. More precisely. we adjust representative level and dead zone parameter 3 of the quantizer in each layer to minimize the overall output error of the quantized network given total bit budget constraint.
4.1 PARETO-OPTIMAL BIT ALLOCATION
We classical Lagrangian rate-distortion formulation find the optimum bit allocation Shoham & Gersho (988). Our Lagrangian cost function is
J D DD Σ (R.: +
(6)
By setting the partial derivatives of I with respect to RW: and RA, to zero, we obtain
aD
aD
0R =\
ORW.
for all 1 S i,j < L. The above equation expresses the intuitively pleasing insight that the slopes of output error versus rate functions for weights and activations of each layer have to equal. This exactly the well-known Pareto condition for optimal resource allocation in economics. 7 holds, as long the output eтror contributions Dj: and D are additive, rates RW, and RA: are additive, the functions D. (RW:) and D (RAi) convex and the resulting RW; RA: >> 0. does make assumptions about the specific quantization or coding scheme used.
4.2 OPTIMIZATION UNDER PARETO CONDITION
We find optimum bit allocation under Pareto condition according to first generate rate- distortion curves of (RW; D) and (RA,, DA by choosing different values for representative level f and dead zone parameter 3.. Then we enumerate À and select the point in each curve with slope equal to A. The set of selected points is one solution of Pareto-optimal bit allocation. We may enumerate many values of À to find the best solution with minimal output error under the bit budget constraint. Assume that we have N curves and M points in each curve. The whole search space of the solution is O(M By utilizing the Pareto condition, the time complexity find optimum bit allocation is reduced to O(K M N). where K the total number evaluated. In practice, we can find the optimum bit allocation for ResNet-50 5e al. (2016) and Mobile Howard al.
5

---

Under review as a conference paper at ICLR 2020
Nfrbbero Weights
Nunte Nerbo
Namber Activiiooss Lesers
Avergge Ditral acroes Lagers
Avengg Kittis somss ligee
15
15
16
16 & 4 & Layers of Weights
4 4
Layers of
Figure 4: Pareto-optinge bit allocation of weights and activations across layers ResNet-50 when compressed into 2 bits on average.
(017) in several seconds on standard CPU (AMD Ryzen threadripper 1950x) using the algorithm described above.
Figure g illustrates the optimization steps under Pareto condition. Each blue curve in Figure denotes a rate-distortion curve in one layer. We totally show rate-distortion curves in Figure The optimization starts from enumerating the value of slope A. Given À (dotted red lines in Figure then find the point with slope equal to for each curve. Specifically, we compute the intercepts on Y-axis of the lines with slope equal to X, which pass the points the curve. The point with minimal intercept on Y-axis will be selected as solution. Note that we only select one point with minimal Y-axis intercept for each curve. The selected point corresponds to the bitrate in the related layer. We start À from very small value and increase À gradually until the total rate reaches the target. The example in Figure totally evaluates three different values of X.
4.3 OBSERVATIONS ON OPTIMUM BIT ALLOCATION
Figure 4 shows the optimum bit allocation of weights and activations according to on ResNet-50 He et al. (2016), when weights and activations are compressed to 2 bits on average. Two observa- tions stand out. First, weights receive much larger bitrates (average bits per layer) than activations in general. Second, the layers with larger number of weights receive relatively lower bitrates; con- versely, the layers with small number of weights receive relatively high bitrates. Similar trends observed on MobileNet-vi Howard et al. (017)) 7), which are illustrated in the supplementary material. We noticed that such bit allocation pattern weights can have positive impact on inference rate. As we allocate relatively small bitrates to large-weight layers, the corresponding memory time will be reduced which in turn reduces compute idle time and improves the overall inference rate (see Section 6).
5 EXPERIMENTS
We evaluate our method on ResNet-50 al. (2016) and MobileNet-vl Howard et al. (2017) over the ImageNet Deng et al.] (2009) validation dataset with 224x224 input image resolution. The total number of weights and activations of ResNet-50 is 32.3M with 22.7M weights and 9.6M activations, and the total number of weights and activations of MobileNet-vi is 9.3M with 4.2M weights and 5.0M activations. We adopt the Tensorflow implementation of ResNet-50 and MobileNet-vl pro- vided by the Tensorpack and SLIM library, respectively. The original ResNet-50 and MobileNet-vi have 76.4% and 70.9% top-1 image classification accuracy on the ImageNet validation dataset, re- spectively.
We compare our method with 15 sate-of-theear methods, including Binarized Weight Network (BWN) et al. (2016), Ternary Weight Network (TWN) et al. (2016), Incremental Network Quantization (INQ) Zhou et al. (2017). Fine-grained Quantization Mellemudli et al. (017))
Integer Arithmetic-only Inference (IAOI) Jacob et al. (2018), Adaptive Quantization (AQ) Zhou
ct al. (0018). Compression Learning by In-parallel Quantization (CLIP-Q) Tung & Mori (2018).
Symmetric Quantization (SYQ) Faraone et al. (2018), Liw-bitwidth CNN (LBCNN) Zhuang et al.
(2018), Extremely Low Bit Neural Network (ELBNN) Leng et al. (017)) Learned Quantization
(LQ-Nets) Zhang al. 2018)) Deep Compression (DC) Han et al. (2016), Hardware-Awaan Au- tomated Quantization (HAQ) Wang et al. (2019), Parameterized Clipping Activation (PACT) Choi
ltt al. (2018b), DooeFa-Netthhou et al. (0116), and Joint Bit Allocation (JBA) The al. 2019 It

---

Under review as a conference paper at ICLR 2020
75
60
30
Top- Accuracy
OFigall Ronk
80
Sepllli lir Alaites Fiarayy
-0-0P Oydked Albicaas
<3 (peial lat Alxaton Seneyy Collng
6- Optidd Alboiin Eaier Eoligg bud
Overal Abeace Ternyy Coday Deal Rerestr
20
16
Total Sizee of Weights Activations (MB)
Total Size of Weights and Activations (MB)
(a) Results
(b) Rexults on MobileNetovl
Figure 5: Effectiveness of each component in the proposed compression framework. Results are shown over the ImageNet validation dataset when weights and activations are compressed differ- ent sizes. The size of original ResNet-50 and MobileNet-vl are 116 MB and 35 MB, respectively.
73
72
Top-1 Accursey (%)
67
TWN
INO
LADE
Original Resul:
A1)
(LISO
4M-MO (Latescy costsicy-osttiieee
SYQ
LBCNN
-GHAA (langy-mmenins))
LQ-No
Rlmbiiggg
42 -2-mn
57
43
16
32
16
Total Size Weights and Activations (MB) (a) Results on ResNet-50
Total Size Weights and Activations (MB)
(b) Results on MobileNet-ri
Figure 6: Tradeoff between size and accuracy on ResNet-50 and MobileNet-vl over ImageNet validation dataset comparing with state-of-thhe-art methods from the literature.
worth noting that the top performing comparison methods all include retraining recover accuracy loss due to quantization error introduced.
5.1 EFFECTIVENESS OF THE PROPOSED COMPRESSION FRAMEWORK
In this section, we evaluate the effectiveness of each component in our compression framework.
Optimal Bit Allocation. As illustrated in Figure show the performance of equal bit allocation as a baseline to evaluate the effectiveness the proposed optimal bit allocation scheme. Equal bit allocation assigns same quantization level for all layers' weights and activations. In Figure there is a large gap between the results of optimal bit allocation (green curves) and the results of equal bit allocation (blue curves). Optimal bit allocation is much more effective.
Entropy Coding and Dead Zone. Applying entropy coding to quantization index can further reduce the size by 20% to 30%. We use the entropy of the quantization index to estimate the code size after entropy coding. In Figure [ the dashed green and blue curves show the results of quantized networks with equal and optimal bit allocation before entropy coding, respectively. After entropy coding, the size of quantized networks can be further reduced (see the solid green and blue curves in Figure 5). Besides, with dead zone, we also observe large performance improvement on ResNet- 50 and noticeable improvement on MobileNet-vl. Adding dead zone to the quantizers forces more weights to zero. This benefits entropy coding, as it makes the distribution of quantized weights more peaky. A dead zone is similar to pruning Han et al. (2015) 2016); both set small weights to zero. The key difference is pruning completely removes the small weights from networks while dead zone still keeps them.

---

Under review as a conference paper at ICLR 2020
Table Comparing our method with -fthe-off methods on ResNet--50 and MobileNet-vl over ImageNet validation dataset when weights and activations are compressed to low bitrates.
Original
2 Bits
4 Bits
Models
Methods
Top
Top 5
Top
Top
Тop 1
Top 5
DoRel -eee Zhou et al. 2016)
75.6
92.2
67.3
84.3
74.5
91.5
LBCNN Zhuang et 2018
75.6
92.2
70.0
87.5
75.7
92.0
ResNet-50
LQ-Nets Zhang al. 2018)
76.4
93.2
71.5
90.3
75.1
92.4
JBA Zhe et al. (2019)
75.2
92.2
74.2
91.5
Ours (No Retraining) Ours
76.4
93.2
66.7
87.5
76.2
93.1
76.4
93.2
75.7
92.8
76.4
93.3
PACT Choi et al. (2018b)
70.8
89.9
62.4
84.2
HAQ (energy)
Wang et al.
2019)
70.8
89.9
64.8
85.9
MobileNet-vi
HAQ (latency) Wang et al. Ours (no retraining)
(2019)
70.8
89.9
67.5
87.9
70.8
89.9
2.1
6.0
68.9
88.6
Ours
70.9
89.9
27.3
50.7
70.1
89.2
Retraining. Like other schemes, compression framework be enhanced by retraining, espe- cially at very low bitrates. We adopt the traight-thhougg estimator (ETS) Bengio et al. (2013) to perform back-propagation through non-differentiabll quantization functions. We retrain 10 epochs for both ResNet-50 He et al. (2016) and MobileNet-vil Howard et al. (0117). For ResNet-50, we set the learning rate as 0.1 if bitrate is more than 3 bits, otherwise, learning rate 0.001. For MobileNet- v1 Howard et al. (0177) learning rate is 0.45, Without retraining, our method obtains 66.7% 7% on
ResNet-50 2 bits on average. The simple retraining stage improves accuracy 75. 75.7% close to that of the original pre-trained model (76.4%).
5.2 COMPARISON WITH STATE-OF-THE-ART METHODS
The proposed compression framework achieves excellent performance on ResNet-50 and MobileNet-vl, compared recent state-of-the-art methods. Figure [ shows the tradeoff between size of weights/activations and classification accuracy the ImageNet validation dataset. On MobileNet-vI, our method without retraining outperforms statc-of-the-a approaches equipped with retraining. Similar observation can be made for ResNet-50.
Table further compares our method with ttate-of-the-a that report results at very low bitrates (average 2 and 4 bits). At 4-bit average, without retraining, method only loses 0.2% Top-1 accuracy on ResNet-50. This is strong evidence of the effectiveness of the proposed compression framework. With simple retraining, it outperforms the best performing baseline by 4.2% Top-1 accuracy at 2 bits. On MobileNet-vI, our method outperform the best performing baseline by 2.6% Top-1 accuracy at 4 bits. Further improvements are possible if combining our unequal bit allocation scheme with advanced quantized model retraining approaches like progressive training and knowledge distillation training Zhuang et al. 2018).
method shows additional accuracy gains. On ResNet-50,
We compare our method with the very recent work JBA Zhe et al. 2019 At bits on ResNet-50, our method without retraining outperforms JBA by 2.0% (76.2% 2% 74.2). As observed in Figure from the paper /Ze et al. 20199, the results JBA Zhe et al. 2019) drops more than 15% bits. While, our method without retraining drops 9.7% accuracy 2 bits, which significantly outperforms JBA, which is attributed to the dead zone quantization. With retraining, our method only loses 0.7% accuracy at bits, outperforming JBA by more than 10.0%.
We also report our results 1.5 bits. We noticed that the quantized networks with equal bit allocation at bits without entropy coding can be further compressed to around 1.5 bits after entropy coding. Our method obtains 72.5 Top-1 accuracy at 1.5 bits on ResNet-50, which still outperforms other state-of-the-ee methods 2 bits by 1.0%.
8

---

Under review as a conference paper at ICLR 2020
Table 2: Architecture parameters of considered deep learning hardware platforms.
Paramter
MIT Eyeriss
Google TPU
On-chip memory
181.5 KBytes
28 MBytes
Off-chip memory- access bandwidth (BW)
GByte/sec
13 GBytes/sec
Computing performance (Perf)
34 GOPs
96 TOPs
Table 3: Inference rates of equal and unequal bit allocation schemes, as well as full precision, on ResNet-50 and MobileNet-vl, TPU uses 8-bit precision while MIT Eyeriss uses 16-bit precision.
2 Bits
4 Bits
Speedup vs. full precision
Platform
Model
equal
ours
speedup
equal
ours
speedup
2 bits
4 bits
ResNet
833
1250
1.5x
625
666
1.07x
4x 2.1x
TPU
Mobile
892
1175
1.3x
979
979
1x
1.8x
1.5x
ResNet
8
8
8
1x
1.05x
1.05x
Eyeriss
Mobile
12
12
1x
12
12
1x
1.05x
1.05x
DISCUSSIONS
6.1 ANALYSIS OF COMPUTATIONAL COMPLEXITY
We compare the computational complexity of method and the equal bit allocation method, by counting the number of arithmetic operations required for neural network inference with batch size 1. Following the protocol defined by MicroNet challenge ! consider 32-bit multipli- cation/addition as one operation. For sexeedprecision computations (e.g.. bit weight times 5 bit input), we count the number of of operations be the maximum bit-width of the 2 operands of this operation, divided by 32. For example, a multiplication operation between 3-bit and 5-bit operand will count as 5/32 of an operation.
On ResNet-50, our method has less amount of operations than the equal bit allocation method when weights and activations are compressed bits (7.0 x 10 vs 7.2 10") and bits (9.4 x 10 vs 10.8 x 108). At 2 bits, our method has 1.4x more operations than the equal bit allocation method (5.1 10 3.6 x 10%).
In practice, the inference time on hardware platforms is constrained by both compute and memory access. The Pareto-optimal allocation tends to allocate fewer bits per weight for layers that have lot of weights, which helps reduce the corresponding memory-access time which in turn reduces compute idle time and improves the overall inference speed. We show the simulation results on hardware platforms in the following section.
6.2 IMPACT ON INFERENCE RATE
We define inference rate as the maximum number of images that a particular implementation of neural network can process per unit time. Inference rate is mainly driven by memory- bandwidth and compute throughput (number of compute operations per unit time). In our work, weights and activations are fetched from off-chip to on-chip memory, where they are decoded and fed into the processing units.
We have explored the impact our approach has on the inference rate for different hardware archi- tectures inspired by current embedded and high- high-performance DNN- -NN-ttageted hardware accelerators (i.e., MIT Eyeriss Chen et al. (0016) and Google TPU Jouppi et al.) (2017), summarized in Table such platforms have shown superior inference rate compared to GPUs Jouppi et al. (217)) We assume that accessing off- memory can be done in parallel while computing is performed (i.c., weights be prefetched from off-chip memory).
Using SCALE-sim Samajdar et al. (2018) software, we simulate the inference rate of ResNet-50 and MobileNet-vl, when mapped into the two considered hardware platforms in Table 2 We report the
https: wwwnettoontteeeeeeeeee challenger lenge.

---

Under review as a conference paper at ICLR 2020
inference speed-up resulting from unequal bit allocation scheme versus the equal bit allocation scheme (all weights and activations are represented with fixed number of bits), when running an inference single 224 x 224 input image. We also report the speedup compared to a full precision allowed on each platform (i.e., 8-bit on TPU, 16-bit on Eyeriss).
Our unequal bit allocation achieves 1.5x and 1.07x higher inference rate for ResNet-50 (1.3x and 1x for MobileNet-vI) at bits and 4 bits, as summarized in Table when compared with their equal bit allocation counterparts, respectively. Higher speed-ups, up to 4x, are achieved when compared with 8-bit full precision. One may note that the speedup is much larger if we consider inference time contributed by memory access only.
These improvements are mainly attributed to the unequal bit allocation pattern -i minimizes the layer-wise bitrates for weights/activations and also balances the size of data moving between off- chip and on-chip memory, which leads to reduced access times to off-chip memory. It is worth mentioning that with higher memory-access bandwidth, lower speed-ups are observed. For instance, at 128 GBytes/s, no speedup is observed. However, such hardware modules consume high energy and are rather costly, where are our approach provides an attractive alternative to increased memory bandwidth. Furthermore, platforms with low computation rate (i.e., few GOPs, as in the case of Eyeriss in Table 2), are constrained by the computational throughput, hence may not benefit from any speed-up.
7 CONCLUSIONS
In this paper, we have proposed novel unequal bit allocation framework for compression of both weights and activations deep CNNs. The output error due to the quantization of individual layer's weights and activations is additive, as the bits used to represent each layer. Building on this observation, we formulate a Lagrangian optimization framework to find the optimum bit allocation. be efficiently solved by utilizing the Pareto condition,
The optimal bit allocation problem ducing the search space from exponential complexity to linear complexity. Our method achieves excellent results ResNet-50 and MobileNet-vl. In particular, our method can compress ResNet- 50 into 4 bits on average without retraining, which is computationally very expensive and might not even be possible, if the original training data are not available. The proposed unequal bit allocation scheme has very positive impact on inference rate. It is able to improve inference rate by up to 4x compared with full precision, as well as 1.5x compared with its equal bit allocation counterpart, on Google TPU hardware accelerator.
REFERENCES
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid-deployment. In arXiv, 2019.
Y. Bengio. N. Leonard, and A. Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. In arXiv: 300..333, 2013.
Yu-Hsin Chen, Tushar Krishna, Joel S Emer, and Vivienne Sze. Eyeriss: An energy-efficient re- configurable accelerator for deep convolutional neural networks. IEEE Journal of Solid-State Circuits, 52(1): 2016.
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang. Swagath Venkataramani, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural net- works (qnn). In arXiv,, 2018a.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. In arXiv, 2018b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
Faraone, N. Fraser, M. Blott, and P. Leong. SYQ: Learning symmetric quantization for efficient deep neural networks. In CVPR, 2018.
10

---

Under review as a conference paper at ICLR 2020
A. Fawzi, S. Moosavi-Dezfoolli and P. Frossard. Robustness of classifiers: from adversarial to random noise. In NIPS, 2016.
S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural network. In NIPS, 2015.
Han, H. M., and W. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. In ICLR, 2016.
K. He, X. Zhang, Ren, and J. Sun. Deep residual learning for image recognition. CVPR, 2016.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In arXiv: /704.04001, 2017.
I. Hubara, M. Courbariaux, D. Soudry, R. El- Yaniv, and Y. Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. In (roX.0::609070000 2016.
B. Jacob, S. Kligys, B. Chen, M. Zhu, M. Tang, A. Howard, H. Adam, and D. Kalenichenko. Quantization and training of neural networks for efficient intege--rrthmetii-only inference. In CVPR, 2018.
N. Jouppi et al. In-datacenter performance analysis of a tensor processing unit. In Proceedings of the 44th Annual International Symposium on Computer Architecture, ISCA '17, pp. 1-12, New York, NY, USA, 2017. ACM. ISBN 978-1-4003-4892-88 doi: 00011445007985633000446 URL http:: acm. htt......r/.....1153377350000500
3080246)
S. Khoram and J. Li. Adaptive quantization of neural networks. In ICLR, 2018.
A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In NIPS, 2012.
C. Leng, H. Li, S. Zhu, and R. Jin. Extremely low bit neural network: Squeeze the last bit out with admm. In CVPR, 2017.
Li, B. Zhang, and B. Liu. Ternary weight networks. In arXiv preprint 1605.0/605.041/1 2016.
N. Mellempudi, A. Kundu, D. Mudigere, D. Das, B. Kaul, and P. Dubey. Ternary neural networks with fine-grained quantization. In arXiv preprint 17X.v:7705.01463 2017.
Eunhyeok Park, Dongyoung Kim, Sungjoo Yoo, and Peter Vajda. Precision highway for ultra low- precision quantization. In arXiv, 2018.
Ananda Samajdar, Yuhao Zhu, Paul N. Whatmough, Matthew Mattina, and Tushar Krishna. Scale- sim: Systolic CNN accelerator. CoRR, abs/18111.02 1.02883, 2018. URL h http:://rxxvv.rrgg
abs 02883
Y. Shoham and A. Gersho. Efficient bit allocation for an arbitrary set of quantizers (speech coding).
In IEEE Transactions on Acousties, Speech, and Signal Processing, 1988.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In ICLR, 2015.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Anguelov, D. Erhan, V, Vanhoucke, and A. Rabinovich. Going deeper with convolutions. In CVPR, 2015.
F. Tung and G. Mori. CLIP-Q: Deep network compression learning by in in-parallel pruning quantiza- tion. In CVPR, 2018.
Kuan Wang Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. HAQ: Hardware-aware automated quantization with mixed precision. In CVPR, 2019.
D. Zhang, J. Yang, D. Ye, and G. Hua. LQ-Nets: Learned quantization for highly accurate and compact deep neural networks. In ECCV, 2018.

---

Under review as a conference paper at ICLR 2020
Wang Zhe, Jie Lin, Vijay Chandrasekhar, and Bernd Girod. Optimizing the bit allocation for com- pression weights and activations of deep neural networks. In ICIP, 2019.
A. Zhou, A. Yao, Y. Guo, L. Xu, and Y. Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. In arXiv preprint arXiv:1702.03044, 2017.
Zhou, Y. Wu, Z. Ni, Zhou, H. Wen, and Y. Zou. Dorefa-net: Training low bitwidth convolu- tional neural networks with low bitwidth gradients. In arXiv preprint 1606.066 2016.
Y. Zhou, Seyed-Mohsen Moosavi-Dezfoolii Ngai-Man Cheung, and P. Frossard. Adaptive quanti- zation for deep neural network. In AAAI, 2018.
X. Zhu, W. Zhou, and H. Li. Adaptive layerwise quantization for deep neural network compression. In IEEE International Conference Multimedia and Expo (ICME), 2018.
B. Zhuang, C. Shen, M. Tan, L. Liu, and Reid. Towards effective low-bitwidth convolutional neural networks. In CVPR, 2018.
12

---

Under review as a conference paper at ICLR 2020
SUPPLEMENTARY MATERIAL TOWARDS EFFECTIVE 2-BIT QUANTIZATION: PARETO-OPTIMAL BIT ALLOCATION FOR DEEP CNNS COMPRESSION
Anonymous authors Paper under double-blind review
ABSTRACT
In this supplementary material, we provide the mathematical analysis for the ad- ditivity output error, and the additional results of Pareto-optimal bit allocation. Section [e provides the proof of Proposition [] Section 2 provides the results of Pareto-optimal bit allocation on ResNet-50 He et al. (2016) and MobileNet-vl Howard et al. (2017) at different bitrates.
ADDITIVITY OF OUTPUT ERROR
X
Fx W1)
a1
F2(x, W2)
y
Neural Network F
72
n2
T2
(1)
+
Fx,,, W2
+
a1
a1 + n
a2
72
ag
Modified Neural Network F
Figure Examples of neural network F and modified neural network F.
Proposition 1 The output error Dy. caused by quantizing all layers' weights and activations, equals the of all output error due the quantization of individual layer's weights and acti- vations
Dy = +.. DDA +... D
(1)
if the neural network is continuously difserentiable in every layer, the quantization errors in different layers independently distributed with mean.
Proof Let F(W1...,,W! WL) denote neural networks and F(W1,..., WL, Di, n2) denote modified neural networks of F where an element-wise add layer with parameter n, is followed for each activation a. Based this definition, have
J(W,,.. Wt. = F(W,, WT, 0, O,..,
(2)
Let X (W,,... Wr, ...., 0) and AX (AW,,... AW,, Ani,... Am)) Assume that the quantization error can be considered as small deviation. We apply the Taylor series expansion up to first order term on F a Xo.
L OF - = Σ AW, +
L OF Ow,
F(Xo + +AX) F(X,)
Ani-
(3)
Ong
1=1

---

Under review as a conference paper at ICLR 2020
Then, [IF(Xu + +AX= - 1(? L
can be written
OF
T
OF
aF
OF
Σ AW,
+
An.
AW; +
An:)
(4)
OW;
On;
aw;
On:
Because we assume that quantization errors in different layers are independently distributed with zero mean, the cross terms of (4) disappear when taking the expectation. That is:
OF OF aw;
OF
OF
B(AW,
AW. = E(AW,))
E(AW.) = (i + j)
aw,
aW,
OW.
as is the case also for the cross products between W; and ny (all i, j), and and nj (i # j). Then, we can obtain
- E OF aw; AWal.)
OF Am.ll?).
= ΣE
E AWal*)+
+
(5)
On:
(5) is the result we want because, again, according to the Taylor series expansion (ignoring the higher order terms),
0F
AW, = F(W,. W. + AW, ...... 0) F(W,, O,.0,....0 0),
aw;
OF
An, F(W,,,,.... WL. a.. An..... 0) FFWFF Wz, O).
Om:
Therefore, after dividing both sides of (5) by the dimensionality of the output vector of the neural network, the left side of (5) becomes Dy and the right side of (5) becomes the sum of all output error due to the quantization of individual layer's 's weights and activations.
2 PARETO-OPTIMAL BIT ALLOCATION ON RESNET-50 AND MOBILENET-VI
Figure shows the Pareto-optimal bit allocation of weights and activations on ResNet-50 He et al. (216)) and MobileNet-v1 Howard et al. (201)) when weights and activations are compressed bits and 4 bits average. Results are generated on ImageNet dataset Deng et al. (2009). Two obser- al. (2016) and MobileNet-v Howard al. (2017). First, weights receive much larger bitrates (average bits per layer) than activations in general. Second, the layers with larger number of weights receive relatively lower bitrates; conversely, the layers with small number of weights receive relatively high bitrates.
vations stand out on both ResNet-500
The bit allocation pattern for weights related to the variance of weights across layers. According to the classical Lagrangian rate-distortion formulation, the optimal bit allocation has positive corre- lation with the variance of the information source. Figure 3 shows layer-wise relationships between bitrate and variance of weights on ResNet-50 and MobileNet-vl. One 50 and MobileNet-vl, the layers with larger number of weights typically have smaller variances, and thus these layers receive smaller bitrates (and vice-versa).
that, for both ResNet-
REFERENCES
Jia Deng. Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In CVPR, 2009.
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
A. G. Howard, M. Zhu, B. Chen, D. Kalenichenko, W. Wang, T. Weyand, M. Andreetto, and H. Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. In arXiv:1704.04480 2017.
2

---

Under review as a conference paper at ICLR 2020
10
0
Vvvergee Bitrate across Layers
Namberr W nights acissa Lapees
20
sagunny
Number of Activators acrosa Layets
varigge vaross Layers
20
Number of Acdivatioess
6
15
Bitrates
15
10
% 1asa
de by ?o 2/
6 to
16 10
Layers of Weights (a) Bit Allocation of Weights and A ctivations on ResNet-50 at Bits on Average
Layers of Activations
10
NNotbbrr of Welgits somss Logen
pp Mquuin
Namber Adtivations
mAvenge
20
Eitrates
15 (100K)
& Layers of Weights (b) Bit A llocation of Weights and Activations on ResNet-50 Bits on Aver Average
4
Layers Activations
10
18
i.Ntmber of Wiiigtt actso actos Lapers Avvengge Layees
mAvange Brzzzo cames Lapers
70
Numbe of Activations
Eitrates
12 (10EK)
6 Layers of Weights (c) Bit A llocation Weights and Activations on MobileNet-vi at 2 on Average
Layers Activations
10
in Nunuarr of Weggs aro Lajes Avenage acta
WNuberr Loges
Number Addivatios
10
mAvrsge Brzza Layar
70 -5
Fitralos
Eltrates
(0K) 1a970
(10KK)
R
15
Layers of Weights
Lay of Activations
(d) Bit A Alocation of Weights and Activations on MobileNet-v1 at Bits on Average
Figure 2: Pareto-optimal bit allocation of weights and activations across layers on ResNet-50 and MobileNet-vl when weights and activations are compressed to 2 bit and 4 bits on average.
0.1
10
10
Average Bitrate across Layers
Average Bitrate across Layers Variance W Weights acress Layers
Vuriance of Weights across Layers
0.01
Bitrates
Bitrates
0.1
Variance
0.001
0.01
0.001
0.0001
0.0001
0.00001
0.00001
1 %
12 ly 2 30
Layers
Layers (b) MobileNet-vl
(a) ResNet-50
Figure 3: The relationship between the variances and the bitrates of weights across layers ResNet- 50 and MobileNet-vl when compressed to 2 bits average.
3


Reviews:
[{"review_id": "rylTYKApFB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Very good paper that studies the error rate of low-bit quantized networks and uses Pareto condition for optimization to find the best allocation of weights over all layers of a network. The theoretical claims are strongly supported by experiments, and the experimental analysis covers state-of-the-art architectures and demonstrates competitive results. The paper in addition also analyzes the inference cost of their approach (in addition to the accuracy results), and shows positive results on ResNet and MobileNet architectures.\n\nThe paper primarily shows that the mean squared error of the final output of a quantized network has the additive property of being equal to the sum of squared errors of the outputs obtained by quantizing each layer individually. Although there is no reason why this should be case, experimental results from the authors on AlexNet and VGG-16 validate this. Based on this assumption, the authors then use a Lagrangian based constrained optimization to minimize the sum of squared errors of outputs when individual weights/activations are quantized, with the constraint being the total bit budget for weights and activations. The authors show that this can be optimized under the Pareto condition easily.\n\nThe experimental section is quite detailed and covers the popular architectures instead of toy ones. The accuracy results compared to other 2-bit and 4-bit approaches are competitive. It's also nice to see analysis of inference cost where unequal bitrate allocation performs better than other methods.\n\nThe authors show that given the constrained optimization, layers that have a large number of weights receive lower bitrates and vice-versa. While it makes sense that this would contribute to stronger inference speedup compared to methods with either equal bitrate allocation across layers or those that allocate higher bitrate to layers with large number of weights, it's not entirely clear why the optimization would produce this allocation in the first place. Do the authors mean to conclude that layers with large number of weights hold a lot of redundancy and don't have a significant impact on the overall accuracy of the model? This needs to be clarified further.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "rJx2sPqAFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work nicely proposes a new theoretically-sound unequal bit allocation algorithm, which is based on the Lagrangian rate-distortion formulation. Surprisingly, the simple Lagrange multiplier on the constraints leads us to the convenient conclusion that the rate distortion curves for the weight quantization and the activation quantization have to match. Based on this conclusion, the authors claim that their search for the best bit allocation strategy is with a less complexity.\n\nI found this paper interesting and enjoyed reading it. However, I wish the paper could address some issues that are a little bit confusing to me. \n\nFirst of all, the paper is not about 2bit quantization. It seeks an “average” 2bit quantization. It means that some weights in some layers can be quantized with higher or lower bits per weight.  Same story goes on for the activation quantization. I don’t exactly know the implication of this, but it seems that the hardware implementation of a convolution layer could be either too complicated to benefit from this quantization scheme, or doesn’t really improve the efficiency of, say 4bit quantization for all layers. Is it really more efficient to do multiplication-and-addition between 2 bit weights and 5 bit input (the output of the previous activation) than between 4bit weights and 4bit input? I’m not a hardware person, but this part needs to be clearly addressed. Storage-wise, lowering the bitrate might be a clear benefit (I guess)\n\nI wish the actual optimization part briefly mentioned in section 4.2 could be elaborated more. It is a crucial part but somewhat understated. \n\nI also wonder what’s the effect or limitation of using MSE for this optimization, where cross-entropy is a more suitable choice. I know that the objective function in eq 5 is just to find the best combination of bit allocations per layer, but still, the error space might not be the best for this classification problem. ", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeAzEJQqS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This works presents a method for inferring the optimal bit allocation for quantization of weights and activations in CNNs. The formulation is sound and the experiments are complete. My main concern is regarding the related work and experimental validation being incomplete, as they don't mention a very recent and similar work published in ICIP19 https://ieeexplore.ieee.org/document/8803498: \"Optimizing the bit allocation for compression of weights and activations of deep neural networks\". A reference in related work as well as a comparison in experimental validation would be necessary  and the novelty of this work is rather weak given the above mentioned 2019 publication.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This works presents a method for inferring the optimal bit allocation for quantization of weights and activations in CNNs. The formulation is sound and the experiments are complete. However, the main concern is that the paper is very similar to a recent work by the authors, which is not cited.
Decision: Reject