DNN Quantization with Attention
Authors: Ghouthi BOUKLI HACENE
Lukas Mauch
Shubhankar Chowdhury
Stefan Uhlich
Fabien Cardinaux
Venue: ICLR 2022 Submitted (2022)
Keywords: Deep learning
Computer vision
Quantization

Abstract: Low-bit quantization of network weights and activations can drastically reduce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). Many different quantization methods like min-max quantization, Statistics-Aware Weight Binning (SAWB) or Binary Weight Network (BWN) have been proposed in the past. However, they still cause a considerable accuracy drop, in particular when applied to complex learning tasks or lightweight DNN architectures. In this paper, we propose a novel training procedure that can be used to improve the performance of existing quantization methods. We call this procedure \textit{DNN Quantization with Attention} (DQA). It relaxes the training problem, using a learnable linear combination of high, medium and low-bit quantization at the beginning, while converging to a single low-bit quantization at the end of the training. We show empirically that this relaxation effectively smooths the loss function and therefore helps convergence. Moreover, we conduct experiments and show that our procedure improves the performance of many state-of-the-art quantization methods on various object recognition tasks. In particular, we apply DQA with min-max, SAWB and BWN to train $2$bit quantized DNNs on the CIFAR10, CIFAR100 and ImageNet ILSVRC 2012 datasets, achieving a very good accuracy comparing to other conterparts.

Under review as a conference paper at ICLR 2022
DNN Quantization with Attention
Anonymous authors Paper under double-bling review
Abstract
Low-bit quantization of network weights and activations can drastically re- duce the memory footprint, complexity, energy consumption and latency of Deep Neural Networks (DNNs). Many different quantization methods like min-max quantization, Statistics-Aware Weight. Binning (SAWB) or Binary Weight Network (BWN) have been proposed in the past. However, they stiil cause considerable accuracy drop, in particular when applied to complex learning tasks or lightweight DNN architectures. In this paper, we propose a novel training procedure that can be used to improve the performance of existing quantization methods. We call this procedure DNN Quantization with Attention (DQA). relaxes the training problem, using a learnable linear combination of high, medium and low-bit quantization at the begin- ning, while converging to single low-bit quantization at the end of the training. We show empirically that this relaxation effectively smooths the loss function and therefore helps convergence. Moreover, we conduct ex- periments and show that our procedure improves the performance of many stato-of-the-art quantization methods on various object recognition tasks. In particular, we apply DQA with min-max, SAWB and BWN to train 2bit quantized DNNs on the CIFARIO, CIFAR100 and ImageNet ILSVRC 2012 datasets, achieving very good accuracy comparing to other conterparts.
1 Introduction and Related Work
In the last. decade, Deep Neural Networks (DNNs) in general and Convolutional Neural Networks (CNNs) in particular became state-of-the-art. many computer vision lasks, such as image classification segmentation, object detection and face recognition (LeCun
let al. 1998 y landola et al. (2016); Simonyan & Zisserman (2014); Graham 2014); Szegedy
lt al. 2015 However, to be the state-of-tho-art, DNNs often contain large number
of trainable parameters and require considerable computational power. Therefore, duc to
their large power and memory consumption, implementing DNNs on embedded systems with limited resources can be a real challenge. To alleviate this problem, à large number of different network compression methods that reduce the resource requirements of DNNs has been proposed in the past. Among them are for example pruning, distillation or quantization methods.
Pruning methods have been first introduced by LeCun et al. (1990). They identify and remove the most insignificant DNN parameters and yield networks with a reduced memory footprint and smaller computational complexity. As reported by Yamamoto & Maeno (2018); Ramakrishhan
al. (2020); He et al. (2020), pruning methods can either remove single DNN parameters, intermediate inputs even whole network layers that are, according to a specific criteria, irrelevant for a good network performance.
Another line of work is distillation. As introduced by Hinton et al. 2015 it aims at training a small student DNN to reproduce the output of bigger teacher network. While distillation methods initially only matched the final outputs of the teacher and student networks, mothods evolved to take into account intermediate representations (Romero et al. (2014); Koratana et al. (018)).
Our work is about DNN quantization, where weights and activations are represented with smaller number of bits << 32. Quantization reduces the memory footprint of DNNs

---

Under review as a conference paper at ICLR 2022
Qi(w,ng n1)
91
91 01
P2(w.n2)
42
92
f(x q)
Qk(wnTk) aK
4K .k
Figure 1: Quantizing the weights of a single network layer, using a linear combination of K different quantizers Q (Q1. Q2, Qx). Note, that each quantizer uses a different bitwidth, i.e. n = {MI, 12,, - nx). The resulting quantized weights qu are multiplied with attention values 0ę e [0, 1] that reflect the importance of the corresponding quantization function Qx(-). The attention values optimized during training according to algorithm
because the number of bits that are required store their parameters is reduced. However, it also reduces their computational complexity, because low bit operations can be implemented more efficiently on dedicated low precision hardware (Merolla et al. (2014); Farabet et al.
22011) Cowan et al. (2020): Han et al. (2020); Hacene et al. [0188).
Following Gupta et (0115). many works have experimentally demonstrated that neural networks do not lose lot of performance when their parameters are restricted to small set of possible values. For instance, Choi et al. (2018) introduced PArameterized Clipping ac- Tivation (PACT) in combination with Statistics-Awant Weight Binning (SAWB) -a method that aims at uniformly quantizing both weights and activations to n bit. Learned Step Size Quantization (T.QQ) presented by [Esser et al. (2019) is quantization method that learns the quantization steps during training. Unlike other methods, it scales the gradient dur- ing backpropagation to speed up the training. Gradient scaling is important especially at transition points.
Nikolić et al. (202 proposed Bit-Pruning, quantization method to learn the number of bits that each layer requires to represent its parameters and activations. In the same vein, Differentiable Quantization of Deep Neural Networks proposed by [Uhlich al. (2019) They propose
(DQDNN) tries to combine the features of both LSQ and Bit- quantization technique where both the number of bits and the quantization steps are learned simultaneous. Other more aggressive quantization methods proposed to use low-bit precision down to binarization (resp. ternarization) with only two (resp. three) possible values and one (resp. two) bit storage for each parameter and/or activation (Hubara et al. (0066) Courbariaux et al. (015)) Li et al. (2016b): Zhu et al. (2016); ; Li ct al. (2016a]).
Zhou et al. (0017 observed that training quantized networks to low precision benefits from incremental training. Rather than quantizing all the weights at once, they are quantiz- ing them incrementally in groups, with some training iterations between each quantization step. In practice, 50% the weights are quantized in the first step, then 75%, 87.5% and finally 100%. Another method for incremental quantization is Binary-Relax (BR) (Yin al. (2018)). Rather than splitting the parameters into groups, it uses a weighted linear combi- nation of quantized and full-precision parameters and adapts a strategy push the weights towards the quantized state, by gradually increasing the scaling factor corresponding to the quantized parameters. However, their strategy how move from full-precision quantized parameters is handcrafted and may not be optimal.
In comparison to most of the previously mentioned works, we do not propose an improved quantization method, but a way how to train a quantized DNN with any existing quantizer. We rely on the fact that the DNN performance can be increased if the network has the ability to learn other features in addition to its own parameters. In the context of DNN quantization, this has already been discussed by Elsken et al. (2020); Hacene al. (2019); UUlich et al. (2019); Nikolić et al.
(2019): Ramakrishnan et al.
2020 In this contribution,
we introduce DNN Quantization with Attention (DQA), an attention mechanism- based
learnable approach for DNN quantization (Vaswani et. al. 2017 As shown in Figure
2

---

Under review as a conference paper at ICLR 2022
rather than quantizing with just single bitwidth DQA performs quantization with multiple different bitwidths in parallel. At each stage of the training, DQA can select the quantizer with the optimal bitwidth by putting more attention weight on it. In particular, by starting with a uniform initialization of the attention weights at the beginning of the training, DQA can move smoothly from high precision to low precision quantization. We demonstrate in this paper that this learnable approach results in better DNN accuracy for the exact same complexity and number of bits.
The outline of the paper is as follows. In Section 3 we introduce the proposed method. Section presents experiments results and compares our method with other state-of-the approaches on challenging computer vision datasets. Finally, we conclude in Section 5
2 Background
In the following, we explain how differentiable quantization with attention (DQA) can be used to train quantized DNNs. For simplicity reasons, we only discuss how to train DNNs with quantized weights. However, DQA is general and can also be used to train DNNs with quantized weights and activations. In particular, will also provide experimental results for this case in Section !
Training DNNs with quantized weights is challenging. Especially when considering low bitwidths n << 32, quantization can cause a severe accuracy degradation compared to the full precision networks. This is mainly caused by the capacity reduction and by ad- ditional optimization issues that go hand in hand with the quantization. In particular, quantization yields non-smooth loss functions with gradients that zero almost every- where, As discussed by Chlich et al. (2019), that effectively stops gradient backpropagation and therefore harms training.
For these reasons, quantized DNNs are usually not trained with standard gradient based training procedures, but require some tricks that allow for gradient backpropagation and that stabilize the training. The most commonly used trick is to apply straight through gradient estimates (STE) and ignore the quantization during backpropagation. STE yield non-zero gradients that are suited for DNN training. However, at the same time they introduce a mismatch between the forward (FW) and backward (BW) pass, what often causes training instabilities and oscillations. More specifically, mismatch means that the gradients are calculated at the position of the quantized parameters, assuming the original float32 loss surface.
Note, that because of this FW/BW mismatch, training with STE does not necessarily converge to the optimum. This mismatch is for example problematic, if the gradient of the cost function changes signs within one quantization step. For this case, gradient descent with STE would start to oscillate near the quantization thresholds and would not converge to the optimum, even if the cost function is convex. Of course, this problem is most pronounced for low bitwidhts.
A training procedure that can alleviate this problem to some degree is Binary-Relax (BR). As proposed by Yin al. (2018). BR does not only apply STE, but also uses a linear combination of the quantized and the full-precision parameters. This effectively reduces the FW /WW mismatch at the beginning, while still enabling the DNN to use the exact low-bit quantized loss at the end of the training.
3 Methodology
DQA builds on a similar approach. As shown in figure I let f(x; Q(w;n)) be the transfer function of a quantized DNN layer, where x R I and Q)w;n) are the layer input, the full-precision weights and the quantized weights, respectively. Similar to the idea of Binary-Relas (BR), DQA relaxes the quantization problem and combines different quanti- zation schemes during training. More specifically, instead of using just one single Q( n), propose to train quantized DNN with set of K different quantization functions that
3

---

Under review as a conference paper at ICLR 2022
are combined linearly during training follows:
y f(x: q)
(1)
q Q"a
(2)
Qi Q2(w:
Q
Qx Qk(w;
where q is the weighted sum of K quantized weight vectors, Q e RKXM is a matrix whose row vectors are the quantized weight vectors and a [0, is the attention vector on the quantization functions. Note, that each row of Q is calculated, using a different quantization function Qx(w; and bitwidth ny E N. In particular, we assume that the quantization functions in Q are sorted by the bitwidth, i.e., < R2 < < пk- In general, DQA is agnostic the choice of the actual quantization method and can be used with any existing method like min-max, SAWB, binary or ternary quantization. In the following section, we review and define popular quantization methods that we used in our experiments.
The attention a is computed from a soft attention vector a e RK using a softmax function with temperature T' RR+ i.e.
C
E ERK
(4)
2k-1
In particular, a reflects the importance of the K quantization methods Qs. During training, the soft attention a is treated as a trainable parameter that is optimized in parallel the weights w. Note that, increasing ag will also increase the corresponding attention weight and therefore the importance of Qs (w; ny). In this manner, the quantized DNN can learn which bitwidth should be used at which stage, during the training.
DQA exponentially cools down the temperature T
T(b) = T(O)
(5)
Here, 6 1,, B is the batch index for batch-wise training, T(0) R+ is the initial temperature and V E [0, 1[ is the decay rate. Because of that schedule, DQA progressively moves from the full mixture of quantization functions at the beginning of the training to just one single ization function at the end of training.
Note, that BR can be seen as a special case of DQA, where q the case that we only use two quantizers with n = 2bit and n = 32bit, and for the case that we use a fixed schedule to change the attention vector a. However, DQA has two advantages: 1) The way how change a and move from high to low precision quantization is learned and data dependent. Hence, DQA can choose the optimal mixture the quantizers at each training iteration. 2) As shown in Figure 2 DQA gives smoother transition from high to low-precision parameters. Here, we plot the absolute quantization error for a fixed tempera- ture based schedule for a. In particular, we choose for BR and DQA with 2, 4, 8bit quantization, respectively. Then, we start with a large tem- perature T + 00, for which we effectively take the average K Σ (x(w; and move towards 2bit quantization for T O. Note, that for the whole interval that we consider for T, DQA results in lower quantization error, meaning that il also yields a smaller FW/BW mismatcch
w: Q(w; 32) i.c. for
[3/4 4, 1/4JT and a 2//7,2/4, 1/7]"
1
In general, training quantized DNNs with such mixture of different weight quantizations and decaying T will not necessarily result in a quantized DNN that uses a low bitwidth. To enforce a low-bit quantized DNN, therefore add to the loss function separate regularizer for each layer
Ag Ag"a(())
r(a)=
(6)
S
4

---

Under review as a conference paper at ICLR 2022
Quantization Érror
-10-2
8
BR
6
DQA
4
2
101
110
10-2
10-1
10
10
10
10a
Temperature T
Figure 2: The quantization error of BR and DQA for different temperatures T. In particular, a low temperature corresponds to 2bit quantization, while a high temperature means mixture of float32 and 2bit quantization mixture of 2/4/8bi quantization, respectively.
where S is the number of weights in the whole network. Note, that the normalization by S makes the regularizer, and therefore the choice of X, independent of the actual network size. g = [0,, 92. "99] is a penalty vector, where 9b is increasing with growing k. Because we assume, that the quantization functions (s(w; ng) are sorted by the bitwidth, ie., 11 quantizers with a large bitwidth. Hence, it helps the method converge the lowest-bit. quantization. Algorithm summarizes the DQA training. To quantize a given value æ, we use min-max, PnatisticsAwwary Weight Binning SAWB, Binary Weight Network (BWN) or Ternary Weight Network (TWN) as defined by Nikolić et al. (2020); Choi et al. Rastegari et al. 20166) et al. 2016aa respectively, and detailed in appendix A.1
< < adding g"a(a) penalizes if large attention values are assigned to
[
2018
Algorithm 1 DQA algorithm for single network layer
Inputs: Input vector x, initial softmax temperature T(0), final softmax temperature T(B),
number of training iterations B, and layer transfer function Output: Output tensor y
TYB
P:OT
1/
<
for each 1,2,
B do
((b) T(O)
a std(ar)
< softmax(a:T(())
q = Q a (linear combination)
y (x.
Update w and a via backpropagation.
end for
4 Experiments
In this section we will first introduce the benchmark protocol that we use evaluate our method, then we report different results obtained by DQA and compare them with other training procedures.
4.1 Benchmark Protocol
To evaluate our method DNN Quantization with Attention (DQA), perform experiments on the three object recognition datasets CIFAR10, CIFAR100 and ImageNet ILSVRC 2012. For each dataset, use DQA to train low-bit quantized versions of the ResNet18 (He et al. (2016)) and MobileNetV2 (andler et al. (2018)) network architectures. Low-bit means, that consider networks that. only
2bit for quanonzation.

---

Under review as a conference paper at ICLR 2022
Atbentian quaatictiion auB(unn
03
Albenkin valar
-05
Epoch
Fgoud
Eporda 300
4-hte
200
300
Epods
Quarrization
Flust wiight
(a) Attention values
(b) Resulting quantization function
Figure 3: The evolution of the attention values dg and the resulting quantization function for the first layer of quantized R8 trained CIFAR100.
For CIFARIO and CIFAR100, we start from randomly initialized parameters and train the quantized networks for 300 epochs. As an optimizer, we use SGD with an initial learning rate 7 0.1, which is divided by 10 every 100 epochs. The training batch size is 128.
On the ImageNet ILSVRC 2012 dataset, we train the quantized networks for 90 epochs, using& batch size of 256 images. As initial learning rate, again use 0.1 which is divided by 10 every 30 epochs. This way, we again apply two equally spaced learning rate drops over the full 90 epochs.
For all our experiments, we either quantize only the weights or both weights and activations using DQA with three different quantization functions (Qi, Q2, Qa)- More specifically, either consider a mixture of three min-max quantization functions that use ' = 2bit, n2 4bit and na Sbit, respectively or mixture of BWN, TWN and 8bit min-max quantization. For all experiments, we use an exponential temperature schedule with an initial temperature T(0) 100 that is cooled down to the final value of T(B) 0.03. The soft attention vector is initialized according to
I OA =j=1, ml
Σj=1 73j
Note, that since the quantization functions (w; are assumed to be sorted by the bitwidth, i.e. My < n2 < ... < Ak: this initialization assigns the highest attention to the quantization function with the lowest bitwidth. The initialization therefore acts as prior that favours low-bit quantized DNNs and helps us to converge to small bit widths early during training. To further encourage low-bit quantized DNNs, we use the penalty values = [1,4, 16] that penalize quantization functions with large bitwidth. Note, that we always compare networks that are quantized to the same bitwidth and thus have the same memory footprint and the same computational complexity.
4.2 Results
In the first experiments, we report the obtained accuracy achieved by DQA and compare it to three different basclines: 1) The full-precision network with float32 parameters. 2) The quantized network that uses 2bit quantized parameters and uses vanilla training without any relaxation scheme. 3) Binary-Relax (BR). To have fair comparison BR, we also report some results where we consider BR with the same mixture of quantization functions, i...
q wQi(w, m1) + Q2(w, + na)
w 2
where w is initialised to 1 and multiplied by 1.02 after each epoch. In other words, we use fixed schedule to move from 8hit. to 2bit. quantization.

---

Under review as a conference paper at ICLR 2022
10
10
10
1.1
1.3
1.1
-1
2
0
2
1-1
(a) float32 weights
(b) 2/4/8bit weights
(c) 2bit weights
Figure 4: The loss surfaces for randomly initialized ResNet18, with float32,, 2bit mixed 2,4,8bit quantized weights, evaluated on the CIFAR10 dataset. 2bit quantized weights yield non-smooth loss surface with many local minima, what is undesirable for optimization. Mixing 2,4, quantizers alleviates this problem.
Table 1 and show the experimental results for the CIFAR10 and CIFAR100 datascts when we quantize only weights and when we quantize both weights and activations, respectively. We report the final validation accuracy of the quantized DNNs for different choices of the quantization functions (Q1. Q2, Qs). In general, all reported validation accuracios are the result of single training run. Only for the experiments that use BWN quantization, report the average validation accuracy computed over runs, because the convergence of BWN quantized networks proved to be noisy, which shadowed the effects of DQA. Our proposed method archives accuracies that are comparable to the full-precision baseline, while outperforming the 2bit quantized baseline and BR.
The second experiment aims at studying the behavior of the attention values as during training. Figure 3 shows the evolution of the attention values ag and the corresponding quantization function. We can observe from 3a that all attention values are equal at the beginming of the training but due to the penalty term and the temperature schedule they slowly converge towards a maximum attent ion value for the 2bit quantization. This evolution can also be seen 3h where we show how the resulting quantization function evolves during training. Note, that the quantization function is smoothed out at the beginning and converges more and more towards the 2bit quantization curve at the end of the training. This smooth transition is the reason why DQA yields better results than training with just a single fixed quantization.
Interestingly, compared to single low-bit quantization, DQA yields smoother loss surfaces. Figure visualizes the loss surface of a ResNet18 with randomly initialized weights on the CIFARTO dataset. Here, we apply the method proposed by Li et al. (2017) that samples two random directions in the parameter space DNN and visualizes the loss along these directions. Obviously, the loss surface is the smoothest for float32 network. In comparison, the same ResNet18 with 2bit min-max quantized weights yields a very rough loss surface. For the 2bit case, optimization can get stuck easily in one of the numerous local minima. Moreover, it yields gradients that change quickly in direction and magnitude, causing severe oscillations and effectively making the training unstable. However, we apply DQA and use a mixture of 2, 4 and 8bit. min-max quantization to quantize the network weights, the loss surface smoothed out. Therefore, compared to DNNs that are trained with only one liw-bit quantization scheme, quantized DNNs trained with DQA typically converge faster at the beginning of the training and reach a better final optimum.
The third experiment compares DQA with other methods for quantized DNNs trained the ImageNet ILSVRC 2012 dataset. Table (parameter quantization only) and Table (param- eter and activation quantization) show that DQA outperforms the quantized 2bit baseline and BR when considering different quantization approaches. Moreover, DQA causes a sig- nificantly smaller drop in accuracy when quantizing VobileNetV2. Thus, it may represent.
7

---

Under review as a conference paper at ICLR 2022
Data
R1
Q1
T22
Q22
n33
Q3
Acc
RI8
C10
32
FP
95.2%
R18
C10
min-max
91.5%
RI8+BR
C10
2
min-max
32
FP
93.0%
RIS+BR
C10
2
min-max
4
min-max
8
mifl-max
93.7%
R18+Ours
C10
min-max
min-max
8
min-maax
94.8%
R18
C10
2
SAWB
94.8%
RIS+BR
C10
SAWB
4
SAWB
8
SAWB
95.1%
RI8+Ours
C10
SAWB
4
SAWB
8
SAWB
95.4%
R18
C10
BWN
93.8%
RI8+BR
C10
BWN
2
TWN
32
FP
94.2%
RI8+Ours
C10
BWN
2
TWN
32
FP
94.5% 5
RI8
C10
TWN
94.3%
RI8+BR
C10
2
TWN
4
min-max
8
min-max
94.5%
R18+Ours
C10
2
TWN
4
min-max
8
min-max
94.8%
RI8
C100
32
FP
77.9%
RI8
C100
2
max
70.0%
RIS+BR
C100
min-max
32
FP
72.9%
R18+BR
C100
min-max
4
min-max
8
min-max
74.0%
R18+Ours
C100
2
min-mac max
4
min-max
8
min-max max
10
76.4%
RI8
C100
SAWB
77.0%
BR
C100
2
SAWB
4
SAWB
8
SAWB
77.3%
R18+Ours
C100
SAWB
4
SAWB
8
SAWB
78.1%
RI8
C100
BWN
75.0%
RIS+BR
C100
BWN
2
TWN
32
FP
75.3%
R18+Ours
C100
BWN
2
TWN
32
FP
30
75. 75.9%
RIS
C100
TWN
76.1%
RIS+BR
C100
TWN
min-max
8
min-max
76.3%
C100
TWN
4
min-max
8
min-max
20
76.7%
Table 1: Obtained accuracy of ResNet18 (R18) trained on CIFAR10 (C10) and CIFAR100 (C100) for quantized weights, only. We consider numerous quantization functions (min-max, SAWB, BWN and TWN). Note, that FP refers to full precision (i.c. Q(w, 32) w).
promising training procedure that makes existing quantization methods more powerful and, hence, helps us to train lightweight DNN architectures.
Conclusion
In this paper, we introduced DQA, novel learning procedure for training low-bit quantized DNNs, using existing quantization methods. Instead of using only a single quantization precision during training, DQA relaxes the problem and uses mixture of high, medium and low-bit quantization functions. Our experiments on popular object recognition datasots, such as CIFARIO, CIFAR100 and ImageNet ILSVRC 2012, show that DQA can be used to train highly accurale low-bit quantized DNNs that achieve good accuracy compared with state-of-the-as counterparts.
If we compare to the full- aall-precision networks, DQA yields significantly lower accuracy drop than other training procedures that only use a single quantization precision and bitwidth during training. This is especially true when quantizing DNN architectures that are already designed to be lightweight and efficient, such as the MobileNetV2. Because such architec- tures are already small, they are naturally harder to compress.
DQA also compares favourably to Binary-Rees Relax (BR), another training procedure for quan- tized DNNs that applies mixture of quantized and full-precision weights during training.

---

Under review as a conference paper at ICLR 2022
Data
R1
Q1
T22
Q22
n33
Q3
X
Acc
RI8
C10
32
FP
95.2%
R18
C10
min-max
87.8%
RI8+BR
C10
2
min-max
32
FP
89.5%
RIs+Ours
C10
2
min-max
4
min-max
8
miml-max
90.4% 5
R18
C10
PS
94.4%
RIS+BR
C10
2
PS
32
FP
94.3%
R18+Ours
C10
PS
4
PS
8
94.7%
RI8
C100
32
FP
77.9%
RI8+BR
C100
min-max
32
FP
65, 65.2%
R18+Ours
C100
min-max
4
min-max
8
min-max
10
68.3%
RIS
C100
2
PS
75. 75.2%
RI8+BR
C100
PS
32
FP
75.9%
R18+Ours
C100
PS
4
PS
8
PS
5
78.1%
Table 2: Obtained accuracy of ResNet18 (R18) trained on CIFARIO (C10) and CIFAR100
(C100), when quantizing both weights and activations 2bit. Note, that PS refers to PACT-SAWB.
T1
Q
122
Q2
Q3
Top-1 (Top-5)
R18
32
FP
69.9%
(89.1%)
R18
2
min-max
58.7% (81.9%)
R18+Ours
2
min-max
4
min-max
min-max
66.9% (87.4%)
MV2
32
FP
69.0% (89.0%)
MV2
2
min-max
44.2% (69.8%)
Mv2+Ours
2
min-max
4
min-max max
min-max max
52.2% (77.1%)
R18
BWN
61.0%
(83.5%)
R18 Ours
BWN
2
TWN
min max
10
61.4% (83. 7%)
Table 3: Experiments on the ImageNet dataset, using the ResNet18 (R18) and the Mo- V2 (MV2) networks with quantized weights, only. Quantized DNNs trained with DQA consistently outperform quantized DNNs that have been trained with just a single quantization method. It also drastically reduces the accuracy drop when quantizing Mo- V2.
f1
Q1
f2
Q2
723
Q3
Top-1 (Top-5)
R18
32
FP
69.9%
(89.1%)
RI8
2
min-max
40.7%
(69.9%)
RI8+BR
2
min-max
FP
57.7%
(81.5%)
Rur+Ours
2
min-max
4
min-max
min-max
0.5
60.4%
(83 33.4%)
Table 4: Experiments on the ImageNet dataset, when quantizing both weights and activa- tions of ResNet18 (R18).
However, while BR uses fixed scheme to mix the network weights of different precision, DQA can learn how to mix them in an optimal way and how to gradually move from high precision to low precision. In practice, this helps training and results in quantized DNNs with higher accuracy.
DQA is agnostic to and can be used with many different existing quantization methods, such as min-max, PACT-SAWB, Sinary-Weight and Ternary Weight quantization. Therefore, DQA is a very promising extension existing DNN quantization methods.

---

Under review as a conference paper at ICLR 2022
References
Jungwook Choi, Pierce I-Jen Chuang, Zhuo Wang, Swagath Venkataramani, Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Bridging the accuracy gap for 2-bit quantized neural networks (qnn). arXiv preprint arXiv:1807.06664 2018.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems, pp. 3123 3131,, 2015.
Meghan Cowan, Thierry Morcau, Tianqi Chen, James Bornholt, and Luis Coze. Automatic generation of bege-performancc quantized machine learning kernels. In Procoedings of the 18th ACM//EED International Symposium on Code Generation and Oplimization, pp. 305 316, 2020.
Thomas Elsken, Jan Hendrik Metzen, Frank Hutter, et al. Neural architecture search: A survey. J. Mach. Learn. Res., 20(55):1-21, 2019.
Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra S Modha. Learned step size quantization. arXiv preprint arXiv:1902 1902.08153, 2019.
Clément Farabet, Berin Martini, Benoit Corda, Polina Akselrod, Eugenio Culurciello, and Yann LeCun. Neuflow: A runtime reconfigurable dataflow processor for vision. In Cvpr 2011 Workshops, pp. 109-116. IEEE, 2011.
Benjamin Graham. Fractional max-pooling. CoRR, abs/ 141/1441,,677, 2014. URL http:// auxiv.or//bbs Sorivorg/abbs/4412.00711
Suyog Gupta, Ankur Agrawal, Kailash Gopalakrishnan, and Pritish Narayanan. Deep learn- ing with limited numerical precision. In International Conference on Machine Learning, pp. 1747-1746, 2015.
Ghouthi Boukli Hacene, Vincent Gripon, Matthieu Arzel, Nicolas Farrugia, and Yoshua Bengio. Quantized guided pruning for efficient hardware implementations of convolutional neural networks. arXiv preprint arXiv:1812.11337, 2018.
Ghouthi Boukli Hacene, Carlos Lassance, Vincent Gripon, Matthien Courbariaux, and Yoshua Bengio. Attention based pruning for shift networks. arXiv proprint arXiv:1905.12300 2019.
Qingchang Han, Yongmin Hu, Fengwei Yu, Hailong Yang, Bing Liu, Peng Hu, Ruihao Gong, Yanfei Wang. Rui Wang, Zhongzhi Luan, et al. Extremely low-bit convolution optimization for quantized neural network modern computer architecturos. In 49th International Conference on Parallel Processing-ICPP, pp. 1-12, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Yang Hc, Yuhang Ding, Ping Liu, Linchao Zhu, Hanwang Zhang, and Yi Yang. Learning fil- ler pruming criteria for deep convolutional neural networks acceleration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2009 2018, 2020.
Gcoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.0255: 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El-Yaniv, and Yoshua. Bengio. Binarized neural networks. In Advances in neural information processing systems, pp. 4107 4115, 2016.
Forrest N landola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and < 0.5 mb model size. arXiv preprint arXiv:1602.07366 2016.
10

---

Under review as a conference paper at ICLR 2022
Animesh Koratana, Daniel Kang, Peter Bailis, and Matei Zaharia. Lit: Block-wise inter- mediate representation training for model compression. arXiv preprint arXiv:1810.01937, 2018.
Yann LeCun, John S Denker, and Sara A Solla. Optimal brain damage. In Advances in neural information processing systems, pp. 598 605, 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 2324, 1998.
Fengfu Li, Bo Zhang, and Bin Liu. Ternary weight networks.
arXiv preprint
arXiv:1605. 10011,, 2016a.
Hao Li. Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein, Visualizing the loss landscape of neural nets. arXiv preprint arXiv:1712.09913. 2017.
Yangqing Li, Saurabh Prasad, Wei Chen, Changchuan Yin, and Zhu Han. An approximate message passing approach for compressive hyperspectral imaging using a simultaneous low-rank and joint-sparsity prior. In Hyperspectral Image and Signal Processing: Evolu- lion in Remote Sensing (WHISPERS), 2016 8th Workshop on, pp. 5. IEEE, 2016b.
Paul A Merolla, John V Arthur, Rodrigo Alvarez-lcaza, Andrew S Cassidy, Jun Sawada, Filipp Akopyan, Bryan L Jackson, Nabil Imam, Chen Guo, Yutaka Nakamura, et al. A million spiking-neuron integrated circuit with scalable commuiccation network and interface. Science, 345(6197):668 673, 2014.
Miloš Nikolić, Ghouthi Boukli Hacene, Ciaran Bannon, Alberto Delmas Lascorz, Matthieu Courbariaux, Yoshua Bengio, Vincent Gripon, and Andreas Moshovos. Bitprum- ing: Learning bitlengths for aggressive and accurate quantization. arXiv preprint arXiv:2002.03090, 2020.
Ramchalam Kinattinkara Ramakrishnan, Eyyub Sari, and Vahid Partovi Nia. Differentiable mask for pruning convolutional and recurrent networks. In 2020 17th Conference on Computer and Robot Vision (CRV), pp. 222-229. IEEE, 2020.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Tm- agenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525 542. Springer, 2016.
Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. arXiv preprint arXiv:1412.6550,
2014.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chinh Chen. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. CoRR, abs/ Abs/1449.1556, 2014. URL httpp://arxvv.rgg/
405.1556
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jonathon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. arXiv preprint arXiv:1512.00567, 2015.
Stefan Uhlich, Lukas Mauch, Kazuki Yoshiyama, Fabien Cardinaux, Javier Alonso Garcia, Stephen Tiedemann, Thomas Kemp, and Akira Nakamura. Differentiable quantization of deep neural networks. arXiv preprint arXiv:1905.11452, 2(8), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems, pp. 5998-6008, 2017.
Kohei Yamamoto and Kurato Maeno. Pcas: Pruning channels with attention statistics. arXiv preprint arXiv:1806.05336 2018.
11

---

Under review as a conference paper at ICLR 2022
Penghang Yin, Shuai Zhang, Jiancheng Lyu, Stanley Osher, Yingyong Qi, and Jack Xin. Binaryrelax: A relaxation approach for training deep neural networks with quantized weights. SIAM Journal on Imaging Sciences, 11(4):2205-23) 2223, 2018.
Aojum Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental net- work quantization: Towards lossless cnns with low-precision weights. arXiv preprint arXiv:1702.03044 2017.
Chenzhuo Zhu, Song Han, Huizi Mao, and William J Dally, Trained ternary quantization. arXiv preprint 1r11v41612.01644, 2016.
A Appendix
A.1 Choosing the Quantization Functions
Quantization describes the process of representing value X' with corresponding quantized value q 2, using a quantization function Q Q: - Q. Here, Q = (q) 92...., 12.)
is the set of quantization steps that is much smaller than X, i.c., |Q] << |. For a given w and Q the quantization function minimizes the distance between w and q, i.e.,
Q(:; n)) arg min x gl.
(9)
Q
where || Il is the Euclidean norm. There different methods how to construct Q that yield different quantization schemes, like uniform or non-uniform quantization.
The first method may consider is the introduced in Nikolić al. (020 For X [umin Tmaz] they define
q: = Imin + (i 1) Imax Imin
1,2,...,
(10)
In particular, the values are uniformly distributed between the values Imin and Zmazs what known as max quantization.
The second method we use with our proposed training procedure is Statistics Aware Weight Binning (SAWB) Choi et al. (2018). The quantization values are again distributed uniformly over a given interval. However, instoad of using the limits Tmin and Fmany SAWB introduces a limit a, i.c.,
2a
q: - (i
2n 1n' i= 2,,,...."" 2"
(11)
The optimal a can calculated in a calibration step, using data. particular, we minimize the mean-square quantization error
arg min Ex-p(x) - (Z;
(12)
with respect to a. After calibration, we can use Q(x; n) Q(2; n, a*) for quantization.
For both min-man max and SAWB quantization, the solution of Eq. equation 9 is straight-forwant to obtain. is a uniform quantization function with equally spaced quantization steps that is defined by
91
91 others
Quw; n)
91
32m
2n
round (I
2"-1 9gn
(13)
42n
x 42"
Another quantization function worth to mention was introduced for the Binary Weight Network (BWN) Rastegari et. al. (2016). It uses scaling factor 8 E((xl) and constrains
12

---

Under review as a conference paper at ICLR 2022
the quantized values to be binary (n 1). In particular, with Q {-8, 9), the quantization function is defined as
BB - others
Qu,, 1) Bw sign(x)
(14)
In the same vein, Ternary Weight Network (TWN) Li et al. (2016a) introduces a third quantization step improve the accuracy. A TWN uses bitwidth of n 2 and symmetric B, 0, B). Similar to the BWN, the range parameter 3 is calibrated with data. More specifically,
Q {-
compute the optimal range 8 E.g p(=|| where å 0.7R[[=]] is the symmetric threshold that is used for quantization during calibration. The resulting quantization function is defined
3
1 8
((x;2)
0
|2|
& 6
(15)
8
8
13


Reviews:
[{"review_id": "qat_K3Zj7RE", "reviewer": "Reviewer_H2zp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n- The paper is well written and the idea is easy to follow.\n- Extensive ablation studies are provided to evaluate different components of the proposed method.\n\nCons:\n- More parameters are introduced in the training stage, such as α. This will increase the computation and storage cost. More theoretical and experimental analysis should be given to study that.\n- Multiple quantizers with different bit-width are conducted in the proposed method, which will increase the storage and computation cost for quantization.\n- In the experiments, the authors compare their method with the corresponding counterparts with the same bit of n1. However, the proposed method has three quantizers with different bit-width, and n1 is the lowest bit-width. Therefore, this comparison seems unfair. For a fair comparison, the baselines and the proposed method should be compared under the same computation and storage cost.\n- The proposed method has not been compared with state-of-the-art approaches, which cannot comprehensively evaluate the proposed method.", "summary_of_the_paper": "This paper attempt to address a challenging quantization problem, i.e., low-bit quantization. This work utilizes a learnable linear combination of high, medium, and low-bit quantization at the beginning while converging to a single low-bit quantization at the end of the training. In the quantization procedure, multiple quantizers and the corresponding attention matrices are adopted to fuse the quantized weights or activations.", "summary_of_the_review": "This work utilizes a learnable linear combination of high, medium, and low-bit quantization at the beginning while converging to a single low-bit quantization at the end of the training. In the quantization procedure, multiple quantizers and the corresponding attention matrices are adopted to fuse the quantized weights or activations, which will increase the computation and storage cost. Some experiments are conducted to evaluate the proposed method. However, it lacks some comprehensive and fair comparison.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WD_nn2A1zz5", "reviewer": "Reviewer_4bsX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. According to the experiment, DOA performs better than the naive quantization strategy and Binary-Relax consistently across the experimented datasets and networks.\n\nWeaknesses:\n1. The presentation needs to be improved. There are grammar errors and typos in the paper.\n2. The paper compares the performance of DOA with only one related quantization work (Binary-Relax). It is not sufficient to demonstrate the effectiveness of the proposed work. There are many quantization works, both quantization-aware training and post-training quantization. It seems that some of them may have better performance in the experimented settings. For example, the work LQ-Nets reports an accuracy of 68% with 2-bit weights and 32 bits activation with ResNet-18 on ImageNet, but DOA proposed in this paper only achieves an accuracy of 66.9%. \n\nMinor comments or questions:\n1. How to decide what quantization method to use (e.g., min-max, SAWB, BWN, TWN) when using DAQ in practice? Appendix A only defines each quantization method but doesn't give any guidance on how to choose them.\n2. For experiments on the ImageNet dataset using ResNet18, why not report R18+BR and MV2+BR results?\n3. In section 4.2, the paper mentions all reported validation accuracies are the results of a single training run. It might be better to report averaged results across several runs even if the convergence of the networks is not noisy. \n4. In Table 1, it seems that DQA using SWAB consistently gives better results than the FP version. Do you have any insight regarding this?", "summary_of_the_paper": "This paper proposes DNN quantization with Attention (DQA), which uses a learnable linear combination of high, medium, and low-bit quantization at the beginning. It gradually converges to a single low-bit quantization at the end of training. Experiments show that DQA outperforms the naive quantization and the Binary-Relax method consistently across three datasets and two networks.", "summary_of_the_review": "Overall, I think the paper is not good enough due to aforementioned nontrivial weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "LPSt7RwLppj", "reviewer": "Reviewer_1Jav", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n-\tAuthors shows the proposed multi-biwidth training effectively reduces quantization error and helps smooth loss landscape with some sample cases and visualizations.\n\nWeaknesses:\n-\tOverall performance of the given approach is not satisfying. Most recent quantization papers mainly conduct experiments on the large-scale ImageNet dataset since the CIFAR datasets are prone to easy overfitting. Almost all papers I know doing low-bit quantization has better results than this one. \n\nOn weight-only 2-bit quantization with MobilenetV2:\nSAT: 66.8 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)\nDeepComp 58.1 (Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. ICLR 2016)\nThis work: 52.2\n\nOn both weight and activation 2-bit quantization with ResNet18:\nPACT 64.4 (Pact: Parameterized clipping activation for quantized neural networks. arXiv 18)\nLQNet 64.9 (Lq-nets: Learned quantization for highly accurate and compact deep neural networks. ECCV 18)\nSAT 65.5 (Neural Network Quantization with Scale-Adjusted Training BMVC 2020)\nThis work: 60.4\n\n-\tSome technical details are not clear. The authors use a penalty term (equation 6) to regularize the attention weights of different bitwdith. However, it is not known whether all bitwidths in the network will be converged to the lowest bit which is the target. If some bitwidths are not property converged, will there be any issue on the performance?\n-\tWriting needs improvement. There are a lot of grammar errors and typos:\n\nPage 2 last paragraph: a way how to train a (delete how)\n\nPage 3 background 4th paragraph: Note, that because (no comma)\n\nPage 3 Background 4th paragraph: this problem is most pronounced for low bitwidhts (typo)\n", "summary_of_the_paper": "This work presents a training method for low-bit network quantization. While training, it employs a multi-bitwidth paradigm in order to alleviate the nonsmooth optimization landscape with lower bitwidth. It uses a temperature parameter and a penalty term to force the network to gradually converge to the target low bit. Experiments are conducted on CIFAR10, CIFAR100, ImageNet Classification with ResNet 18 and MobilenetV2.", "summary_of_the_review": "Overall, this work presents a new approach to help improve the training convergence of low-bit quantization for neural network. Due to the weak results on large-scale datasets and unclear technical details, I do not think it meets our bar at ICLR.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IP7lhqGn1b1", "reviewer": "Reviewer_UAi4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n1) In my understanding, what the paper call attention is simply a value weighting the importance of the different quantization functions (see Eqn. 2-3). Therefore this terminology is misleading in my opinion, as the vector of attention a only depends on the “trainable parameter alpha”, and not does not depend on the input (either or activation) as one would imagine with this name. Maybe I misunderstood something, but this is what I infer from Eqn (4). Eqn (2-3) and Figure 1. \n\n2) The paper does not consier in the literature review techniques for quantizing neural networks [A,B,C] that, to my knowledge, are state-of-the-art for quantizing popular neural networks. \n[A] Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks, Martinez et al., CVPR’2021\n[B] And the bit goes down: Revisiting the quantization of neural networks, Stock et al. ICLR’2020\n[C] Training with Quantization Noise for Extreme Model Compression, Fan et al. ICLR’2021\nAt a high level, some elements are similar to these approaches, even though the details may differ.  One noticeably similarity is the various of precision, which is already present in the work by Fan et al [C] when they consider both blocks that are not quantized (i.e., 32 bits) versus some that are quantized with low-precision, and randomly choose. In my own experience, choosing randomly a choice is better that relaxation, so I guess the paper should have included such a comparison. In any case the paper should be better positioned against the recent literature. \n\n3) The paper is compared to poor baselines. As a result, the paper reports results that do not look competitive the state of the art [A,B,C] on Imagenet ILSVRC 2012, which is the most significant benchmark in the paper. For instance, Stock et al. [B] report some results with R18  (Figure 3 left, table 3) that are as follow: compression factors of x20 for top-1 accuracy at 67.87. In the submitted paper. Additionally, more recent papers [A] and [C] are compared to [B] on larger networks and show that they further improve results. Therefore I conclude that the proposed approach is not competitive, while additionally requiring a more engaged scheduling that may not generalize as well to other training settings. The paper states that it could be combined with any compression method, therefore in this context it would be worth using the same or similar quantization as in [A,B,C] and show how the method compares to these methods. \n\n4) Formally I have nothing against having in the same section the introduction and the related work. In the case of this paper, I found that the introduction is actually more a related work than a formal introduction providing the motivation and rationale of the paper content at the core of the initial discussion.  This discussion appears later in the background section. While I know well this area and therefore the problem at stake, I would advise to re-work jointly these two sections. \n\n5) The paper  mentions that the method could be used for quantization activation, but only addresses the case of weight quantization. I think that a lot of practical considerations would appear with activation quantization, so I would suggest either to support this claim with experiments,  or to suppress or soften this unsupported claim. \n\n6) The paper needs some polishing. Some mistakes indicate that the paper was not analyzed by a spell-checker, for instance in the abstract: conterparts -> counterparts. The term bitwidth is not established (and occurred at least once with a typo). \n\n\n", "summary_of_the_paper": "The papers addresses the problem of compression of neural networks. The paper builds upon binary-relax prior work. The precision is adapted during training with a mixture-based quantization method through temperature cooling and a set of an ``attention’’ vector a. The idea of the method, called DQA, is to progressively moves from a mixture of quantization functions (mixing high with low precision, for instance 32 bit with 2 bit training), to a single one (low precision) towards the end of the training. The paper states that the method can be used with several types of quantization methods. The evaluation is carried on computer vision architectures (ResNet18, MobileNet) for image recognition tasks (Cifar-10, Cifar-100, Imagenet ILSVRC 2012). \n", "summary_of_the_review": "The paper does not demonstrate that the method is a significant contribution to the state of the art in quantizing neural networks. The experiments are only applied on image classification task with small architectures, and in the setting that I found comparable in the literature, the results do not look great, which questions the significance of this work. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new learning procedure for quantizing neural networks. Basically, DQA method proposed in this paper uses attention to obtain a linear combination of the existing network quantization techniques and uses it to pursue more efficient quantization.

Overall, it seems the submission was written in haste, so there are many typos and errors. Above all, the motivation that it can be applied to various existing techniques could not be proved experimentally at all since it only covers one somehow obsolete work. In addition, as in [1], it seems necessary to quantize not only weights but also activations, or to verify in lightweight networks such as MobileNetV2 rather than ResNet.

[1] Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss, ICCV 2021
Decision: Reject