QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization
Authors: Xiuying Wei
Ruihao Gong
Yuhang Li
Xianglong Liu
Fengwei Yu
Venue: ICLR 2022 Poster (2022)

Abstract: Recently, post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining. Despite the low cost, current PTQ works always fail under the extremely low-bit setting. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. To deeply understand the inherent reason, a theoretical framework is established, which inspires us that the flatness of the optimized low-bit model on calibration and test data is crucial. Based on the conclusion, a simple yet effective approach dubbed as \textsc{QDrop} is proposed, which randomly drops the quantization of activations during reconstruction. Extensive experiments on various tasks including computer vision (image classification, object detection) and natural language processing (text classification and question answering) prove its superiority. With \textsc{QDrop}, the limit of PTQ is pushed to the 2-bit activation for the first time and the accuracy boost can be up to 51.49\%. Without bells and whistles, \textsc{QDrop} establishes a new state of the art for PTQ.

Published as a conference paper at ICLR 2022
QDROP: RANDOMLY DROPPING QUANTIZATION FOR EXTREMELY LOW-BIT POST- TRAINING QUANTIZATION
Xiuying Weil. Ruihao Gongl.2, Yuhang Li², Xianglong Liu Fengwei Yu²
Stttt Key Lab of Software Development Environment, Beihang University, "SenseTime Research (weixiuying, gongruihao, Lorlso,,yyyie Liyuhangl) @sensetime com, x1 wiliuubuua... edd.
ABSTRACT
Recently, post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time time retraining. Despite its low cost, cur- rent PTQ works tend to fail under the extremely low-bit setting. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. To deeply understand the in- herent reason, a theoretical framework is established, indicating that the flatness the optimized low-bit model on calibration and test data is crucial. Based on the conclusion, simple yet effective approach dubbed as QDROP is proposed, which randomly drops the quantization of activations during PTQ. Extensive ex- periments various tasks including computer vision (image classification, ob- ject detection) and natural language processing (text classification and question answering) prove its superiority. With QDRÓP, the limit of PTQ is pushed the 2-bit activation for the first time and the accuracy boost can be up to 51.49% Without bells and whistles, QDROP establishes a new state of the art for PTQ. Our code is available at https: : //ghthbb. com/ wimh966 /Drrop and has been integrated into MQBench (https: : / / thub. ICm//ModeTC//MMMMeee
INTRODUCTION
In recent years, deep learning has been applied to all walks of life and offered substantial con- venience for people's production and activities. While the performance of deep neural networks continues to increase, the memory and computation cost also scale up fastly and bring new chal- lenges for edge devices. Model compression techniques such as network pruning (Han et al., 2015), distillation (Hinton al., 2015), network quantization (Jacob et al., 2018) and neural architecture search (Zoph & Le, 2016) etc., are dedicated to reduce calculation and storage overhead. In this pa- per, we study quantization which adopts low-bit representation for weights and activations to enable fixed-point computation and less memory space.
Based on the cost of a quantization algorithm, researchers usually divide the quantization work into two categories: (I) Quantization-Aware Training (QAT) and (2) Post- Post-Training Quantization (PTQ). QAT finetunes pre-trained model by leveraging the whole dataset and GPU effort. On the contrary, PTQ demands much less computation to obtain quantized model since it does not require end-to- end training. Therefore, much attention has recently been paid to PTQ (Cai et al., 2020; Wang et al., 2020; Hubara et al., 2021; Banner et al., 2019; Nahshan et al., 2019; Zhang et al., 202 Li et al., 2021c) due to its low cost and easy-to-use characteristics in practice.
Traditionally, PTQ pursues accuracy by performing the rounding-to-ne neearesstt operation, which fo- cuses on minimizing the distance from the full-preeision (FP) model in parameter space. In recent progress, Nagel et al. (2020); Li al. (2021a) considered minimizing the distance in model space, i.e. the final loss objective. They use Taylor Expansion to analyze the change of loss value and derive method to reconstruct the pre-trained model's feature by learning the rounding scheme. Such methods are efficient and effective in 4-bit quantization and can even push the limit of weight quantization to b-b However, the extremely low-bit activation quantization, which faces more challenges, still fails to achieve satisfactory accuracy. We argue that one key reason is that existing
"Equal contribution, Corresponding author.

---

Published as a conference paper at ICLR 2022
theoretical analyses only model the weight quantization perturbation while ignoring activation ss This will lead to the same optimized model no matter which bit the activations use, which is obvi- ously counter-intuitive and thus causes sub-optimal solution.
In this work, the effecc of activation quantization in PTQ is deeply investigated for the first time. We empirically observe that perceiving the activation quantization benefits the extremely low-bit PTQ reconstruction and surprisingly find that only partial activation quantization more preferable. An intuitive understanding is that incorporating activation will lead to a different optimized weight. Inspired by this, we conduct theoretical studies on how activation quantization affects the weight tuning, and the conclusion that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data. Motivated by both empirical and theoretical findings, we propose QDROP that randomly drops quantization during the PTQ reconstruction to pursue the flatness from a general perspec- tive. With this simple and effective approach, we set up new state of the art for PTQ on various tasks including image classification, object detection for computer vision, and text classification and question answering for natural language processing.
To this end, this paper makes the following contributions:
We confirm the benefits unprecedentedly from involving activation quantization in the PTQ re- construction and surprisingly observe that partial involvement of activation quantization performs better than the whole.
2. A theoretical framework is established deeply analyze the influence of incorporating activation quantization into weight tuning. Using this framework, we conclude that the flatness of the optimized low-bit model on calibration data and test data is crucial for the final accuracy.
3. Based on the empirical and theoretical analyses, we propose a simple yet effective method QDROP that achieves the flatness from general perspective. QDROP is easy to implement and can consistently boost existing methods as plug-and-ple -add module for various neural networks including CNNs like ResNets and Transformers like BERT.
4. Extensive experiments on a large variety of tasks and models prove that our method set up new state of the art for PTQ. With QDROP, the 2-bit post-training quantization becomes possible for the first time.
2 PRELIMINARIES
Basic Notations. Throughout this paper, matrices (or tensors) are marked as X, whereas the vectors are denoted by x. Sometimes we use w represent the flattened version of the weight matrix W. Operator is marked as scalar multiplication, @ is marked as element-wise multiplication for matrices or vectors. For matrix multiplication, we denote Wa a as matrix-vector multiplication or W as matrix-matrix multiplication.
For a feedforward neural network with activation function, we denote it as g(wr æ)) and the loss function as L(w, x), where and w are the network inputs and weights, respectively. Note that we assume æ is sampled from training dataset D,, thus the final loss is denoted by En [Lw x)]. For the network forward function, we can write it as:
((+1) Σ LJ a f(z, ((2(( a (e+1)
where Wa, denotes weight connecting the jth activation neuron and the tth output. The bracket superscript (E) is the layer index. ʃ() indicates the activation function.
Post-training Quantization. Uniform quantizer maps continuous values x E IR into fixed-point integers. For instance, the activation quantization function can be written as à 11a 1.1 denotes the rounding-to-nearees operator, s is the step size between two subsequent quantization levels. While rounding-to-nearee operation minimizes the mean squared error between å and the minimization of parameter space certainly cannot equal to the minimization in final task loss (Li et al., 2021a), i.c., Ez-D, [L(ui x)). However, in the post-training setting, we only have tiny subset D. E D, that only contains 1k images. Thus, it is hard to minimize the final loss objective with limited data.
where
2

---

Published as a conference paper at ICLR 2022
Recently, a series of works (Nagel et al., 2020; Li et al., 2021a) learn to either round up or down and view the new rounding mechanism as weight perturbation, i.c., v = w + Aw. Take pre-trained network G example, they leverage Taylor Expansion to analyze the target, which reveals the quantization interactions among weights:
min E [T((tin a:) [uw, .:) min w
Aw H Am
where H Eg-D, W ((w, x) is the expected second-order derivative. The above objective could be transformed into the change of output weighted by the output Hessian.
min [Aw" "H" Aw] min E [Aa" H Aa]
About the above minimization, they finetune only the weight by reconstructing each block/layer output (See Fig. 1). But they did not explore the activation quantization during output reconstruc- tion with only modeling weight quantization noise. The step size for activation quantization is determined after the reconstruction stage.
Intuitively, when quantizing the activations of a full- cull-precision model 2-bit 3-bit, there should be different suitable weights. However, the existing works result in the same optimized weight due to the neglect of activation quantization. Therefore, argue that when quantizing the neural network, the noise caused by activation quantization should be considered coherently with weights.
3 METHODOLOGY
In this section, to reveal the influence of introducing activation quantization before output recon- struction, we first conduct empirical experiments and present two observations. Then a theoretical framework is built to investigate how the activation quantization affects the optimized weights. Last, equipped with the analysis conclusions, a simple yet effective method dubbed QDROP is proposed.
Casc
block
block k-1
block
Case
3
ResNet-18
18.88
45.74
48.07
Case 2
block
block k-1
block
ResNet-50
4.34
46.98
49.07
MobileNerV2
5.83
50.71
51.20
RegNet-600MF
42.77
60.94
62.07
Case
block
block k-1
block
MnasNet
26.62
58.79
60.19
Figure 1: cases involve activation quantization when optimiz- ing the kich block's weight rounding. Activations are quantized inside the blue block and not quantized inside the orange block.
Table I: 2-bit 3-bit post-training quanti- zation accuracy on ImageNet dataset across different cases and different models.
3.1 EMPIRICAL OBSERVATIONS
To investigate the influence of activation quantization when reconstructing the layer/block output, we conduct preliminary experiments on the ImageNet (Russakovsky et al., 2015) dataset. Our exper- iments are based on the open-sourced code Li et al. (2021a) except that will introduce activation quantization from 1 to & blocks hefore the kęg. block's 's reconstruction. We give a simple visual- ization in Fig. 1 show 3 cases for putting activation quantization in different stages. Case means that all activations are kept in 32-bit full-precision during the reconstruction of block output, which also adopted in existing work Nagel al. (2020); Li et al. (2021a). Case and Case 3 are used for incorporating activation quantization into the reconstruction stage. However, Case 3 will omit the current block's 's quantization while Case 2 will not. The detailed results of these three cases are listed in Table (Comparisons on 2-bit (W2A2) quantization for ResNet- 18, ResNer-50 and W3A3 for others for the sake of the crashed results on 2-bit.) and the algorithm is put in algorithm 2. According to the comparison,
obtain two observations:
1. For extremely low-bit quantization (e.g., W2A2), there will be huge accuracy improvement when considering activation quantization during weight tuning. This is confirmed by comparing with Case and Case We find Case barely converges while Case achieves good accuracy. reveals that separate optimization of weights and activations cannot find an optimal solution. After introducing the activation quantization, the weights will learn to diminish the influence of activation quantization.
3

---

Published as a conference paper at ICLR 2022
2. Partially introducing block-wise activation quantization surpasses introducing the whole activa- tion quantization. Case does not quantize the activations inside the current tuning block but achieves better results than Case 2. This inspires that how much activation quantization introduce for weight tuning will affect the final accuracy.
3.2
How DOES ACTIVATION QUANTIZATION AFFECT WEIGHT TUNING
The empirical observations have highlighted the importance of activation quantization during the PTQ pipeline. To further explore how activation quantization will affect the weight tuning, we build theoretical framework that analyzes the final loss objective with both weights and activations being quantized, which presents clues of high accuracy for extremely low-bit post-training quantization.
Conventionally, the activation quantization could be modeled as injecting some form of noise im- posed on the full-precision counterpart, defined as (ā (a. To remove the influence of activation range on e, we translate the noise into multiplicative form, i.e., â = a (1 + u). where the range of is affected by bit-widhe hitth and rounding error. Detailed illustration of the new form noise can be found in Appendix A.
Here, 1 (+x) is adopted present the activation noise since it is related to specific input data point a. Equipped with the noise, we add another argument in calculating the loss function and define our optimization objective in PTQ as:
min Ba [L(w + Aw, æ, 1 + u(()) - L(w, a, 1)]-
(4)
We hereby use transformation that absorb the noise activation and transfer to weight, where the perturbation on weight is denoted as + v(z) (V(x) is used matrix multiplication format).
Consider simple matrix-vector multiplication W a in forward pass, have ((a (1-u(x)))) (W @ (1 + ((x))) )a, given by
+ ui (x)
I + ui (x)
+ u2(2)
... 1 un())
+ (z(x))
1 + u1 (1)
+ u2(x))
1 u, (z)
W(a +
= ( o
)
(5)
1 + un (x)]
1 + u1 (x:)
+ (2:))
un (xr)
By taking Vi,(()) = (z) quantization noise on the activation vector (1 + u(x)) can be trans- planted into perturbation weight (1 v(x)). Note that for specific input data point æ, there two distinct uxx) and v Proof is available at Sec, B.1.
Also note that for a convolutional layer, we cannot apply such transformation since the input to convolution is a matrix and will cause different V. Nonetheless, we can give a formal lemma that absorbs u(a and holds corresponding v(x) (See the Appendix Sec. B.2 for rigorous proof):
Lemma 1. For quantized (convolutional) neural network, the inftuence of activation quantization on the final loss objective in post-traiiing quantization can be transformed into weight perturbation.
EnnD, (L x, + u(x)) L(w, x, 1)] Eg [L(u @ (1 v(x)), x, L(w, x. 1)] (6)
By interpolating T(i æ. 1) into Lemma 1, we can obtain the final theorem: Theorem 1. For neural network G with quantized weight io and activation perturbution (a)),,
we have:
ExDD [L(1, x, 1+ u(x)) - L(w, x. 1)] x
Ez-D, [L(r,, x, 1) (a0)0L(wwwpp
(L(ι c (1 + v(x)), x. L(aū, x. 1))]
(7-1)
(7-2)
Here, Theorem I divides optimization objective into two terms. Term (7-1) is the same as Eq. (2) explored in (Nagel et al., 2020; Li et al., 2021a), which reveals how weight quantization interacts with loss function. Term (7-2) is the additional loss change by introducing activation quantization. In another way to interpret Eq. (7), the term (7-2) stands for the loss change with jitters on the weight quantized network G(i,, 2.) This type of noise correlates with certain kinds of robustness.
4

---

Published as a conference paper at ICLR 2022
Calibration Data
Data
perturbation magnitude
0,010
Case
0.009
Case
Case
0,008
0.005
0.007
0.807
0.806
9.005
500
350
400
450
500
sharpness
sharpness
Figure 2: Measure sharpness on different data distributions among three cases. We adopt the measurement defined in (Keskar et al., 2016). With the same degree of loss change ratio, those who can tolerate a larger perturbation magnitude enjoy a flatter loss landscape.
As stated in some works about generalization and flatness (Dinh et al., 2017; Hochreiter & Schmid- huber, 1997), intuitively, flat minimum means relatively small loss change under perturbation in the parameters, otherwise, the minimum is sharp. In this paper, we follow the notion of flatness defined in (Neyshabur et al., 2017), which considers loss change from the perspective of statistical expectation. And (Neyshabur et al., 2017) and (Jiang et al., 2019) refer we consider the mag- nitude of the perturbation with respect to the magnitude parameters and take the formulation as L(fw)], where each element of is random variable sampled from a noise distribution DD and L represents for optimization objective on the training set. From this perspective, the term (7-2) be interpreted the flatness with perturbation related input data, and thereby we could achieve the following corollary.
Eu-D (.f.DLL(fwcaa (mooo(1++)))
On calibration data æ, with activation quantization noise u(x). there exists the corre- sponding weight perturbation v(x) which satisfies that the trained quantized model is flatter under perturbation (az).
Corollary 1.
With Corollary 1, Case and 3 discussed in Sec. 3.1 enjoy flatter loss landscape benefited from perceiving the activation quantization. This explains their superiority compared with Case The measurement of sharpness calibration data (left part) in Fig. 2 further validates this point. With similar perturbation magnitude, Case 2 and suffer less loss degradation than Case
3.3
QRROP
As aforementioned. introducing activation quantization theoretically proved to produce a flatter model than existing works and the directions of flatness depend the data distribution. Since the PTQ is especially sensitive to calibration data (Yu et al., 2021), we need to transfer the investigations in Sec. 3.2 calibration data into the test setting for thorough understanding. In specific, consider Eq. (7) on test set and inspect two terms separately in the following. Based on the analyses, our method QDROP will be derived pursue an excellent performance on test data.
Term (7-1) on test set. As suggested in Sec. 3.2, with both quantized activations and weights, we additionally optimize the term (7-2) representing the flatness on calibration data. This term will encourage the quantized model to learn flat minimum. As result, the tradi- tional objective of AdaRound (term (7-1)) can naturally generalize better for test data (i.e., En-D, (L(10, æ, 1) L(w, 1))).
Term (7-2) on test set. Furthermore, we should also concern about the term (7-2) on test data, iee. EL-D, @ (1 + vxx)). x, 1) L((i, x. 1)]. As revealed in Sec. 3.2, the term (7-2) implies flatness where its situation on calibration data has been exploited. Here, we further investigate the flatness on test samples. Note that v(x) ) is converted from u(x) and this activation quantization noise varies with input data. Fig. shows that there is gap between the test data and calibration data for the flatness of the 3 cases. According to Corollary 1, these cases actually introduce different mathematically and thus will result in different flatness directions, given by
à
Case u { 0,
1, block block--1
Case 1: u Case2: u
(8)
a
blocks
5

---

Published as a conference paper at ICLR 2022
Algorithm 1: QDROP in one block for one batch
Input: the ky, block from layer to layer j. minibatch of quantized block input à' FP32 block input a'-1 FP32 block output a², quantization dropping probability p. {1. Forward propagation: During training phase, substitute à'-l with corresponding a'-l neuron-level with probability and mark the replaced input a'- for to do
a' t W'a'1; à' f- Quantize(a');
During training phase, randomly drop some a' with at as defined in Eq. (9) and get ā'
(2. Backward propagation:]
Compute Aad @' a²;
Tune the weight by gradient descent;
return Quantized block;
For Case there is activation quantization during calibration without taking flatness into account. Case 2 suggests the activation perturbation totally and therefore enjoys a good flatness on calibration data. However, due the mismatch calibration data and test one, it is highly possible that Case 2 causes overfitting (See Table 8 for more details). Case 3, in fact achieves the best performance by dropping some activation quantization along with little different weight perturbation and might be restricted to flatness on calibration data (More evidence can be found in Table 9). This inspires us to pursue a flat minimum from a general perspective, that only optimizing the target on calibration set is suboptimal to test set.
QDROP. Inspired by this, we propose QDROP to further increase the flatness on as many directions as possible. In particular, we randomly disable and enable the quantization of the activation each forward pass:
0 with probability p with probability 1 -
QDROP: : u
(9)
We name it QDROP because it randomly drops the quantization of activation. Theoretically, by masking some u(x) randomly, QDROP can have more diverse v(æ)) and cover more directions of flatness thus flatter on test samples, which contributes to the final high accuracy. Fig. 3 support our analysis where QDROP has smoother loss landscape than Case 3, the winner among cases on test data. Mcanwhile, it is indeed fine-gra fine-grained version of Case since Case 3 drops the quantization in a block-wise manner, whereas our QDROP operates in an element-wise way.
generalized form of the existing schemes. Case and respectively corresponds to the dropping probability of p = and p = O. Case equivalent to setting the block being optimized with dropping probability 1 and remains the quantization of
Discussions. QDROP can be viewed
L
4Losa
1
-.2
-0.1
0.0
-0.2
-0.1
-0.1
0.0
-0.2
-0.1
0.0
0.1
-0.2
0.1
-0.2
0.1
-0.2
(a) Case
(b) Case
(c) QDROP
Figure 3: Loss surface of the quantized weight for QDROP, Case and 3 on test data and ResNet- 18 W3A3. To better distinguish Case I and 3, we zoom into the local loss surface with perturbation i; and 82 magnitude |-0.025,0.025).

---

Published as a conference paper at ICLR 2022
Method
Bits (W/A)
ResNet-18
ResNet-50
MobileNetV2
RegNet-600MP
RegNet-3.2GP
MNasNet-2.0
No Drop
2/2
46.64
47.90
4.55
25.52
39.76
9.51
QDROP
2/2
51.14
54.74
8.46
38,90
52.36
22.70
No Drop
2/4
64.16
69.60
51.61
61.52
70.29
60.00
QDROP
2/4
64.66
70.08
52.92
63.10
70.95
62.36
Table 2: Effect of QDROP.
other parts. Note that the p obeys Bernoulli distribution and thus can be 0.5 for the maximal entropy (Qin et al., 2020), which is helpful for flatness across various directions.
QDROP is easy to implement for various neural networks including CNNs and Transformers, and Blug-and-pu with little additional computational complexity. With QDROP, the complicated prob- lem of choosing optimization order, i.e. different cases in Sec. 3.1, can be avoided.
4 EXPERIMENTS
In this section, we conduct two sets of experiments to verify the effectiveness of QDROP. In Sec. 4.1, first conduct an ablation study for the impact with and without dropping quantization and ana- lyze the option of distinct dropping rates. In Sec. 4.2, we compare our method with other existing approaches across vision and language tasks including image classification on ImageNet, object detection on MS COCO, and natural language processing on GLUE benchmark and SQuAD.
Implementation Details. Our code based on PyTorch Paszke et al. (2019). We set the default dropping probability p 0.5, except we explicitly mention it. The weight tuning method is the same with Nagel et al. (2020); Li et al. (2021a). Each block or layer output is reconstructed for 20k iterations. For ImageNet dataset, we sample 1024 images calibration set, while COCO we use 256 images. In NLP, we sample 1024 examples. We also keep the first and the last layer in 8-bit except NLP tasks and adopt per-channel weight quantization. We use W4A4 represent 4-bit weight and activation quantization. More model choices and other setting is described in Appendix E.
But to be noted, regular first and last layer 8-bit means 8-bit weight and input in first and last layer while BRECQ uses another setting which not only keeps the first layer's input 8-bit but also the first layer's output (second layer's input). This would indeed perform better than the regular leaving one more layer's input 8-bit but may not be practical on the hardware. Therefore, we both compare with BRECQ'S setting to show the superiority of our approach and experiment on the usual one to provide practical baseline. Symbol f is used to mark BRECQ's 's setting.
with
4.1 ABLATION STUDY
Effeet of QDROP. We propose QDROP and here we would like to test the effect of PTQ with or without QDROP. We use ImageNet classification benchmark and quantize the weight parameters to 2- and quantize activation to 2/4-bit. As shown in Table 2, QDROP improves the accuracy across all bit settings evaluated for 6 models on ImageNet. Furthermore, the gains more obvious when applying QDROP to lightweight network architecture: 2.36% increment for MNasNet under W2A4 and 12.6% for RegNet-3.2GG 2 with W2A2.
Effect of Dropping Probability. We also explore the dropping probability in PTQ. We choose p and test on MobileNetV2 and RegNet-600MF. The results are summa- rized in Fig. 5. We find 0.5 generally performs best among 5 candidates. Although there could be a fine-grained best solution for each ar- chitecture, we shall avoid cumbersome hyper- parameter search and continue using 0.5.
MohileNen-V2 WIAT
RegNr--iitMil W1A2
in (0.0.25.0.5.0.755
Tell Accurey
35.00
Probubility
Protabiliit
Figure 5: Impact of dropping probability on ImageNet.
4.2 LITERATURE COMPARISON
ImageNet. We choose ResNet-18 and -50 (He et al., 2016), MobileNetV2 (Sandler et al., 2018), searched MNasNet (Tan al., 2019) and RegNet (Radosavovic al., 2020). We summarize the
7

---

Published as a conference paper at ICLR 2022
Methods
Bits (W/A)
Res18
Res50
MNV2
Reg600
Reg3.2G
MNasx2
Full Prec.
32/32
71.06
77.00
72.49
73.71
78.36
76.68
ACIQ-Mix (Banner et al,, 2019)
4/4
67.00
73.80
ZeroQ (Cai et al., 2020)*
4/4
21.71
2.94
26.24
28.54
12.24
3.89
LAPQ (Nahshan al., 2019)
4/4
60.30
70.00
49,70
57.71
55.89*
65.32*
AdaQuant (Hubara et al., 2021)
4/4
69.60
75.90
47.16*
Bit-Split (Wang et al., 2020)
4/4
67.56
73.71
AdaRound (Nagel et al., 2020y*
4/4
67.96
73.88
61,52
68.20
73.85
68.86
QDROP (Ours)
4/4
69.10
75.03
67.89
70.62
76,33
72.39
AdaRound (Nagel ct al,, 2020)*
4/4
69.36
74.76
64.33
BRECQ (Li et al., 2021a)
4/4
69.60
75.05
66.57
68.33
74.21
73.56
QDRPP (Ours)
4/4
69.62
75.45
68.84
71.18
76.66
73.71
LAPQ (Nahshan al., 2019)*
2/4
0.18
0.14
0.13
0.17
0.12
0.18
AdaQuant (Hubara al., 2021)*
2/4
0.11
0.12
0.15
AdaRound (Nagel al., 2020)*
2/4
62.12
66.11
36.31
57.00
63.89
46.73
QDROP (Ours)
2/4
64.66
70.08
52,92
63.10
70.95
62.36
AdaRound? (Nagel al., 2020)*
2/4
64.14
68.40
41.52
59.27
65.33
53.77
BRECQ (Li et al., 2021a)
2/4
64.80
70.29
53.34
59.31
67.15
63.01
QDROPt (Ours)
2/4
65.25
70.65
54,22
63.80
71.70
64.24
AdaQuant (Hubara et al., 2021)*
3/3
60.09
67.46
2.23
QDROP (Ours)
3/3
65.56
71.07
54.27
64.53
71.43
63.47
AdaRoundf (Nagel et al., 2020)*
3/3
64.66
66.66
15,20
$1.01
56.79
BRECQ (li 2021a)*
3/3
65.87
68.96
23.41
55.16
57.12
49.78
QDROPF (Ours)
3/3
66.75
72.38
57.98
65.54
72.51
66.81
QDROP (Ours)
2/2
51.14
54.74
8.46
38,90
$2.36
22.70
BRECQ* (Li et al., 2
2/2
42.54
29.01
0.24
3.58
3.62
0.61
QDROPf (Ours)
2/2
54.72
58.67
13.05
41.47
55.11
28.77
Table 3: Comparison among different post-training quantization strategies with low-bit activation in terms of accuracy ImageNet. * represents for implementation according open-source codes and means using BRECQ's first and last layer 8-bit setting, which keeps first layer's output 8-bit besides input and weight in the first and last layer.
results in Table 3. First, the W4A4 quantization is investigated. It can be observed that QDROP provides 0 3% accuracy uplift when compared to strong baselines including AdaRound, BRECQ As for the gap between our method and AdaQuant on W4A4, we argue that there are some discrep- ancies on settings such as positions quantization nodes and put this explaination Sec. C.3. With W2A4 quantization, QDROP can improve the accuracy of ResNet-50 by 0.5%, and RegNet-3.2GG by 4.6%. addition, to fully exploit the limit of QDROP, we conduct more challenging cases with 2/3-bit weights and activations. According to the last two rows of Table 3, proposed QDROP consistently achieves good results while existing methods suffer from non-negligible accuracy drop. For W3A3, the difference is even larger MobileNetV2, where method reaches 58% accuracy and BRECQ only gets 23%. In the W2A2 setting, the PTQ becomes much harder. QDROP out- performs the competing method by large margin: 12.18% upswings for ResNet-18, 29.66% for ResNet-50 and 51.49% for RegNet-3.2GF.
MS COCO. In this part, we validate the performance of QDROP on object detection task using MS COCO dataset. We use both two-stage Faster RCNN (Ren et al., 2015) and one-stage RetinaNet (Lin al., 2017) models. Backbone are selected from ResNet-18, ResNet-50, and MobileNetV2. Note that we set the first layer and the last layer to 8-bit and do not quantize the head of the model, however, the neck (FPN) is quantized. Experiments show that W4A4 quantization using QDROP nearly do not affect Faster RCNN's mAP. For RethinaNet, our method has 5 mAP improvement on MobileNetV2 backbone. low bit setting W2A4, our method shows great improvement both on Faster-RCNN and RetinaNet, up 6.5 mAP.
GLUE benchmark and SQuAD. We test QDROP in NLP tasks including the GLUE benchmark and 1QuADI.1. They all conducted on the typical NLP model, i.e. BERT (Devlin et al., 2018). Compared with those QAT methods (Bai et al., 2020), which usually adopt data augmentation trick only randomly extract 1024 examples without any
to achieve dozens of times the original data, extra data processing. Besides AdaQuant and B BRECQ, which suffer a huge accuracy degradation, our ODROP surpasses No Drop all the tasks, specifically QNLI(8.7%) QQP(4.6%) and RTE( (7.2%).
8

---

Published as a conference paper at ICLR 2022
Faster RCNN
RetinaNet
Method
Bits (W/A)
ResNet-18
ResNet-50
MobileNatV2
ResNet-18 18
ResNet-S0
MobileNetV2
Full Prec.
32/32
34.60
38.56
33.47
33.22
36.80
32.63
AdaRound*
4/4
32.57
34.47
26.11
31.04
33.51
24.99
BRECQ**
4/4
32.58
34.59
26.58
31.21
33.47
24.84
QDROP
4/4
33.37
36.96
30.88
31.99
35.67
29.75
AdaRound*
2/8
30.54
33.15
25.35
29.30
32.22
24.22
BRECQF
2/8
31.82
34.23
27.54
31.42
34.75
27.59
QDROP
2/8
32,20
36.14
28.48
31.03
34.84
27.42
BRECOF*
2/4
29.92
30.23
19.35
28.73
29.47
18.46
QDROP
2/4
31.01
34.23
25.04
29.69
33.01
24.89
Table Comparison among typical post-training quantization strategies in terms of mAP MS COCO. Note that refer BRECQ, we didn't quantize head and keep first and last layer in backbone 8-bit. Other notations align with Table 3.
Method
SST-2
QNLI
QQP (El/ace)
STS-B
MNLI (ace m//mm)
MRPC (acc)
CoLA (Matthews corr)
SQUAD1.1
(ace)
(acc)
Cecuason/Spcarman coI))
(a00)
(I])
Prec.
37.81/90.91
84.57/84.46
AdaQuant*
5.17
BRECQ"
4.47/62.22
5.94/
31.91/81.81
No DROP
87.94
68.05
68. 09.09976.69
82 #2.24/81.6 1.68
69. 19/77/21.2
77.39
53.43
40.17
75.97
Table 5: Performance on NLP tasks compared to other methods on E8W4A4. Here, we use symbol EeWwAa additionally express the embedding bit and conduct experiments on GLUE and SQuADI.I.
As for SST-2, despite little enhancement by dropping quantization, it is indeed close to the FP32 value within 4.4%. And for STS-B, we argue that the original fine-tuned model trained with limited data, which might not be very representative.
4.3 ROBUSTNESS OF QDROP
In this part, we discuss the effectiveness of QDROP under more challenging situations including even less data and cross-domain ones. Concerning about size of calibration data, we consider another 4 options. It can be observed that dropping some quantization behaves better under each setting and is even comparable with No Drop with half of the original calibration data. Motivated by Yu et al. (2021), we also reconstruct block output by 1024 examples from out-of-domain data, i.e, CIFARION 100 (Krizhevsky al., 2009), MS COCO, and test on ImageNet. Results are available in Table 6, where our QDROP still works steadily.
ResNet-18 W3A3
RegNet-600MF W2M
Calibration
Bits (W/A)
No Drop
QDROP
Accoracy
64
MS COCO
4/4
68.64
68.94
62
MS COCO
3/3
64.12
65.15
QUrop
QUrop
No Drap
No Deap
10FAR100
4/4
46.83
52.88
CIFARI00
3/3
21.74
28.58
Calibration Amount
Callbration
Figure Impact calibration data size on ImageNet.
Table Cross domain data.
5 CONCLUSION
In this paper, we have introduced QDROP, a novel mechanism for post-training quantization. QDrop aims achieve good test accuracy given tiny calibration set. This is done by optimization towards a flat minima. We dissect the PTQ objective theoretically into a flatness problem and improve the flatness from general perspective. We comprehensively verify the effectiveness of QDROP large variety of tasks. can achieve a nearly lossless 4-bit quantized network and can significantly improve the 2-bit quantization results.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENT
We sincerely thank the anonymous reviewers for their serious reviews and valuable suggestions to make this better. And we thank Xiangguo Zhang and Sheng Chen for their kind of help of this work. This work was supported in part by the National Natural Science Foundation of China under Grant 62022009 and Grant 2182021,, the SenseTime Research Fund for Young Scholars, and the Beijing Nova Program of Science and Technology under Grant Z191 100001 119050.
REFERENCES
Haoli Bai, Wei Zhang. Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit bert quantization. arXiv preprint 2rXiv:2002..5500) 2020.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances Neural Information Processing Systems, 2019.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: novel zero shot quantization framework. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13169-13178, 2020.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural net- works for efficient inference. In ICCV Workshops, pp. 3009-3018, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.0000 04800, 2018.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp minima can generalize for deep nets, 2017.
Zhen Dong, Zhewei Yao, Yaohui Cai, Daiyaan Arfeen, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq-v2: Hessian aware weighte quantization of neural networks. arXiv preprint arXiv:191.00385 2019.
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S Modha. Learned step size quantization. arXiv preprint 19020:1902.0815: 2019.
Angela Fan, Pierre Stock, Benjamin Graham, Edouard Grave, Rémi Gribonval, Herve Jegou, and Armand Joulin. Training with quantization noise for extreme model compression. arXiv preprint MXXi:2000.003220 2020.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. arXiv preprint arXiv:2010.01412, 2020.
Yonggan Fu, Qixuan Yu. Meng Li, Vikas Chandra, and Yingyan Lin. Double-win quant: Aggres- sively winning robustness of quantized deep neural networks via random precision training and inference. In International Conference on Machine Learning, pp. 3492-3504 PMLR, 2021.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint arXiv:1510.0014 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 02531, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 9(1):1-4,, 1997.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. International Conference on Machine Learning, pp. 4475- PMLR, 2021.
10

---

Published as a conference paper at ICLR 2022
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wil- son. Averaging weights leads to wider optima and better generalization. arXiv preprint arXin:18033.5540,,2200 2018.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient intege-arritmmeiioonlly inference. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2704 2713, 2018.
Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. arXiv preprint 191200 2019.
Prad Kadambi, Karthikeyan Natesan Ramamurthy, and Visar Berisha. Comparing fisher information regularization with distillation for dnn quantization. 2020.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv:1609 160904460 2016.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non- uniform discretization for neural networks. arXiv preprint arXiv: 13/44, 2019.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv: 102.0520, 2021a.
Yuhang Li, Mingzhu Shen, Jian Ma. Yan Ren, Mingxin Zhao, Zhang, Ruihao Gong, Fengwei Yu, and Junjie Yan. MQBench: Towards reproducible and deployable model quantization benchmark. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 1), 2021b. URL httts:: hppp///peeeeeeeeeeeeeeeeeeeeeeee net/ Corum?id=TUplll OnF 8DsM.
Yuhang Li, Feng Zhu, Ruihao Gong, Mingzhu Shen, Xin Dong, Fengwei Yu, Shaoqing Lu, and Shi Gu. Mixmix: All you need for data-free compression feature and data mixing, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4410-4419, 2021c
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on compuler vision, pp- 2980-2988, 2017.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEF/CVF International Conference Computer Vision, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort, Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning. pp. 7197-7206. PMLR, 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint arXiv: 2019.
Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nathan Srebro. Exploring gener- alization in deep learning, 2017.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32: 8026-8037, 2019.
11

---

Published as a conference paper at ICLR 2022
Haotong Qin, Ruihao Gong, Xianglong Liu, Mingzhu Shen, Ziran Wei, Fengwei Yu, and Jingkuan Song. Forward and backward information retention for accurate binary neural networks. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2020.
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV,, 11(3))211-252, 2015. doi: 1007/s (0007/s11263-000 1263-015-0816-y.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Ching-Chieh Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Mingzhu Shen, Feng Liang. Ruihao Gong, Yuhang Li, Chuming Li, Chen Lin, Fengwei Yu, Junjie Yan, and Wanli Ouyang. Once quantization-awes training: High performance extremely low- bit architecture search. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5340-5349, 2021.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2820-2823 2019.
Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate vost-training network quantization via bit-split and stitching. In Proc. 37nd Int. Conf. Mach. Lear.(ICML), 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener-
alization. arXiv preprint 2xXiv:2004.05844 2020.
Guandao Yang, Tianyi Zhang, Polina Kirichenko, Junwen Bai, Andrew Gordon Wilson, and Chris De Sa. Swalp: Stochastic weight averaging in low precision training. In International Conference on Machine Learning, pp. 7015-7024. PMLR, 2019.
Haichao Yu, Linjie Yang, and Humphrey Shi. Is in-domain data really needed? pilot study on cross-domain calibration for network quantization. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR) Workshops, pp. 3043-000333 June 2021.
Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quan- tization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2021a.
Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan, Renshuai Tao, Yuhang Li, Fengwei Yu, and Xianglong Liu. Diversifying sample generation for accurate data-free quantiza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 15658-15667, 2021b.
Yaowei Zheng, Richong Zhang, and Yongyi Mao. Regularizing neural networks via adversarial model perturbation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8156-81 8165, 2021.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv:1611.01578 2016.
12

---

Published as a conference paper at ICLR 2022
A NOISE FORM CHOICE
As mentioned in Sec. 2, we refer the quantizer to = L 8, where is the step size that can be learned or be determined by collecting activation distribution in PTQ. By marking the rounding error as c, which subjects to 21[ (4.5 0.5), the additive noise e = à - a can be denoted as c s. However, this traditional noise does not decouple the noise from step size and the range of activations. The range of noise will vary when the range of activations change, making it complicated to analyze across the whole network in unified form. Some existing papers (Jiang et al., 2019) also indicate the problem that the additive noise doesn't 't take parameters' magnitude into account and suggest multiplicative form form (Keskar et al., 2016). To be specific, they claim that:
Perturbing the parameters without taking their magnitude into account can cause many of them to switch signs. Therefore, one cannot apply large perturbations the model without changing the loss significantly. One possible modification to improve the perturbations is to choose the perturbation magnitude based on the magnitude of the parameter. In that case, it is guaranteed that if the magnitude perturbation less than the magnitude of the parameter, then the sign the parameter does not change.
To eliminate the influence induced by the range of activations, we take the noise of multiplicative form: à (1 + u) In the quantization background, we denote the fixed-point integer value with respect to as (a = (ā + c) 8 and à = ā s). And then u can be denoted as:
(a + c)
(10)
ā a+c c
ā c
To be noted, и is derived from the definition of the quantizer and can be equivalently transformed from the generalized noise here.
From this formulation, we can find that the range of u is not related to the activation range or step size and only influenced by the rounding erгor and the bit-width. In word, the multiplicative form has its pysical meaning in quantization background and is beneficial as discussed above.
B PROOF OF LEMMA AND THEOREM
We demonstrate them by considering the situation (1) fully connected and (2) convolutional net- works separately. As transformation with fully connected ones has been revealed in main body, add some extra illustration here and mainly target at the convolutional layers.
B. FULLY CONNECTED NETWORKS
Here, we give the proof of Eq. (5) by leveraging the definition of FC layers in Eq. (1). We first look at each input sample and temporarily omit æ in the notation below for simplicity.
With activation noise tt,
(e+1) Σ w(O (1 u ) ,0
(11)
2u+ w a.a
By taking v:O = "0 we have
(e+1)
[1 v2(0)
w (2) 0
z
(12)
13

---

Published as a conference paper at ICLR 2022
Thus, with proper constructed v, the noise activation (u) can be viewed as the perturbation weight (v).
For every layer, ing on the quantized weight and calibration data, optimizing E EEDD, [L([] æ, 1 + w(x)) can be
conduct this transformation from
Therefore, considering operat-
approximated to optimizing EgzD, [L(ri (1 v(x)). x. 1)}. Then we have
Eg (Lú æ, + u(x)) L(w, æ, 1)] EELD, [L(r @ (1 vax)), æ. 1) L(w, a, 1)]. (13)
Now, Lemma is proved for the case of fully connected networks
B.2 CONVOLUTIONAL NETWORKS
We first look at each input sample and temporarily omit æ in the notation below for simplicity, One layer in the network G can be interpreted
((+1) = f( = I w (()
Atpps+g)
(14)
where the /()) is the activation function, (p, q) pair represents for one element in weight matrix and (1,5) pair represents for one element in activation matrix.
Due to the complicated design of convolutional structure, based on we introduce two networks G1, G2 to make proofs more clearly.
Definition 1. 91 means inserting random noise on activations, G2 means sticking random variables into weights.
91 A (e+1) 5(Z
f( f
W p.o
(() (1
(tp.stt) A
(15)
G2 A (1+1) |(z
S E V(O)
W (0) pig (fpp+ag))
p-2
Definition 2. We mark losses of network 91 and G2 in the following way:
Loss of G1 L(w: x. u) Loss of G2 L(w @ (1 v),
when taking
it becomes L(w, x. 1)
(16)
when taking a = 0, it becomes L(w @ 1,
With the definitions, we first prove that 91 and G2 share common parts in their first-order derivative to its noise in Lemma 2.
Lemma 2. Assuming the same weight and taking 0 and v for Gi and G2. we have
θL(w, 1,
(e)
aU!
(17)
@L(w @ 1.
T (
0v (2)
(r.))
i.j
where
T (1)).6)))))
0L(w, x, 1)
(' [e+1) (
W(!)
0A ((+1)
1-pj-g
(18)
0L(w @ 1,
('(
W(O 1
A (++1)
i-po-g
Note that with = 0 and 0, the activations are the same for the two networks we don't different notations here.
Proof.
aL(w, ar,
@L
fZ (
WC A
(19)
(8) OU!
0A
(f-1)
P.g
4.
P.g
14

---

Published as a conference paper at ICLR 2022
@L( @ 1)
aL(w + 1,
W P. itput4
0A 1.
(E+1)
i.j
(20)
( O 1,
f(Z
W A
0A (++1) P.jー
Because of taking the same weight, and calculating the derivatives to U. at and derivatives at O, we have the same activation values for these two networks. Therefore, Eq. (21) holds.
(E)
0 -Ai-nj-q (e+1) l' (++1) (p3--)) W A aL(w O 1,x) -Ai-p.j-o (t+1) (Z W A (21)
0L @L(w, 1)
By marking the above value as T (i.))(na)) we can get Lemma 2.
Based on Lemma 2, we further derive Theorem 2 as the pre-condition of the final Lemma 1.
Theorem 2. By taking
Eis U. (1.))
(e)
V p.g
(22)
T(0)) T Ens
(15).(p)))
we have
u VuLLw,, v V.Llw 11 1.
(23)
Proof. According to Lemma 2,
aL(w, x. 1)
V I(w, x. 1)
ΣΣ U
au!
i.j
P-g
(n)
(24)
(i.i))P)))
P-
Σ 1
U. (L.))(P)))
(E) -1
(
Eis T:()) (P)
P-
(1.))))))))
Σn 0 T,(
(L.))a))
aL(w 1,
Σng T(()) (
OV 9.9
P-
By taking (2(0 Ec, (1()) T(()
E,, (:()) (1.)..c.))
0L(w 1,
T VuL,w x.
ΣΣ V p.y
pVp (0)
(25)
P.N
v TV V.L(W O 1, 1,
Thus Theorem 2 affirmed.
We now prove Lemma equipped with Taylor Expansion technique and Theorem 2.
Proof. First, by adopting Taylor Expansions O, get that:
L( x. u. T( x. 1) L(w,, ac, 1) u Vn Tu, ær, I(w, xr, 1).
(26)
15

---

Published as a conference paper at ICLR 2022
Then, according to Theorem 2, the above equation can be rewritten L(ti, æ, 1 u) L(w, az 1) ((w @ 1, V L @ 1, L(w, az 1).
(27)
Again, by adopting Taylor Expansions at 0 for the right part of Eq. (27), we arrive at: L(i æ; 1 + u L(w, æ. 1) ~ L(w @ (1 v)) - L(w, æ, 2, 1).
(28)
Finally, apply expectation on Eq. (28), and Lemma proved:
E Be-D. (L æt 1 + u())) L(w, 2, 1)] ~ EE-D. L(r o (1 v(x)), - L(w, x, 1)].
(29)
With the Lemma proved, we easily derive Theorem 1 by the following transformation:
Em-D, D. [L + (x:) L(w, æ, 1)]
EgnD. [[u @ [1 + v vxx)) ([)x))03 x) L(w, x. 1)]
(30)
L(w, xr 1) + L( x. 1) I( m, 1)] B [L v, 1) - (w, x, x. + L( o v (x)) L( x. 1)].
BEND, : [T @ (1 + v(m) (. x,
C EXPERIMENTS
C.I SUPPLEMENTARY EXPERIMENTS OF SEC. 3.1
To explore the upper limit of PTQ, we concern two key parts of the algorithm (weight quantization and activation one). As is generally known, QAT is a popular way to produce favor quantization results, thus we use QAT's learning mechanism to replace each of the two parts in PTQ and record outcomes in Table 7. In detail, about weight quantization, QAT's settings we used include the whole ImageNet, STE learning strategy and end-to-end training for 40 epochs while PTQ' settings only cover 1024 samples, training block by block. But we both keep the punding-up-or-dowe parameter optimization space. About activation quantization, QAT' settings we used include the whole ImageNet, LSQ (Esser et al., 2019) scheme and end-to-end training for 5 epochs.
However, results Table 7 are surprising that although the optimization space in weight kept very restricted, leveraging the whole data to implement weight tuning can achieve a large accuracy boost. And the activation quantization step size might not be as important as weight. These findings indicate that exploring quantization -uuntiiiaiionffrennllll weight may be the fresh insight for the accurate PTQ, where this work dedicates to.
C.2 SUPPLEMENTARY EXPERIMENTS OF SEC. 3.3
Overfitting phenomenon. To analyze the overfitting problem refered in Sec. 3.3, have ducted experiments to compare the accuracy on test data and calibration data, respectively. The table below is example of ResNet-18 W2A2. can be seen that with extremely low-bit quantiza- tion, both Case 2 and Case 3 perform well on calibration data but on test data Case 2 performs worse than Case 3. This is shred of clear evidence that Case 2 suffers a more severe overfitting problem.
Activation setting
Weight setting
Accuracy
Methods
Test accuracy
Train accuracy
PTQ
PTQ
46.64
Case
18.88
19.82
Case 2
45.77
69.54
QAT
PTQ QAT
49.53 57.49
Case 3
48.07
69,92
PTQ
QDROP 51.14
66.11
Table 7: Ablation study quantization settings on R8 W2A2.
Table Train and test accuracy on ResNet-18 W2A2.
Rethinking Theorem 1, both Case 2 and Case 3 introduce the term (7-2) that implies flatness against the perturbation u (x). However, Case 2 completely introduces the quantization noise u(x) accord- ing to the calibration data. Thus the resulting flatness fits calibration data but does not generalize
16

---

Published as a conference paper at ICLR 2022
well test data. Case drops partial of ux)) and improves the possibility of flatness test data instead. The two accuracy of QDROP also confirmed this phenomenon with more diverse directions of flatness. As for Case 1, its accuracy on both calibration data and test data is low. This is be- cause that it does not introduce activation quantization during weight tuning and thus behaves worst without taking flatness term even calibration data into account.
Hessian information. As Hessian infor- mation is known to be a metric to char- acterize flatness property (Keskar et al., 2016: Dong et al., 2019), we also calcu- late the top-1, top-5 Hessian eigenvalues (A1; Aς) and the Hessian trace (Tr) among 3 cases and QDROP to better support our analysis.
Method
A5
Tr
Case 1
14770
6746
122894
Case 2
8423
4050
86287
Case 3
8258
3821
84044
QDROP
6850
3044
66371
table Table 9, QDROP has the smallest value of Hessian information, which matches with our theoretical frame- work and observations of loss landscape.
Table Hessian information of the ResNet-18 W3A3 model. represents for the top-1 Hessian eigenvalue, Às for top-5 Hessian eigenvalues and Tr for Hessian trace.
C.3 SUPPLEMENTARY EXPERIMENTS OF SEC.
To clarify the gap between AdaQuant and our method in classification task on 4-bit ResNet- and -50, we analyze the difference on settings between these two algorithms. There are two major discrepancies which would influence the final accuracy. One is the position of activation quanti- zation nodes, the other is the FP32 accuracy of pretrain-models. As shown in Fig. 7, by insert- ing the quantizer like the right side one, AdaQuant would introduce two different quantizers for the same input with learned quantization parameters such step size. We find this can bring 0.4% upswings on ResNet-18 and -50 W4A4 and improve more on ultra low-bit with exper- iments of our algorithm. However, this way to insert quantization nodes is not practical in real deployment, because two different quantizers for the same input would make it impossible to fol- low the so-called Requantize procedure (Li et al., 2021b). As for the prodrain-models, they 71.97% and 77.2%, higher than ours (71.06% and 77.0%), which can indeed improve the accuracy. Replacing settings with the two adopted in AdaQuant, finally reach 71.07% and 76.67%.
X
PalceQuant
Fake
Method
(W/A)
ResNet-18
SWAze
Coov
Downdampl
Cocv
Downample
FP32
71.06
71.50
8/8
70.94
71.40
ReLU
ReLU
FakeQuant
Min-Max
4/8
52.33
65.26
PaiceQuant
PuiceQuam
Conyy
44.37
32/4
64.15
65.99
Conv
Conv
OMSE#
(Choukroun et al., 2019)
4/4
38.32
55.86
BRECy+*
65.35
66.33
ReLU
ReLU
(li al., 2021a)
2/2
41.74
44.30
Experiments between ResNet-18 and SWA20 when adopting different PTQ methods. SAA is acquired by finetuning ResNet-18 using SWA technique for 20 epochs.
Table 10:
Figure 7: Different ways of inserting activation quanti- zation nodes. Our method oheys the left side one while AdaQuant adopts the right side one.
C.4 FLATNESS AND POST- TRAINING QUANTIZATION
While there are plenty of works exploring the correlation between flatness and generalization, the interaction between quantization and flatness has not been exploited much. In this paper, first connect flatter quantized weight with activation quantization from PTQ view, as implied in Sec. 3.2. From this, conjecture that flatness and quantization might be helpful to each other.
By leveraging some mechanisms devoting to produce a smoother loss surface for better generaliza- tion, such (lzmailov al., 2018), the improved FP32 model is obtained and used validate the
17

---

Published as a conference paper at ICLR 2022
performance on quantization compared with the naive one. In Table 10, model SWAz0 is enabled by applying the SWA technique (lzmailov et al., 2018) to ResNet-18 with 20 epochs fintuning process on the whole ImageNet. From the table, the observation is that the promotion not fully represent the enhancement induced by SWA on quantization. With even lower bits thus larger noise, SWA20 surpasses the naive one by a large margin. It also reveals that distinct FP32 models with analogous accuracy might contribute to surprisingly disparate outcomes after PTQ, particularly for those naive methods without any weight tuning.
generalization can
In turn, there have been some studies that delve into robustness boost by applying quantization. (Fu et al., 2021) advocates that quantization can be properly leveraged to enhance DNNs' robust- ness, even beyond their full full-precision counterparts. They propose a random bit training strategy to accomplish it, where our work illustrates the correlation with bit and perturbation in Appendix A.
D RELATED WORKS
Post-training quantization. Unlike QAT Esser et al. (2019); Li et al. (2019); Shen al. (2021) where the quantized model is finetuned with full training dataset and over 100 epochs training, PTQ much more faster. Rounding-to-nees nearesst operation is known to be the direct and easy way for quantizing parameters or activations in PTQ. Although there is almost no accuracy drop when quan- tizing to 8-bit, lower bit quantization is yet a hard task and worth exploring. (Choukroun et al., 2019) transforms quantization to Minimum Mean Squared Error problem both for weights and activations. (Nagel et al., 2019) equalizes weight ranges among channels thus be more favorable to per-layer quantization and employs bias correction absorb the output error induced by quanti- zation. However, such methods neglect the task loss thus lead to a sub-optimal. AdaRound (Nagel et al., 2020), which proposes to learn the rounding mechanism by reconstructing output layer by layer brings more opportunities for 4-bit quantization. Besides layer reconstruction, BRECQ (Li al., 2021a) discusses more choices and advises to do block reconstruction with better accuracy at 2-bit weight quantization. Nonetheless, we argue that AdaRound and BRECQ isolate weight quan- tization and activation one theoretically and experimentally, which might be a key point of failures on extremely low-bit quantization. Recently, there is another trend of utilizing synthetic data for PTQ. which explicitly do backpropagation on the learned input tensor Cai et al. (2020); Zhang et al. (2021b); Li et al. (2021c).
Flatness. The idea of "flat" minima might date back to (Hochreiter & Schmidhuber, 1997), where the benefits are recognized in recent years, such generalization (Jiang et al., 2019; Keskar et al., 2016) and adversarial training (Wu et al., 2020; Zheng et al., 2021). Some previous works (Izmailov et al., 2018; Foret et al., 2020) devote to improve the flatness of the trained weight for robustness under perturbation or distribution shift. Other ideas try to model flatness or sharpness formally by visualization of loss landscape or mathematical formulas. And for quantization, which could be viewed as some kind of noise, a flat model has been implied to be preferable, (Dong et al., 2019: 9;; Yang et al., 2019; Kadambi et al., 2020). Despite the natural fact that flatness helps with weight quantization, how does activation quantization involves with smoother loss surface has not been dis- cussed deeply, particularly for post- post-training quantization. In this work, we introduce noise scheme by randomly dropping activation quantization and achieve general flatness. Another paper (Fan al., 2020) also utilizes randomness by adding noise to weight for the simulation of weight quan- tization. But they target at reducing the induced bias of Straight Through Estimation (STE) in QAT and also have different motivations and solve different problems from us.
E IMPLEMENTATION DETAILS
Observation. Here, we give the concrete implementation of experiments in Sec. 3.1. We actually consider three ways of introducing activation for Case 2, but we find the differences of outcomes among them are negligible thus employing one of them for clearer clarification.
ImageNet. We randomly extract 1024 training examples from ImageNet as calibration dataset based on the standard pre-process. vectrain-model are downloaded from BRECQ'S open source code. we keep it BRECQ, such batch size 32, learning rate for activation step size 4c-5, learning rate for weight tuning 1e-3, iterations 20000. Following BRECQ we first fold batch normalization layer into convolution then reconstruct output block-wise to learn the weight
18

---

Published as a conference paper at ICLR 2022
Algorithm 2: Implementations of three cases in Sec. 3.1
Input: Model with K blocks.
if Case 2 then
for k to do
parameterize activation step size in this block;
for k 1 to K do
Tuning weight like (Li et al., 2021a) by reconstructing block output if Case 3 then
parameterize activation step size in this block ;
Case then
for k ==140 to K do
parameterize activation step size in this block ;
return Quantized model;
rounding policy and meanwhile use LSQ (Esser et al., 2019) parameterize activation step size. For QDROP, we learn the weight and activation parameters together and use 50% rate to drop some activation quantization.
Object detection. Here, we also obey BRECQ' settings and use the same pretrain pretrain-models with 256 training samples taken from MS COCO dataset for calibration. Parameters about resolution set to 800 (max size 1333) and 600 (max size 1000) for ResNets and MobileNetV2, respectively and batch size is set while others are the same with classification task. To be noted, we didn't 't quantize the head but applied block reconstruction to backbone and layer reconstruction to neck like BRECQ.
GLUE benchmark and SQuAD. The BERT fine-tuned models are taken from huggingface group (httss://ugginnffcce.//// And we sampled 1024 examples from training set. We keep the maximum sequence length to 128 for GLUE benchmark but maximum sequence length 384 with doc stride 128 for SQuAD1.1. Also, the attention module with only affirming the embedding weight to 8-bit. Other settings and hyper- parameters are chosen in the same way with ImageNet experiments.
quantize all the part in BERT as well the internal structure of
Baselines. We run baseline methods from open-source codes, such as AdaQuant, BRECQ and Adaround. And try best align some optimal settings like per-channel quantization for fair comparisons.
19


Reviews:
[{"review_id": "uYcOxJnAMp", "reviewer": "Reviewer_gKCS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n1. Do exploration on activation quantization and transforming the activation quantization into weight perturbations is a good idea.\n\n2. Experimental results on many tasks and models show the QDROP's advantages.\nAblation studies also support the analysis.\n\n3. This paper is generally well organized.\n\nWeakness:\n\n1. The terms of 'flatness' and 'sharpness' are not well defined, it might make readers confused on these and hard to follow.\nCan the authors provide more detailed explanations on these two terms? \n\n2. About Figure 3, the reviewer did not find any related explanation about these figures.\n \n3. The reviewer expected a discussion on how does activation quantization involves with smoother loss surface,\nwhile it looks missing in this work.\n\nMinor issues:\n\n1. Impact of dropping probability on ImageNet is performed on two network structures. \nCan the authors provide more experiments on other networks? That will be more convincing.\n\n2. On page 3, it should be H^{w} in the line below the equation (3)?", "summary_of_the_paper": "The authors proposed a random dropping quantization method at the post-training stage\nto achieve a low bit quantization network.\nBy observing the performance on partial activation quantization, \nthe authors analyze the influence of incorporating activation quantization into weight tuning.\nExperimental results show that their QDROP methods have benefits on several scenarios, such as detection and NLP tasks.", "summary_of_the_review": "Overall, I think this will be an instructive work if the authors can tackle the reviewer's concerns. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "RqmCsOR7prS", "reviewer": "Reviewer_iK8W", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper presents a strikingly simple approach to activation quantization. It boils down to a per layer modified dropout of the quantized activation, where either the network backdrops through the quantized activation or the full precision activation. Since approach is developed and implemented via layer wise fine-tuning, we could say that the approach is \"greedy\" but with the benefit of being palpably easy to implement in practice. \n\nAlthough the empirical results provided by the authors show that in practice the approach works well enough, I would like to see the authors to push the envelope further and try to use 1st or 2nd order gradient information (i.e. gradient or hessian) to better inform when applying their dropout is actually necessary (and to be even more clear, I'm thinking things like the Hessian criteria presented in HAWQ-v2 which the authors reference in their argumentation in favor of flatter models in Section 2 or the Gradient loss presented by Lee et al. in \"Data-free mixed-precision quantization using novel sensitivity metric\" https://arxiv.org/abs/2103.10051).\n\nI would also like to author to explain how their approach relates and differs to the one introduced by Fan et al. at ICLR21 in \"TRAINING WITH QUANTIZATION NOISE FOR EXTREME MODEL COMPRESSION\" since both approaches seemed to boil down fine-tuning the model under a noise quantization modeling scheme. I don't necessarily need a numerical comparison, but simply a brief mention in the related work section would be sufficient to better inform the reader of the potential connection between these 2 works.\n\nOverall, I'm satisfied with the paper, and I think it's in a good enough shape to be published at ICLR.\n", "summary_of_the_paper": "The authors conduct theoretical studies on how activation quantization affects weight tuning, and their conclusion is that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data. They present both empirical and theoretical findings, and propose the QDROP algorithm to exploit this phenomenon. QDROP randomly drops quantization during post-training quantization (PTQ) reconstruction to pursue the flatness from a general perspective. The authors also claim a new state of the art for PTQ on various tasks including image classification, object detection for computer vision, and text classification and question answering for natural language processing.", "summary_of_the_review": "The paper presents a simple approach to post training activation quantization that should be easy to implement and test by other researchers. The authors present both a mathematical treatment and empirical results to back up their claims and I wasn't able to find any glaring error. Hence, I think the paper is good and should be accepted. I expect their code to be jointly published with their paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q4v6tEBvTkB", "reviewer": "Reviewer_Y2p4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Post-rebuttal\nThe response well address my concerns, and I would like to increase the score.\n-----------------------------------------\nPros:\n1. The idea of considering the activation quantization during reconstruction of PTQ is significant. As the authors demonstrate in Table that Case 1 underperforms Case 2 obviously, assigning more optimal weight calibration is highly demanded.\n\n2. The observation of loss landscape flatness is also beneficial to the design of PTQ. Although the property of loss landscape of neural networks have been widely studied, the importance to PTQ remains to be explored.\n\n3. The excremental results clearly show the presented methods outperform the SOTA especially in lightweight architectures such as MobileNet-V2, and the accuracy of 4-bit PTQ seems to be promising in large-scale dataset such as ImageNet.\n\nCons:\n1. I think the core problem of this paper is the technical soundness of Eq. (7). I do not think the approximation in Eq. (14) and (15) is accurate Since the summation of the production in (19) is assumed to be exchangeable, is this always true? Otherwise, the form of Eq. (7) will be very complex.\n\n2. The writing needs to be improved. The notations should be well defined before use. For example, what do $u(x)$ and $V(x)$ exactly mean? The authors only say  $1+u(x)$ represent the noise and this is not the formal definition.\nMeanwhile, the connection between Eq. (5) and (6) should be specified.\n\n3. What is the relationship between Corollary 1 and Eq (8)?\n\n4. Figure 3 is not explained in the mainbody. What information can this figure convey? Meanwhile, why case 1 and case 3 seems to be similar?\n\n5. The connection between flatness and randomly activation quantization dropping needs to be explained, i.e., the physical meaning of Eq. (9).", "summary_of_the_paper": "This paper proposes the post-training quantization for extremely low-bit neural networks. By considering the activation quantization during reconstruction, the presented QDrop randomly drops the quantization of activations with higher loss flatness that adapts the activations with various activation bitwidths well. Experimental results have demonstrated the superiority of the presented method.", "summary_of_the_review": "Please see the Cons in the main review.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "GLZCTHcu7et", "reviewer": "Reviewer_5Zj1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well-organized, and its goal is well-oriented by providing a good motivational experiment. Theoretical analyses are well-balanced with empirical observations to offer a valuable insight that the flatness of the model is crucial to the extremely low-bit PTQ. However, there seem to be some glitches in formulating and presenting a theoretical framework. The novelty of the QDrop algorithm itself is moderate and simple, but the intuition behind it is critical. Experimental results look promising. Here are a few concerns and questions: \n\n1. An essential basis of the given theoretical framework is an assumption that the activation quantization could be modeled as injecting a multiplicative form of perturbation, say a^ = a + e = a + au = a(1+u) = as, where s is a scaling factor. Is this assumption reasonable in an actual situation? If this assumption is not available, is the conclusion still valid? Is it possible to be represented by any generalized form of perturbation?\n\n2. An operator used in the vector/matrix multiplication should be defined. Is it element-wise multiplication or ordinary matrix-vector multiplication? Does it have a commutative property? For example, in Eq.(5) and the equation above Eq.(5), could you check the dimension of vectors and matrices? Could you provide any proof of Eq.(5)? It seems to be crucial to prove the following Lemma 1 and Theorem 1. The proof of Sec. 4.2 provided in Appendix A is a bit confusing to the reviewer. Could you clean up the critical statements by setting up Corollary or Lemma ones?\n\n3. Several vague and confusing terminologies were used in Corollary 1. For example, the trained quantized weights are usually(?) flatter on some(?) directions and more robust under some(?) perturbations for calibration data. Is there any other case that the trained quantized weights are not flattering? Again, technical terms should be used to give some intuitions. \n\n4. In Section 4.3, the authors claim that it is highly possible(??) that Case 2 causes overfitting due to the mismatch on calibration and test data. However, Case 2 and Case 3 seem to perform activation quantization based on Eq.(7) using the same calibration data. Could you prove that the relatively low performance of Case 2 compared with Case 3 is due to the overfitting? Their performance gap is just 2% in most of the models in Table 1. If overfitting occurs during the weight tuning process, it seems to be suitable to arise in Case 1, which does not perform activation quantization. There should be a supplementary explanation for this by comparing the results before and after weight tuning.\n\n5. When equivalent compression levels are applied by quantization, it seems more evident that the proposed method can improve generalization performance while preventing expected compression losses. Could you evaluate the average precision(bit-width) of QDrop in the experiments? If the average precision is not critical to this case, could you explain the reason?\n\n6. It is generally known that the performance of QAT is relatively good compared to PTQ. Are some ablation studies necessary to provide supplementary experiments with a QAT setting in Appendix B?\n\nMinor Comments: \n1. (Singh et al., 2019) is wrongly cited on page 2. Please refer to (Nagel et al., 2019) and (Banner et al., 2019) instead. \n2. There are several typos: (1) are->is on page 2, (2) for each architecture, we ~~ on page 7, (3) origin->original on page 9, (4) Fig. 5.3 should be replaced with Table 6 on page 9, (5) Table ?? on page 15.\n3. In Table 2 and Table 6, please unify the same terms, say No Drop vs. No QDROP. Does it correspond to Case 2?\n4. The caption title is missing in the table on page 15. In the table, what is SWA_20? and 32/4 is right in OMSE? It seems that the order of rows in OMSE should be sorted in this case.\n", "summary_of_the_paper": "This paper aims to analyze how activation quantization affects the PTQ process and provides some theoretical analysis and experimental results on the effects of activation quantization. The authors argue that previous studies only model the weight quantization as perturbation while ignoring activation quantization, causing a sub-optimal solution by missing out on the main factor in the performance degradation of quantized models in low-precision PTQ environments. Based on the empirical and theoretical analyses, this paper proposes QDrop to pursue flatness and demonstrate that partial activation quantization is more beneficial.\n\nContributions: \n1. Observed the benefits of activation quantization in low precision PTQ.\n2. Conducted theoretical studies on how activation quantization affects weight tuning.\n3. Presented QDrop by showing that both integrating activation quantization into PTQ reconstruction and dropping partial activation quantization may help the flatness of the model, which is vital to the final accuracy.\n4. Established a new SOTA for W2/A2 PTQ with QDrop.", "summary_of_the_review": "To sum up, the paper is well-written and generally interesting even though there are a few glitches. Therefore, I may change my rating according to the authors' rebuttal to resolve the above concerns and questions. \n\n***** Post-Rebuttal Comments *****\nI appreciate the authors for their detailed response. The newly updated manuscript addresses most of my concerns. Therefore, I am happy to raise my rating up and genuinely recommend this paper to be accepted. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple, theoretically motivated approach for post-training quantization. The authors justify its effectiveness with both a sound theoretical analysis, and strong empirical results across many tasks and models, including a state-of-the-art result for 2-bit quantized weights/activations. All reviewers agreed the paper is worth accepting, with 3/4 rating it as a clear accept following the discussion period, and the fourth reviewer not giving strong reasons not to accept.
Decision: Accept (Poster)