QDROP: RANDOMLY DROPPING QUANTIZATION FOR EXTREMELY LOW-BIT POST-TRAINING QUANTIZATION
Year: 2022

Abstract:
Recently, post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining. Despite its low cost, current PTQ works tend to fail under the extremely low-bit setting. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. To deeply understand the inherent reason, a theoretical framework is established, indicating that the flatness of the optimized low-bit model on calibration and test data is crucial. Based on the conclusion, a simple yet effective approach dubbed as QDROP is proposed, which randomly drops the quantization of activations during PTQ. Extensive experiments on various tasks including computer vision (image classification, object detection) and natural language processing (text classification and question answering) prove its superiority. With QDROP, the limit of PTQ is pushed to the 2-bit activation for the first time and the accuracy boost can be up to 51.49%. Without bells and whistles, QDROP establishes a new state of the art for PTQ. Our code is available at https://github.com/wimh966/QDrop and has been integrated into MQBench (https://github.com/ModelTC/MQBench).

heading: 1 INTRODUCTION
text: In recent years, deep learning has been applied to all walks of life and offered substantial convenience for people’s production and activities. While the performance of deep neural networks continues to increase, the memory and computation cost also scale up fastly and bring new challenges for edge devices. Model compression techniques such as network pruning (Han et al., 2015), distillation (Hinton et al., 2015), network quantization (Jacob et al., 2018) and neural architecture search (Zoph & Le, 2016) etc., are dedicated to reduce calculation and storage overhead. In this paper, we study quantization which adopts low-bit representation for weights and activations to enable fixed-point computation and less memory space.
Based on the cost of a quantization algorithm, researchers usually divide the quantization work into two categories: (1) Quantization-Aware Training (QAT) and (2) Post-Training Quantization (PTQ). QAT finetunes a pre-trained model by leveraging the whole dataset and GPU effort. On the contrary, PTQ demands much less computation to obtain a quantized model since it does not require end-toend training. Therefore, much attention has recently been paid to PTQ (Cai et al., 2020; Wang et al., 2020; Hubara et al., 2021; Banner et al., 2019; Nahshan et al., 2019; Zhang et al., 2021a; Li et al., 2021c) due to its low cost and easy-to-use characteristics in practice.
Traditionally, PTQ pursues accuracy by performing the rounding-to-nearest operation, which focuses on minimizing the distance from the full-precision (FP) model in parameter space. In recent progress, Nagel et al. (2020); Li et al. (2021a) considered minimizing the distance in model space, i.e. the final loss objective. They use Taylor Expansion to analyze the change of loss value and derive a method to reconstruct the pre-trained model’s feature by learning the rounding scheme. Such methods are efficient and effective in 4-bit quantization and can even push the limit of weight quantization to 2-bit. However, the extremely low-bit activation quantization, which faces more challenges, still fails to achieve satisfactory accuracy. We argue that one key reason is that existing
∗Equal contribution, Corresponding author.
theoretical analyses only model the weight quantization as perturbation while ignoring activation’s. This will lead to the same optimized model no matter which bit the activations use, which is obviously counter-intuitive and thus causes a sub-optimal solution.
In this work, the effect of activation quantization in PTQ is deeply investigated for the first time. We empirically observe that perceiving the activation quantization benefits the extremely low-bit PTQ reconstruction and surprisingly find that only partial activation quantization is more preferable. An intuitive understanding is that incorporating activation will lead to a different optimized weight. Inspired by this, we conduct theoretical studies on how activation quantization affects the weight tuning, and the conclusion is that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data. Motivated by both empirical and theoretical findings, we propose QDROP that randomly drops quantization during the PTQ reconstruction to pursue the flatness from a general perspective. With this simple and effective approach, we set up a new state of the art for PTQ on various tasks including image classification, object detection for computer vision, and text classification and question answering for natural language processing.
To this end, this paper makes the following contributions:
1. We confirm the benefits unprecedentedly from involving activation quantization in the PTQ reconstruction and surprisingly observe that partial involvement of activation quantization performs better than the whole. 2. A theoretical framework is established to deeply analyze the influence of incorporating activation quantization into weight tuning. Using this framework, we conclude that the flatness of the optimized low-bit model on calibration data and test data is crucial for the final accuracy. 3. Based on the empirical and theoretical analyses, we propose a simple yet effective method QDROP that achieves the flatness from a general perspective. QDROP is easy to implement and can consistently boost existing methods as a plug-and-play module for various neural networks including CNNs like ResNets and Transformers like BERT. 4. Extensive experiments on a large variety of tasks and models prove that our method set up a new state of the art for PTQ. With QDROP, the 2-bit post-training quantization becomes possible for the first time.

heading: 2 PRELIMINARIES
text: Basic Notations. Throughout this paper, matrices (or tensors) are marked as X , whereas the vectors are denoted by x. Sometimes we use w to represent the flattened version of the weight matrix W . Operator · is marked as scalar multiplication, ⊙ is marked as element-wise multiplication for matrices or vectors. For matrix multiplication, we denote Wx as matrix-vector multiplication or WX as matrix-matrix multiplication.
For a feedforward neural network with activation function, we denote it as G(w,x) and the loss function as L(w,x), where x and w are the network inputs and weights, respectively. Note that we assume x is sampled from training dataset Dt, thus the final loss is denoted by Ex∼Dt [L(w,x)]. For the network forward function, we can write it as:
z (ℓ+1) i = ∑ j W (ℓ) i,j · a (ℓ) j , f(z (ℓ+1) i ) = a (ℓ+1) i , (1)
where Wi,j denotes weight connecting the jth activation neuron and the ith output. The bracket superscript (ℓ) is the layer index. f(·) indicates the activation function. Post-training Quantization. Uniform quantizer maps continuous values x ∈ R into fixed-point integers. For instance, the activation quantization function can be written as x̂ = ⌊xs ⌉ · s, where ⌊·⌉ denotes the rounding-to-nearest operator, s is the step size between two subsequent quantization levels. While rounding-to-nearest operation minimizes the mean squared error between x̂ and x, the minimization of parameter space certainly cannot equal to the minimization in final task loss (Li et al., 2021a), i.e., Ex∼Dt [L(ŵ,x)]. However, in the post-training setting, we only have a tiny subset Dc ∈ Dt that only contains 1k images. Thus, it is hard to minimize the final loss objective with limited data.
Recently, a series of works (Nagel et al., 2020; Li et al., 2021a) learn to either round up or down and view the new rounding mechanism as weight perturbation, i.e., ŵ = w + ∆w. Take a pre-trained network G as an example, they leverage Taylor Expansion to analyze the target, which reveals the quantization interactions among weights:
min ŵ E [L(ŵ,x)− L(w,x)] ≈ min ŵ
E [ 1
2 ∆w⊤Hw∆w
] , (2)
where Hw = Ex∼Dt∇2wL(w,x) is the expected second-order derivative. The above objective could be transformed into the change of output weighted by the output Hessian.
min ŵ
E [ ∆w⊤Hw∆w ] ≈ min ŵ E [ ∆a⊤Ha∆a ] (3)
About the above minimization, they finetune only the weight by reconstructing each block/layer output (See Fig. 1). But they did not explore the activation quantization during output reconstruction with only modeling weight quantization as noise. The step size for activation quantization is determined after the reconstruction stage.
Intuitively, when quantizing the activations of a full-precision model to 2-bit or 3-bit, there should be different suitable weights. However, the existing works result in the same optimized weight due to the neglect of activation quantization. Therefore, we argue that when quantizing the neural network, the noise caused by activation quantization should be considered coherently with weights.

heading: 3 METHODOLOGY
text: In this section, to reveal the influence of introducing activation quantization before output reconstruction, we first conduct empirical experiments and present two observations. Then a theoretical framework is built to investigate how the activation quantization affects the optimized weights. Last, equipped with the analysis conclusions, a simple yet effective method dubbed QDROP is proposed.

heading: 3.1 EMPIRICAL OBSERVATIONS
text: To investigate the influence of activation quantization when reconstructing the layer/block output, we conduct preliminary experiments on the ImageNet (Russakovsky et al., 2015) dataset. Our experiments are based on the open-sourced code Li et al. (2021a) except that we will introduce activation quantization from 1 to k − 1 blocks before the kth block’s reconstruction. We give a simple visualization in Fig. 1 to show 3 cases for putting activation quantization in different stages. Case 1 means that all activations are kept in 32-bit full-precision during the reconstruction of block output, which is also adopted in existing work Nagel et al. (2020); Li et al. (2021a). Case 2 and Case 3 are used for incorporating activation quantization into the reconstruction stage. However, Case 3 will omit the current block’s quantization while Case 2 will not. The detailed results of these three cases are listed in Table 1 (Comparisons on 2-bit (W2A2) quantization for ResNet-18, ResNet-50 and W3A3 for others for the sake of the crashed results on 2-bit.) and the algorithm is put in algorithm 2. According to the comparison, we can obtain two observations:

heading: 1. For extremely low-bit quantization (e.g., W2A2), there will be huge accuracy improvement when
text: considering activation quantization during weight tuning. This is confirmed by comparing with Case 1 and Case 2. We find Case 1 barely converges while Case 2 achieves good accuracy. It reveals that a separate optimization of weights and activations cannot find an optimal solution. After introducing the activation quantization, the weights will learn to diminish the influence of activation quantization.
2. Partially introducing block-wise activation quantization surpasses introducing the whole activation quantization. Case 3 does not quantize the activations inside the current tuning block but achieves better results than Case 2. This inspires us that how much activation quantization we introduce for weight tuning will affect the final accuracy.

heading: 3.2 HOW DOES ACTIVATION QUANTIZATION AFFECT WEIGHT TUNING
text: The empirical observations have highlighted the importance of activation quantization during the PTQ pipeline. To further explore how activation quantization will affect the weight tuning, we build a theoretical framework that analyzes the final loss objective with both weights and activations being quantized, which presents clues of high accuracy for extremely low-bit post-training quantization.
Conventionally, the activation quantization could be modeled as injecting some form of noise imposed on the full-precision counterpart, defined as e = (â−a). To remove the influence of activation range on e, we translate the noise into a multiplicative form, i.e., â = a · (1 + u), where the range of u is affected by bit-width and rounding error. Detailed illustration of the new form noise can be found in Appendix A.
Here, 1+u(x) is adopted to present the activation noise since it is related to specific input data point x. Equipped with the noise, we add another argument in calculating the loss function and define our optimization objective in PTQ as:
min ŵ Ex∼Dc [L(w +∆w,x,1+ u(x))− L(w,x,1)]. (4)
We hereby use a transformation that can absorb the noise on activation and transfer to weight, where the perturbation on weight is denoted as 1 + v(x) (V (x) is used in matrix multiplication format). Consider a simple matrix-vector multiplication Wa in forward pass, we have W (a⊙(1+u(x))) = (W ⊙ (1+ V (x)))a, given by
W (a⊙ 1 + u1(x)1 + u2(x)... 1 + un(x) ) = (W ⊙ 1 + u1(x) 1 + u2(x) ... 1 + un(x)1 + u1(x) 1 + u2(x) ... 1 + un(x)... 1 + u1(x) 1 + u2(x) ... 1 + un(x) )a. (5) By taking Vi,j(x) = uj(x), quantization noise on the activation vector (1 + u(x)) can be transplanted into perturbation on weight (1+ v(x)). Note that for a specific input data point x, there are two distinct u(x) and v(x). Proof is available at Sec. B.1.
Also note that for a convolutional layer, we cannot apply such transformation since the input to convolution is a matrix and will cause different V . Nonetheless, we can give a formal lemma that absorbs u(x) and holds corresponding v(x) (See the Appendix Sec. B.2 for rigorous proof): Lemma 1. For a quantized (convolutional) neural network, the influence of activation quantization on the final loss objective in post-training quantization can be transformed into weight perturbation.
Ex∼Dc [L(ŵ,x,1+ u(x))− L(w,x,1)] ≈ Ex∼Dc [L(ŵ ⊙ (1+ v(x)),x,1)− L(w,x,1)] (6)
By interpolating L(ŵ,x,1) into Lemma 1, we can obtain the final theorem: Theorem 1. For a neural network G with quantized weight ŵ and activation perturbation 1+u(x), we have:
Ex∼Dc [L(ŵ,x,1+ u(x))− L(w,x,1)] ≈ Ex∼Dc [(L(ŵ,x,1)− L(w,x,1))︸ ︷︷ ︸
(7−1)
+(L(ŵ ⊙ (1+ v(x)),x,1)− L(ŵ,x,1))︸ ︷︷ ︸ (7−2) ]. (7)
Here, Theorem 1 divides optimization objective into two terms. Term (7-1) is the same as Eq. (2) explored in (Nagel et al., 2020; Li et al., 2021a), which reveals how weight quantization interacts with loss function. Term (7-2) is the additional loss change by introducing activation quantization. In another way to interpret Eq. (7), the term (7-2) stands for the loss change with jitters on the weight quantized network G(ŵ,x). This type of noise correlates with certain kinds of robustness.
As stated in some works about generalization and flatness (Dinh et al., 2017; Hochreiter & Schmidhuber, 1997), intuitively, flat minimum means relatively small loss change under perturbation in the parameters, otherwise, the minimum is sharp. In this paper, we follow the notion of flatness defined in (Neyshabur et al., 2017), which considers loss change from the perspective of statistical expectation. And as (Neyshabur et al., 2017) and (Jiang et al., 2019) refer to, we consider the magnitude of the perturbation with respect to the magnitude of parameters and take the formulation as Ev∼D[L(fw⊙(1+v))−L(fw)], where each element of v is a random variable sampled from a noise distributionD and L represents for optimization objective on the training set. From this perspective, the term (7-2) can be interpreted as the flatness with perturbation related to input data, and thereby we could achieve the following corollary. Corollary 1. On calibration data x, with activation quantization noise u(x), there exists the corresponding weight perturbation v(x) which satisfies that the trained quantized model is flatter under the perturbation v(x).
With Corollary 1, Case 2 and 3 discussed in Sec. 3.1 enjoy a flatter loss landscape benefited from perceiving the activation quantization. This explains their superiority compared with Case 1. The measurement of sharpness on calibration data (left part) in Fig. 2 further validates this point. With similar perturbation magnitude, Case 2 and 3 suffer less loss degradation than Case 1.

heading: 3.3 QDROP
text: As aforementioned, introducing activation quantization is theoretically proved to produce a flatter model than existing works and the directions of flatness depend on the data distribution. Since the PTQ is especially sensitive to calibration data (Yu et al., 2021), we need to transfer the investigations in Sec. 3.2 on calibration data into the test setting for a thorough understanding. In specific, we consider Eq. (7) on test set and inspect two terms separately in the following. Based on the analyses, our method QDROP will be derived to pursue an excellent performance on test data.
Term (7-1) on test set. As suggested in Sec. 3.2, with both quantized activations and weights, we additionally optimize the term (7-2) representing the flatness on calibration data. This term will encourage the quantized model to learn a flat minimum. As a result, the traditional objective of AdaRound (term (7-1)) can naturally generalize better for test data (i.e., Ex∼Dt (L(ŵ,x,1)− L(w,x,1))). Term (7-2) on test set. Furthermore, we should also concern about the term (7-2) on test data, i.e. Ex∼Dt [L(ŵ ⊙ (1+ v(x)),x,1)− L(ŵ,x,1)]. As revealed in Sec. 3.2, the term (7-2) implies the flatness where its situation on calibration data has been exploited. Here, we further investigate the flatness on test samples. Note that v(x) is converted from u(x) and this activation quantization noise varies with input data. Fig. 2 shows that there is a gap between the test data and calibration data for the flatness of the 3 cases. According to Corollary 1, these 3 cases actually introduce different u mathematically and thus will result in different flatness directions, given by
Case 1: u = 0; Case2: u = â
a − 1; Case 3: u = { â a − 1, block1 ∼ blockk−1 0, blockk . (8)
Algorithm 1: QDROP in one block for one batch Input: the kth block from layer i to layer j, a minibatch of quantized block input âi−1, FP32 block input ai−1, FP32 block output aj , quantization dropping probability p. {1. Forward propagation:} During training phase, substitute âi−1 with corresponding ai−1 at neuron-level with probability p and mark the replaced input as ãi−1 ; for l = i to j do al ←W lãl−1; âl ← Quantize(al); During training phase, randomly drop some âl with al as defined in Eq. (9) and get ãl ;
{2. Backward propagation:} Compute ∆aj = ãj − aj ; Tune the weight by gradient descent ; return Quantized block ;
For Case 1, there is no activation quantization during calibration without taking flatness into account. Case 2 suggests the activation perturbation totally and therefore enjoys a good flatness on calibration data. However, due to the mismatch on calibration data and test one, it is highly possible that Case 2 causes overfitting (See Table 8 for more details). Case 3, in fact achieves the best performance by dropping some activation quantization along with a little different weight perturbation and might not be restricted to flatness on calibration data (More evidence can be found in Table 9). This inspires us to pursue a flat minimum from a general perspective, that only optimizing the target on calibration set is suboptimal to test set.
QDROP. Inspired by this, we propose QDROP to further increase the flatness on as many directions as possible. In particular, we randomly disable and enable the quantization of the activation each forward pass:
QDROP : u = { 0 with probability p â a − 1 with probability 1− p . (9)
We name it QDROP because it randomly drops the quantization of activation. Theoretically, by masking some u(x) randomly, QDROP can have more diverse v(x) and cover more directions of flatness thus flatter on test samples, which contributes to the final high accuracy. Fig. 3 support our analysis where QDROP has smoother loss landscape than Case 3, the winner among 3 cases on test data. Meanwhile, it is indeed a fine-grained version of Case 3 since Case 3 drops the quantization in a block-wise manner, whereas our QDROP operates in an element-wise way.
Discussions. QDROP can be viewed as a generalized form of the existing schemes. Case 1 and 2 respectively corresponds to the dropping probability of p = 1 and p = 0. Case 3 is equivalent to setting the block being optimized with dropping probability p = 1 and remains the quantization of

heading: Method Bits (W/A) ResNet-18 ResNet-50 MobileNetV2 RegNet-600MF RegNet-3.2GF MNasNet-2.0
text: other parts. Note that the p obeys Bernoulli distribution and thus can be set as 0.5 for the maximal entropy (Qin et al., 2020), which is helpful for flatness across various directions.
QDROP is easy to implement for various neural networks including CNNs and Transformers, and plug-and-play with little additional computational complexity. With QDROP, the complicated problem of choosing optimization order, i.e. different cases in Sec. 3.1, can be avoided.

heading: 4 EXPERIMENTS
text: In this section, we conduct two sets of experiments to verify the effectiveness of QDROP. In Sec. 4.1, we first conduct an ablation study for the impact with and without dropping quantization and analyze the option of distinct dropping rates. In Sec. 4.2, we compare our method with other existing approaches across vision and language tasks including image classification on ImageNet, object detection on MS COCO, and natural language processing on GLUE benchmark and SQuAD.
Implementation Details. Our code is based on PyTorch Paszke et al. (2019). We set the default dropping probability p as 0.5, except we explicitly mention it. The weight tuning method is the same with Nagel et al. (2020); Li et al. (2021a). Each block or layer output is reconstructed for 20k iterations. For ImageNet dataset, we sample 1024 images as calibration set, while COCO we use 256 images. In NLP, we sample 1024 examples. We also keep the first and the last layer in 8-bit except NLP tasks and adopt per-channel weight quantization. We use W4A4 to represent 4-bit weight and activation quantization. More model choices and other setting is described in Appendix E.
But to be noted, regular first and last layer 8-bit means 8-bit weight and input in first and last layer while BRECQ uses another setting which not only keeps the first layer’s input 8-bit but also the first layer’s output (second layer’s input). This would indeed perform better than the regular one with leaving one more layer’s input 8-bit but may not be practical on the hardware. Therefore, we both compare with BRECQ’s setting to show the superiority of our approach and experiment on the usual one to provide a practical baseline. Symbol † is used to mark BRECQ’s setting.

heading: 4.1 ABLATION STUDY
text: Effect of QDROP. We propose QDROP and here we would like to test the effect of PTQ with or without QDROP. We use ImageNet classification benchmark and quantize the weight parameters to 2-bit and quantize activation to 2/4-bit. As shown in Table 2, QDROP improves the accuracy across all bit settings evaluated for 6 models on ImageNet. Furthermore, the gains are more obvious when applying QDROP to lightweight network architecture: 2.36% increment for MNasNet under W2A4 and 12.6% for RegNet-3.2GF with W2A2.
Effect of Dropping Probability. We also explore
the dropping probability in PTQ. We choose p in [0,0.25,0.5,0.75,1] and test on MobileNetV2 and RegNet-600MF. The results are summarized in Fig. 5. We find 0.5 generally performs best among 5 candidates. Although there could be a fine-grained best solution for each architecture, we shall avoid cumbersome hyperparameter search and continue using 0.5.

heading: 4.2 LITERATURE COMPARISON
text: ImageNet. We choose ResNet-18 and -50 (He et al., 2016), MobileNetV2 (Sandler et al., 2018), searched MNasNet (Tan et al., 2019) and RegNet (Radosavovic et al., 2020). We summarize the

heading: Methods Bits (W/A) Res18 Res50 MNV2 Reg600M Reg3.2G MNasx2
text: results in Table 3. First, the W4A4 quantization is investigated. It can be observed that QDROP provides 0 ∼ 3% accuracy uplift when compared to strong baselines including AdaRound, BRECQ. As for the gap between our method and AdaQuant on W4A4, we argue that there are some discrepancies on settings such as positions of quantization nodes and put this explaination in Sec. C.3. With W2A4 quantization, QDROP can improve the accuracy of ResNet-50 by 0.5%, and RegNet-3.2GF by 4.6%. In addition, to fully exploit the limit of QDROP, we conduct more challenging cases with 2/3-bit weights and activations. According to the last two rows of Table 3, our proposed QDROP consistently achieves good results while existing methods suffer from non-negligible accuracy drop. For W3A3, the difference is even larger on MobileNetV2, where our method reaches 58% accuracy and BRECQ only gets 23%. In the W2A2 setting, the PTQ becomes much harder. QDROP outperforms the competing method by a large margin: 12.18% upswings for ResNet-18, 29.66% for ResNet-50 and 51.49% for RegNet-3.2GF.
MS COCO. In this part, we validate the performance of QDROP on object detection task using MS COCO dataset. We use both two-stage Faster RCNN (Ren et al., 2015) and one-stage RetinaNet (Lin et al., 2017) models. Backbone are selected from ResNet-18, ResNet-50, and MobileNetV2. Note that we set the first layer and the last layer to 8-bit and do not quantize the head of the model, however, the neck (FPN) is quantized. Experiments show that W4A4 quantization using QDROP nearly do not affect Faster RCNN’s mAP. For RethinaNet, our method has 5 mAP improvement on MobileNetV2 backbone. In low bit setting W2A4, our method shows great improvement both on Faster-RCNN and RetinaNet, up to 6.5 mAP.
GLUE benchmark and SQuAD. We test QDROP in NLP tasks including the GLUE benchmark and SQuAD1.1. They are all conducted on the typical NLP model, i.e, BERT (Devlin et al., 2018). Compared with those QAT methods (Bai et al., 2020), which usually adopt data augmentation trick to achieve dozens of times the original data, we only randomly extract 1024 examples without any extra data processing. Besides AdaQuant and BRECQ, which suffer a huge accuracy degradation, our QDROP surpasses No Drop all the tasks, specifically on QNLI(8.7%), QQP(4.6%) and RTE(7.2%).
As for SST-2, despite little enhancement by dropping quantization, it is indeed close to the FP32 value within 4.4%. And for STS-B, we argue that the original fine-tuned model is trained with limited data, which might not be very representative.

heading: 4.3 ROBUSTNESS OF QDROP
text: In this part, we discuss the effectiveness of QDROP under more challenging situations including even less data and cross-domain ones. Concerning about size of calibration data, we consider another 4 options. It can be observed that dropping some quantization behaves better under each setting and is even comparable with No Drop with half of the original calibration data. Motivated by Yu et al. (2021), we also reconstruct block output by 1024 examples from out-of-domain data, i.e, CIFAR100 (Krizhevsky et al., 2009), MS COCO, and test on ImageNet. Results are available in Table 6, where our QDROP still works steadily.
64 128 256 512 1024 Calibration Data Amount
60
62
64
A cc
ur ac
y
ResNet-18 W3A3
QDrop No Drop
64 128 256 512 1024 Calibration Data Amount
55
60
RegNet-600MF W2A4
QDrop No Drop
Figure 6: Impact of calibration data size on ImageNet.

heading: 5 CONCLUSION
text: In this paper, we have introduced QDROP, a novel mechanism for post-training quantization. QDrop aims to achieve good test accuracy given a tiny calibration set. This is done by optimization towards a flat minima. We dissect the PTQ objective theoretically into a flatness problem and improve the flatness from a general perspective. We comprehensively verify the effectiveness of QDROP on a large variety of tasks. It can achieve a nearly lossless 4-bit quantized network and can significantly improve the 2-bit quantization results.

heading: ACKNOWLEDGMENT
text: We sincerely thank the anonymous reviewers for their serious reviews and valuable suggestions to make this better. And we thank Xiangguo Zhang and Sheng Chen for their kind of help of this work. This work was supported in part by the National Natural Science Foundation of China under Grant 62022009 and Grant 61872021, the SenseTime Research Fund for Young Scholars, and the Beijing Nova Program of Science and Technology under Grant Z191100001119050.

heading: A NOISE FORM CHOICE
text: As mentioned in Sec. 2, we refer the quantizer to â = ⌊as ⌉ · s, where s is the step size that can be learned or be determined by collecting activation distribution in PTQ. By marking the rounding error as c, which subjects to U [−0.5, 0.5], the additive noise e = â− a can be denoted as c · s. However, this traditional noise does not decouple the noise from step size s and the range of activations. The range of noise e will vary when the range of activations change, making it complicated to analyze across the whole network in a unified form. Some existing papers (Jiang et al., 2019) also indicate the problem that the additive noise doesn’t take parameters’ magnitude into account and suggest a multiplicative form form (Keskar et al., 2016). To be specific, they claim that:
Perturbing the parameters without taking their magnitude into account can cause many of them to switch signs. Therefore, one cannot apply large perturbations to the model without changing the loss significantly. One possible modification to improve the perturbations is to choose the perturbation magnitude based on the magnitude of the parameter. In that case, it is guaranteed that if the magnitude of perturbation is less than the magnitude of the parameter, then the sign of the parameter does not change.
To eliminate the influence induced by the range of activations, we take the noise of multiplicative form: â = (1 + u) · a. In the quantization background, we denote the fixed-point integer value with respect to a as ā (a = (ā+ c) · s and â = ā · s). And then u can be denoted as:
u = â
a − 1
= ā · s
(ā+ c) · s − 1
= ā
ā+ c − 1
= −c ā+ c .
(10)
To be noted, u is derived from the definition of the quantizer and can be equivalently transformed from the generalized noise here.
From this formulation, we can find that the range of u is not related to the activation range or step size s and is only influenced by the rounding error and the bit-width. In a word, the multiplicative form has its pysical meaning in quantization background and is beneficial as discussed above.

heading: B PROOF OF LEMMA 1 AND THEOREM 1
text: We demonstrate them by considering the situation of (1) fully connected and (2) convolutional networks separately. As transformation with fully connected ones has been revealed in main body, we add some extra illustration here and mainly target at the convolutional layers.

heading: B.1 FULLY CONNECTED NETWORKS
text: Here, we give the proof of Eq. (5) by leveraging the definition of FC layers in Eq. (1). We first look at each input sample and temporarily omit x in the notation below for simplicity.
With activation noise u, z (ℓ+1) i = ∑ j W (ℓ) i,j · (1 + u (ℓ) j ) · a (ℓ) j
= ∑ j (1 + u (ℓ) j ) ·W (ℓ) i,j · a (ℓ) j .
(11)
By taking V (ℓ)i,j = u (ℓ) j , we have
z (ℓ+1) i = ∑ j (1 + V (ℓ) i,j ) ·W (ℓ) i,j · a (ℓ) j . (12)
Thus, with proper constructed v, the noise on activation (u) can be viewed as the perturbation on weight (v).
For every layer, we can conduct this transformation from u to v. Therefore, considering operating on the quantized weight and calibration data, optimizing Ex∼Dc [L(ŵ,x,1 + u(x))] can be approximated to optimizing Ex∼Dc [L(ŵ ⊙ (1+ v(x)),x,1)]. Then we have
Ex∼Dc [L(ŵ,x,1+u(x))−L(w,x,1)] ≈ Ex∼Dc [L(ŵ⊙ (1+v(x)),x,1)−L(w,x,1)]. (13)
Now, Lemma 1 is proved for the case of fully connected networks.

heading: B.2 CONVOLUTIONAL NETWORKS
text: We first look at each input sample and temporarily omit x in the notation below for simplicity. One layer in the network G can be interpreted as:
A (ℓ+1) i,j = f(Z (ℓ+1) i,j ) = f( ∑ p,q W (ℓ)p,q ·A (ℓ) i+p,j+q), (14)
where the f(·) is the activation function, (p, q) pair represents for one element in weight matrix and (i, j) pair represents for one element in activation matrix.
Due to the complicated design of convolutional structure, based on G we introduce two networks G1,G2 to make proofs more clearly. Definition 1. G1 means inserting random noise on activations, G2 means sticking random variables into weights.
G1 :A(ℓ+1)i,j = f(Z (ℓ+1) i,j ) = f( ∑ p,q W (ℓ)p,q · (1 +U (ℓ) i+p,j+q) ·A (ℓ) i+p,j+q)
G2 :A(ℓ+1)i,j = f(Z (ℓ+1) i,j ) = f( ∑ p,q (1 + V (ℓ)p,q ) ·W (ℓ)p,q ·A (ℓ) i+p,j+q))
(15)
Definition 2. We mark losses of network G1 and G2 in the following way:
Loss of G1 : L(w,x,1+ u) when taking u = 0, it becomes L(w,x,1) Loss of G2 : L(w ⊙ (1+ v),x) when taking v = 0, it becomes L(w ⊙ 1,x).
(16)
With the definitions, we first prove that G1 and G2 share common parts in their first-order derivative to its noise in Lemma 2.
Lemma 2. Assuming the same weight and taking u = 0 and v = 0 for G1 and G2, we have
∂L(w,x,1)
∂U (ℓ) i,j
= ∑ p,q T (ℓ) (i,j),(p,q)
∂L(w ⊙ 1,x) ∂V (ℓ) p,q = ∑ i,j T (ℓ) (i,j),(p,q),
(17)
where
T (ℓ) (i,j),(p,q) =
∂L(w,x,1)
∂A (ℓ+1) i−p,j−q
· f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j
= ∂L(w ⊙ 1,x) ∂A
(ℓ+1) i−p,j−q
· f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j .
(18)
Note that with u = 0 and v = 0, the activations are the same for the two networks so we don’t use different notations here.
Proof. ∂L(w,x,1)
∂U (ℓ) i,j
= ∑ p,q ∂L(w,x,1) ∂A (ℓ+1) i−p,j−q · f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j (19)
∂L(w ⊙ 1,x) ∂V (ℓ) p,q = ∑ i,j ∂L(w ⊙ 1,x) ∂A (ℓ+1) i,j · f ′(Z(ℓ+1)i,j ) ·W (ℓ) p,q ·A (ℓ) i+p,j+q
= ∑ i,j ∂L(w ⊙ 1,x) ∂A (ℓ+1) i−p,j−q · f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j
(20)
Because of taking the same weight, and calculating the derivatives to U (ℓ)i,j at u = 0 and derivatives to V (ℓ)i,j at v = 0, we have the same activation values for these two networks. Therefore, Eq. (21) holds.
∂L(w,x,1)
∂Ai−p,j−q (ℓ+1)
·f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j = ∂L(w ⊙ 1,x) ∂Ai−p,j−q (ℓ+1) ·f ′(Z(ℓ+1)i−p,j−q) ·W (ℓ) p,q ·A (ℓ) i,j (21)
By marking the above value as T (ℓ)(i,j),(p,q), we can get Lemma 2.
Based on Lemma 2, we further derive Theorem 2 as the pre-condition of the final Lemma 1. Theorem 2. By taking
V (ℓ)p,q =
∑ i,j U (ℓ) i,j · T
(ℓ) (i,j),(p,q)∑
i,j T (ℓ) (i,j),(p,q)
, (22)
we have u⊤∇uL(w,x,1) = v⊤∇vL(w ⊙ 1,x). (23)
Proof. According to Lemma 2,
u⊤∇uL(w,x,1) = ∑ ℓ ∑ i,j U (ℓ) i,j · ∂L(w,x,1) ∂U (ℓ) i,j
= ∑ ℓ ∑ i,j U (ℓ) i,j · ∑ p,q T (ℓ) (i,j),(p,q)
= ∑ ℓ ∑ i,j ∑ p,q U (ℓ) i,j · T (ℓ) (i,j),(p,q)
= ∑ ℓ ∑ p,q ∑ i,j U (ℓ) i,j · T (ℓ) (i,j),(p,q) = ∑ ℓ ∑ p,q ∑ i,j U (ℓ) i,j · T (ℓ) (i,j),(p,q)∑ i,j T (ℓ) (i,j),(p,q) · ∑ i,j T (ℓ) (i,j),(p,q) = ∑ ℓ ∑ p,q ∑ i,j U (ℓ) i,j · T (ℓ) (i,j),(p,q)∑ i,j T (ℓ) (i,j),(p,q) · ∂L(w ⊙ 1,x) ∂Vp,q (ℓ) .
(24)
By taking V (ℓ)p,q = ∑ i,j U (ℓ) i,j ·T (ℓ)
(i,j),(p,q)∑ i,j T (ℓ) (i,j),(p,q) ,
u⊤∇uL(w,x,1) = ∑ ℓ ∑ p,q V (ℓ)p,q · ∂L(w ⊙ 1,x) ∂Vp,q (ℓ)
= v⊤∇vL(w ⊙ 1,x). (25)
Thus Theorem 2 is affirmed.
We now prove Lemma 1 equipped with Taylor Expansion technique and Theorem 2.
Proof. First, by adopting Taylor Expansions at u = 0, we can get that:
L(ŵ,x,1+ u)− L(w,x,1) ≈ L(ŵ,x,1) + u⊤∇uL(ŵ,x,1)− L(w,x,1). (26)
Then, according to Theorem 2, the above equation can be rewritten as:
L(ŵ,x,1+ u)− L(w,x,1) ≈ L(ŵ ⊙ 1,x) + v⊤∇vL(ŵ ⊙ 1,x)− L(w,x,1). (27)
Again, by adopting Taylor Expansions at v = 0 for the right part of Eq. (27), we arrive at: L(ŵ,x,1+ u)− L(w,x,1) ≈ L(ŵ ⊙ (1+ v),x)− L(w,x,1). (28)
Finally, apply expectation on Eq. (28), and Lemma 1 is proved: Ex∼Dc [L(ŵ,x,1+ u(x))− L(w,x,1)]
≈ Ex∼Dc [L(ŵ ⊙ (1+ v(x)),x)− L(w,x,1)]. (29)
With the Lemma 1 proved, we can easily derive Theorem 1 by the following transformation: Ex∼Dc [L(ŵ,x,1+ u(x))− L(w,x,1)]
≈ Ex∼Dc [L(ŵ ⊙ (1+ v(x)),x)− L(w,x,1)] = Ex∼Dc [L(ŵ ⊙ (1+ v(x)),x,1)− L(w,x,1) + L(ŵ,x,1)− L(ŵ,x,1)] = Ex∼Dc [L(ŵ,x,1)− L(w,x,1) + L(ŵ ⊙ (1+ v(x)),x,1)− L(ŵ,x,1)]. (30)

heading: C EXPERIMENTS

heading: C.1 SUPPLEMENTARY EXPERIMENTS OF SEC. 3.1
text: To explore the upper limit of PTQ, we concern two key parts of the algorithm (weight quantization and activation one). As is generally known, QAT is a popular way to produce favor quantization results, thus we use QAT’s learning mechanism to replace each of the two parts in PTQ and record outcomes in Table 7. In detail, about weight quantization, QAT’s settings we used include the whole ImageNet, STE learning strategy and end-to-end training for 40 epochs while PTQ’ settings only cover 1024 samples, training block by block. But we both keep the rounding-up-or-down parameter optimization space. About activation quantization, QAT’ settings we used include the whole ImageNet, LSQ (Esser et al., 2019) scheme and end-to-end training for 5 epochs.
However, results in Table 7 are surprising that although the optimization space in weight is kept very restricted, leveraging the whole data to implement weight tuning can achieve a large accuracy boost. And the activation quantization step size might not be as important as weight. These findings indicate that exploring a quantization-friendly weight may be the fresh insight for the accurate PTQ, where this work dedicates to.

heading: C.2 SUPPLEMENTARY EXPERIMENTS OF SEC. 3.3
text: Overfitting phenomenon. To analyze the overfitting problem refered in Sec. 3.3, we have conducted experiments to compare the accuracy on test data and calibration data, respectively. The table below is an example of ResNet-18 W2A2. It can be seen that with extremely low-bit quantization, both Case 2 and Case 3 perform well on calibration data but on test data Case 2 performs worse than Case 3. This is a shred of clear evidence that Case 2 suffers a more severe overfitting problem.
Rethinking Theorem 1, both Case 2 and Case 3 introduce the term (7-2) that implies flatness against the perturbation v(x). However, Case 2 completely introduces the quantization noise u(x) according to the calibration data. Thus the resulting flatness fits calibration data but does not generalize
well on test data. Case 3 drops partial of u(x) and improves the possibility of flatness on test data instead. The two accuracy of QDROP also confirmed this phenomenon with more diverse directions of flatness. As for Case 1, its accuracy on both calibration data and test data is low. This is because that it does not introduce activation quantization during weight tuning and thus behaves worst without taking flatness term even on calibration data into account.
Hessian information. As Hessian information is known to be a metric to characterize flatness property (Keskar et al., 2016; Dong et al., 2019), we also calculate the top-1, top-5 Hessian eigenvalues (λ1, λ5) and the Hessian trace (Tr) among 3 cases and QDROP to better support our analysis. In table Table 9, QDROP has the smallest value of Hessian information, which matches with our theoretical framework and observations of loss landscape.

heading: C.3 SUPPLEMENTARY EXPERIMENTS OF SEC. 4
text: To clarify the gap between AdaQuant and our method in classification task on 4-bit ResNet-18 and -50, we analyze the difference on settings between these two algorithms. There are two major discrepancies which would influence the final accuracy. One is the position of activation quantization nodes, the other is the FP32 accuracy of pretrain-models. As shown in Fig. 7, by inserting the quantizer like the right side one, AdaQuant would introduce two different quantizers for the same input with learned quantization parameters such as step size. We find this can bring ∼ 0.4% upswings on ResNet-18 and -50 W4A4 and improve more on ultra low-bit with experiments of our algorithm. However, this way to insert quantization nodes is not practical in real deployment, because two different quantizers for the same input would make it impossible to follow the so-called Requantize procedure (Li et al., 2021b). As for the pretrain-models, they use 71.97% and 77.2%, higher than ours (71.06% and 77.0%), which can indeed improve the accuracy. Replacing our settings with the two adopted in AdaQuant, we finally reach 71.07% and 76.67%.

heading: C.4 FLATNESS AND POST-TRAINING QUANTIZATION
text: While there are plenty of works exploring the correlation between flatness and generalization, the interaction between quantization and flatness has not been exploited much. In this paper, we first connect flatter quantized weight with activation quantization from PTQ view, as implied in Sec. 3.2. From this, we conjecture that flatness and quantization might be helpful to each other.
By leveraging some mechanisms devoting to produce a smoother loss surface for better generalization, such as (Izmailov et al., 2018), the improved FP32 model is obtained and used to validate the
performance on quantization compared with the naive one. In Table 10, model SWA20 is enabled by applying the SWA technique (Izmailov et al., 2018) to ResNet-18 with 20 epochs fintuning process on the whole ImageNet. From the table, the observation is that the promotion on generalization can not fully represent the enhancement induced by SWA on quantization. With even lower bits thus larger noise, SWA20 surpasses the naive one by a large margin. It also reveals that distinct FP32 models with analogous accuracy might contribute to surprisingly disparate outcomes after PTQ, particularly for those naive methods without any weight tuning.
In turn, there have been some studies that delve into robustness boost by applying quantization. (Fu et al., 2021) advocates that quantization can be properly leveraged to enhance DNNs’ robustness, even beyond their full-precision counterparts. They propose a random bit training strategy to accomplish it, where our work illustrates the correlation with bit and perturbation in Appendix A.

heading: D RELATED WORKS
text: Post-training quantization. Unlike QAT Esser et al. (2019); Li et al. (2019); Shen et al. (2021) where the quantized model is finetuned with full training dataset and over 100 epochs training, PTQ is much more faster. Rounding-to-nearest operation is known to be the direct and easy way for quantizing parameters or activations in PTQ. Although there is almost no accuracy drop when quantizing to 8-bit, lower bit quantization is yet a hard task and worth exploring. (Choukroun et al., 2019) transforms quantization to a Minimum Mean Squared Error problem both for weights and activations. (Nagel et al., 2019) equalizes weight ranges among channels thus be more favorable to per-layer quantization and employs bias correction to absorb the output error induced by quantization. However, such methods neglect the task loss thus lead to a sub-optimal. AdaRound (Nagel et al., 2020), which proposes to learn the rounding mechanism by reconstructing output layer by layer brings more opportunities for 4-bit quantization. Besides layer reconstruction, BRECQ (Li et al., 2021a) discusses more choices and advises to do block reconstruction with better accuracy at 2-bit weight quantization. Nonetheless, we argue that AdaRound and BRECQ isolate weight quantization and activation one theoretically and experimentally, which might be a key point of failures on extremely low-bit quantization. Recently, there is another trend of utilizing synthetic data for PTQ, which explicitly do backpropagation on the learned input tensor Cai et al. (2020); Zhang et al. (2021b); Li et al. (2021c).
Flatness. The idea of “flat” minima might date back to (Hochreiter & Schmidhuber, 1997), where the benefits are recognized in recent years, such as generalization (Jiang et al., 2019; Keskar et al., 2016) and adversarial training (Wu et al., 2020; Zheng et al., 2021). Some previous works (Izmailov et al., 2018; Foret et al., 2020) devote to improve the flatness of the trained weight for robustness under perturbation or distribution shift. Other ideas try to model flatness or sharpness formally by visualization of loss landscape or mathematical formulas. And for quantization, which could be viewed as some kind of noise, a flat model has been implied to be preferable, (Dong et al., 2019; Yang et al., 2019; Kadambi et al., 2020). Despite the natural fact that flatness helps with weight quantization, how does activation quantization involves with smoother loss surface has not been discussed deeply, particularly for post-training quantization. In this work, we introduce noise scheme by randomly dropping activation quantization and achieve a general flatness. Another paper (Fan et al., 2020) also utilizes randomness by adding noise to weight for the simulation of weight quantization. But they target at reducing the induced bias of Straight Through Estimation (STE) in QAT and also have different motivations and solve different problems from us.
E IMPLEMENTATION DETAILS
Observation. Here, we give the concrete implementation of experiments in Sec. 3.1. We actually consider three ways of introducing activation for Case 2, but we find the differences of outcomes among them are negligible thus employing one of them for clearer clarification.
ImageNet. We randomly extract 1024 training examples from ImageNet as calibration dataset based on the standard pre-process. Pretrain-models are downloaded from BRECQ’s open source code. Hyper-parameters we keep it as BRECQ, such as batch size 32, learning rate for activation step size 4e-5, learning rate for weight tuning 1e-3, iterations 20000. Following BRECQ, we first fold batch normalization layer into convolution then reconstruct output block-wise to learn the weight
Algorithm 2: Implementations of three cases in Sec. 3.1 Input: Model with K blocks. if Case 2 then
for k = 1 to K do parameterize activation step size in this block ;
for k = 1 to K do Tuning weight like (Li et al., 2021a) by reconstructing block output ; if Case 3 then parameterize activation step size in this block ; if Case 1 then
for k = 1 to K do parameterize activation step size in this block ;
return Quantized model ;
rounding policy and meanwhile use LSQ (Esser et al., 2019) to parameterize activation step size. For QDROP, we learn the weight and activation parameters together and use 50% rate to drop some activation quantization.
Object detection. Here, we also obey BRECQ’ settings and use the same pretrain-models with 256 training samples taken from MS COCO dataset for calibration. Parameters about resolution is set to 800 (max size 1333) and 600 (max size 1000) for ResNets and MobileNetV2, respectively and batch size is set to 2 while others are the same with classification task. To be noted, we didn’t quantize the head but applied block reconstruction to backbone and layer reconstruction to neck like BRECQ.
GLUE benchmark and SQuAD. The BERT fine-tuned models are taken from huggingface group (https://huggingface.co/). And we sampled 1024 examples from training set. We keep the maximum sequence length to be 128 for GLUE benchmark but maximum sequence length 384 with doc stride 128 for SQuAD1.1. Also, we quantize all the part in BERT as well as the internal structure of the attention module with only affirming the embedding weight to 8-bit. Other settings and hyperparameters are chosen in the same way with ImageNet experiments.
Baselines. We run baseline methods from open-source codes, such as AdaQuant, BRECQ and Adaround. And we try our best to align some optimal settings like per-channel quantization for fair comparisons.

Review summaries:
This paper proposes the post-training quantization for extremely low-bit neural networks. By considering the activation quantization during reconstruction, the presented QDrop randomly drops the quantization of activations with higher loss flatness that adapts the activations with various activation bitwidths well. Experimental results have demonstrated the superiority of the presented method.

This paper aims to analyze how activation quantization affects the PTQ process and provides some theoretical analysis and experimental results on the effects of activation quantization. The authors argue that previous studies only model the weight quantization as perturbation while ignoring activation quantization, causing a sub-optimal solution by missing out on the main factor in the performance degradation of quantized models in low-precision PTQ environments. Based on the empirical and theoretical analyses, this paper proposes QDrop to pursue flatness and demonstrate that partial activation quantization is more beneficial.  Contributions:  1. Observed the benefits of activation quantization in low precision PTQ. 2. Conducted theoretical studies on how activation quantization affects weight tuning. 3. Presented QDrop by showing that both integrating activation quantization into PTQ reconstruction and dropping partial activation quantization may help the flatness of the model, which is vital to the final accuracy. 4. Established a new SOTA for W2/A2 PTQ with QDrop.

The authors proposed a random dropping quantization method at the post-training stage to achieve a low bit quantization network. By observing the performance on partial activation quantization,  the authors analyze the influence of incorporating activation quantization into weight tuning. Experimental results show that their QDROP methods have benefits on several scenarios, such as detection and NLP tasks.

The authors conduct theoretical studies on how activation quantization affects weight tuning, and their conclusion is that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data. They present both empirical and theoretical findings, and propose the QDROP algorithm to exploit this phenomenon. QDROP randomly drops quantization during post-training quantization (PTQ) reconstruction to pursue the flatness from a general perspective. The authors also claim a new state of the art for PTQ on various tasks including image classification, object detection for computer vision, and text classification and question answering for natural language processing.