QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization

1 INTRODUCTION . In recent years , deep learning has been applied to all walks of life and offered substantial convenience for people ’ s production and activities . While the performance of deep neural networks continues to increase , the memory and computation cost also scale up fastly and bring new challenges for edge devices . Model compression techniques such as network pruning ( Han et al. , 2015 ) , distillation ( Hinton et al. , 2015 ) , network quantization ( Jacob et al. , 2018 ) and neural architecture search ( Zoph & Le , 2016 ) etc. , are dedicated to reduce calculation and storage overhead . In this paper , we study quantization which adopts low-bit representation for weights and activations to enable fixed-point computation and less memory space . Based on the cost of a quantization algorithm , researchers usually divide the quantization work into two categories : ( 1 ) Quantization-Aware Training ( QAT ) and ( 2 ) Post-Training Quantization ( PTQ ) . QAT finetunes a pre-trained model by leveraging the whole dataset and GPU effort . On the contrary , PTQ demands much less computation to obtain a quantized model since it does not require end-toend training . Therefore , much attention has recently been paid to PTQ ( Cai et al. , 2020 ; Wang et al. , 2020 ; Hubara et al. , 2021 ; Banner et al. , 2019 ; Nahshan et al. , 2019 ; Zhang et al. , 2021a ; Li et al. , 2021c ) due to its low cost and easy-to-use characteristics in practice . Traditionally , PTQ pursues accuracy by performing the rounding-to-nearest operation , which focuses on minimizing the distance from the full-precision ( FP ) model in parameter space . In recent progress , Nagel et al . ( 2020 ) ; Li et al . ( 2021a ) considered minimizing the distance in model space , i.e . the final loss objective . They use Taylor Expansion to analyze the change of loss value and derive a method to reconstruct the pre-trained model ’ s feature by learning the rounding scheme . Such methods are efficient and effective in 4-bit quantization and can even push the limit of weight quantization to 2-bit . However , the extremely low-bit activation quantization , which faces more challenges , still fails to achieve satisfactory accuracy . We argue that one key reason is that existing ∗Equal contribution , Corresponding author . theoretical analyses only model the weight quantization as perturbation while ignoring activation ’ s . This will lead to the same optimized model no matter which bit the activations use , which is obviously counter-intuitive and thus causes a sub-optimal solution . In this work , the effect of activation quantization in PTQ is deeply investigated for the first time . We empirically observe that perceiving the activation quantization benefits the extremely low-bit PTQ reconstruction and surprisingly find that only partial activation quantization is more preferable . An intuitive understanding is that incorporating activation will lead to a different optimized weight . Inspired by this , we conduct theoretical studies on how activation quantization affects the weight tuning , and the conclusion is that involving activation quantization into the reconstruction helps the flatness of model on calibration data and dropping partial quantization contributes to the flatness on test data . Motivated by both empirical and theoretical findings , we propose QDROP that randomly drops quantization during the PTQ reconstruction to pursue the flatness from a general perspective . With this simple and effective approach , we set up a new state of the art for PTQ on various tasks including image classification , object detection for computer vision , and text classification and question answering for natural language processing . To this end , this paper makes the following contributions : 1 . We confirm the benefits unprecedentedly from involving activation quantization in the PTQ reconstruction and surprisingly observe that partial involvement of activation quantization performs better than the whole . 2 . A theoretical framework is established to deeply analyze the influence of incorporating activation quantization into weight tuning . Using this framework , we conclude that the flatness of the optimized low-bit model on calibration data and test data is crucial for the final accuracy . 3 . Based on the empirical and theoretical analyses , we propose a simple yet effective method QDROP that achieves the flatness from a general perspective . QDROP is easy to implement and can consistently boost existing methods as a plug-and-play module for various neural networks including CNNs like ResNets and Transformers like BERT . 4 . Extensive experiments on a large variety of tasks and models prove that our method set up a new state of the art for PTQ . With QDROP , the 2-bit post-training quantization becomes possible for the first time . 2 PRELIMINARIES . Basic Notations . Throughout this paper , matrices ( or tensors ) are marked as X , whereas the vectors are denoted by x . Sometimes we use w to represent the flattened version of the weight matrix W . Operator · is marked as scalar multiplication , ⊙ is marked as element-wise multiplication for matrices or vectors . For matrix multiplication , we denote Wx as matrix-vector multiplication or WX as matrix-matrix multiplication . For a feedforward neural network with activation function , we denote it as G ( w , x ) and the loss function as L ( w , x ) , where x and w are the network inputs and weights , respectively . Note that we assume x is sampled from training dataset Dt , thus the final loss is denoted by Ex∼Dt [ L ( w , x ) ] . For the network forward function , we can write it as : z ( ℓ+1 ) i = ∑ j W ( ℓ ) i , j · a ( ℓ ) j , f ( z ( ℓ+1 ) i ) = a ( ℓ+1 ) i , ( 1 ) where Wi , j denotes weight connecting the jth activation neuron and the ith output . The bracket superscript ( ℓ ) is the layer index . f ( · ) indicates the activation function . Post-training Quantization . Uniform quantizer maps continuous values x ∈ R into fixed-point integers . For instance , the activation quantization function can be written as x̂ = ⌊xs ⌉ · s , where ⌊·⌉ denotes the rounding-to-nearest operator , s is the step size between two subsequent quantization levels . While rounding-to-nearest operation minimizes the mean squared error between x̂ and x , the minimization of parameter space certainly can not equal to the minimization in final task loss ( Li et al. , 2021a ) , i.e. , Ex∼Dt [ L ( ŵ , x ) ] . However , in the post-training setting , we only have a tiny subset Dc ∈ Dt that only contains 1k images . Thus , it is hard to minimize the final loss objective with limited data . Recently , a series of works ( Nagel et al. , 2020 ; Li et al. , 2021a ) learn to either round up or down and view the new rounding mechanism as weight perturbation , i.e. , ŵ = w + ∆w . Take a pre-trained network G as an example , they leverage Taylor Expansion to analyze the target , which reveals the quantization interactions among weights : min ŵ E [ L ( ŵ , x ) − L ( w , x ) ] ≈ min ŵ E [ 1 2 ∆w⊤Hw∆w ] , ( 2 ) where Hw = Ex∼Dt∇2wL ( w , x ) is the expected second-order derivative . The above objective could be transformed into the change of output weighted by the output Hessian . min ŵ E [ ∆w⊤Hw∆w ] ≈ min ŵ E [ ∆a⊤Ha∆a ] ( 3 ) About the above minimization , they finetune only the weight by reconstructing each block/layer output ( See Fig . 1 ) . But they did not explore the activation quantization during output reconstruction with only modeling weight quantization as noise . The step size for activation quantization is determined after the reconstruction stage . Intuitively , when quantizing the activations of a full-precision model to 2-bit or 3-bit , there should be different suitable weights . However , the existing works result in the same optimized weight due to the neglect of activation quantization . Therefore , we argue that when quantizing the neural network , the noise caused by activation quantization should be considered coherently with weights . 3 METHODOLOGY . In this section , to reveal the influence of introducing activation quantization before output reconstruction , we first conduct empirical experiments and present two observations . Then a theoretical framework is built to investigate how the activation quantization affects the optimized weights . Last , equipped with the analysis conclusions , a simple yet effective method dubbed QDROP is proposed . 3.1 EMPIRICAL OBSERVATIONS . To investigate the influence of activation quantization when reconstructing the layer/block output , we conduct preliminary experiments on the ImageNet ( Russakovsky et al. , 2015 ) dataset . Our experiments are based on the open-sourced code Li et al . ( 2021a ) except that we will introduce activation quantization from 1 to k − 1 blocks before the kth block ’ s reconstruction . We give a simple visualization in Fig . 1 to show 3 cases for putting activation quantization in different stages . Case 1 means that all activations are kept in 32-bit full-precision during the reconstruction of block output , which is also adopted in existing work Nagel et al . ( 2020 ) ; Li et al . ( 2021a ) . Case 2 and Case 3 are used for incorporating activation quantization into the reconstruction stage . However , Case 3 will omit the current block ’ s quantization while Case 2 will not . The detailed results of these three cases are listed in Table 1 ( Comparisons on 2-bit ( W2A2 ) quantization for ResNet-18 , ResNet-50 and W3A3 for others for the sake of the crashed results on 2-bit . ) and the algorithm is put in algorithm 2 . According to the comparison , we can obtain two observations : 1 . For extremely low-bit quantization ( e.g. , W2A2 ) , there will be huge accuracy improvement when . considering activation quantization during weight tuning . This is confirmed by comparing with Case 1 and Case 2 . We find Case 1 barely converges while Case 2 achieves good accuracy . It reveals that a separate optimization of weights and activations can not find an optimal solution . After introducing the activation quantization , the weights will learn to diminish the influence of activation quantization . 2 . Partially introducing block-wise activation quantization surpasses introducing the whole activation quantization . Case 3 does not quantize the activations inside the current tuning block but achieves better results than Case 2 . This inspires us that how much activation quantization we introduce for weight tuning will affect the final accuracy . 3.2 HOW DOES ACTIVATION QUANTIZATION AFFECT WEIGHT TUNING . The empirical observations have highlighted the importance of activation quantization during the PTQ pipeline . To further explore how activation quantization will affect the weight tuning , we build a theoretical framework that analyzes the final loss objective with both weights and activations being quantized , which presents clues of high accuracy for extremely low-bit post-training quantization . Conventionally , the activation quantization could be modeled as injecting some form of noise imposed on the full-precision counterpart , defined as e = ( â−a ) . To remove the influence of activation range on e , we translate the noise into a multiplicative form , i.e. , â = a · ( 1 + u ) , where the range of u is affected by bit-width and rounding error . Detailed illustration of the new form noise can be found in Appendix A . Here , 1+u ( x ) is adopted to present the activation noise since it is related to specific input data point x . Equipped with the noise , we add another argument in calculating the loss function and define our optimization objective in PTQ as : min ŵ Ex∼Dc [ L ( w +∆w , x,1+ u ( x ) ) − L ( w , x,1 ) ] . ( 4 ) We hereby use a transformation that can absorb the noise on activation and transfer to weight , where the perturbation on weight is denoted as 1 + v ( x ) ( V ( x ) is used in matrix multiplication format ) . Consider a simple matrix-vector multiplication Wa in forward pass , we have W ( a⊙ ( 1+u ( x ) ) ) = ( W ⊙ ( 1+ V ( x ) ) ) a , given by W ( a⊙ 1 + u1 ( x ) 1 + u2 ( x ) ... 1 + un ( x ) ) = ( W ⊙ 1 + u1 ( x ) 1 + u2 ( x ) ... 1 + un ( x ) 1 + u1 ( x ) 1 + u2 ( x ) ... 1 + un ( x ) ... 1 + u1 ( x ) 1 + u2 ( x ) ... 1 + un ( x ) ) a . ( 5 ) By taking Vi , j ( x ) = uj ( x ) , quantization noise on the activation vector ( 1 + u ( x ) ) can be transplanted into perturbation on weight ( 1+ v ( x ) ) . Note that for a specific input data point x , there are two distinct u ( x ) and v ( x ) . Proof is available at Sec . B.1 . Also note that for a convolutional layer , we can not apply such transformation since the input to convolution is a matrix and will cause different V . Nonetheless , we can give a formal lemma that absorbs u ( x ) and holds corresponding v ( x ) ( See the Appendix Sec . B.2 for rigorous proof ) : Lemma 1 . For a quantized ( convolutional ) neural network , the influence of activation quantization on the final loss objective in post-training quantization can be transformed into weight perturbation . Ex∼Dc [ L ( ŵ , x,1+ u ( x ) ) − L ( w , x,1 ) ] ≈ Ex∼Dc [ L ( ŵ ⊙ ( 1+ v ( x ) ) , x,1 ) − L ( w , x,1 ) ] ( 6 ) By interpolating L ( ŵ , x,1 ) into Lemma 1 , we can obtain the final theorem : Theorem 1 . For a neural network G with quantized weight ŵ and activation perturbation 1+u ( x ) , we have : Ex∼Dc [ L ( ŵ , x,1+ u ( x ) ) − L ( w , x,1 ) ] ≈ Ex∼Dc [ ( L ( ŵ , x,1 ) − L ( w , x,1 ) ) ︸ ︷︷ ︸ ( 7−1 ) + ( L ( ŵ ⊙ ( 1+ v ( x ) ) , x,1 ) − L ( ŵ , x,1 ) ) ︸ ︷︷ ︸ ( 7−2 ) ] . ( 7 ) Here , Theorem 1 divides optimization objective into two terms . Term ( 7-1 ) is the same as Eq . ( 2 ) explored in ( Nagel et al. , 2020 ; Li et al. , 2021a ) , which reveals how weight quantization interacts with loss function . Term ( 7-2 ) is the additional loss change by introducing activation quantization . In another way to interpret Eq . ( 7 ) , the term ( 7-2 ) stands for the loss change with jitters on the weight quantized network G ( ŵ , x ) . This type of noise correlates with certain kinds of robustness . As stated in some works about generalization and flatness ( Dinh et al. , 2017 ; Hochreiter & Schmidhuber , 1997 ) , intuitively , flat minimum means relatively small loss change under perturbation in the parameters , otherwise , the minimum is sharp . In this paper , we follow the notion of flatness defined in ( Neyshabur et al. , 2017 ) , which considers loss change from the perspective of statistical expectation . And as ( Neyshabur et al. , 2017 ) and ( Jiang et al. , 2019 ) refer to , we consider the magnitude of the perturbation with respect to the magnitude of parameters and take the formulation as Ev∼D [ L ( fw⊙ ( 1+v ) ) −L ( fw ) ] , where each element of v is a random variable sampled from a noise distributionD and L represents for optimization objective on the training set . From this perspective , the term ( 7-2 ) can be interpreted as the flatness with perturbation related to input data , and thereby we could achieve the following corollary . Corollary 1 . On calibration data x , with activation quantization noise u ( x ) , there exists the corresponding weight perturbation v ( x ) which satisfies that the trained quantized model is flatter under the perturbation v ( x ) . With Corollary 1 , Case 2 and 3 discussed in Sec . 3.1 enjoy a flatter loss landscape benefited from perceiving the activation quantization . This explains their superiority compared with Case 1 . The measurement of sharpness on calibration data ( left part ) in Fig . 2 further validates this point . With similar perturbation magnitude , Case 2 and 3 suffer less loss degradation than Case 1 .

Summary: This paper proposes the post-training quantization for extremely low-bit neural networks. By considering the activation quantization during reconstruction, the presented QDrop randomly drops the quantization of activations with higher loss flatness that adapts the activations with various activation bitwidths well. Experimental results have demonstrated the superiority of the presented method.