SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation

1 INTRODUCTION . With the widespread application of DNN , more and more DNN models are deployed on both computation-constrained and memory-constrained environments , e.g. , smartphones , IoT devices , and self-driving cars . The desire for lightweight and energy-efficient DNN deployment solutions is increasing . Quantization is one of the most promising techniques to convert weights and activations to lower bit formats and simultaneously reduce computational time and memory consumption . There are two kinds of quantization : Post-training quantization ( PTQ ) ( Banner et al. , 2018 ; Choukroun et al. , 2019 ; Zhao et al. , 2019 ; Nagel et al. , 2020 ) and Quantization-aware training ( QAT ) ( Gupta et al. , 2015 ; Jacob et al. , 2018 ; Wang et al. , 2019 ; Zhuang et al. , 2021 ) . QAT requires to simulate quantization in the training process , which invokes time-consuming retraining and hyper-parameter tuning . In contrast , PTQ directly quantizes well-trained models without retraining . However , they still need training datasets to calibrate ( Nagel et al. , 2020 ) quantized models but are often unavailable due to privacy and security issues , such as medical and confidential scenarios . ∗Jingwen Leng and Minyi Guo are corresponding authors of this paper . 1https : //github.com/clevercool/SQuant In contrast , data-free quantization ( DFQ ) has recently been presented as a promising way to quantize models without original datasets ( Nagel et al. , 2019 ; Cai et al. , 2020 ; Zhang et al. , 2021 ; Xu et al. , 2020 ; Liu et al. , 2021 ; Qin et al. , 2021 ; Choi et al. , 2020 ) . From a deployment perspective , DFQ is the most attractive quantization method since we can apply it to any trained models as a black box postprocessing step . However , current DFQ methods can not achieve high accuracy and fast processing time simultaneously . Traditionally , DFQ ( Nagel et al. , 2019 ) uses rounding quantization , leading to the rounding-to-nearest strategy . Such a strategy causes significant accuracy loss , especially in low-bit settings . To bridge the accuracy gap between data-free and data-driven quantization , researchers propose a series of data-generative DFQ methods . They use gradient-based methods to generate fake datasets for trained models . With the synthetic data , they can employ a data-driven calibration and fine-tuning strategy to improve accuracy . However , data generation typically adopts the time-consuming gradient-based methods , which require multiple iterations to generate each input . For example , prior works often spend hours generating a calibration dataset and fine-tuning the network ( Xu et al. , 2020 ; Liu et al. , 2021 ; Zhang et al. , 2021 ) . To solve this dilemma , we propose SQuant , a fast and accurate data-free quantization framework for convolutional neural networks , employing the constrained absolute sum of error ( CASE ) of weights as the rounding metric . By leveraging Hessian information of network loss due to quantization , we propose a novel diagonal Hessian approximation , which decomposes the optimization objective into three data-free sub-items : element-wise , kernel-wise , and output channel-wise , each of which corresponds to a single or a set of dimensions of the weight tensor . We progressively compose and optimize these three sub-items in the discrete space . The final approximate objective eliminates the requirement of data generation . We propose a progressive algorithm with linear complexity to solve the optimization objective , further accelerating DFQ time to a sub-second level . For example , SQuant only needs an average of 4 ms and 84 ms for quantizing a layer and the overall network of ResNet18 , respectively . As it does not require back-propagation nor fine-tuning , SQuant can run on inference-only devices with limited computation and memory resources on the fly . That opens up new opportunities and scenarios for adopting quantization . Compared with state-of-the-art DFQ methods , SQuant achieves higher accuracy on all evaluated models under the 4/6/8-bit settings . SQuant only introduces 0.1 % accuracy loss on average under the 8-bit setting . Under fewer bit precisions , the advantage of SQuant further expands . SQuant only introduces 1.8 % accuracy loss on average under the 6-bit setting . Under the 4-bit setting , SQuant can achieve more than 30 % accuracy improvement compared with data-free PTQ methods . In a word , SQuant pushes the accuracy and processing time of DFQ to a new frontier . 2 PRELIMINARIES . 2.1 NOTATIONS . We specifically use x , y and w to denote the input , output , and weight variables , respectively . Constant and scalar are denoted by italic letters , e.g. , c , M . Column vector and flattened matrix are denoted by bold lowercase letters , e.g. , w , and matrices ( or tensors ) are represented by uppercase letters , e.g. , W. The subscript and superscript can further represent the element indices and the layer of a network , respectively , e.g. , W ` i , j. E [ · ] denotes the expectation operator , and the network loss function is represented by L ( · ) . For convenience in this paper , we call the row of FC ( fully connected layer ) weight as the output channel and the column of FC weight as the input channel , which are the counterparts to Conv ( convolution layer ) weight . We use M , N , and K to denote output channel size , input channel size , and kernel height × kernel width , respectively . Specifically , FC has the shape of ( M , N,1 ) . 2.2 QUANTIZATION . Most previous works adopt the rounding-to-nearest approach for quantizing deep neural networks by rounding elements w to the nearest quantization grid values with a fixed-point data type . The quantization and dequantization for a quantized element ŵ can be described as ŵ = s · clip ( bws e , min , max ) , where s denotes the quantization scale parameter and , min and max are the lower and upper thresholds for the clipping function clip ( · ) . The operator b·e represents the rounding-to-nearest , i.e. , minimizing the mean squared error ( MSE ) between the quantized and the original value . 2.3 HESSIAN-BASED OPTIMIZATION FOR NEURAL NETWORKS . The Hessian-based approach is one of the most promising optimizations to further improve the quantization ( Dong et al. , 2019b ; a ; Nagel et al. , 2020 ; Shen et al. , 2020 ; Qian et al. , 2020 ; Wu et al. , 2020 ; Hubara et al. , 2020 ; Li et al. , 2021 ; Yao et al. , 2021 ) and pruning ( Yu et al. , 2021 ) performance for DNN models . Some of those works exploit the Hessian matrix to approximate loss degradation due to the quantization perturbation of weight , ∆W , by E [ L ( X , Y , W+∆W ) −L ( X , Y , W ) ] ≈E [ ∆W ·gW + 1 2 ∆W ·HW ·∆WT ] , ( 1 ) where the equation comes from second-order Taylor series expansion , gW is the gradient and HW is the full network Hessian matrix w.r.t . original weight , W. Since a well-trained model has already converged , the gradient term will be close to 0 and thus can be safely ignored . However , computing HW is infeasible because of the large memory overhead and computation complexity . To tackle this problem , we approximate HW as a layer-wise Hessian matrix HW ` under the assumption of cross-layer independence ( Dong et al. , 2017 ; Nagel et al. , 2020 ) , i.e. , HW ` = x ` x ` T ⊗∇2y ` L , where ⊗ denotes Kronecker product of two matrices , ∇2y ` L is the Hessian of the task loss w.r.t . y ` . For the m-th output channel of Conv or FC , HW ` can be approximatively simplified into output channel-wise ( Nagel et al. , 2020 ; Yu et al. , 2021 ; Wu et al. , 2020 ; Qian et al. , 2020 ) , HW ` m ≈ ∇2y ` Lm , m ·x ` x ` T = lm ·x ` x ` T , ( 2 ) where ∇2y ` L is approximately a diagonal matrix . Then the final optimization objective is ∆Ŵ ` m , : =argmin ∆W ` m , : ∆W ` m , :E [ HW ` m ] ∆W ` m , : T ( 3 ) =argmin ∆W ` m , : ∆W ` m , :E [ x ` x ` T ] ∆W ` m , : T = argmin ∆W ` m , : E [ ( ∆W ` m , :x ` ) 2 ] , ( 4 ) which is the MSE between the output activation produced from original and quantized weights . Each sub-problem deals with a single output channel ∆W ` m , : . We will further approximate Eq . ( 4 ) to remove any input data dependency from the optimization objective in Sec . 3.2 . 3 METHODOLOGY . 3.1 OVERVIEW . Although we can obtain a good quantization strategy by minimizing MSE for each output channel , it is an NP-hard combinatorial optimization problem . Even approaching an acceptable local minimum requires significant effort and involves input activations without the data-free promise . To avoid the combinatorial optimization problem and eliminate the requirement of data , we propose the SQuant framework . First , SQuant approximates Eq . ( 4 ) with three diagonal Hessian matrices corresponding to the dimensions of weight , in Sec . 3.2 . Due to the quantization with a fixed-point data type , SQuant transforms the problem into a data-free optimization problem in the discrete domain . SQuant dedicates to optimizing each layer ’ s weight employing a flipping approach ( Nagel et al. , 2020 ) without any input activation . To achieve our proposed optimization objective , minimizing CASE ( Constrained Absolute Sum of Error ) , SQuant progressively composes three approximate sub-items under constraint relaxation , introduced in Sec . 3.3 . Finally , SQuant needs to work out a flipping set f to minimize the CASE of each kernel and output channel . We design an efficient algorithm with a linear computation complexity to find a proper f based on Eq . ( 8 ) , in Sec . 3.4 . 3.2 DIAGONAL HESSIAN APPROXIMATION . In this work , we propose a new approximation of the Hessian matrix to cover non-diagonal elements and decompose Eq . ( 4 ) into three sub-items that correspond to the three dimensions of the weight tensor as illustrated in Fig . 1 : SQuant-E for element-wise optimization covers the diagonal elements of Hw ` m ( H-E ) ; SQuant-K for kernel-wise optimization covers the diagonal blocks of Hw ` m ( H-K ) ; SQuant-C for output channel-wise optimization covers the whole Hw ` m ( H-C ) . The E [ x ` x ` T ] can be approximated by the following equation : E [ x ` x ` T ] ≈ E+K+C , ( 5 ) where C = cmJNK , K = k1JK . . . kNJK , and E = e1,1 . . . eN , K . In the above equations , JNK is an all-one matrix with the dimension of N×K ( denoted as NK ) , and cm is a constant value for m-th output channel . K is a diagonal block matrix , where JK represents an all-one matrix with the dimension of K×K . The n-th diagonal block corresponds to n-th kernel in convolution and has its own constant value kn . E is a diagonal matrix with the diagonal elements of en , i , each of which is a constant value corresponding to i-th element of n-th kernel . Eq . ( 5 ) provides an approximation that preserves as much information from three different levels of E [ x ` x ` T ] as possible , which we explain in Appendix A.1 . The matrix C catches the common component of the Hessian matrix , while the matrix E reserves the individual components in the diagonal line of the Hessian matrix . In addition , we consider kernel-wise approximation for convolution layers by using matrix K. For each inference , the weights of a kernel , W ` m , n , : , scan the same feature map . As a result , the corresponding x has nearly the same expectation values in the center area , with a small perturbation in the marginal area due to padding . Therefore , knJK as a kernel-wise approximation achieves a low approximate error for convolution . For any E [ x ` x ` T ] , we can always find a decomposition that satisfies en , i , kn , cm > 0 , for which we present the decomposition method in Appendix A.2 . Substituting Eq . ( 5 ) into Eq . ( 4 ) yields the following equation . ∆W ` m , :E [ x ` x ` T ] ∆W ` m , : T ≈∑ n , i en , i∆W ` m , n , i 2 +∑ n kn∆W ` m , n , :JK∆W ` m , n , : T + cm∆W ` m , :JNK∆W ` m , : T . ( 6 )

Summary: This paper proposes a new data-free quantization method that does not require back-propagation nor fine-tuning. The key idea is adopting Hessian-based optimization that can be decomposed into three parts (SQuant-E, K, and C) corresponding to the three dimensions of the weight tensor (using a few approximations, such as cross-layer independence to simplify the optimization). Then, instead of MSE, the authors introduce CASE (constrained ASE) of weight perturbation. The experimental results show that the proposed DFQ method outperforms even GDFQ that is basically a kind of QAT. The proposed technique is especially useful for a low-bit quantization.