SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation

1 INTRODUCTION . With the widespread application of DNN , more and more DNN models are deployed on both computation-constrained and memory-constrained environments , e.g. , smartphones , IoT devices , and self-driving cars . The desire for lightweight and energy-efficient DNN deployment solutions is increasing . Quantization is one of the most promising techniques to convert weights and activations to lower bit formats and simultaneously reduce computational time and memory consumption . There are two kinds of quantization : Post-training quantization ( PTQ ) ( Banner et al. , 2018 ; Choukroun et al. , 2019 ; Zhao et al. , 2019 ; Nagel et al. , 2020 ) and Quantization-aware training ( QAT ) ( Gupta et al. , 2015 ; Jacob et al. , 2018 ; Wang et al. , 2019 ; Zhuang et al. , 2021 ) . QAT requires to simulate quantization in the training process , which invokes time-consuming retraining and hyper-parameter tuning . In contrast , PTQ directly quantizes well-trained models without retraining . However , they still need training datasets to calibrate ( Nagel et al. , 2020 ) quantized models but are often unavailable due to privacy and security issues , such as medical and confidential scenarios . ∗Jingwen Leng and Minyi Guo are corresponding authors of this paper . 1https : //github.com/clevercool/SQuant In contrast , data-free quantization ( DFQ ) has recently been presented as a promising way to quantize models without original datasets ( Nagel et al. , 2019 ; Cai et al. , 2020 ; Zhang et al. , 2021 ; Xu et al. , 2020 ; Liu et al. , 2021 ; Qin et al. , 2021 ; Choi et al. , 2020 ) . From a deployment perspective , DFQ is the most attractive quantization method since we can apply it to any trained models as a black box postprocessing step . However , current DFQ methods can not achieve high accuracy and fast processing time simultaneously . Traditionally , DFQ ( Nagel et al. , 2019 ) uses rounding quantization , leading to the rounding-to-nearest strategy . Such a strategy causes significant accuracy loss , especially in low-bit settings . To bridge the accuracy gap between data-free and data-driven quantization , researchers propose a series of data-generative DFQ methods . They use gradient-based methods to generate fake datasets for trained models . With the synthetic data , they can employ a data-driven calibration and fine-tuning strategy to improve accuracy . However , data generation typically adopts the time-consuming gradient-based methods , which require multiple iterations to generate each input . For example , prior works often spend hours generating a calibration dataset and fine-tuning the network ( Xu et al. , 2020 ; Liu et al. , 2021 ; Zhang et al. , 2021 ) . To solve this dilemma , we propose SQuant , a fast and accurate data-free quantization framework for convolutional neural networks , employing the constrained absolute sum of error ( CASE ) of weights as the rounding metric . By leveraging Hessian information of network loss due to quantization , we propose a novel diagonal Hessian approximation , which decomposes the optimization objective into three data-free sub-items : element-wise , kernel-wise , and output channel-wise , each of which corresponds to a single or a set of dimensions of the weight tensor . We progressively compose and optimize these three sub-items in the discrete space . The final approximate objective eliminates the requirement of data generation . We propose a progressive algorithm with linear complexity to solve the optimization objective , further accelerating DFQ time to a sub-second level . For example , SQuant only needs an average of 4 ms and 84 ms for quantizing a layer and the overall network of ResNet18 , respectively . As it does not require back-propagation nor fine-tuning , SQuant can run on inference-only devices with limited computation and memory resources on the fly . That opens up new opportunities and scenarios for adopting quantization . Compared with state-of-the-art DFQ methods , SQuant achieves higher accuracy on all evaluated models under the 4/6/8-bit settings . SQuant only introduces 0.1 % accuracy loss on average under the 8-bit setting . Under fewer bit precisions , the advantage of SQuant further expands . SQuant only introduces 1.8 % accuracy loss on average under the 6-bit setting . Under the 4-bit setting , SQuant can achieve more than 30 % accuracy improvement compared with data-free PTQ methods . In a word , SQuant pushes the accuracy and processing time of DFQ to a new frontier . 2 PRELIMINARIES . 2.1 NOTATIONS . We specifically use x , y and w to denote the input , output , and weight variables , respectively . Constant and scalar are denoted by italic letters , e.g. , c , M . Column vector and flattened matrix are denoted by bold lowercase letters , e.g. , w , and matrices ( or tensors ) are represented by uppercase letters , e.g. , W. The subscript and superscript can further represent the element indices and the layer of a network , respectively , e.g. , W ` i , j. E [ · ] denotes the expectation operator , and the network loss function is represented by L ( · ) . For convenience in this paper , we call the row of FC ( fully connected layer ) weight as the output channel and the column of FC weight as the input channel , which are the counterparts to Conv ( convolution layer ) weight . We use M , N , and K to denote output channel size , input channel size , and kernel height × kernel width , respectively . Specifically , FC has the shape of ( M , N,1 ) . 2.2 QUANTIZATION . Most previous works adopt the rounding-to-nearest approach for quantizing deep neural networks by rounding elements w to the nearest quantization grid values with a fixed-point data type . The quantization and dequantization for a quantized element ŵ can be described as ŵ = s · clip ( bws e , min , max ) , where s denotes the quantization scale parameter and , min and max are the lower and upper thresholds for the clipping function clip ( · ) . The operator b·e represents the rounding-to-nearest , i.e. , minimizing the mean squared error ( MSE ) between the quantized and the original value . 2.3 HESSIAN-BASED OPTIMIZATION FOR NEURAL NETWORKS . The Hessian-based approach is one of the most promising optimizations to further improve the quantization ( Dong et al. , 2019b ; a ; Nagel et al. , 2020 ; Shen et al. , 2020 ; Qian et al. , 2020 ; Wu et al. , 2020 ; Hubara et al. , 2020 ; Li et al. , 2021 ; Yao et al. , 2021 ) and pruning ( Yu et al. , 2021 ) performance for DNN models . Some of those works exploit the Hessian matrix to approximate loss degradation due to the quantization perturbation of weight , ∆W , by E [ L ( X , Y , W+∆W ) −L ( X , Y , W ) ] ≈E [ ∆W ·gW + 1 2 ∆W ·HW ·∆WT ] , ( 1 ) where the equation comes from second-order Taylor series expansion , gW is the gradient and HW is the full network Hessian matrix w.r.t . original weight , W. Since a well-trained model has already converged , the gradient term will be close to 0 and thus can be safely ignored . However , computing HW is infeasible because of the large memory overhead and computation complexity . To tackle this problem , we approximate HW as a layer-wise Hessian matrix HW ` under the assumption of cross-layer independence ( Dong et al. , 2017 ; Nagel et al. , 2020 ) , i.e. , HW ` = x ` x ` T ⊗∇2y ` L , where ⊗ denotes Kronecker product of two matrices , ∇2y ` L is the Hessian of the task loss w.r.t . y ` . For the m-th output channel of Conv or FC , HW ` can be approximatively simplified into output channel-wise ( Nagel et al. , 2020 ; Yu et al. , 2021 ; Wu et al. , 2020 ; Qian et al. , 2020 ) , HW ` m ≈ ∇2y ` Lm , m ·x ` x ` T = lm ·x ` x ` T , ( 2 ) where ∇2y ` L is approximately a diagonal matrix . Then the final optimization objective is ∆Ŵ ` m , : =argmin ∆W ` m , : ∆W ` m , :E [ HW ` m ] ∆W ` m , : T ( 3 ) =argmin ∆W ` m , : ∆W ` m , :E [ x ` x ` T ] ∆W ` m , : T = argmin ∆W ` m , : E [ ( ∆W ` m , :x ` ) 2 ] , ( 4 ) which is the MSE between the output activation produced from original and quantized weights . Each sub-problem deals with a single output channel ∆W ` m , : . We will further approximate Eq . ( 4 ) to remove any input data dependency from the optimization objective in Sec . 3.2 . 3 METHODOLOGY . 3.1 OVERVIEW . Although we can obtain a good quantization strategy by minimizing MSE for each output channel , it is an NP-hard combinatorial optimization problem . Even approaching an acceptable local minimum requires significant effort and involves input activations without the data-free promise . To avoid the combinatorial optimization problem and eliminate the requirement of data , we propose the SQuant framework . First , SQuant approximates Eq . ( 4 ) with three diagonal Hessian matrices corresponding to the dimensions of weight , in Sec . 3.2 . Due to the quantization with a fixed-point data type , SQuant transforms the problem into a data-free optimization problem in the discrete domain . SQuant dedicates to optimizing each layer ’ s weight employing a flipping approach ( Nagel et al. , 2020 ) without any input activation . To achieve our proposed optimization objective , minimizing CASE ( Constrained Absolute Sum of Error ) , SQuant progressively composes three approximate sub-items under constraint relaxation , introduced in Sec . 3.3 . Finally , SQuant needs to work out a flipping set f to minimize the CASE of each kernel and output channel . We design an efficient algorithm with a linear computation complexity to find a proper f based on Eq . ( 8 ) , in Sec . 3.4 . 3.2 DIAGONAL HESSIAN APPROXIMATION . In this work , we propose a new approximation of the Hessian matrix to cover non-diagonal elements and decompose Eq . ( 4 ) into three sub-items that correspond to the three dimensions of the weight tensor as illustrated in Fig . 1 : SQuant-E for element-wise optimization covers the diagonal elements of Hw ` m ( H-E ) ; SQuant-K for kernel-wise optimization covers the diagonal blocks of Hw ` m ( H-K ) ; SQuant-C for output channel-wise optimization covers the whole Hw ` m ( H-C ) . The E [ x ` x ` T ] can be approximated by the following equation : E [ x ` x ` T ] ≈ E+K+C , ( 5 ) where C = cmJNK , K = k1JK . . . kNJK , and E = e1,1 . . . eN , K . In the above equations , JNK is an all-one matrix with the dimension of N×K ( denoted as NK ) , and cm is a constant value for m-th output channel . K is a diagonal block matrix , where JK represents an all-one matrix with the dimension of K×K . The n-th diagonal block corresponds to n-th kernel in convolution and has its own constant value kn . E is a diagonal matrix with the diagonal elements of en , i , each of which is a constant value corresponding to i-th element of n-th kernel . Eq . ( 5 ) provides an approximation that preserves as much information from three different levels of E [ x ` x ` T ] as possible , which we explain in Appendix A.1 . The matrix C catches the common component of the Hessian matrix , while the matrix E reserves the individual components in the diagonal line of the Hessian matrix . In addition , we consider kernel-wise approximation for convolution layers by using matrix K. For each inference , the weights of a kernel , W ` m , n , : , scan the same feature map . As a result , the corresponding x has nearly the same expectation values in the center area , with a small perturbation in the marginal area due to padding . Therefore , knJK as a kernel-wise approximation achieves a low approximate error for convolution . For any E [ x ` x ` T ] , we can always find a decomposition that satisfies en , i , kn , cm > 0 , for which we present the decomposition method in Appendix A.2 . Substituting Eq . ( 5 ) into Eq . ( 4 ) yields the following equation . ∆W ` m , :E [ x ` x ` T ] ∆W ` m , : T ≈∑ n , i en , i∆W ` m , n , i 2 +∑ n kn∆W ` m , n , :JK∆W ` m , n , : T + cm∆W ` m , :JNK∆W ` m , : T . ( 6 )

Summary: In this paper, the authors propose a data-free quantization algorithm of deep neural networks called SQuant. The main idea of SQuant is to decompose the Hessian-based optimization objective into three components: element-wise, kernel-wise and channel-wise components. In order to jointly optimize these three objective functions, a constrained absolute sum of error (CASE) is studied and a progressive algorithm is used. Experiment results show that SQuant algorithm is able to keep higher accuracy with the same number of bits, compared to several baseline algorithms.