BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction

1 INTRODUCTION . The past decade has witnessed the rapid development of deep learning in many tasks , such as computer vision , autonomous driving , etc . However , the issue of huge computation cost and memory footprint requirements in deep learning has received considerable attention . Some works such as neural architecture search ( Zoph & Le , 2016 ) try to design and search a tiny network , while others , like quantization ( Hubara et al. , 2017 ) , and network pruning ( Han et al. , 2015 ) are designed to compress and accelerate off-the-shelf well-trained redundant networks . Many popular quantization and network pruning methods follow a simple pipeline : training the original model and then finetune the quantized/pruned model . However , this pipeline requires a full training dataset and many computation resources to perform end-to-end backpropagation , which will greatly delay the production cycle of compressed models . Besides , not all training data are always ready-to-use considering the privacy problem . Therefore , there is more demand in industry for quantizing the neural networks without retraining , which is called Post-training Quantization . Although PTQ is fast and light , it suffers from severe accuracy degeneration when the quantization precision is low . For example , DFQ ( Nagel et al. , 2019 ) can quantize ResNet-18 to 8-bit without accuracy loss ( 69.7 % top-1 accuracy ) but in 4-bit quantization , it can only achieve 39 % top-1 accuracy . The primary reason is the approximation in the parameter space is not equivalent to the approximation in model space thus we can not assure the optimal minimization on the final task loss . Recent works like ( Nagel et al. , 2020 ) recognized the problem and analyzed the loss degradation by Taylor series expansion . Analysis of the second-order error term indicates we can reconstruct each layer output to approximate the task loss degeneration . However , their work can not further quantize the weights into INT2 because the cross-layer dependency in the Hessian matrix can not be ignored when the perturbation on weight is not small enough . In this work , we analyze the second-order ∗Equal contribution , Corresponding author . error based on the Gauss-Newton matrix . We show that the second-order error can be transformed into network final outputs but suffer from bad generalization . To achieve the best tradeoff , we adopt an intermediate choice , block reconstruction . In addition , our contributions are threefold : 1 . Based on the second-order analysis , we define a set of reconstruction units and show that block reconstruction is the best choice with the support from theoretical and empirical evidence . We also use Fisher Information Matrix to assign each pre-activation with an importance measure during reconstruction . 2 . We incorporate genetic algorithm and the well-defined intra-block sensitivity measure to generate latency and size guaranteed mixed precision quantized neural networks , which fulfills a general improvement on both specialized hardware ( FPGA ) and general hardware ( ARM CPU ) . 3 . We conduct extensive experiments to verify our proposed methods . We find that our method is applicable to a large variety of tasks and models . Moreover , we show that post-training quantization can quantize weights to INT2 without significant accuracy loss for the first time . 2 PRELIMINARIES . Notations Vectors are denoted by small bold letters and matrices ( or tensors ) are denoted by capital bold letters . For instance , W and w represent the weight tensor and its flattened version . Bar accent denotes the expectation over data points , e.g . ā . Bracketed superscript w ( ` ) indicates the layer index . For a convolutional or a fully-connected layer , we mark its input and output vectors by x and z . Thus given a feedforward neural network with n layers , we can denote the forward process by x ( ` +1 ) = h ( z ( ` ) ) = h ( W ( ` ) x ( ` ) + b ( ` ) ) , 1 ≤ ` ≤ n , ( 1 ) where h ( · ) indicates the activation function ( ReLU in this paper ) . For simplicity , we omit the analysis of bias b ( ` ) as it can be merged into activation . || · ||F denotes the Frobenius norm . Quantization Background Uniform symmetric quantization maps the floating-point numbers to several fixed-points . These points ( or grids ) have the same interval and are symmetrically distributed . We denote the set that contains these grids asQu , symb = s× { −2b−1 , . . . , 0 , . . . , 2b−1− 1 } . Here , s is the step size between two grids and b is the bit-width . Quantization function , denoted by q ( · ) : R → Qu , symb , is generally designed to minimize the quantization error : min ||ŵ −w||2F . s.t . ŵ ∈ Q u , sym b ( 2 ) Solving this minimization problem , one can easily get the q ( · ) by leveraging the rounding-to-nearest operation b·e . Rounding-to-nearest is a prevalent method to perform quantization , e.g . PACT ( Choi et al. , 2018 ) . However , recently some empirical or theoretical evidence supports that simply minimizing the quantization error in parameter space does not bring optimal task performances . Specifically , Esser et al . ( 2020 ) propose to learn the step size s by gradient descent in quantization-aware training ( QAT ) . LAPQ ( Nahshan et al. , 2019 ) finds the optimal step size when the loss function is minimized without re-training the weights . Their motivations are all towards minimizing a final objective , which is the task loss , i.e. , minE [ L ( ŵ ) ] , s.t . ŵ ∈ Qu , symb . ( 3 ) While this optimization objective is simple and can be well-optimized in QAT scenarios , it is not easy to learn the quantized weight without end-to-end finetuning as well as sufficient training data and computing resources . In post-training quantization settings , we only have full precision weights that w ? = arg minw E [ L ( w ) ] where w ∈ R and a small subset of training data to do calibration . Taylor Expansion It turns out that the quantization imposed on weights could be viewed as a special case of weight perturbation . To quantitatively analyze the loss degradation caused by quantization , Nagel et al . ( 2020 ) use Taylor series expansions and approximates the loss degradation by E [ L ( w + ∆w ) ] − E [ L ( w ) ] ≈ ∆wTḡ ( w ) + 1 2 ∆wTH̄ ( w ) ∆w , ( 4 ) where ḡ ( w ) = E [ ∇wL ] and H̄ ( w ) = E [ ∇2wL ] are the gradients and the Hessian matrix and ∆w is the weight perturbation . Given the pre-trained model is converged to a minimum , the gradients can be safely thought to be close to 0 . However , optimizing with the large-scale full Hessian is memoryinfeasible on many devices as the full Hessian requires terabytes of memory space . To tackle this problem , they make two assumptions : 1 . Layers are mutual-independent , thus the Hessian is in the form of layer-diagonal1 and Kroneckerfactored , i.e. , H̄ ( w ( ` ) ) = E [ x ( ` ) x ( ` ) , T ⊗ H ( z ( ` ) ) ] , where ⊗ is the Kronecker product . 2 . The second-order derivatives of pre-activations are constant diagonal matrix ( H ( z ( ` ) ) = c × I ) which is independent of input data points . At last , the objective is transformed into a practical proxy signal , the change in feature-maps ( z = Wx ) , and the quantized model can be obtained by a layer-by-layer feature map reconstruction algorithm ( with few calibration images ) . Recent works , like Bit-Split ( Wang et al. , 2020 ) and AdaQuant ( Hubara et al. , 2020 ) , also take this layer-wise objective to improve the post-training quantization . However , they failed to quantize weights to INT2 . We think the inherent reason is that when ∆w grows higher , the former assumptions do not hold and an accurate signal is required . 3 PROPOSED METHOD . 3.1 CROSS-LAYER DEPENDENCY . Denote the neural network output z ( n ) = f ( θ ) , the loss function can be represented by L ( f ( θ ) ) where θ = vec [ w ( 1 ) , T , . . . , w ( n ) , T ] T is the stacked vector of weights in all n layers . The Hessian matrix can be computed by ∂2L ∂θi∂θj = ∂ ∂θj ( m∑ k=1 ∂L ∂z ( n ) k ∂z ( n ) k ∂θi ) = m∑ k=1 ∂L ∂z ( n ) k ∂2z ( n ) k ∂θi∂θj + m∑ k , l=1 ∂z ( n ) k ∂θi ∂2L ∂z ( n ) k ∂z ( n ) l ∂z ( n ) l ∂θj , ( 5 ) where z ( n ) ∈ Rm . Since the pretrained full precision model is converged to a local minimum , we can assume the Hessian is positive-semidefinite ( PSD ) . Specifically , the converged model has ∇z ( n ) L close to 0 so the first term in Eq . ( 5 ) is neglected and Hessian becomes the Gauss-Newton ( GN ) matrix G ( θ ) . GN matrix can be written in matrix form ( Botev et al. , 2017 ) as H ( θ ) ≈ G ( θ ) = Jz ( n ) ( θ ) TH ( z ( n ) ) Jz ( n ) ( θ ) , ( 6 ) where Jz ( n ) ( θ ) is the Jacobian matrix of the network output with respect to the network parameters . However , in practice , we can not explicitly compute and store the Jacobian for each input data point in such a raw form . To reduce the computation and memory budget , we will transform the secondorder error into the network output , as shown in the following theorem . Theorem 3.1 . Consider an n-layer feedforward neural network with ReLU activation function . Assuming all weights are quantized , the second-order error optimization can be transformed by : arg min θ̂ ∆θTH̄ ( θ ) ∆θ ≈ arg min θ̂ E [ ∆z ( n ) , TH ( z ( n ) ) ∆z ( n ) ] . ( 7 ) Remark 3.1 . The same transformation is also applicable for activation quantization . The quadratic loss is defined as E [ ∆γTH ( γ ) ∆γ ] where ∆γ = vec [ ∆x ( 1 ) , T , . . . , ∆x ( n ) , T ] T. We prove the theorem using the quadratic form , details can be found in Appendix A.1 . Here we provide a sketch of the proof by matrix form . The product of perturbation and Jacobian can be thought as the first-order Taylor approximation of the change in network output ∆z ( n ) : ∆z ( n ) = ẑ ( n ) − z ( n ) ≈ Jz ( n ) ( θ ) ∆θ . ( 8 ) Therefore , combining Eq . ( 8 ) and Eq . ( 6 ) we can transform the large-scale second-order error into the change in network outputs characterized by the output Hessian H ( z ( n ) ) . The theorem indicates a simple observation , suppose a well-trained teacher model and an initialized student model , we can minimize their discrepancy by reconstructing the network ’ s final output z ( n ) , which coincides with and generalizes the distillation ( Hinton et al. , 2015 ; Polino et al. , 2018 ) . LAPQ ( Nahshan et al. , 2019 ) also considers the dependency but their optimization does not rely on second-order information . However , we should emphasize that distillation requires the same computation and data resources as in normal training procedure , which is impractical for PTQ with limited data . 1To prevent ambiguity , we hereby use layer-diagonal Hessian to replace the common name “ block-diagonal Hessian ” because the block in this paper means a building block in the CNNs .

Summary: This paper explores the post-training inference quantization. Based on second-order quantization error analysis, it proposes to reconstruct quantized model in a block level to achieve SOTA accuracy for INT2 weight quantization, which distinguish this paper from previous reported layer-wise reconstruction approach. The proposed approach is intuitive and supported by extensive experiments across a wide range of image classification and object detection tasks.