Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming

1 INTRODUCTION . The pursuit of advanced Deep Neural Networks ( DNNs ) causes researchers to construct deeper and wider networks , making them expensive to use in terms of power and time . This increases the need for efficient implementations of these networks . Efficient networks reduce cloud-vendor costs and make it possible to run them on low-power devices such as smartphones and wearable devices . The most common off-the-shelf approach to improving network efficiency is quantization , which reduces the numerical precision of the network and its complexity and memory footprint . DNN quantization techniques can be classified as either post-training or quantization-aware training ( QAT ) techniques ( Han et al. , 2015 ; Courbariaux et al. , 2015 ; Hubara et al. , 2017 ; Zhou et al. , 2016 ) . Although QAT techniques , in general , achieve better results , there are important real-world scenarios in which they are not applicable . These are the cases where the training data is sensitive or simply unavailable at the time of deployment . For instance , when off-the-shelf or legacy models are being used , or when medical records are involved . Therefore , much attention has recently been dedicated to post-training quantization methods ( Nagel et al. , 2019 ; Banner et al. , 2018 ; Zhao et al. , 2019 ) , which can be more easily applied in practice . These methods allow for network quantization to happen seamlessly when deployed , without requiring additional information from the user except a small unlabeled calibration set . Unfortunately , post-training quantization below 8-bit always incurs significant accuracy degradation and in some cases even higher numerical precision is required . In this paper , our goal is to break this barrier by distilling all the information the pre-trained model and calibration set encode . Our goal is to find an optimal scheme for current state of the art hardware which usually support 16,8,4 bits data types with per-channel quantization of the weights . To that end , we suggest a three-stage pipeline that consists of methods applied solely on a small calibration set to reduce the local error introduced during the quantization process ( e.g. , round-off errors ) followed by integer programming to determine the bit-width of different layers so that the overall accuracy degradation is minimized . Even without using mixed-precision , the suggested method is much less prone to over-fitting than current methods and yields best in class results for 8-bits Mobilenet-V2 and BERT-base trained on ImageNet and SQuAD1.1 datasets , respectively . Our paper suggests several contributions for mixed-precision post-training quantization : 1 . AdaQuant : A layer-by-layer optimization method that minimizes the error between the quantized layer output and the full-precision layer output . This method can consume only a small calibration dataset from training data without overfitting . In a comprehensive study , we show that AdaQuant defines a new state-of-the-art for post-training quantization on several networks and tasks , including vision models ( Resnet18 , Resnet50 , MobilenetV2 ) and language ( BERT ) . 2 . Integer programming : As some parts of the network may allow lower precision compared to other layers , we suggest an integer-linear programming based approach for determining the precision level of different layers . This method aims at maximizing either the expected speedup or savings in power consumption without violating a predefined constraint on network accuracy degradation or compression . 3 . Batch-norm tuning : Following quantization we observe an inherent bias in the mean and the variance of batch norm statistics . We show that by employing the re-estimated statistics in batch normalization , much of the quantized network degradation can be recovered . 4 . Light and Advanced pipelines : We analyze the advantages and disadvantages of each of the given methods and suggest two pipelines : ( 1 ) light pipeline that does not require a backward pass , thus can be invoked even on inference-only hardware ; and ( 2 ) Advanced pipeline that includes also AdaQuant and bias tuning . 2 RELATED WORK . There has been a significant effort to accelerate inference via quantization ( Courbariaux et al. , 2015 ; Han et al. , 2015 ; Rastegari et al. , 2016 ; Zhou et al. , 2017 ) . These works involve re-training in order to compensate for the degradation due to the quantization process . Post-training quantization , on the other hand is applied to a model after it was trained . Thus , it avoids re-training and as such it is much simpler to use . However , naively quantizing a full-precision model to INT4 or lower to accelerate computation usually incurs significant accuracy degradation ( Krishnamoorthi , 2018 ; Jacob et al. , 2018 ) . AdaQuant : A recent post-training quantization method ( Nagel et al. , 2020 ) , termed AdaRound , suggested optimizing the rounding policy . Instead of using the predominant rounding-to-nearest approach , they suggest formulating a per-layer quadratic optimization problem to optimize the roundoff error . Our proposed method , AdaQuant , takes another step and relaxes AdaRound ’ s implicit constraint which forces the quantized weights to be within ±1 of their round-to-nearest value . This is done by optimizing the weights and quantization parameters of each layer separately , over the calibration set , to minimize the MSE between the layer ’ s original and quantized outputs . As oppose to AdaRound we apply AdaQuant to find optimal quantization not only to weights but also to activations . In addtion we suggest two flavors for AdaQuant : ( 1 ) parallel-AdaQuant suited for mixed precision setting ; ( b ) sequential-adaquant which suited for fixed configuration . Integer programming : Early work by Lin et al . ( 2016 ) used a convex optimization formulation which results in a simple greedy compression scheme . Aflalo et al . ( 2020 ) used a combinatorial optimization approach for network pruning . Their problem was formulated as a Knapsack problem that optimizes the trade-off between the channels importance and their associated computational cost . Cai et al . ( 2020 ) finds a mixed-precision configuration with a guaranteed Pareto efficient allocation with respect to model size and accuracy degradation . While this provides a ” best-effort ” standard ( e.g. , the configuration can not be further compressed without hurting accuracy ) , it does not suggest which of all possible outcomes is best . To the best of our knowledge , this work is the first to formalize a generic integer program , which can easily be adapted to various types of models and requirements with a clear objective and constraints . Batch norm tuning : Finkelstein et al . ( 2019 ) were the first to recognize that a significant source of degradation is a shift in the mean activation value . They show a simple method to compensate for this bias by updating the bias terms . Nagel et al . ( 2019 ) suggest to equalize the weight ranges in the network and correct biases in the error that are introduced during quantization .Recently Sun et al . ( 2019 ) suggested batch norm tuning for FP8 models . Here we detail how to perform this procedure on a per-channel quantized ( PCQ ) model with fused batch-norm layers . The procedure is light as it only requires to invoke the quantized model few times ( on the calibration set ) and adjust the quantization parameters.Moreover after retuning the BN layers can be reabsorbed which reduces the inference complexity . To the best of our knowledge we are the first to suggest it . 3 OPTIMIZING THE QUANTIZATION PIPELINE . In most post-training quantization settings , a model and a small unlabeled calibration set are given . To avoid overfitting the calibration set , most studies utilize it only to extract the network ’ s internal statistics , which is later used to set the quantization parameters . Here we suggest using the calibration set much more extensively to tune the model while avoiding over-fitting the data . In the following subsections , we detail three different optimization methods over the calibration set : ( 1 ) AdaQuant , a layerwise optimization of weights and quantization parameters ; ( 2 ) an integer programming formulation for a mixed-precision setting ; and ( 3 ) Batch Normalization Tuning ( BNT ) , for tuning the model ’ s internal statistics to match the numerical precision setting . We discuss the strengths and weaknesses of each method and suggest an optimization flow that exploits all the additive merits and leads to state-of-the-art results . 3.1 ADAQUANT - LAYERWISE OPTIMIZATION OVER THE CALIBRATION SET . Several researchers suggested per-tensor optimization to reduce quantization error by minimizing some form of MSE objective between the quantized and the full-precision tensor X ( either weights or activations ) . They look for an optimized quantization step size ∆̂ obtained by ∆̂ = arg min ∆ ||X −Q∆ ( X ) ||2 ; Q∆ ( X ) = ∆ · ⌊ X ∆ ⌉ , ( 1 ) where Q ( · ) is the quantization function . Although these methods are fast and easy to use , they often result in an inferior solution — the loss in eq . 1 is sub-optimal , as it penalizes all the quantization errors equally . However , the loss should penalize more quantization errors which affect the classification . Accordingly , researchers suggested Quantization-Aware-Training ( QAT ) methods to fix this error by training the entire model at once . However , those methods have three limitations : ( a ) they require the large training set to avoid over-fitting , ( b ) they approximate the back-propagation gradients through discrete function ( the quantizer ) and ( c ) they have high computational and memory footprints . We suggest a modified objective for per-layer joint optimization of the weights and quantization parameters . ( ∆̂w , ∆̂x , V̂ ) = arg min ∆w , ∆x , V ||WX −Q∆w ( W + V ) ·Q∆x ( X ) ||2 , ( 2 ) where V is a continuous variable added to W and the quantized network weights are defined as Wq = Q∆̂w ( W + V̂ ) . In this new objective the quantized tensor is not required to be ” close ” to the original tensor , as in eq . 1 , thus benefiting from the flexibility that Quantization-Aware-Training methods have . Yet , it can be executed in parallel over all layers and is much less prone to over-fitting . Moreover , under a fixed configuration we can optimize the model globally and infer the error between layers . Thus , instead of running AdaQuant on all layers in parallel we can run it sequentially and fix the error induced by quantaizing former layers . Thus , Eq . 2 changes to : ( ∆̂wl , ∆̂xl , V̂l ) = arg min ∆wl , ∆xl , Vl ||WlXl −Q∆wl ( Wl + Vl ) ·Q∆xl ( X q l ) || 2 , ( 3 ) Xq = σ ( Q∆wl−1 ( Wl−1 + Vl−1 ) ·Q∆xl ( X q l−1 ) ) ( 4 ) where σ ( · ) is some activation function . Note , that sequential AdaQuant should not be applied before the bit allocation was set as it optimize over noisy input obtain from predecessor quantized layers . We evaluate both flavors of adaquant ( named , AdQuant and sequential-AdaQuant and detail our finding in section 5.1 . We note that AdaQuant also optimizes over biases and offsets and optimized fused conv-bn-relu layers when present ; these were removed from the formulation in Equation 2 for simplicity . Size of calibration set Perhaps surprisingly , although we experiment with a very small calibration set , no over-fitting is observed . Let us examine a simple fully connected layer W ∈ RM×N . The input and output are of sizes N and M , respectively . For each output we have B equations and N separate parameters ( i.e. , with no overlap in parameters between different outputs ) . Therefore if B N we generically have an infinite amount of solutions and we can overfit the data . If B N then we might underfit the data . Thus , the size of the calibration set required for AdaQuant should roughly be O ( N ) . A similar derivation for convolution layers reveals that the calibration size should have B ≥ Ci·k 2 HW samples to avoid over-fitting , where B is the number of unique samples , k is the convolution ’ s kernel size , Ci and Co is the number of input and output channels respectively and H , W represent height and width . In fig . 1 we compare AdaQuant to current state-ofthe-art methods including QAT with knowledge distillation ( QAT-KLD ) ( Kim et al. , 2019 ) and AdaRound ( Nagel et al. , 2020 ) . For each method , we measured the top-1 accuracy with respect to the number of samples in the calibration set over five runs and present the mean and standard deviation . As can be seen , AdaQuant is superior to previous methods and specifically excels on small calibration sets . Remarkably , AdaQuant does not overfit even when optimized on a single image . Additional details can be found in section A and D of the Appendix .

Summary: The paper studies the problem of Post-Training Quantization of NNs, where no fine-tuning is performed to quantize the model. In particular, the authors focus on sub-8 bit quantization and propose a novel integer linear programming formulation to find the optimal bit width for a given model size. Additional approaches are proposed to minimize accuracy degradation after quantization. These include