Improving Post Training Neural Quantization: Layer-wise Calibration and Integer Programming
Authors: Itay Hubara
Yury Nahshan
Yair Hanani
Ron Banner
Daniel Soudry
Venue:  (2021)
Keywords: Efficient Deep Learning
Quantization
Compression

Abstract: Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only a small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only use the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except on small datasets). Here we aim to break the 8-bit barrier. To this end, we minimize the quantization errors of each layer separately by optimizing its parameters over the calibration set.  We empirically demonstrate that this approach is: (1) much less susceptible to over-fitting than the standard fine-tuning approaches, and can be used even on a very small calibration set; and (2) more powerful than previous methods, which only set the activations' dynamic ranges. Furthermore, we demonstrate how to optimally allocate the bit-widths for each layer, while constraining accuracy degradation or model compression by proposing a novel integer programming formulation. Finally, we suggest model global statistics tuning, to correct biases introduced during quantization. Together, these methods yield state-of-the-art results for both vision and text models. For instance, on ResNet50, we obtain less than 1\% accuracy degradation --- with 4-bit weights and activations in all layers, but the smallest two. Our code is available at, https://github.com/papers-submission/CalibTIP

Under review as a conference paper at ICLR 2021
IMPROVING POST TRAINING NEURAL QUANTIZATION: LAYER-WISE CALIBRATION AND INTEGER PROGRAM- MING
Anonymous authors Paper under double-bling blind review
ABSTRACT
Lately, post-training quantization methods have gained considerable attention, as they are simple to use, and require only small unlabeled calibration set. This small dataset cannot be used to fine-tune the model without significant over-fitting. Instead, these methods only the calibration set to set the activations' dynamic ranges. However, such methods always resulted in significant accuracy degradation, when used below 8-bits (except small datasets). Here aim to break the 8-hit barrier. To this end, we minimize the quantization errors each layer separately by optimizing its parameters over the calibration that this approach is: (1) much less susceptible to over-fitting than the standard fine-tuning approaches, and can be used even on very small calibration set; and (2) more powerful than previous methods, which only set the activations' dynamic ranges. Furthermore, we demonstrate how to optimally allocate the bit-widths for each layer, while constraining accuracy degradation or model compression by proposing a novel integer programming formulation. Finally, we suggest model global statistics tuning, to correct biases introduced during quantization. Together, these methods yield state-of-the-ar results for both vision and text models. For instance, ResNet50, we obtain less than 1 1 accuracy degradation with 4-bit weights and activations in all layers, but the smallest two. Our code is publicly available https://gituub..o//ppppp htw.:/pppppessssiisisiiiiiioi wwwwissioo/Caiiiil/////
We empirically demonstrate
INTRODUCTION
The pursuit of advanced Deep Neural Networks (DNNs) causes researchers to construct deeper and wider networks, making them expensive use in terms of power and time. This increases the need these networks. Efficient networks reduce cloud-vendor costs and
for efficient implementations make it possible to them low-power devices such as smartphones and wearable devices. The most common off-the-shelf approach to improving network efficiency is quantization, which reduces the numerical precision of the network and its complexity and memory footprint.
DNN quantization techniques can be classified as either post-training or quantization-ave training (QAT) techniques (Han et 2015 Courbariaux al. 2015 Hubara et al.) 2017 Zhou et 2016). Although QAT techniques, in general, achieve better results, there are important real real-world scenarios in which they are not applicable. These are the cases where the training data is sensitive simply unavailable at the time of deployment. For instance, when off-the-shelf or legacy models are being used, or when medical records are involved. Therefore, much attention has recently been dedicated to post-training quantization methods (Nagel et al. 2019 Banner et al. 2018 Zhao et al. 2019), which can more easily applied in practice. These methods allow for network quantization to happen seamlessly when deployed, without requiring additional information from the user except small unlabeled calibration set.
Unfortunately, post-training quantization below 8-bit always incurs significant accuracy degradation and in some cases even higher numerical precision required. In this paper, our goal is to break this barrier by distilling all the information the pre- pre-trained model and calibration set encode. Our goal is to find an optimal scheme for current state of the art hardware which usually support 16,8,4 bits data types with per-channel quantization of the weights To that end, suggest three-stage

---

Under review as a conference paper at ICLR 2021
pipeline that consists of methods applied solely on small calibration set reduce the local error introduced during the quantization process (e.g., round-off errors) followed by integer programming to determine the bit- tidth of different layers that the overall accuracy degradation is minimized. Even without using mixed-precision, the suggested method is much less prone to over-fitting than current methods and yields best in class results for 8-bits Mobilenet-V2 and BERT-base trained on ImageNet and SQuADI.1 datasets, respectively. Our paper suggests several contributions for mixed-precision post-training quantization:
AdaQuant: A layer-by-layed optimization method that minimizes the error between the quantized layer output and the fpee-preiision layer output. This method can consume only small calibration dataset from training data without overfitting. In a comprehensive study, we show that AdaQuant defines new state-of-the-art for post-training quantization on several networks and tasks, including vision models (Resnet18 8, Resnet50, Mobilenet and language (BERT).
2. Integer programming:
As some parts of the network may allow lower precision compared to other layers, we suggest an integer-linear programming based approach for determining the precision level of different layers. This method aims at maximizing either the expected speedup or savings in power consumption without violating a predefined constraint on network accuracy degradation or compression.
3. Batch-norm tuning:
Following quantization we observe an inherent bias in the mean and the variance of batch norm statisties. We show that by employing the re-estimated statistics in batch normalization, much of the quantized network degradation can be recovered.
4. Light and Advanced pipelines: We analyze the advantages and disadvantages of each of the given methods and suggest two pipelines: (1) light pipeline that does not require backward pass, thus can be invoked even on inference-only hardware; and (2) Advanced pipeline that includes also AdaQuant and bias tuning.
RELATED WORK
There has been significant effort to accelerate inference via quantization (Courbariaux et al. 2015
Han et al. 2015 Rastegari et al., [016 Zhou et al.) 2017)) These works involve re- re-training in order to compensate for the degradation due to the quantization process. Post-training quantization, on the other hand is applied to model after it was trained. Thus, it avoids re-training and as such it is much simpler to use. However, naively quantizing a full-precision model to INT4 or lower to accelerate computation usually incurs significant accuracy degradation (Krishnamoorthi 2018 2018).
Jacob et al.)
AdaQuant: A recent post-training quantization method (Nagel et al. 2020), termed AdaRound, suggested optimizing the rounding policy. Instead of using the predominant rounding-to-Bn neaaresst approach, they suggest formulating per-layer quadratic optimization problem to optimize the round- off error. Our proposed method, AdaQuant, takes another step and relaxes AdaRound's 's implicit constraint which forces the quantized weights be within +1 of their round-to-nearest value. This is done by optimizing the weights and quantization parameters of each layer separately, over the calibration set, to minimize the MSE between the layer's s original and quantized outputs. As oppose to AdaRound we apply AdaQuant to find optimal quantization not only to weights but also to activations. In addtion we suggest two flavors for AdaQuant: (1) parallel-Adauam suited for mixed precision setting; (b) soquential-adaquuee
which suited for fixed configuration.
Integer programming: Early work by Lin al. (016) used convex optimization formulation which results in a simple greedy compression scheme. Affalo et al. (2020) used combinatorial optimization approach for network pruning. Their problem was formulated as Knapsack problem that optimizes the trade-off between the channels importance and their associated computational cost.
Cai et al. (2020) finds a mixed-prees preecison configuration with a guaranteed Pareto efficient allocation with respect to model size and accuracy degradation. While this provides "best-effort" standard (e.g., the configuration cannot be further compressed without hurting accuracy), it does not suggest which of all possible outcomes is best. To the best of our knowledge, this work is the first to formalize a generic integer program, which can easily be adapted to various types of models and requirements with clear objective and constraints.
2

---

Under review as a conference paper at ICLR 2021
Batch norm tuning: Finkelstein et al. (2019) were the first to recognize that significant souree of degradation is a shift in the mean activation value. They show a simple method to compensate for this bias by updating the bias terms. Nagel et al. (2019) suggest to equalize the weight ranges in the network and correct biases in the error that are introduced during quantization RRceetly [un et al. (2019) suggested batch norm tuning for FP8 models. Here detail how to perform this procedure on a per-channel quantized (PCQ) model with fused batch-norm layers. The procedure is light as it only requires to invoke the quantized model few times (on the calibration set) and adjust the quantization parameters. Moreover after retuning the BN layers can be reabsorbed which reduces the inference complexity. To the best of our knowledge we the first to suggest
OPTIMIZING THE QUANTIZATION PIPELINE
In most post- post-training quantization settings, model and small unlabeled calibration set given. To avoid overfitting the calibration set, most studies utilize it only to extract the network's 's internal statistics, which is later used set the quantization parameters.
Here we suggest using the calibration set much more extensively to tune the model while avoiding over-fitting the data. In the following subsections, detail three different optimization methods over the calibration set: (1) AdaQuant, a layerwise optimization weights and quantization parameters; (2) an integer programming formulation for a mixed-preeision setting; and (3) Batch Normalization Tuning (BNT), for tuning the model's internal statistics to match the numerical precision setting. We discuss the strengths and weaknesses of each method and suggest an optimization flow that exploits all the additive merits and leads to state-of-th -ott-----tt---tttt- results.
3.1 ADAQUANT LAYERWISE OPTIMIZATION OVER THE CALIBRATION SET
Several researchers suggested per-tensor optimization to reduce quantization error by minimizing some form of MSE objective between the quantized and the full-precision tensor X (either weights or activations). They look for optimized quantization step size Á obtained by
X A
arg min X Qx(x)1
Qx(X) A
where Q(-) is the quantization function. Although these methods are fast and easy to use, they often result in an inferior solution -i the loss in eq. i sub-optimal, as it penalizes all the quantization errors equally. However, the loss should penalize more quantization errors which affect the classification. Accordingly, researchers suggested Auantizattonn-Aware- Training (QAT) methods to fix this error by training the entire model at once. However, those methods have three limitations: the large training through discrete function (the quantizer) and (c) they have high computational and memory footprints. We suggest a modified objective for per-layer joint optimization of the weights and quantization parameters.
(a) they require avoid over-fitting, (b) they approximate the back pack-propagation gradients
(mw: A.s A.r V) arg min || - Qa (W + VV (X)II?
A.,VV
where V is a continuous variable added to W and the quantized network weights are defined as Wg Qs (W + V). In this new objective the quantized tensor not required to be "close" to the original tensor, in eq. [ thus benefiting from the flexibility that Quantization-Awant -ariraatiin-Awaa---rrTnnnnn methods have. Yet, it can be executed in parallel over all layers and is much less prone to over-fitting. Moreover, under a fixed configuration we can optimize the model globally and infer the error between layers. Thus, instead of running AdaQuant on all layers in parallel we can run it sequentially and fix the error induced by quantaizing former layers. Thus, Eq. changes to:
Ami Amyy vi) = arg min || - Qs (Wr + Vi) QD (X²)11
(3)
Auy AV
Xg = =0 (Wi-1 + Vi-1) Qs (XP_i)
(4)
where (.) is some activation function.
Note, that sequential AdaQuant should not be applied before the bit allocation was set as optimize over noisy input obtain from predecessor quantized layers. We evaluate both flavors of adaquant
3

---

Under review as a conference paper at ICLR 2021
(named, AdQuant and requential-AdaQueet and detail our finding in section 5.1. We note that AdaQuant also optimizes over biases and offsets and optimized fused conv-bn-relu layers when present; these were removed from the formulation in Equation 2 for simplicity.
Size of calibration set Perhaps surprisingly, although we experiment with very small cal- ibration set, no over-fitting observed. Let us examine simple fully connected layer W E RMXN_ The input and output are of sizes N and M, respectively. For each output have B equations and separate parameters (i.e., with overlap in parameters between different outputs). Therefore have infinite amount of solutions and we can overfit the data. B > N then we might un-
60
Accuracy
40
FP32
< N we generically
20
Naive min-max
AdaQuant
AdaRound
derfit the data. Thus, the size of the calibra- tion set required for AdaQuant should roughly be O(N). A similar derivation for convolution layers reveals that the calibration size should C.-k²
0
QAT-KLD
10º
Calibration (log scale)
HW samples avoid over-fitting, where B is the number of unique samples, the convolution's kernel size, C, and C, is the number of input and output channels respec- tively and H, W represent height and width. In fig. [[we we compare AdaQuant to current state-of- the-art methods including QAT with knowledge distillation (QAT-KLD) (Kim et al.) (019) and AdaRound (Nagel et al. 2020). For each method, we measured the top-1 accuracy with respect to the number of samples in the calibration set over five runs and present the mean and standard deviation. As can be seen, AdaQuant is superior to previous methods and specifically excels on small calibration sets. Remarkably, AdaQuant does not overfit when optimized single image. Additional details can found in section on the Appendix.
have B >
Comparison of different optimization methods over ResNet-50 quantized 4 bit except the first and the last layers which were kept in 8bit. Even optimizing on single image drastically improves the results but as expected have a high variance (red bar). The variance decreases rapidly as calibration set size increases.
Figure 1:
3.2 PER-LAYER BIT ALLOCATIONS WITH INTEGER PROGRAMMING
AdaQuant significantly enhances network accuracy lower bit widths. However, it is often not sufficient by itself to attain acceptable accuracy. Therefore, in practical use cases, the user would like balance between accuracy and performance (e.g., power and speed), by setting several layers to higher precision. Our high-level goal this section would be to optimize the overall network performance while maintaining predefined accuracy degradation
model compression constraint.
In the following, we provide an allocations. Depending the needs, our performance metrics P would be either the execution time of the network or its power consumption. Also, with every layer quantization, there is an associated quantization error that affects the training loss L. We chose the latter to our penalty metric. Integer programming is applied in those situations where a given problem can clearly be represented in the form of linear relationship between different decision variables. Unlike other previous works on compression, it attains a global optimum. For example, Lin et al. (2016) suggested convex optimization problem, but the constraints and the objective are not linear. This typically has drastic impact convergence time, and the quality of the results since the Simplex method can no longer be applied Van Doormaal Raithby 1984).
(IP) formulation for optimizing per-layer bit
Basic formulation We are given a neural network with L layers. For each layer I, we have weights W; that need to be multiplied with activations of the previous layer X,-1- Such lower bit width multiplications can be executed by quantizing the weights and activations to achieve higher throughput and energy-effien efficieen solutions. Let W and X represent a quantized version of Wi and X,-1 to and n bits, respectively. For each layer i, a low-bit width multiplication W X results loss degradation ACk.." and in performance improvement APÊ." with respect the original product
4

---

Under review as a conference paper at ICLR 2021
Wi- -X-11 This performance improvement measure needs to be additive and sum up total benefit in end-to-end network performance (c.g., power, model size, etc.). Our goal would be to maximize the total performance improvement without exceeding the total network degradation AC
We now turn solve the above problem using an integer program. We define a binary variable 1," which is set one if and only if the weights Wi are multiplied with the activations X layer I; otherwise we set the indicator to zero i.e., 1 fk.n = 0. Then, the basic bit allocation problem can be formulated as follows:
Maximize
API
(5a)
1=0
Subject to E
ACI S AC,
(5b)
E fl..... L) APF
Σ I, APT. ALC Σ k,n
If. Ach.
(5c)
k
E (1,... L)
Σ k,n
/ 1, F E {0,1}
(5d)
The objective function (3a) maximizes the total performance improvement. Constraints (3b) and (3c) ensure that the total degradation loss and the total improvements in performance due to the quantization of layer I to k-bit-weights and n-bit-activations would be A&₁ and AP,, respectively. Equation (3d) states that the restriction on total degradation of AC is obeyed and ensures that only one configuration (of quantized weights and activation) per layer is selected.
3.3 BATCH NORMALIZATION TUNING
A common practice is fusing BN layers into their predecessor weight layers before applying post- training quantization to reduce the amount of reduction in bit-width after quantization can cause the model's internal statistics to deviate further from those of the full precision model. To compensate for this deviation, we suggest updating BN statistics. First, we need to reconstruct the BN layers then re-tune the BN layers' statistics (by few iterations running-mean to re-collect the statistics). Finally, re-absorb (re-fuse) the BN layers into the weight layers (this is possible only in a per-channel weights quantization setting, which is the current standard). Next, we give more details on each phase.
(MAC) operations. However, the
Reconstructing BN layers Assume the original (pre-fusing) BN parameters Yo. 9. and are known, as is usually the case. We would like to initialize Ju, o², as well as the BN parameters 7, and 8, (r for "reconstructed") that the reconstructed BN
BN) -
x 11
3, R
will re-adjust the model statistics. To do so, first we initialize the reconstructed BN layers by setting the following parameters (denoted by r):
/ 3, = 3,:
2 ;7, = V12 +
so that (x) = x. Then, we update and o² by collecting running mean and running variance on the calibration data. We stress that the BN parameters, Tr, Br, do not change while applying BN tuning, as we only invoke forward propagation.
Re-fusing BN layers Due to the per-channel quantization setting we use, the collected statistics can be fused back into the current quantization scale as follows:
W = W; 2r
6 2r (b. - (4) + Brg
A
Yr
Aw:
(8)
Thus, in addition to the regular BN fusion, the quantization step is adjusted by y,a-! Additional details given in section B of the Appendix.
5

---

Under review as a conference paper at ICLR 2021
Bias tuning Much like Finkelstein et al. 2019), suggest apply a global bias-tuning procedure on the final mixed-prec -exxedpprecsion model by applying quantization aware training to minimize the Knowl- edge Distillation (KD) loss (which does not require labels). Since restrict the trainable variables to be the biases only, we can train only on the calibration set without experiencing overfitting.
4 QUANTIZATION FLOW
Past years have seen the rapid development of efficient deployment techniques (Nagel et al. 2019 Haroush et al.] 2019). Deployment flows can vary based the user setting such hardware constraints, deployment time and task/dataset availability. While some users are willing to pay at initialization the time and effort to gain another fraction of accuracy, others require simple and fast solution. We address this by suggesting two novel pipelines, light and advanced. Our pipelines are designed to the current, most common setting: per-channel quantization with a small calibration set.
Our light pipeline requires three steps: (1) Fuse layers and define quantization parameters; (2) Find optimal mixeddprecissio configuration using IP; and (3) Use BN tuning to correct the internal statistics. We note that all steps do not require prooppropggttion and thus are very light and fast. In addition to the light setting, in the advanced pipeline we apply AdaQuant to reduce each layer's output distortion from its full precision counterpart before invoking the IP algorithm. A detail comparison between the two pipeline is given in table-1. Models that were optimized using AdaQuant to different bit-widths can be seamlessly stitched thus having the ability create optimized model in mixed precision setting. Subsequently, global methods such as tuning both BN statistics and the layers' biases can be applied to reduce Knowledge Distillation loss. Although there additional post-training quantization techniques that could be potentially combined with our methods, such as bias correction (Banner et al., 20188) equalization (Meller al. (2019), and outlier channel splitting (Zhao al.) 2019), we did not find it necessary: our results demonstrate that our relatively simple pipeline yields state of the art accuracy on both vision and text models, even without combining such methods. In the following sections we show our findings and give an ablation study that highlights the importance of each method and their combination.
AdaQuant
Mixed- (IP) BN tuning
Bias Tuning
Sight-Pipeling Piceey-Pipelinne
x
Table 1: Comparison between light and advanced pipelines.
5 EXPERIMENTS
In this section, we demonstrate our methods and pipelines on several models and datasets. We first start by analyzing image recognition models such as ResNet 8/500 MobileNet-V2, which were trained over the ImageNet dataset. Next, we demonstrate our method robustness by applying it on question answering task using the popular BERT model (Devlin et al., 2018), which was fine-tuned on SQuAD1.1 11 dataset (Rajpurkar et al. 2016). In all our experiments, set taken from the training dataset. Unless stated otherwise, we applied asymmetric per-channel quantization (i.e. GEMLOWP Wu et al. (0166) with quantized offset (i.e., zero point). Next, analyze each method's s strengths and weaknesses separately and argue for its validity. Additional implementation details can be found in section and the code are given in sections D of the Appendix.
used a small calibration
5.1 ADAQUANT
Recently several researchers suggested different types of MSE optimization. In most cases, the optimization was done per-tensor (i.e., for the weights and activations separately). Here we argue that by optimizing both quantization parameters and the weights jointly we can reduce the MSE even further and hence improve the accuracy as demonstrated in fig. 2b In contrast to AdaRound (Nagel et al., 2020) which restricted the change of the weights to be within #1 we allow the weights to change as needed. As can be seen in fig. 2a the weights indeed change their quantized value by more than one. Since our pipeline is focused on the mixed-precision setting we optimize each layer

---

Under review as a conference paper at ICLR 2021
separately to enable maximum flexibility when stitching the optimized models. Under that setting AdaQuant can be performed in parallel across all layers. However, since most recent papers do not curves and only few attempt 4-bit compression, flavor. While sequential
show full compression-acente our results to common fixed configurations using our rrquential-AdaQuuaaa AdaQuant cannot be parallelized or used for the mixed-precision setting yields best-in-class results for all models tested as can be seen in table-2 and 3. For instance, on the extensively studied 8bit MobileNet-V2 (MobileNet-V2) topology we achieved 71.6% top-1 accuracy less than 0.5%
also compare
degradation compared to the full precision counterparts (71.9%).
RN-18
RN-34
RN-50
RN-101
RNext-50
Inc-
ACIQ* (Banner et al. 20188)
64.5%
69.1
68.1%
68.1
N/A
60.4
DFQ (Nagel et al. 2019)
57.1%
N/A
64.5%
N/A
N/A
N/A
AdaQuant
67.4%
70.3%
73.7%
74.4%
74.0
72.6%
Sequential-AaaQuan
69.4%
71.7%
75.1%
75,5%
75.6%
73.4%
FP32
71.97%
73.3%
77.2%
77.3%
79.22%
77.4%
Table 2: INT-4 quantization of weights and activations. Top-1 score over imagenet dataset for different post-training quantization methods. All layers were quantized to 4-bit except first and last layers which were set to 8-bit. Methods marked with (*) were implemented according to the paper. In all our experiments we apply per-channel quantization of the weights.
MobileNet-V2 (top-1)
BERT Pase-SQuadLL.
(FI)
min-max
70.9%
87.83%
DFQ (Nagel et al.
2019)
71.2%
N/A
ZeroQ (Ca et al. 2020)
72.91%
N/A
AdaQuant
73.03%
88.35%
Sequential-AdaQuaant
72.94%
88.45%
FP32
73.03%
88.81% %
Table 3: INT-8 quantization of weights and activations. A comparison with DFQ and naive quan- tization methods (which uses the channel's full dynamic range). In all our experiments we apply per-channel quantization of the weights and quantized all layers to 8-bit.
Testing the strength of this method on both vision and text topologies resulted in state-of-the-arr results. As can be seen in table 3 on BERT-base model over SQuADI. dataset (BERT-Base- Base-SQuADL.J managed to obtain 88.45% F1 score using just AdaQuant less than 0.5% of its full precision counterpart (81.3%). Throughout our experiments, we avoided using any augmentation technique and follow the known validation set prepossessing.
5.2 INTEGER PROGRAMMING
Our Integer programming formulation requires us to have two quantities per-layer: (1) loss degrada- tion and; (2) performance improvement. Obtaining those quantities requires invoke the model over a small calibration set L times (once per layer) and measure the loss degradation and the performance gain. In our experiments, we set the performance value be the number of parameters, but this measure could be changed to any additive measure. In all experiments, we used 1000 samples from the training set as our calibration set. Our setting considers only a mixture of 8-bit and 4-bit layers; to further test IP capabilities, we investigate a mixture of 2-4-8 bits as well. Unfortunately, since 2-bits quantization in post-training setting results in high degradation, the IP algorithm chose only mixture of 4-8 bits for compression ratio higher than 12.5%. Yet for 12.5% compression ratio, IP method found that by setting one layer to 2-bits while setting 8 smaller layers to 8-bits accuracy gains over 5.5% with respect to uniform 4-bit quantization. Also, by allowing less hardware friendly setting where numerical precision can have the form of any integer between 2-8, yields the highest compression-accuracy ratio (fig. relaxed advanced pipeline).
7

---

Under review as a conference paper at ICLR 2021
102
80.0
77.5
10
75.0
10
70.0
10a
67.5
10
62.5
10a
-4
-2
60.0
AW quantized
bias weight+
qparams
Figure 2: AdaQuant vs. AdaRound. (a) A histogram of AW distribution. AdaRound restricts this additive term be AW =1. Relaxing this constraint provides more powerful optimization. (b) Ablation study on parameters optimization for ResNet50 over ImageNet. AdaRound is based exclu- sively on weight optimization, while AdaQuant optimizes the weights, biases, and other quantization parameters jointly.
5.3 BATCH-NORM TUNING
Batch-Norm Tuning (BNT) has a significant advantage, as does not require weight optimization. Since BNT is applied by invoking the entire model, we must apply it only after setting the mixed- precision bit-width configuration. This is the case for all global optimization methods including bias-tuning. Notably, BNT requires only few (at most 10) forward passes set and yield significant gains (fig. B In this study, we applied BNT on models trained with BN layers only. However, it might be possible extend this method models without BN layers by reconstructing it from the statistics. We encourage the reader to investigate this path.
the calibration
5.4 FULLL PIPELINE AND ABLATION STUDY
Although several researchers suggested different methods for post-training preec-speeisoon quantiza- tion, none offer their code. Each paper focuses on a different quantization setting (e.g., quantizing only the weights, per-tensor quantization, etc.). Therefore, to demonstrate pipeline strength, created two different baselines based on common practices:
Greedy accuray recent studies suggested measuring each layer sensitivity and, based on the compression target, reduce the precision for the most robust layers.
Grreddy-copprssionn the complementary greedy approach (Lin et al. (016) to sort the layers by their number of parameters and increase the precision of the layers from the smallest to the largest layer until the compression budget is reached.
Surprisingly, although the size of the layer should correlate with its sensitivity to quantization, the two greedy methods yield entirely different configurations. Investigating the configuration greedy- compression found that sorting by compression correlates with the location of the layers in the model. In most vision models, the layers closer the input have fewer parameters. This aligns with current common practice (Banner et al. 2018). Notably, even when not combined with any other technique, the IP method obtained the best bit-width configurations stressing importance.
Next, we turn to consider the light and advanced pipelines. Under challenging compression rates, our light-pipeline results highlight the importance BN tuning. As can be seen our experiment fig. 3 by merely invoking the model at inference mode for few iterations and fixing the intermediate statistics, one can recover more than 1.5% of the accuracy (73.7% v.s 75.37%). As expected, obtain state-of-the-ze accuracy. Arguably, our most
by applying the advanced pipeline, impressive results are at 0.13% compression rate in which we managed to stay within 1% of the full precision accuracy while converting 96% of the model to 4-tit. For the challenging MobileNet-V2
8

---

Under review as a conference paper at ICLR 2021
managed to switch 25% of the layers 4bit (weights and activations) while maintaining less than 2% degradation; Additionally, we achieved, for the first time, reasonable top-1 accuracy of 65% when almost the entire model is in 4-bit.
Accuracy
Accuracy
68
67
advanced pipeline
advanced pipeline light pipeline
light pipeline
66
1p
greedy (compression)
greedy (compression) greedy (accuracy)
greedy (accuracy)
65
0.16
0.22
0.14
0.16
0.20
0.24
compression ratio
compression ratio
(a) Mo
(b) ResNet-18
76
75
Accuracy
74
73
advanced pipeline
light pipeline
ip
72
greedy (compression)
greedy (accuracy) relaxed advanced pipeline
71
0.12
0.14
0.16
0.18 0.20
0.22
0.24
compression ratio
(c) ResNet-50
Figure 3: Ablation study over ResNet-50018 and MobileNet-V2 compression-accurac curves. Our advanced pipeline is consist of AdaQuant, IP-mixed- precisson,,sionn BN-tuming and bias-tuning. Our light pipeline is consist of only IP-mi mixedd-reccision. BN-tuning. The relaxed advanced pipeline appears in O is similar to the advance pipeline but allows the pnteger-programming to choose any bit-width between 2-8 and just 4-bit 8-bit. The compression ratio is measured the ratio between the compressed model and the full-po full-precision (32-bit) mode thus 0.25 compression rate indicate that the entire model uses 8-bit precision and respectively for 4-bit the compression rate is 0.5
REFERENCES
Yonathan Aflalo, Asaf Noy, Ming Lin, Itamar Friedman, and Lihi Zelnik, Knapsack pruning with inner distillation. arXiv preprint arXiv:2002.08258 2020.
Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Aciq: Analytical clipping for integer quantization of neural networks. 2018.

---

Under review as a conference paper at ICLR 2021
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. arXiv preprint arXiv:2001.0028/1 2020.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural networks for efficient inference. In 2019 IEEE/CVE International Conference on Computer Vision Workshop (ICCVW), pp. 3009-3018. IEEE, 2019.
Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Binaryconnect: Training deep neural networks with binary weights during propagations. In Advances in neural information processing systems. pp. 2223-111, 2015.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1rXiv:1810.04805, 2018.
Alexander Finkelstein, Uri Almog, and Mark Grobman. Fighting quantization bias with bias. arXiv preprint ww.iv:1996.03/11 2019.
Song Han, Huizi Mao, and William Dally, Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv preprint 1111v:1510.00149, 2015.
Matan Haroush, Itay Hubara, Elad Hoffer, and Daniel Soudry. The knowledge within: Methods for data-free model compression. arXiv preprint arXiv: 1oXivv/912.000 1912.01244 2019.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran HI- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1):68696888, 2017.
Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient anteger-arithmetic-onll inference. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2704-2713, 2018.
Jangho Kim, Yash Bhalgat, Jinwon Lee, Chirag Patel, and Nojun Kwak. Qkd: Quantization-aware knowledge distillation. arXiv preprint arXiv: 1911.1291, 2019.
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 10X.v:1806.08342. 2018
Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy, Fixed point quantization of deep convolu- tional networks. In International conference on machine learning. pp. 2849-2858, 2016.
Eldad Meller, Alexander Finkelstein, Uri Almog, and Mark Grobman. Same, same but different- recovering neural network quantization error through weight factorization. arXiv preprint arXiv:1902.01917, 2019.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE International Conference Computer Vision, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. arXiv preprint arXiv:2004. 105668 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint 1rXiv:1606.0522 2016.
Mohammad Rastegari, Vicente Ordonez, Joseph Redmon, and Ali Farhadi. Xnor-net: Imagenet classification using binary convolutional neural networks. In European conference on computer vision, pp. 525-542. Springer, 2016.
Xiao Sun, Jungwook Choi, Chia- Yu Chen. Naigang Wang, Swagath Venkataramani, Vijayalak- shmi Viji Srinivasan, Xiaodong Cui, Wei Zhang, and Kailash Gopalakrishnan. Hybrid 8- floating point (hfp8) training and inference for deep neural networks. In Advances in Neural Information Processing Systems, pp. 4901-4910, 2019.
10

---

Under review as a conference paper at ICLR 2021
Jeffrey P Van Doormaal and George D Raithby. Enhancements of the simple method for predicting incompressible fluid flows. Numerical heat transfer, 7(2):147-163, 1984.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation sys- tem: Bridging the gap between human and machine translation. arXiv preprint arXiv: 1609.08144, 2016.
Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International Conference on Machine Learning, pp. 7543-7552, 2019.
Aojun Zhou, Anbang Yao, Yiwen Guo, Lin Xu, and Yurong Chen. Incremental network quantization: Towards lossless cnns with low-precision weights. arXiv preprint
2017.
Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Train- ing low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint arXiv: 1606 06000 2016.
Appendix
A SIZE OF CALIBRATION SET
Fully Connected layers Let's assume that we have weights of size W RM and input and output are of sizes N and M respectively. Recalling Eq. 2 and setting Y WX and W = W + V results in:
(Awr v) = arg min || - Q (W") (x)]1.
A..A..V
For simplicity we assume that A₂ is fixed and define X, = Qa. Ws = Qs (W'). Therefore, we have B unique samples, then the problem we aim to solve have the following structure:
W11
WIN
I11
T1B
911
/1B
WM1
EMN
IN1
ENB
YMi
UMB
which translates to:
W11
WINUNI
U11
W11811
WINNN2
912
WMIEIB
WMNNNA
JMB
Notice that in the above equations, for each output we have a different set of parameters, therefore can examine each output separately. For single output in scalar linear regression with N parameters and B equations. If B 2 N we are under-parameterized, and if B < N we are over-parameterizee
Convolution layers layers Similarly, for convolution layers with C, output channels, C, input channels, and kernel size k each element of the output
dot product of C, k A parameters. We have in total Co x H x W outputs where H, W is the output height and width. Thus we need C.-A² HW samples avoid over-fitting where B is the number of unique samples.
B
B RECONSTRUCTION AND RE-FUSING OF BATCH NORMALIZATION
In this section, we provide more details on Batch Normalization reconstruction and re-fusing proce- dure.
11

---

Under review as a conference paper at ICLR 2021
Reconstructing BN layers: Consider Batch Normalization layer with parameters Yo. So that fused into previous convolutional layer weight and bias. Fusing batch normalization layer transforms weights and bias as following:
W W; 7o
83 To (b. 11) + So:
o
To reconstruct the batch normalization, would like to initialize μt, o² well as the BN parameters Ty and 3, (r for "reconstructed") so that the reconstructed BN is approximately identity fig. 4
x 1
(N) = " V02 +
+ 3r x
(10)
Yr
B r
Conv2d
BN,
Yo W
Yo (bi 11) + Bo
σ
σ
Figure 4
To do so, first we initialize the reconstructed BN layers by setting the following parameters (denoted by r):
1 = 3r Bo;
0² vo
7
+
(11)
so that BN, =
Now, we can update μ and 0² by collecting running mean and running variance on the calibration data. We stress that the BN parameters, 7r, B,. do not change while applying BN tuning, as we only invoke forward propagation.
Re-fusing BN layers: After BNT phase need to fuse Batch Normalization layer again into convolution weights and bias. Regular batch normalization fusing will cause degradation due to quantization of the weights. To resolve this issue we can leverage per-channel quantization setting
we use.
Denote 8 Zun; scale and zero point of the weigh, the quant/dequant operation defined as:
W
Zunz
Zura
W
Swi
+
(12)
Sws
Swe
8ws
We fuse parameters of the hatch normalization layer following:
W = W: Yr.
by = 7r (b: - Hz) + 3,
σr
(13)
2/1 Yr zure
Suni
12

---

Under review as a conference paper at ICLR 2021
Finally can show that transformations eq. (13
equivalent Wa
dr
W
W
s'wz
Sang
(14)
Ir
W
Zwy
Zuna
Swi
+
8ms;
8wz
Bong
Ir Wg
C ADDITIVE LOSS ASSUMPTION FOR PNTEGER-PROGRAMMIN
Suppose the loss function of the network C depends on a certain set of variables (weights, activations, etc.), which denote by a vector v. We would like to measure the effect of adding quantization noise to this set vectors.
Since the quantization is emulated with additive noise, the loss is smooth and thus can be expanded to the Taylor series:
AC = L(v + +) - -((v)
(15)
OCT
02C 0 gº e + O (IE]]*)
Ov +
(16)
One can see from Eq 16 that when the quantization error is sufficiently small, the overall degradation AC can be approximated as sum of N independent degradation processes by neglecting the quadratic terms e?:
OCT dv
n
ac
AL =
E Σ
Ov
Es
(17)
We note that Lin et al. (016); Choukroun et al. (2019) used similar assumption with respect to the additivity of quantization noise.
D EXPERIMENTAL DETAILS
In all experiments, used small subset of the training to methods. Specifically, for vision models, we used 1000 unlabeled images from the ImageNet training set (single image for each class) calibration set. For the Bert model, All presented methods AdaQuant, BNT, BT, and IP, performed well on such small calibration set producing SOTA results. Next we detail our setting for each of the technique in our pipelines
used one paragraph from the training set.
D. ADAQUANT
AdaQuant optimization problem defined as following except zero-point of the quantizer which we omitted from eq. (18):
(Aw, (Aw, Day Vw, Vo))
| + 6 - Qa. (W + A.. A.. Vo
arg min
Vir - Q(b Q(6 Vo) (18)
Technically find solution for eq. 188
Adam optimizer with different learning rates per type of parameters. We set different learning rates for weight, bias, and quantization parameters of input and weights. After experimenting with different models, parameters worked for each model. The learning rates are le - 5, le - 3, le le 3 for weight, bias, quantization parameters of the inputs, and weights, respectively.
found that the set of LR
For vision models, we used 1000 unlabeled images from the ImageNet training set (single image for each class), running Adam optimizer for 100 iterations and batch-size of 50 unless otherwise stated.
13

---

Under review as a conference paper at ICLR 2021
80
60
Accuracy
40
FP32
Naive min-max
AdaQuant
20
AdaRound
Early stop KLD
0
Simple KLD
10
10
102
10
Calibration set size (log scale)
Figure 5: Calibration size ablation study with additional early-stop plot.
For BERT-base model, we used one paragraph from the training set, running Adam optimizer for 50 100 iterations depending on the type of layer. Learning rates and batch size are the same as vision models.
In fig. [ we aimed answer the following question: Assuming you have a small calibration set and no resources constraints (time,power) which method is the most accurate and robust Out method were evaluated by running each experiments five times and reporting mean and standrad deviation. Here, in fig. [5] we add an additional naive early-stop plot on top of QAT-KLD experiment. We split the calibration data into two equal sets and train on half the examples while evaluation our performance the other half Both KLD experiments used an SGD optimizer over 10 epochs; starting with learning rate of 0.1 and decreasing it by le-2 factor after 2 and 8 epochs. We also conducted KLD experiments with Adam optimizer and learning rate of 1e-3 where performed but their results were inferior. As can be seen in the plot AdaQuant is superior to other methods and remarkably excels small calibration QAT-KI.D as they use much smaller training set. However, other types training-validation splits (c.g. 80-20) may boost the results.
As be seen in
fig. 5 the early exit results were inferior to the
D.2 INTEGER PROGRAMMING
Our IP method requires two steps, the first is measuring the properties each layer, and the second is applying the program based on these measurements with user defined constraint. As reference, we measure the loss (can also be accuracy) of the base precision model on the calibration set. Next, measure the sensitivity of each layer by evaluating model where all layers qunatize the base-precision but one layer that is quantized to lower precision (e.g., all 8-bit but one layer with 4-bit). The A in Eq. 3 is defined as the difference between the reference model loss and the measured loss. If a layer is robust to quantization, AC, will be small, and if layer is sensitive to quantization, A&; will be large. The performance gain in the case of compression, simply the model parameters size difference when lowering the precision of the examined layer. Hence, if layer has N parameters, the performance gain when changing from 8-bit to 4-bit result compression gain of AP; N 8 N * 4N.. In the second stage, we run the integer program based on the sensitivity and compression measured on each layer along with the user defined constraint.
D.3 BATCH NORMALIZATION AND BIAS TUNING
The Batch Norm tuning phase the most lightweight phase of the pipeline. We found empirically less than ten iterations of statistics update sufficient. We also found that compression growth,
14

---

Under review as a conference paper at ICLR 2021
more iterations of batch norm tuning are required. At the bias tuning phase, perform 200 iterations of fine-tuning with the learning-rate of 0.1.
E CODE
For all our vision dataset we used the default torchvision pre-trained model. For BERT-base experi- ment fined-tuned on SQUADI.1 dataset and provide the script for that part of our repository. Our code can be found at:
15


Reviews:
[{"review_id": "gKjagp4qWvK", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\nThis paper proposed a set of methods for post-training quantization of dnns. The methods include AdaQuant (which jointly optimizes quantization steps for weight and activation per output activation of each layer), Integer Programming (which determines bit-precision for all the layers), and the batchnorm tuning. The authors presented promising experimental results on various neural networks to support the proposed methods. \n\nHowever, there are serious concerns about these claims as follows:\n1) AdaQuant\n- It is straightforward to think that the joint optimization of quantization step size for weight and activation would result in better quantization results. But this joint optimization would also increase the search space (at least) quadratically, resulting in significant computational cost. Note that the biggest merit of post-training quantization is its simplicity (cf., QAT incurs full-blown training epochs); thus increased cost for post-training quantization is not desirable. Since AdaQuant is the major claim, the authors should provide more discussion on how they dealt with this increased complexity.\n\n- The authors claim that AdaQuant avoids overfitting, but the reason does not seem to be clear. There is no clear explanation of how AdaQuant increases the generality of the quantized model, and the discussion about the sample size (B) is hard to understand (why there's infinite solution when B << N? how B>= Ck^2/(HW) is derived for the convolution case?)\n\n- Also, it seems that the \"per-channel\" quantization method is utilized in this work, but the formulation in (2) seems to be for \"per-layer\" optimization. Why they are different?\n\n- How much time does it take to solve this joint optimization?\n\n\n2) Integer Programming\n- The authors proposed an Integer Programming formulation, but there seem to be missing information: what is the formulation of the penalty function, \"deltaL\"? The authors described it simply as \"Loss\", but it is not clear what the exact method it is calculated. In fact, deltaL can be pretty complex functions, which might not be independent terms for each layer; thus the formulation like (3) might not be correct. Note that the impact of quantization in the earlier layers affect the quantization impact in the current layer. Without clear explanation and justification about it, the proposed IP formulation does not make sense.\n\n- The authors mentioned that deltaP should be additive and sum up to the total benefit. How can one guarantee it? \n\n- Also, it seems that the complexity of the IP optimization increases as the number of layers increases. How much computation time increases if the number of layers are large? \n\n\n3) Batch normalization tuning\n- Unfortunately, there is a very similar idea proposed by [Sun et al., NeurIPS 19]. Cf. \"Sec.3 Trans-Precision Inference in FP8\".\n\n\nAlso, there are several suggestions to improve understanding of readers.\n- Currently, the ablation study looks very confusing. It is not clear which of the pipeline options (light, advanced?) include what kinds of techniques. Please do specify (maybe in a separate table) the list of techniques covered by different pipeline options. \n\n- The proposed method is not much evaluated by various neural networks. It would be desirable to expand the coverage of neural nets as much as the prior work did. \n\n- Currently, the proposed methods only utilized \"per-channel\" quantization. How much accuracy the proposed methods can maintain if they adopt \"per-layer\" quantization?\n\n- What is the definition of \"compression ratio\"? (typically compration RATIO is like 12:1, and compression rate is like 2X, 3X...)\n\n- \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "VH858vlzD7x", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "\n\nSummary: The paper studies the problem of Post-Training Quantization of NNs, where no fine-tuning is performed to quantize the model. In particular, the authors focus on sub-8 bit quantization and propose a novel integer linear programming formulation to find the optimal bit width for a given model size. Additional approaches are proposed to minimize accuracy degradation after quantization. These include\n(i) AdaQuant in which the parameters are quantized layer-by-layer to match the full precision output,\n(ii) a batch norm tuning approach to re-adjust the statistics to the quantized model, and\n(iii) an advanced pipeline for cases where backpropogation can be performed.\n\nExperiments are performed on ResNet18/50, MobileNetV2, and BERT-SQuaD1.1 showing the feasibility of the proposed method. \n\nI think the approach in the paper is pretty interesting, and specially the integer linear programming solution.\nOverall the paper is strong however, please note the following:\n\n\n- Page 3 last paragraph: it seems there are errors in the results for calibration data.\n1/ it is stated that if \"B>> M then we might underfit the data.\". Do the authors mean \"B>>N\"? Similarly it seems the result given for the convolution (i.e. B> cK^2/HW) needs to be revised.\n2/ The analysis requires specifying the rank of W and in particular the relationship between M and N. In particular, note that the matrix W can at most have a rank of min(M,N) and as a result the rest of the linear equations for calibrating the data would be redundant. This needs to be taken into account in your result.\n\n- Page 4: \"Depending on the needs, our performance metrics P would be either the execution time of the network or its power consumption.\" This is good but no result on either latency or power consumption is provided in the paper.\n\n- Figure 4 (a,b): Please provide the BOPS for the mixed-precision results. BOPS proposed by (https://arxiv.org/pdf/2005.07093.pdf) is a good metric to measure the total reduction in computations for mixed precision quantization.\n\n- The results section uses weak FP32 baseline. For instance, the baseline accuracy for ResNet50 is 77.2\\% and MobileNetV2 is 73\\%, while the paper uses 76.1\\% and 71.8\\%.\n\n- Related to the above, it is stated \"on the extensively studies 8bit MobileNet-V2 topology we achieved 71.6% top-1 accuracy\". This is a good result but please note that other work in the literature (arxiv:2001.00281) reports 72.91\\% for INT8 quantization of MobileNetV2 (this comparison is actually missing from the paper). It is immediately not clear if the lower reported accuracy is due to the weaker FP32 baseline used or if it is an inherent problem with the method (most probably it is the former but it would be to show this).\n\n\n\n\nMinor Comments:\n\n- There were several grammatical/spelling mistakes in the paper. Please proofread the paper thoroughly. Below are some of the errors that I caught:\n\n- 3 OPTIMIZING QUANTIZATION PIPLINE -> PIPELINE\n\n- Page 6:  the our method robustness -> the robustness of our method\n\n- Page 6: manged -> managed\n\n- Page 7: an significant advantage -> a significant advantage\n\n- Page 8: we managed to switched  -> we managed to switch\n\n- Page 8: For instance, on the extensively studies -> For instance, on the extensively studied", "extra_scores": {}, "extra_text": {}}, {"review_id": "LGgSIt4lRt0", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose to use many techniques to push the limit of neural quantization, which shows reasonable improvements in some datasets.\n\n+ good performance.\n+ clear presentation.\n+ easy to read and follow\n\nMy main complaint is that this type of combination is better for a technical report rather than a standalone paper. In more detail, the authors claim four proposed components: AdaQuant, Integer programming, Batch-norm tuning, and two pipelines for neural quantization. However, the problem is, are these four components all original or just adapted from existing works? The authors are failed to present the relation of the proposed components with the exitinging ones. For example, in my understanding, the AdaQuant is only an adapted version of AdaRound, but the discussion about it (in Section 3.1) is too superficial. Furthermore, the experiment part is not convincing and I think can not back up the claim. With so many \"proposed components\", the missing of thorough ablation study is a big problem.", "extra_scores": {}, "extra_text": {}}, {"review_id": "3PQxXLv3lW6", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces a series of techniques to quantize neural networks, and how to combine them:\n* Layer by layer quantization where weights can change as needed (rather than to the nearest quantization error).\n* Integer programming to determine the precision required at every layer.\n* Tuning batch norm weights by re-computing statistics.\n\nPROS\n\n* Main text is well written (see below for other issues though).\n* All components in the proposed method are straightforward.\n* Strong results, with very little performance loss after very aggressive quantization (~4 bits).\n\nCONS\n\n* The organization of the paper can and should be easily improved (see below).\n* Very basic details, such as the metric used in some tables, are omitted.\n* Few experiments/baselines.\n\n---\n\nOn the flow of the paper: Fig. 1 is barely introduced. Fig. 2 is introduced much earlier than it is first referred to in the text, in section 5.2, where it is not explained either. The labels in the figure are not defined either at this point (and ip should be IP). Additionally, the colors for the last two lines are too similar.\n\nThe experiment of Fig.1  should be introduced with some level of detail (there is none outside the caption). The caption is a bit confusing: what are the \"the last layers after\"? Why are variance bars always red? They are also hard to see. The labels are not descriptive (light pipeline, advanced pipeline, relaxed advanced pipeline) and hard to find in the text. MN-V2 and B-SQuad1.1 just drop in Table 1 without any context. It can be inferred from earlier sections what this refers to but it should be explained clearly. Same for \"min-max\" in Fig. 1 and Table 1.\n\nBaselines are barely discussed. Also, I am not very familiar with quantization papers, so I might have missed relevant baselines, but they seem hard to compare. For instance, the authors omit [Nagel et al 2020](https://arxiv.org/pdf/2004.10568.pdf), which seems to do better at similar quantization levels, but I am not sure the results are directly comparable. The authors should discuss this better.\n\nI think section 4 can be titled simply \"Quantization flow\".\n\nCompression ratio in the plots should indicate %?\n\n---\n\nTypos/grammar:\n\n* \"Mobilsnet-V2\" -> \"Mobilenet-V2\n* \"we suggest [an] integer-linear programming\"\n* \"AdaRound['s] implicit constraint\"\n* \"MSE distance\": shouldn't this just be MSE?\n* \"Early work by Lin et al. (2016) used [a] convex optimization formulation which results ~~with~~ [in] a simple greedy compression scheme.\"\n* \"3. OPTIMIZING [THE] QUANTIZATION PIP[E]LINE\"\n* \"model['s] internal statistic[s]\" -> found twice\n* \"often result with an inferior solution\" -> \"often result in an inferior solution\"\n* \"Accordingly, researches suggested\" -> researchers?\n* \"where V is a continuous variable V\" -> redundant\n* \"thus enjoy[ing] some of the flexibility\"? -> \"benefitting from\" might be a better phrasing?\n* \"Quantization-Aware- Training\" -> extra space\n* \"and [is] much less prone to over-fitting\"\n* \"[A] Similar derivation\"\n* \"Even optimizing on [a] single image\"\n* \"MAC operations.\" -> acronym not previously introduced (unless mistaken)\n* Section 4: IP acronym should be introduced in the integer programming section.\n* \"we investigate [a] mixture\"\n* \"results with high degradation\" -> in high degradation\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "29f6RlrgQKc", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work presents a quite comprehensive multi-step scheme for post-training neural quantization that does not rely on large datasets or large computational resources.  \n\nThe work is has significance in the domain of post-training neural quantization, especially in cases where only a small calibration set or limited resources are available. It would be interesting to also think about accumulator quantization.\n\n\nPros:\n\nThe empirical results are relatively strong in this method; 4-bit quantization is a good achievement in the models considered here.\n\nThe quantization process covers nicely the various different parts of the errors that  post-training quantization induces and propose somewhat original solutions to them. \n\nCons:\n\nThe framework is relatively complex and consists of multiple steps.\n\nWhat is the detailed difference of computational resource use between light and advanced pipeline?\n\nAdaQuant seems like a rather straight-forward step from AdaRound by combining with it some related works.\n\nIt is not clear how the BN error compensation differs exactly from related work.\n\nSome details were missing, for example, it is up to the reader to guess how many bits were used in the accumulators.\n\nSome spelling mistakes, e.g., “Optimizing Quantization Pipline”\n\n\nOverall: An engineering oriented paper with some lack of testable hypotheses and analysis of some parts of the methods, but the 4-bit results justify publication. Edit: I have not seen author reply and further reading of the paper has not clarified the main issues found by all reviewers. I have to lower the score. Edit2: new version of the paper and the author reply cleared some concerns, score raised accordingly.\n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper received mixed reviews, 3 positives (7, 6, 6) and 2 negatives (4, 4). Due to the divergence of the reviews, I carefully read the paper and made my best efforts to understand the paper and the review comments. This paper proposes to learn a quantization network using a small calibration set given a network trained with the full precision. The combination of AdaQuant, integer programming, and batch-norm tuning makes sense although they do not have substantial novelty. The three components are reasonably tightly-coupled and comprise a complete algorithm. However, the sequential-AdaQuant distracts the main claim of this work significantly. This is probably added during the review process but looks ad-hoc to me. Sequential AdaQuant seems to be effective to improve accuracy, but cannot be applied before the bit allocation was set, which makes it require integer programming no more. Because of this issue, the overall presentation becomes confusing and the argument sometimes sounds unfair (please refer to the last posting by R5.). 

In addition, the presentation of this paper could be improved, especially for the details of the integer programming formulation. It is not clear how to define some variables mathematically. The discussion about the size of the calibration set together with the overfitting issue is lacking, and rigorous discussion and analysis would make the paper much stronger. The reviewers are not convinced of the novelty of this paper, and they rather believe that this is an engineering-oriented work. Considering this fact,  the evaluation of this paper is not very comprehensive. The ablation study with respect to the size of the calibration set should be conducted more intensively. The experiment fails to show the benefit of mixed precision quantization effectively and it is limited to presenting the compression ratio in Figure 3. The authors used a small calibration set taken from the training dataset, which looks weird because they claim that the post-training quantization requires only a small "unlabeled" calibration set at the beginning of the abstract; it is more desirable to use arbitrary examples in the same domain.

Despite the interesting aspects, I believe that this paper needs a focus and substantial improvement for publication, and, consequently, recommend rejection.
Decision: Reject