BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction
Authors: Yuhang Li
Ruihao Gong
Xu Tan
Yang Yang
Peng Hu
Qi Zhang
Fengwei Yu
Wei Wang
Shi Gu
Venue: ICLR 2021 Poster (2021)
Keywords: Post Training Quantization
Mixed Precision
Second-order analysis

Abstract: We study the challenging task of neural network quantization without end-to-end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Aware Training (QAT). In this work, we propose a novel PTQ framework, dubbed BRECQ, which pushes the limits of bitwidth in PTQ down to INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves a good balance between cross-layer dependency and generalization error. To further employ the power of quantization, the mixed precision technique is incorporated in our framework by approximating the inter-layer and intra-layer sensitivity. Extensive experiments on various handcrafted and searched neural architectures are conducted for both image classification and object detection tasks. And for the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 comparable with QAT and enjoy 240 times faster production of quantized models.  Codes are available at https://github.com/yhhhli/BRECQ.

Published as a conference paper at ICLR 2021
BRECQ: PUSHING THE LIMIT OF POST-TRAINING QUANTIZATION BY BLOCK RECONSTRUCTION
Yuhang Li Ruihao Gong?; ; Xu Tan", Yang Yang", Peng Hu?,
Qi Zhang", Fengwei Yu', Wei Wang, Shi Gu 1z
'University of Electronic Science and Technology of China, SenseeTime Research 1 Liyuhang69 9996 999gmmill.... com, pongruihadesenser ime. com, gus@uest edu.
ABSTRACT
We study the challenging task of neural network quantization without end-to- end retraining, called Post-training Quantization (PTQ). PTQ usually requires a small subset of training data but produces less powerful quantized models than Quantization-Awe Aware Training (QAT). In this work, we propose novel PTQ frame- work, dubbed BRECQ, which pushes the limits of bitwidth PTQ down INT2 for the first time. BRECQ leverages the basic building blocks in neural networks and reconstructs them one-by-one. In a comprehensive theoretical study of the second-order error, we show that BRECQ achieves good balance between cross- layer dependency and generalization error. To further employ the power of quan- tization, the mixed precision technique is incorporated in framework by ap- proximating the inter-layer and intra-layer sensitivity. Extensive experiments on various handcrafted and searched neural architectures are conducted for both im- age classification and object detection tasks. And for the first time we prove that, without bells and whistles, PTQ can attain 4-bit ResNet and MobileNetV2 com- parable with QAT and enjoy 240x faster production of quantized models. Codes are available https: git git.ub.................... watom/hhhhliBBBB
INTRODUCTION
The past decade has witnessed the rapid development of deep learning in many tasks, such com- puter vision, autonomous driving, etc. However, the issue of huge computation cost and memory footprint requirements in deep learning has received considerable attention. Some works such as neural architecture search (Zoph & Le, 2016) try to design and search a tiny network, while oth- ers, like quantization (Hubara et al., 2017), and network pruning (Han al., 2015) are designed to compress and accelerate off-the-shelff well-trained redundant networks.
Many popular quantization and network pruning methods follow simple pipeline: training the original model and then finetune the quantized/pruned model. However, this pipeline requires full training dataset and many computation resources to perform end-to-end backpropagation, which will greatly delay the production cycle of compressed models. Besides, not all training data are always ready-to-use considering the privacy problem. Therefore, there is more demand industry for quan- tizing the neural networks without retraining, which is called Post-training Quantization. Although PTQ is fast and light, it suffers from severe accuracy degeneration when the quantization precision low. For example, DFQ (Nagel al., 2019) quantize ResNet- 8-bit without accuracy loss (69.7% top-1 accuracy) but in 4-bit quantization, can only achieve 39% top-1 accuracy. The primary reason is the approximation in the parameter space is not equivalent the approximation in model space thus we cannot assure the optimal minimization on the final task loss.
Recent works like (Nagel et al., 2020) recognized the problem and analyzed the loss degradation by Taylor series expansion. Analysis of the second-order error term indicates we can reconstruct each layer output to approximate the task loss degeneration. However, their work cannot further quantize the weights into INT2 because the cross-layer dependency in the Hessian matrix cannot be ignored when the perturbation on weight not small enough. In this work, we analyze the second-order
"Equal contribution, Corresponding author.

---

Published as a conference paper at ICLR 2021
error based the Gauss-Newtor matrix. We show that the second-order error can be transformed into network final outputs but suffer from bad generalization. To achieve the best tradeoff, we adopt an intermediate choice, block reconstruction. In addition, our contributions are threefold:
1. Based on the second-order analysis, we define a set of reconstruction units and show that block reconstruction is the best choice with the support from theoretical and empirical evidence. We also use Fisher Information Matrix to assign each pre-activation with an importance measure during reconstruction.
2. We incorporate genetic algorithm and the well-defined intra-block sensitivity measure to generate latency and size guaranteed mixed precision quantized neural networks, which fulfills a general improvement on both specialized hardware (FPGA) and general hardware (ARM CPU). We conduct extensive experiments to verify our proposed methods. We find that our method applicable to a large variety of tasks and models. Moreover, we show that post-training quanti- zation can quantize weights to INT2 without significant accuracy loss for the first time.
PRELIMINARIES
Notations Vectors denoted by small bold letters and matrices (or tensors) are denoted by capital bold letters. For instance, W and w represent the weight tensor and its flattened version. Bar accent denotes the expectation over data points, e.g. a. Bracketed superscript w(f) indicates the layer index. For convolutional or a fully-connected layer, we mark its input and output vectors by and z. Thus given a feedforward neural network with n layers, we can denote the forward process by
((++1) h22(9)) h h(w()x + b( 1 ises
(1)
where h()) indicates the activation function (ReLU in this paper). For simplicity, we omit the anal- ysis of bias ((9) as it can be merged into activation.
||ز denotes the Frobenius norm.
Quantization Background Uniform symmetric quantization maps the floating-point numbers to several fixed-points. These points (or grids) have the same interval and symmetrically dis- tributed. We denote the set that contains these grids as Qt = s x x 2-20-1
0, 2 1}.
Here, s is the step size between two grids and 6 is the bit-width. Quantization function, denoted by q(-) R + 2b is generally designed to minimize the quantization error:
min
w s.t. W 2 2..symn
Solving this minimization problem, one can easily get the q(-) by leveraging the rounding lounding-to-nearee te-near operation 1.7. Kounddingtt-nneares is a prevalent method to perform quantization, e.g. PACT (Choi al., 2018). However, recently some empirical or theoretical evidence supports that simply mini- mizing the quantization error in parameter space does not bring optimal task performances. Specif- ically, Esser et al. (2020) propose to learn the step size s by gradient descent in quantization aware training (QAT). LAPQ (Nahshan et al., 2019) finds the optimal step size when the loss function minimized without re-training the weights. Their motivations are all towards minimizing a final objective, which is the task loss, i.e.,
min [[W(W)] s.t. W Q...ym
(3)
While this optimization objective is simple and can be well-optimized in QAT scenarios, it is not easy to learn the quantized weight without end-to-end finetuning as well as sufficient training data and computing resources. post-training quantization settings, we only have full precision weights that w arg min,, E[L(w) where w E R and a small subset of training data to do calibration.
Taylor Expansion It turns out that the quantization imposed on weights could be viewed as a special case of weight perturbation. To quantitatively analyze the loss degradation caused by quantization, Nagel et al. (2020) use Taylor series expansions and approximates the loss degradation by
E E[L(w + Aw) [[]) = Aw (g(ggw + Aw H (WW)AW
(4)
where g(w) = L] and (w) E LJ are the gradients and the Hessian matrix and Aw is the weight perturbation. Given the pre- pre-trained model is converged to a minimum, the gradients can be safely thought to be close to 0. However, optimizing with the large-scale full Hessian is memory- infeasible on many devices as the full Hessian requires terabytes of memory space. To tackle this problem, they make two assumptions:
2

---

Published as a conference paper at ICLR 2021
1. Layers are mutual nutual-independent, thus the Hessian is in the form of layer-diagonal and Kronecker- factored, i.e., H( (xx(0x(0, @ H(x where & is the Kronecker product.
2. The second-order derivatives of pre-activations are constant diagonal matrix (H =cx which is independent of input data points.
At last, the objective is transformed into a practical proxy signal, the change in feature-maps (z Wx), and the quantized model can be obtained by layer-by-layer feature map reconstruc- tion algorithm (with few calibration images). Recent works, like Bit-Split (Wang et al., 2020) and Ada Auanu (Hubara et al., 2020), also take this layer-wise objective to improve the post-training quantization. However, they failed to quantize weights to INT2. We think the inherent reason is that when Aw grows higher, the former assumptions do not hold and an accurate signal is required.
3 PROPOSED METHOD
3.1 CROSS-LAYER DEPENDENCY
Denote the neural network output ((n) f(0), the loss function can be represented by L(f(0) where 9 vec(w(1)) matrix can be computed by
w(n).TJT is the stacked vector of weights in all layers. The Hessian
L
m ÔL Oz
(m)
m
aL
82
(n)
m
dz
(n)
0L
dz (m)
Σ dz (n)
Σ 00,00,
(5)
00,00,
30,
Ozk (n) 00,
00, dz (m)
Oz
(n) 00,
k,1=1
where z (n) R Since the pretrained full precision model is converged to local minimum, we can assume the Hessian is positive-semine semidefiitte (PSD). Specifically, the converged model has V Lo L close to 0 so the first term in Eq. (5) is neglected and Hessian becomes the Gauss-Newtor (GN) matrix ()), GN matrix can be written in matrix form (Botev et al., 2017)
( ( = (
(..)(()))) (0) (H() (0))))))
(6)
where J Jzc. (0) is the Jacobian matrix of the network output with respect to the network parameters. However, in practice, we cannot explicitly compute and store the Jacobian for each input data point in such raw form. To reduce the computation and memory budget, we will transform the second- order error into the network output, as shown in the following theorem.
Theorem 3.1. Consider an n-layer feedforward neural network with ReLU activation function. Assuming all weights are quantized, the second-order error optimization can be transformed by:
arg min A6 H AA arg min
(n. (H( (1)) Az("))
Remark 3.1. The same transformation is also applicable for activation quantization. The quadratic loss defined as E[Ay H(n) Ar] where Ây vec| (x))) ,..., Ax..,AA(n).T]]]
We prove the theorem using the quadratic form, details be found in Appendix A.I. Here provide a sketch of the proof by matrix form. The product of perturbation and Jacobian can be thought as the first-order Taylor approximation of the change in network output Az(n);
(n) = á (m) (n) R J (0)
Therefore, combining Eq. (8) and Eq. (6) we can transform the large-scale second-order error into the change network outputs characterized by the output Hessian H( The theorem indicates a simple observation, suppose a well-trained teacher model and an initialized student model, we can minimize their discrepancy by reconstructing the network's final output (n)), which coincides with and generalizes the distillation (Hinton al.. 2015: Polino al., 2018). LAPQ (Nahshan et al., 2019) also considers the dependency but their optimization does not rely on second-order information. However, we should emphasize that distillation requires the same computation and data resources as in normal training procedure, which is impractical for PTQ with limited data.
'To prevent ambiguity, we hereby use layer-diagonal Hessian to replace the common name "block-diagonal Hessian" " because the block in this paper means building block in the CNNs.
3

---

Published as a conference paper at ICLR 2021
judl F4
(PS MA
+
head
Stage
block
layer
Iygrediggool
(eb MOp
body
stage?
block 2
layer 2
Bockodiagonal
6n2.12.00
Ielk bl
stem
stage
block 1
layer
Full manie
(a) typical structure of CNN (taken from Radosavovic al. (2020)). Network is composed of stem layer (first convolu- tion on imput images), body and a head layer (average pool- ing with a fully connected layer). A body contains several stages, and a stage contains several blocks. A representative block the bottleneck block with residual path.
(b) An example illustration of Hessian (or Fisher) matrix. Blue sub-block means the layor-diagonal and each layer are mutual- independent; orange sub-block consider the dependency inside building block and green parts measure all dependencies.
Figure I: We define kinds of reconstruction granularity, namely net net-wise, stage-wise, block- wise and layer- wisc optimization, cach of them corresponds an essential component of CNN.
3.2 BLOCK RECONSTRUCTION
Although the network output reconstruction has an accurate estimation of the second-order error, find in practice it is worse than the layer-by-layer reconstruction in PTQ. The primary reason for this is optimizing the whole networks over 1024 calibration data samples leads to over-fitting easily. As Jakubovitz al. (2019) explained, the networks can have perfect expressivity when the number of parameters exceeds the number of data samples during training. but lower training error does not ensure lower test error. We find layer-wise reconstruction acts like regularizer which reduces the generalization error by matching each layer's output distribution. In other words, both layer-wise and network- -wise output reconstruction has their own drawbacks. And there should be better bias-variance trade-off choice to conduct reconstruction at an intermediate granularity.
The wise optimization corresponds to layer-diagonal Hessian (Fig. 1b blue parts) and the network- -wise optimization corresponds to full Hessian (Fig. 1b green parts). Similarly, we can de- fine an intermediate block-diagonal Hessian. Formally, if layer & to layer (where S k < f S n) form a block, the weight vector is defined as 0 = vec(w (k),T
(ê)TTT and the Hessian can be also transformed by A H AØ E ((), ( (0) Such dickkdia Hessian ignores the inter-block dependency and considers the intra-block dependency but it produces less general- ization error. Then can block-by-block reconstruct the intermediate output.
To this end, we define 2 extra kinds of intermediate reconstruction granularity: Stage-wise recon- struction and Block-wise reconstruction. These 4 reconstruction granularities are described below:
1. Liye Reconstruction: Assume the Hessian matrix is layer-diagonal and optimize the layer output one-by-one. It does not consider cross-layer dependency and resemble existing methods (Nagel et al., 2020; Hubara et al., 2020; Wang et al., 2020).
Block-wise Reconstruction: A block the core component in modern CNN, such as the Residual Bottleneck Block as shown in Fig. la. This method assumes the Hessian matrix is block- diagonal and block-by-block perform reconstruction, which ignores inter-block dependencies.
3. Stage-wise Reconstruction: A stage is where the featuremaps will be downsampled and gener- ate more channels, which is believed produce higher-level features. Typical CNN in ImageNet dataset contains 4 or 5 different stages. This method simultaneously optimizes all layers within a stage and thus considers more dependencies than the block-wise method.
4. Network-wiss wise Reconstruction: Optimize the whole quantized network by reconstructing the output of the final layers. This method resembles distillation but does not result in good perfor- mances with few images because of high generalization error.
The relationship between network, stage, block, and layer is illustrated in Fig. la. We test these 4 kinds of reconstruction granularity and find that biock-wise optinization outperforms others. We think this is because the main off-diagonal loss in the Hessian is concentrated in each block, as Fig. 1b orange part illustrated, while the inter-block loss is small and can be ignored in the opti-
4

---

Published as a conference paper at ICLR 2021
Algorithm 1: BRECQ optimization
Input: Pretrained FP model: Calibration dataset, iteration '
for all i 1, 2, ... N-th block in the FP model do
Collect imput data the block x(), the FP output 2 and its gradient g g(x();
for all j = 1, ....... - l'itteration do
Get quantized output (8) and compute Az = z 2
Descend Eq. (10) and update the rounding of all the weights in this block (Eq. (16));
if Activation Quantization is triggered then
Update the activation quantization step size (Eq. (18)).
After optimization, compute the sensitivity for each layer and between layers (2-bit only);
return Quantized model, Sensitivities for mixed precision;
mization. The shortcut connections, which is proposed in (He et al., 2016), may also increase the dependencies within a block. Also, the stage-wise or net-wise optimization suffer from the bad generalization the validation set and degenerate the final performances. We report the quanti- tative comparison in Sec. 4.1. We name our algorithm BRECQ, because we choose block as our base reconstruction unit. It is necessary to point that our analysis does not give the optimal configuration of the reconstruction granularity. The choice of block-wise optimization comes from our experiments and find this choice has two merits. (1) No hyper-parameter' included and (2) applicable for all models and all tasks we tested.
3.3 APPROXIMATING PRE-ACTIVATION HESSIAN
With block- block-diagonal approximated Hessian matrix, we can measure the cross-layer dependency inside each block and transform any block's second-order error to the output of this block ( (z) This objective requires the further computation of the knowledge in the rest of the network, i.c., pre-activation Hessian (x(1), One way is to follow Nagel et al. (2020) and assume (x) x I. Therefore the quadratic loss becomes || 1112 This method might be easy to implement but lose too much information.
We use the diagonal Fisher Information Matrix (FIM) replace the pre-activation Hessian. For- mally, given a probabilistic model p(x] the FIM defined as:
( E V log V log (y) |
E
V (ogr) H (8)
p(z)())
The FIM equal to the negative expected Hessian of the log-likelihood function, therefore, simple corollary is that the Hessian of task loss will become FIM if the model distribution matches the true data distribution (LeCun et al., 2012). Although matching true data distribution seems impossible, this is the best can do since the pretrained model is converged.
The diagonal of the pre-activation FIM is equal to the squared gradients of each elements, which is successfully used in Adam (Kingma & Ba, 2014) for the second momentum. The optimization objective becomes
(
aL (
aL Oz 28)
min E
Az min w
diag Az Az (0. (.Tdiag
( Oz (8)
2
Az
(10)
Compared with the MSE minimization, the above minimization incorporates the squared gradient information. If the output has higher absolute gradients, will receive more attention when being reconstructed. A similar method for pruning the pre-activation has been proposed in Theis et al. (2018).
Note that BRECQ is compatible with any optimization method, like STE (Hubara et al., 2017). Here we adopt adaptive rounding (Nagel et al., 2020) for weights and learned step size (Esser et al., 2020) for activation step size because observe they generally perform better in PTQ, see details in Appendix 1. We formulate the overall calibration algorithm for a unified precision model in algorithm 1. We should emphasize that we only need small subset (1024 in our experiments) of the
5

---

Published as a conference paper at ICLR 2021
whole training dataset to calibrate the quantized model. And we can obtain quantized ResNet- within 20 minutes on a single GTX 1080TI GPU.
3.4 MIXED PRECISION
To further push the limit of post-training quantization, employ mixed precision techniques, which can be formulated by
min c) H(c) S 6, c C (2, 8)
(11)
Here, is the bit-width vector with the shape of number of layers. (-) is hardware performance measurement function, which is used to ensure the mixed precision model has the same or lower hardware performance (e.g., memory and speed) than a predefined threshold 5. We choose 4, 8-bit for mixed precision because they are most common in practical deployment.
Regarding the training loss L, we find that nearly all existing literature (Cai et al., 2020; Hubara al., 2020; Dong et al., 2019) uses layer-wise measurement. They all assume the sensitivity within a layer independent and can be summed together. Therefore, the mixed precision problem becomes an integer programming problem. However, we argue that the loss measurement should contain two parts: diagonal loss and off-diagonal loss, the first the same with previous works and measure the sensitivity of each layer independently, while the off-diagonal loss is used to measure the cross-layer sensitivity. Theoretically, we should examine all permutations, which results in 3 possibilities and prohibits the search algorithm. Our first attempt is to reduce the off-diagonal loss into the block- level as we mentioned that the Hessian can be approximated to block-diagonal matrix. Granted, still find the search space is large, for example, if block has four layers, then have to consider the 3 = 81 permutations for a single block. Based on our preliminary experiments, find that 4-bit and 8-bit quantization nearly do not drop the final accuracy. Hence only take 2-bit permutations into consideration and drastically reduce the search space. We use genetic algorithm (Guo et al., 2020) to search the optimal bitwidth configuration with hardware performance threshold, the algorithm is located in algorithm 2. Due to space limits, we put related works in Appendix 5. Readers can refer to related works for brief discussion quantization and second- order analysis.
4 EXPERIMENTS
In this section, we report experimental results for the ImageNet classification task and MS COCO object detection task. The detailed implementation of the experiments can be found in the Ap- pendix B.4.4. The rest of this section will contain ablation study on reconstruction granularity. classification and detection results, mixed precision results and comparison with quantization-awe aware training. In Appendix B, we conduct more experiments, including the impact of the first and the last layer, the impact of calibration dataset size and data source.
4.1 ABLATION STUDY
We test four kinds of reconstruction granularity: Net Stage-wise, Block-wise, and Layer- Layer-wise reconstruction. We conduct ImageNet experi- ments using MobileNetV2 and ResNet-18 with 2- bit weight quantization for all layers except for the first and the last layer. It can be seen from Table that Block-wiss wise optimization outperforms other methods. This result implies that the general- ization eтror in wet-wise and stage-wise optimiza- tion outweighs their off-diagonal loss. In ResNet- 18, we find the difference is not significant, this can be potentially attributed to that ResNet- 18 only has 19 layers in the body and the block size, as well as the stage size, is small, therefore leading to indistinct results.
Table 1: Ablation study.
Model
Layer
Block
Stage
Net
ResNet-18
65.19
66.39
66.01
54. 15
MobileNetV2
52.13
59.67
54.23
40.76
4.2 IMAGENET
We conduct experiments on a variety of modern deep learning architectures, including ResNet (He al., 2016) with normal convolution, MobileNetV2 (Sandler et al., 2018) with depthwise separa-

---

Published as a conference paper at ICLR 2021
Table 2: Accuracy comparison weight-only quantized post-training models. Activations here quantized kept full precision. We also conduct variance study for our experiments. Bold values indicates best results. indicates our implementation based on open-source codes.
(W/A)
ResNet-18
ResNet-50
MobileNetV2
RegNet-600MF
RogNet-3.2GP
Poll Proc.
32/32
71.08
77.00
72.49
73.71
78.36
76.68
Comection"
OMSBs (Choukroun sl., 2019)
4/32
67.12
74,67
AdaRound (Nagel 2020)
4/32
68.71
75.23
69.78
71,97
77.12*
74,87*
AdaQuant
2020)
4/32
Bit-Split (Wang al., 2020)
4/32
69.11
75.58
BRECO (Ours)
4/32
70.70
76.29:00.4
71.66
73.02
78.04:9.44
76.00:8.02
Bias Correction
3/32
12.85
7.97
10.89
28.82
17.95
10.72
AdaRound (Nagel et al., 2020)*
3/32
68.07
73.42
64.33
67.71
72.31
69.33
Adatquant (Huham 2020)*
3/32
67.61
12.56
Bii- Split (Wang al., 2020)
3/32
66.75
73.24
BRECQ (Ours)
3/32
69.81
0.06
75.61 1:009
69.50
71.48 o007
77.22
74.58 0:06
Correction*
2/32
AdaRound (Nugel al., 2020y*
2/32
$5.96
47.95
32.54
25.66
24.70
30,60
AdaQuant (Hubara et al., 2020y*
2/32
0,30
0.49
0.11
BRECO
66.30
72.40
Table 3: Accuracy comparison on fullyy quantized post-training models. Activations here are quantized to 4-bit. Notations follows the upper table.
Methods
Bits (W/A)
ResNet-18
ResNet-50
MobileNetV2
RegNet-600MP
RegNet-3.2GP
Nets-e
Prec.
71.08
77.00
72.49
73.71
ACIQ-Mix (Ranner al., 2019)
4/4
67.0
73.8
ZeroQ (Cai al. 2020)*
4/1
21.71
2.94
26.24
28.54
12.24
3.89
(Nahshan
65.32*
AdaQuantt (Hubara et al., 2020)
4/4
67.5
73.7
34.95
Bii-Split (Wang al., 2020)
4/4
67.56
73:71
BRECO
66.57
ZeroQ (Cai et al., 2020)*
2/1
0.08
0.08
0.10
0.10
0.05
0.12
LAPQ (Nahshan 2019y*
2/4
0.18
0.14
0.13
0.17
0.12
0.18
Adaiquant (Hubara et al., 20200*
0122
0.12
BRECQ (Ours)
2/1
64.80
:005
70.29
53.34:0.15
59.31 -0.40
67.15
97.15:9.11
63.01
ble convolution and RegNet (Radosavovic et al., 2020) with group convolution. Last but not least important, also investigate the neural architecture searched (NAS) models, MNasNet (Tan al., 2019). In Table 2, we only quantize weights into low-bit integers and keep activations full precision. We compare with strong baselines including Bias Correction, optimal MSE, AdaRound, AdaQuant, and Bit-split. Note that the first and the last layer are kept with 8-bit. While most of the existing methods have good performances 4-bit quantization, they cannot successfully quantize the model into 2-bit. Our method consistently achieves the lowest accuracy degradation for ResNets (within 5%) and other compact models. We further quantize activations into 4-bit make the quantized model run on integer-arithmetic hardware platforms. We find that 4-bit activation quantization can have a huge impact on RegNet and MobileNet. Nonetheless, our methods produce higher perfor- mance than other state-of-the tfrt--o---hh--ts.... To be noted, BRECQ is the first to promote the 2W4A accuracy of PTQ to a usable level while all other existing methods crashed.
4.3 COMPARISON WITH QUANTIZATION- TRAINING
Table 4: Performance well training cost comparison with quantization- aware training (QAT).
Models
Methods
Precision
Accuracy
Model Size
Training Data
GPU hours
ZEROQ (CAI ET AL.,, 2020))
4/4
0.008
BRECQ (OURS)
4/4
69.60
5.81 MB
1024
0.4
ResNet-18
BRECQ (W/ DISTILLED DATA)
4/4
69.32
5.81 MB
0
0.4
FP: 71.08
PACT (CHOl ET AL., 2018)
4/4
69.2
5.81 MB
1.2 M
100
DSQ (GONG ET AL., 2019)
4/4
69.56
MB
1.2
100
LSQ (ESSER ET AI.,, 2020)
4/4
71.1
5.81 MB
1.2 M
100
BRECQ (OURS)
4/4
66.57
2.26 MB
1024
0.8
MobileNerV2 FP: 72.49
PACT (CHOI FT AI.,, 2018)
4/4
61.40
2.26 MB
1.2 M
192
DSQ (GONG ET AL.. 2019)
4/4
64.80
2.26 MB
1.2 M
192
BRECO (OURS)
Mixed/8
70.74
1.38 MB
1024
3.2
HAQ (WANG AI., 2019)
Mixed/8
70.90
1.2
384

---

Published as a conference paper at ICLR 2021
Table Objection detection task (MS COCO) comparison on fully quantized post- past-training models. Activations here are quantized 8-bit. report bounding box mean Average Precision (mAP) metric.
Full Prec.
Blas Correction
AdaRound*
ZeroQ
BRECQ (Ours)
Models
Backbone
32/32
8/8
4/8
4/8
2/8
4
8/8
4/8
2/8
Faster RCNN
ResNet-18
34,55
34.30
0.84
33.96
23.01
34.53
34.34
31.82
(Ren ct ul.,
ResNet-50
38.55
38.25
0.25
37.58
19.63
38.54
38.29
34.23
2015)
MobilcNetV2
33.44
33.24
18.39
32.77
16.35
33.40
33.18
27.54
RetinaNer
ResNet-18
33.20
33.00
0.04
32.59
19.93
33.14
33.01
31.42
iiin et al., 2017)
ResNet-50
36.82
36.68
0.07
36.00
19.97
33.7
36.73
36.65
34.75
MobileNetV2
32.63
32.60
18.47
31.89
32.57
32.31
27.59
Mixed
Unified
2-bit
4-bit
ZeroQ
Rec
71
Te.
16.15
Test Accuracy
Accaracy
Text Accuracy
70
47.09
07
64
3.8
3.5
50
1.1
1.4
1.25
2.80
3.25
1.75
Model Size (MR)
Model Size (MB)
Model Sizee (M
71
73.08
79
Text Accuracy
Accarncy
Test Accuracy
79
64.33
67
F
39
45
50
28.5
25.0
20.5
31.8
31.5
31.0
31.5
22
34
Latency (125
Latency (tns)
Lafency (ms)
Figure 2: Mixed Precision results.
In this section, compare our algorithm (post-training quantization) with some quantization-awee training methods, including PACT (Choi et al., 2018), DSQ (Gong et al., 2019), LSQ (Esser et al., 2020), and mixed precision technique HAQ (Wang et al., 2019). Table 4 shows that although BRECQ a PTQ method with limited available data, it can achieve comparable accuracy results with existing quantization-aware training models. In addition, our method can surpass them in 4- bit MobileNetV2 while using less than one training GPU hours. Our method also has comparable accuracy with HAQ, which is training-based mixed precision method. Note that our GPU hours include unified precision training (2-, 4- 8-bit respectively) and further mixed-precision training only needs to check the lookup table. Instead, HAQ would end-to-end search for each hardware performance threshold from scratch.
4.4
MS COCO
To validate the effectiveness BRECQ on other tasks, we conduct object detection on the two- stage Faster-RCNN (Ren et al., 2015) and the one-stage RetinaNet (Lin et al., 2017). ResNet-18, 50 as well as MobileNetV2 are adopted as backbones for the detection model. Results Table demonstrate method nearly does not drop the performance in 4-bit weight quantization and 8- bit activation. In particular, BRECQ only decreases 0.21 % mAP performance on 4-bit ResNet- 18 backboned Faster RCNN. On 4-bit ResNet-50 backboned RetinaNet, our method is outperforms the mixed precision based ZeroQ model by 3 3 mAP. Even when the weight bit decreases to 2, the model still achieves near-to- car-to-original mAP.
8

---

Published as a conference paper at ICLR 2021
4.5 MIXED PRECISION
In this section, we test (1) model-size guaranteed mixed precision and (2) FPGA latency guar- anteed mixed precision" unleash the potential of mixed precision and further push the limit of PTQ. We choose ResNet-18, MobileNetV2, and RegNetX-600MF to validate the efficacy of our algorithm. Note that this section, we keep activation in 8-bit because we only compare the dis- erepancy between the unified and mixed precision in weights. We omit 3-bit weight quantization in unified precision because it is usually unfriendly to the hardware. Latency settings can be found in Appendix B.4.3. From Fig. 2 we find that (1) mixed precision consistently outperforms unified precision, especially when using extremely low-bit, e.g., up to 10% accuracy increase with the same latency as the 2-bit model. (2) mixed precision can produce many bit configurations that can adapt to plenty of hardware requirements while unified precision can only have 2 fixed models.
5 RELATED WORKS
Quantization Model quantization can be classified into two categories: Quantization-aware Train- ing (QAT) and Post-training Quantization (PTQ). Rounding floating-point numbers to fixed-points numbers will produce 0 gradients almost everywhere. Therefore, most QAT methods employ the Straigh-TThoouge Estimator (STE) for gradients approximation. Gong et al. (2019) uses a differen- tiable tanh function to gradually approach the step function. Choi et al. (2018); Esser et al. (2020) introduces parameterized clipping thresholds to learn by STE. Apart from uniform quantization, some works like Li et al. (2019) argue that non-uniform quantization has better performance than uniform quantization while keeping its efficiency. Despite the promising results given by QAT meth- ods, they usually need more than 100 GPU hours to get it. In that case. PTQ plays an important role which is what we focus on in this paper. Generally, most deep learning models can be safely quan- tized to 8-bit without re- tr-trriniin. Data-Free Quantization Nagel al. (2019) even do layer- wise 8-bit PTQ without any data. However, in 4-bit quantization, most parameter space-based methods cannot obtain good performances. Recently, Nagel al. (2020) propose do layer-wise calibration and made huge progress in 4-bit quantization. Our work continues its analysis on Taylor expansion and considers the off-diagonal loss. Another perspective of quantification the precision allocation scheme. Hardware-aware Quantization (HAQ Wang et al. (2019)) leverages reinforcement learning to search the optimal bitwidth configuration. Hessian-aware Weight Quantization (HAWQ) (Dong et al., 2019) utilizes the sesond-order information to decide the bitwidth. Mixed precision also appears in PTQ, such as the Pareto frontier method in ZeroQ (Cai et al., 2020) and the Integer Programming method in AdaQuant (Hubara et al., 2020).
Second-order Analysis and Optimization The history of second-order information in perturbation analysis can be traced to the 1990s like Optimal Brain Surgeon (Hassibi & Stork, 1993; Dong al., 2017). The Hessian matrix is essential for pruning and quantization. As aforementioned, HAWQ uses the largest eigenvalue of Hessian to determine the sensitivity. Hessian matrix is also impor- tant for second-order optimization like Newton's 's method as it consists of the curvature informa- tion. However, calculating the real full Hessian is prohibitive on today's 's deep learning architectures. Therefore, approximations are made to simplify the calculation and make the storage more flexi- ble, e.g., Causs-Newton optimization with Kronecker-factored recursive approximation Botev al. (2017). -essian-FFree optimization (Martens, 2010) avoids the explicit computation of the Hessian matrix by solving the linear system g Hv. Second-order optimization with FIM is called Natural Gradient Descent (Amari, 1998). K-FAC (Martens & Grosse, 2015) utilizes the layer-diagonal FIM and the approximation of the expected Kronecker product to compute the curvature information.
CONCLUSION
In this paper, we propose BRECQ. a post-training quantization framework by analyzing the second- order error. We show that the reconstruction of quantization at the block granularity arrives at a good balance of cross-layer dependency and first order approximation, especially in 2-bit weight quantization where no prior works succeed to quantize. BRECQ is compatible with mixed precision and can reduce the search cost. To our best knowledge, BRECQ reaches the highest performance in post-training quantization and is the first to be on par with quantization-awar training using 4-bit.
We also test mobile CPU latency guaranteed mixed precision, located in Appendix B.3.

---

Published as a conference paper at ICLR 2021
ACKNOWLEDGMENT
We thank Markus Nagel and anonymous reviewers for their kind help of this work. This project is primarily supported by NSFC 61876032.
REFERENCES
Shun-Ichi Amari. Natural gradient works efficiently in learning. Neural computation, 10(2):251- 276, 1998.
Haoli Bai, Jiaxiang Wu, Irwin King, and Michael Lyu. Few shot network compression via cross distillation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 3203-3210, 2020.
Ron Banner, Yury Nahshan, and Daniel Soudry. Post training 4-bit quantization of convolutional networks for rapid-deployment. In Advances in Neural Information Processing Systems, 2019.
Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical gauss-newton optimisation for deep learning. arXiv preprini arXiv: 1eXiiv/706.000 170603366, 2017.
Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 13169- 2020.
Jungwook Choi, Zhuo Wang. Swagath Venkataramani, Pierce I-Jen Chuang, Vijayalakshmi Srini- vasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. arXiv preprint arXi:: 2018.
Yoni Choukroun, Eli Kravchik, Fan Yang, and Pavel Kisilev. Low-bit quantization of neural net- works for efficient inference. In 20/9 IEEE/CVF International Conference on Computer Vision Workshop (ICCVW), pp. 3009-3018. IEEE, 2019.
Xin Dong, Shangyu Chen, and Sinno Pan. Learning to prune deep neural networks via layer-wise optimal brain surgeon. In Advances in Neural Information Processing Systems, 2017.
Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hes- sian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEE International Conference on Computer Vision, pp. 293-302, 2019.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S. Modha. Learned step size quantization. In International Conference on Learning Representations, 2020. URL https: http:///www.e...ee/mmmm wtt.eeewwwwweeeeeeeeeeeeeeeeeeee wttp://ewwee//eeeeeeeeeeeeee seee//oouumii@Grkgee 1tt.rmmm611rK006660050
Ruihao Gong, Xianglong Liu, Shenghu Jiang. Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantization: Bridging full- full-precision and low-bit neural networks. arXiv preprint arXiv: 1908. 000333 2019.
Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng. Zechun Liu, Yichen Wei, and Jian Sun. Single path one-shot neural architecture search with uniform sampling. In European Conference Computer Vision pp. 544-560. Springer, 2020.
Qingchang Han, Yongmin Hu, Fengwei Yu, Hailong Yang. Bing Liu, Peng Hu, Ruihao Gong, Yanfei Wang, Rui Wang. Zhongzhi Luan, et al. Extremely low-bit convolution optimization for quantized neural network on modern computer architectures. In 49th International Conference on Parallel Processing-ICPP pp. 1-12, 2020.
Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks 2015.
with pruning, trained quantization and huffman coding. arXiv preprint
Babak Hassibi and David G Stork. Second order derivatives for network pruning: Optimal brain surgeon. In Advances in neural information processing systems, pp. 164-171, 1993.
10

---

Published as a conference paper at ICLR 2021
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffryy Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint 1XX3v:5503.22. 2015.
Itay Hubara, Matthieu Courbariaux, Daniel Soudry, Ran EI- Elnva,iv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Jour- nal of Machine Learning Research, 18(1))88996888, 2017.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Layer-wise calibration and integer programming. arXiv preprint arXiv:2006.105/1 2020.
Daniel Jakubovitz, Raja Giryes, and Miguel RD Rodrigues. Generalization error in deep learning. In Compressed Sensing and Its Applications, pp. 153-193. Springer, 2019.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: petp:6922666 2014.
Yann A LeCun, Léon Bottou, Genevieve B Orr, and Klaus-Robert Müller. Efficient backprop. In Neural networks: Tricks of the trade, pp. 9-48. Springer, 2012.
Yuhang Li, Xin Dong, and Wei Wang. Additive powers-of-two quantization: An efficient non- uniform discretization for neural networks. In International Conference on Learning Representa- tions, 2019.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference computer vision, pp- 2980-2988, 2017.
James Martens. Deep learning via hessian-free optimization. 2010.
James Martens and Roger Grosse. Optimizing neural networks with kronecker-factored approximate curvature. In International conference machine learning, pp.
2015.
Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE International Con- ference on Computer Vision, pp. 1325-1334, 2019.
Markus Nagel, Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. arXiv preprint arXiv:2004.1083 2020.
Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. arXiv preprint arXiv:1911.071900 2019.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quanti- zation. arXiv preprint arXiv: 05668, 2018
Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Dollár. Designing network design spaces. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 10428-10436. 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection with region proposal networks. In Advances in neural information processing systems, pp. 91-99. 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018

---

Published as a conference paper at ICLR 2021
Hardik Sharma, Jongse Park, Naveen Suda, Liangzhen Lai, Benson Chau, Vikas Chandra, and Hadi Esmaeilzadeh. Bit fusion: Bit-level dynamically composable architecture for accelerating deep neural network. In 2018 ACM/IEEE 45th Anmual International Symposium on Computer Archi- tecture (ISCA), pp- 764-775. IEHE, 2018.
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2820-2828, 2019.
Lucas Theis, Iryna Korshunova, Alykhan Tejani, and Ferenc Huszár. Faster gaze prediction with dense networks and fisher pruning. arXiv preprint arXiv:1801 (0l1:/8801.0577,, 2018.
Kuan Wang, Zhijian Liu, Yujun Lin, Ji Lin, and Song Han. Haq: Hardware-aware automated quan- tization with mixed precision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8612-8620, 2019.
Peisong Wang, Qiang Chen, Xiangyu He. and Jian Cheng. Towards accurate post-training network quantization via bit-split and stitching. In Proc. 37nd Int. Conf. Mach. Learn.(ICML.), 2020.
Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rX1v:1611.0157,, 2016.
A MAIN PROOFS
A. PROOF OF THEOREM 3.1
Proof. We will prove the theorem using quadratic form. Denote the weight vector shape as 0 e Rd and the network output vector shape (n R The quadratic form of the (H) A0 can be represented by:
1. 003
d
OT. A0,
d Σ i= 1=1
ØT. ) (12)
A
H 0
A0
Σ i=1
A0
+2
10,20; Σ
00,0,
00,0,)
where L is the cross-entropy loss. Based on Eq. (5), we have
LL
dz(m)) (m)
0L Ozi" (n)
(13)
00,0g
00, Oz
(m)
00,
Substituting above equation in Eq. (12), have
(n)
0
Oz (n)
A0 H AO
k
(14a)
00,
Oz
(n)
(n)
00,
dz
(n)
8
Oz (m)
ΣΣΣ
20,20,
(14b)
00
8z
(n)
Z1
(x)
00,
m
02
A Ozi
(m)
Ozk
(n)
(14c)
Ozh
(n)
(n)
aA;
08,
2
(n))
(1)
= (AOJ
(AOJ
(14d)
0
where we define the J[ is the Jacobian matrix of w.r.t. y. this end, we use the first-order Taylor expansion as we did in Eq. (8) to approximate the change in network output, i.c.,
(D)
Az
AOJ
(15)
Therefore, the final objective is transformed to Az (n). (z (x
12

---

Published as a conference paper at ICLR 2021
B EXPERIMENTS
B.1 EFFECT OF THE FIRST AND THE LAST LAYER
Many papers claim that the first and the last layer can have huge impact on the final accuracy. In this section, we investigate this phenomenon as well as the impact of the first and the last layer on hardware performances. We test ResNet-18, MobileNetV2 as well as RegNet-600MF Our observa- tions include:
In terms of accuracy, the 4-bit quantization is essentially good, both these two layers won't drop too much accuracy (with 0.2%) But in 2-bit quantization, the last fully connected layer is far more important than the first layer. We also observe that the first layer in MobileNetV2 and RegNet (3x3 kernels, 32 channels) is slightly more sensitive than that in ResNet- (7 x7 kernel, 64 channels).
2. In terms of model size, the first layer merely has minor impact because the input images only have channels, while the last layer contains many weight parameters and greatly affects the memory footprint. We should point out that just the model size in the first layer is low doesn't t mean the memory burden is low, because the input image will cost huge memory space. In terms of latency, the situation depends the architecture. For example, in ResNet-18 the first layer has huge impact on the latency, while in MobileNetV2 and BigNett600MF the last layer more important than the first layer. This is because the latency is affected by multiple factors, such the input size of the featuremap, the FLOPs, and the weight memory size. The arithmetic intensity (OPs/byte) greatly affects the latency. We find that the operations with high arithmetic intensity, i.e., shallow layers in the network, generate less latency gap between different bit- widths.
In conclusion, we find that keeping the first and the last layer 8-bit is unnecessary. Especially in ResNet- 18, we find that setting all layers 4-bit results in 53.3 ms latency and faster than the 59.8 ms in 2-bit quantization (with first and last layer 8-bit), but the accuracy is even 4% higher. Such phenomenon indicates the potential power of the mixed precision.
B.2 EFFECT OF DATA
We evaluated the influence of the size of calibration dataset and the source of the data ResNet- 18. We test different numbers of input data point and find that the improvement in 4-bit quantization trivial. Yet in 2-bit quantization we can see that the accuracy increases 5% when #data points increase. We also test the distilled data introduced in ZeroQ (Cai et al., 2020). Distilled data is learned from pretrained models' BN statistics, i.e., Xdistilled arg min Σ ((4: ((P=-/i -0)=-41)222 + (.) where and is the original mean and variance in BN statistics of the (i)-th layer. We find that distilled data performs good in 4-bit quantization but still has a large margin with original ImageNet dataset in 2-bit quantization. We also find the final accuracy does not benefit much from the increase number of distilled data, this might because the distilled data are minimized by same objective and has low diversity.
Table 6: Impact of the first and the last layer
No Quantization
Precision 4/8
Precision 2/8
Models
First
Last
Accuracy
Model Size
Latency
Accuracy
Model Size
Latency
70.76
5.81 MB
70.72 ms
66.30
3.15 MB
59.84
ResNet-18
70.66
5.81 MB
53.76
65.95
3.15 MB
31.20
FP: 71.08
70.64
5.57 MB
70.08 ms
64,87
2.79 MB
58.72 ms
X
X
70.58
5.56 MB
53.28 ms
64.53
2.78 MB
30.88 ms
71.80
2.26 MB
32.80 ms
59.59
1.74 MB
30.40 ms
MobileNetV2 FP: 72.49
71.69
2.26 MB
32.64
59.13
1.74 MB
30.24
V
71.42
1.65 MB
31.52
56,29
0.83 MB
28.48
71.42
1.65 MB
31.36 ms
55.58
0.82 MB
28.32 ms
72.98
3.19 MB
31.84 ms
65.66
1.84 MB
23.20 ms
RegNet-600MF
72.89
3.19 MB
31.68 ms
65.83
1.85 MB
22.88 ms
FP: 73.71
72.69
2.94 MB
31.20 ms
62.93
1.47 MB
22.40
72.73
2.94 MB
31.04 ms
63.08
1.47 MB
22.08
13

---

Published as a conference paper at ICLR 2021
1esNet-18
10.00
70.6
70.55
6472
Accuracy
70.5
70,49
Accuracy
03.25
70.3
70.33
70.20
...... 20,9
1.31
08310
99.22
70.1
78.0
Imags/et data
ImageNer dats
Bistilie date
Iistlled dats
125
256
512
1024
138
256
512
1824
data points
data points
Figure 3: Effect of #data points and data source.
B.3 MOBILE CPU LATENCY GUARANTEED MIXED PRECISION
ResNet-18
ResNet-50
70
Test Accuracy
69
Test Accuraey
$7.88
Mixed
Unified
Unifted
65
2-hit
2-hit
64
1150
1175
1200
1225
1250
1378
Latency (ms)
Latency (ms)
Figure 4: Mixed precision results on ResNet-18 and 50.
In this section, test the mobile CPU latency guaranteed mixed precision. The latency lookup table tested using the technique in Gong et al. (2019). We only validate it on ResNet- 18 and ResNet-50 because the current low-bit bit General Matrix Multiply (GEMM) implementation only supports normal convolution. The results concur with Fig. 2. Below 4-bit, the mixed precision can achieve better task performance than the unified precision models. For ResNet-50, the improvement is lower than that for ResNet-18 and any other mixed precision models. We think this is because the sensitivity in ResNet-50 is not distinct and therefore the improvement brought by mixed precision is trivial.
B.4 IMPLEMENTATION
B.4.1 LEARNING STRATEGIES
In this work, we mainly focus developing optimization objective rather than optimization strate- gies. We observe adaptive rounding performs well in post-training quantization. A brief introduction on AdaRound is given below, the detailed algorithm be found in Nagel et al. (2020).
Traditional quantization function is performed by rounding-tor -oeddingtt--eersst operation: w 5 X clip[[//s] n, AdaRound optimizes the rounding policy in post-training quantization. Specif- ically, all weights are initially rounded by floor operation, and a learnable variable v determines the final rounding result to be flooring ceiling. A sigmoid-like function a(-) keeps the learnable variable v moving between and I and a regularization term assures the avv) can converged to either 0 1. The formulation is given by
W clip ( +()),
(16)
The minimization problem together with the regularization is given by
ØL 12
ØL
arg min Az (0.T
(0.T
(O.Tiii
,
A)
+ x x
(vo
- -11), (17)
Oz (0)
Oz
(2)
14

---

Published as a conference paper at ICLR 2021
where progressively decreasing in the calibration ensures the ovv) converged to binary values. The activations cannot be quantized using adaptive rounding because they vary with different input data points. Thus, only adjust its quantization step size Esser et al. (2020). Denoting the quadratic loss in above equation as lg. the gradients of step size is given by
if > n
Ox
bhg as
dhg
x
x
(18)
xrosx<a
0x
if 0
where all step size in the block will be optimized.
B.4.2 GENETIC ALGORITHM FOR MIXED PRECISION
Algorithm 2: Genetic algorithm
Input: Random initialized population Po with population size S; Iteration T, mutation
probability p: Hardware performance threshold o; Hardware measurement function
H()
TopK = 20; 1,2,..., T-th iteration do
for all t
Evaluate fitness value (Eq. (11)) for each individual Update and sort TopK based on fitness function; repeat
New bitwidth configuration by crossover Ceross Ooop) Peroseover := Perossover Ceross if H (Cerose) < 8; until Size of P Perossaver equal to S/2;
repeat
New bitwidth configuration by mutation Cruutate Mutate[TopK, probability = p); Poustate := Pmmutate + Cmutate if H (Cmutate) <
until Size of Pmmtuate equal S/2;
P. = Perossover U U Pmmutatei
Pmnutate =,, Perossover = Ø3;
Get the best fitted entry and then do the overall block reconstruction (ef. algorithm 1); return mixed precision model
B.4.3 LATENCY ACQUISITION
We test the latency of quantized neural networks self-developed simulator of precision- variable accelerator for NN. The basic architecture this accelerator is inspired by typical systolic- matrix multiplication. The accelerator supports the per-channel quantization parameter. The preci- sion of each layer a NN is highly configurable this accelerator, supporting 9 types of precision combination: activation: 2-, 4-, 8-bit weight: 2-, 4- 8-bit, see Fig. 5a. With the support of scal- able function-unit (Sharma et al., 2018), the peak performance of the accelerator is able to achieve corresponding linear improvement the precision decreases. For example, the peak performance of this accelerator is 256 GMAC/s in 8-bit x 8-bit precision, and it scales to 512 GMAC/s in 8- bit x 4-bit precision and TMAC/s in 2-bit 2-bit precision. Although this accelerator provides considerable computation resources especially in low precision, the parallelism of the specific layer (like depthwise convolution) and the bandwidth of on-chip buffer is limited. Consequently, actual performance may not scale accurately along with the peak performance, and the final performance differs according to the size and type of layers. The simulator performs cycle-accurate simulation can get equivalent evaluation
and evaluation for given NN executed on the accelerator, by using this simulator. The simulator is available in the provided source codes.
For the acquisition of mobile ARM CPU latency, adopt the redesigned low-bit GEMM imple- mentation Han et al. (2020). Fig. 5b shows brief overview of the low-bit GEMM implementa- tion. Since there instruction supporting the bit-width below 8 on ARM architecture, we
15

---

Published as a conference paper at ICLR 2021
Weight Buffer
Input Buffer
All
A12
A13
B11
B12
B13
C11
C12
Presision
Multiply
Accumalate
A21
A22
A23
B21
B22
B23
C21
C22
PE Array
B31
B32
Replicate
B12
B13
AlL
A11
AlL
Accu-
C11
C12
C13
Element-
Requant/quant D computation Unit
A21
wiss Multiply
B12
B13
R02
C21
C23
C23
A31
B11
B12
B13
A31 811
A31 B12
B13
C31
C32
C33
Output Buffer
Buffer A
Buffer B
Buffer C
Results Matrix C
(a) Accelcrator design.
(b) Low-bit GEMM implementation on ARM CPU,
Figure 5: FPGA-based and mobile CPU-based latency acquisition.
not get higher computation efficiency for extremely low-bit such as 2-bit and 4-bit. But we can acquire a better memory access efficiency. The primary speedup comes from the reduction of data movement. Specifically, we can conduct more times of addition in the same 8-bit register before we have to move it to 16-bit register to avoid overflow. The lower bit-width is used, the less movement needed. Together with the optimized data packing and data padding, we can run mixed precision quantization on Raspberry Pi 3B, which has a 1.2 GHz 64-bit quad-core ARM Cortex-A53. Note that this implementation not optimized for depthwise separable or group convolution, therefore we only verify the latency on ResNets.
B.4.4 IMPLEMENTATION DETAILS
The ImageNet dataset consists of 1.2M training images and 50K test images. We follows standard pre-process (He et al., 2016) to get 1024 224 224 input images the calibration dataset. We fold the batch normalization layer into convolution and freeze the BN statistics before post- post-training quan- tization. We use Adam optimizer (Kingma Ba, 2014) to learn the weight rounding and activation range to reconstruct the block output. Note that some layers are not component of any block, such as the first convolutional layer and the last fully connected layer and the last convolutional layer in the MobileNetV2. These layers use naive layer reconstruction. The batch size of learning is set to 32 and each block will be optimized for 2 x 10 iterations. The learning rate is set to 10 during the whole learning process. Other papeepparameters such the temperature B are kept the same with Nagel et al. (2020). For activation step size, we also use Adam optimizer and set the learning rate 4e-5. Note that we do not implement the gradient scale as introduced in the original paper (Esser et al., 2020). After reconstruction, we will store the sensitivity measured on the calibration dataset. Note that we will store intra-block sensitivity in 2-bit quantization. The sensitivity, well hard- ware performances for each layer, will be stored in a lookup table. When calculating the fitness value and determining the hardware performances in a genetic algorithm, we will check the lookup table. For genetic algorithm, we set the population size to 50 and evolve 100 iterations to obtain the best individual. The first population is initialized by Gaussian distribution and we round the samples to integers in [O, 1, 2), corresponding to bit-width [2, 4, 8). The mutation probability is set to 0. The genetic algorithm usually completes the evolution only about 3 seconds.
For object detection tasks, we use 256 training images taken from the MS COCO dataset for cal- ibration. The image resolution set to 800 (max size 1333) for ResNet-18 and ResNet-50, while the image resolution for MobilcNetV2 is set to 600 (max size 1000). Note that we only apply block reconstruction in the backbone because other parts of the architecture, such as Feature Pyramid Net, do not have the block structure. Therefore naive layer reconstruction is applied to the rest of the network. Learning hyper-parament parameters are kept the same with ImageNet experiments.
16


Reviews:
[{"review_id": "oOg-qWXmFoe", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Post-training quantization is an important problem, especially for industry. This paper leverages the basic building blocks and conducts a block-wise quantization. Nice results are obtained with the proposed method. \n\nThe proposed method is cheap to implement and pushes the post-training quantization to 2-bit. \nThe measurement problem of mixed precision literature raised in this paper is of insights. This problem may inspire the community to find a better measurement in future work. \n\nExtensive experiments on various methods (handcrafted and designed by NAS), various tasks (classification, detection), various configurations (different bits, latency, model size) are impressive. Various baselines are also included to make the results stronger. \n\n\nQuestions:\nThe block-diagonal scheme is selected according to experimental results. Is it possible to visualize the real Hessian of stage-wise settings? (it may be impossible to the full Hessian matrix for the whole network). If we see a few non-zero elements at the off-diagonal for the block-wise setting, this choice can be better motivated. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "T2_S8rdLTK8", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "I couldn't follow the method described in the paper. The authors are basically trying to address post-training quantization by perturbing the the weights of a trained DNN. The goal is to perturb the weights so that the quantized DNN will behave similar to the original full-precision DNN. The authors draw a link between this optimization problem and optimizing for the \"reconstruction\" of the output activations of a block (see Equation 7). The technique BRECQ, shown in Algorithm 1, is basically to optimize the perturbation of the weights for the right hand side of Equation 7 for each block of a DNN.\n\nThe method only seems to work on ReLU networks, so it's restricted to CNNs, but this is still extremely useful.\n\nThe experimental results are very strong. BRECQ is the first to achieve 4-bit integer post-training weight quantization that nearly matches the fp32 baseline on ImageNet. Even at 2-bit, BRECQ can often get within 5% of the baseline while other methods leave the model with inoperable loss of accuracy.\n\nI couldn't pinpoint any major flaws in the paper, and the results are extremely impressive.", "extra_scores": {}, "extra_text": {}}, {"review_id": "P_FYDuSYqiX", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper explores the post-training inference quantization. Based on second-order quantization error analysis, it proposes to reconstruct quantized model in a block level to achieve SOTA accuracy for INT2 weight quantization, which distinguish this paper from previous reported layer-wise reconstruction approach. The proposed approach is intuitive and supported by extensive experiments across a wide range of image classification and object detection tasks.\n\nOverall, this paper is well written. The idea is straightforward but comes from a detailed theoretical analysis and supported by strong experimental results. The authors also proposed solutions to approximate pre-activation Hessian by using Fisher Information Matrix which sounds reasonable and is easy to implement. The authors did comprehensive comparison with SOTA approaches, not just PTQ, but also quantization aware training and mix-precision frameworks.  \n\n\nI have some questions wish to be clarified.\n\n1). It is not clear to me, in the reconstruction process, how does the short-cut (e.g. in ResNet) are handled.  \n\n2). In the error analysis, batch normalization layers are not considered. Batch norm will have direct impact on the output activation, hence the quantization bias term. Plus, batch norm statistics will be changed during the reconstruction phase. Could the authors comment on the impact of batch norm layers?\n\n3). The authors use 1024 training samples to do the reconstruction. What happens when more samples are used? What determines the number of samples needed?\n\n4). The data presented seems to focus mainly on weight quantization, while leaving activation in relative higher precision. What happens when quantize activations in 2 bits together with the weights in this approach? What are the challenges?\n\n5). It would have been nice to verify this approach on tasks other than vision, such as speech and NLP tasks.\n\n6). Seems a format error on one of the reference: Zichao Guo, Xiangyu Zhang, Haoyuan Mu, Wen Heng, Zechun Liu, Yichen Wei, and Jian Sun. {SINGLE} {path} {one}-{shot} {neural} {architecture} {search} {with} {uniform} {sampling}, 2020. URL https://openreview.net/forum?id=r1gPoCEKvH. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "2YRPVayIn3p", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes BRECQ which is a new Post Training Quantization (PTQ) method. The goal of the paper is to push the limit of PTQ to low bit precision (INT2). They try to address this by considering both inter and intra-layer sensitivity to find the best update to the model parameters so that the output from a block is minimally changed/perturbed. Furthermore, the authors also consider mixed precision quantization setting.\n\nEmpirical results are shown for multiple NNs for image classification and object detection.\n\n\nWhile the paper is trying to address an interesting problem and there are a lot of empirical results, but I had a very hard time to follow the paper's main idea and the \"final\" proposed algorithm. It would be great if the authors could answer the questions below and I will reconsider my score accordingly.\n\n\n- What is the final algorithm? The data provided in Algorithm 1 is very vague and there are no equations. It is not clear if you are actually using any second-order information since based on Eq 11 it seems that the proposed method is only using the gradient, and not the second order information. If so are the discussion from page 2-4 necessary?\n\n- It is not clear if the equality given in Eq 11 is correct. With the softmax layer the Hessian diagonal would not be the same as gradient^2.\n\n\n- Why not use the change in loss in Eq 4 (left) instead of using a second-order Taylor series? Second-order Taylor series is an approximation and does not hold for large perturbations. Also when we have access to evaluating the change in loss, why do we even need to focus on measuring the second order term? This does not seem to be correct/necessary at least for uniform quantization.\n\n- An ablation study is actually needed for the above.\n\n- What is the experimental setting for the ablation study in Table 1? while not mentioned, from the accuracy it seems the results are on ImageNet but this is not mentioned. Did the authors consider other bit precision settings?\n\n\n- In page 3, it is stated that using the Hessian directly is not possible due to memory and computational overhead. However, you can use matrix free methods which do not require forming the Hessian explicitly.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a new method for post-training quantization, achieving very good results. After the author's response, all the reviewers were positive. There were some issues regarding clarity, and about explaining why the methods work better than just optimizing the loss, but I think the reviewers were eventually satisfied.  Following some info after the author's response phase, I'll just ask the authors to verify their published code works with publicly available PyTorch packages, so their method could be easily used.
Decision: Accept (Poster)