Leveraging Inter-Layer Dependency for Post -Training Quantization
Authors: Changbao Wang
DanDan Zheng
Yuanliu Liu
Liang Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: low-bit
post-training quantization
computer vision
CNN
over-fitting
discrete optimization

Abstract: Prior works on Post-training Quantization (PTQ) typically separate a neural network into sub-nets and quantize them sequentially. This process pays little attention to the dependency across the sub-nets, hence is less optimal. In this paper, we propose a novel Network-Wise Quantization (NWQ) approach to fully leveraging inter-layer dependency. NWQ faces a larger scale combinatorial optimization problem of discrete variables than in previous  works, which raises two major challenges: over-fitting and discrete optimization problem. NWQ alleviates over-fitting via a Activation Regularization (AR) technique, which better controls the activation distribution. To optimize discrete variables, NWQ introduces Annealing Softmax (ASoftmax) and Annealing Mixup (AMixup) to progressively transition quantized weights and activations from continuity to discretization, respectively. Extensive experiments demonstrate that NWQ outperforms previous state-of-the-art by a large margin: 20.24\% for the challenging configuration of MobileNetV2 with 2 bits on ImageNet, pushing extremely low-bit PTQ from feasibility to usability. In addition, NWQ is able to achieve competitive results with only 10\% computation cost of previous works.

Leveraging Inter-Layer Dependency for Post -Training Quantization
Changbao Wang Ant Technology Group Co., Ltd. changbao. wcb@antgroup.cc com
Dandan Zheng Ant Technology Group Co., Ltd. wwnndaanzzddaatggruu..coom com
Yuanliu Liu Ant Technology Group Co., Ltd. yuanliu. 1yl@antgroup. com
Liang Li Ant Technology Group Co., Ltd. double.. 11@antgrouup...... com
Abstract
Prior works on Post-training Quantization (PTQ) typically separate a neural ner- work into sub-nets and quamtize them sequentially. This process pays little attention to the dependency across the sub-nets, hence is less optimal. In this paper, we propose novel Network- Wise Quantization (NWQ) approach to fully leveraging inter-layer dependency. NWQ faces a larger scale combinatorial optimization prob- lem of discrete variables than in previous works, which raises two major challenges: over-fitting and discrete optimization problem. NWQ alleviates over fitting via Activation Regularization (AR) technique, which better controls the activation distribution. To optimize discrete variables, NWQ introduces Annealing Softmax (ASoftmax) and Annealing Mixup (A (AMixup) progressively transition quantized weights and activations from continuity to discretization, respectively. Extensive experiments demonstrates that NWQ outperforms prior by a large margin: 20.24% for the challenging configuration of MobileNetV2 with 2 bits ImageNet, pushing extremely low-bit PTQ from feasibility usability. In addition, NWQ is able to achieve competitive or better results with only 10% computation cost of previous works.
approaches
Introduction
Deep learning has seen boom of architectures with ever-increasing capabilities and capacity. Huge capacity, however, always comes with boosting parameters, resulting in large computation cost and memory footprint. Researchers have utilized strategies like neural architecture search (NAS) [4, 11, 29, 49], network pruning [12, 19], and knowledge distillation (KD) [18, 21] to design compact networks and improve their accuracy. Furthermore, neural network quantization [7, 15, 22, 25, 28] is typically applied along with the methods mentioned above to reduce the computation cost and memory footprint. Quantization represents neural networks with lower bit precision, usually 8 bits in practice, and saves 75% of memory while offering 2 ~ 4x speedup/26].
Network quantization techniques typically categorized into Quantization Aware Training (QAT) and Post-training QuantizationPPTQ) based on the dependency on labeled data and the necessity of weight tuning. Generally, PTQ techniques are preferred in industry because they rely less labeled data and take into account privacy concerns as well as compute resources. This paper focuses on PTQ.
PTQ usually suffers from significant accuracy degradation, especially for low-bit quantization which offers much memory reduction and speedup. To improve the accuracy of PTQ, much effort
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

layer 1
block
layer
layer
layer 2
block
layer
layer
model
layer
layer N
block N
layer
layer-wise
(b) block-wise
(c) network-wise
Figure 1: layer-wise block-wise network-wise. Gray nodes frozen while highlighted nodes are learnable (quantization parameters only). term block here denotes residual unit in modern CNN (e.g. Residual Bottleneck for ResNets).
has been devoted [5, 32, 34, 35, 39]. [daRound[32] has recently discovered that round-to-neares. appears to be optimal for single weight but not for the entire layer. The rounding errors of weights in a single layer have the capability of compensating for each other, contributing to smaller layer-wise quantization error. We term this property intra-layer dependency. Motivated by this property, AdaRound[32) proposes to learn the round policy layer by layer to improve accuracy.
Inspired by AdaRound[32], able to compensate for each other, resulting in a reduced quantization error. We refer to this capability inter-layer dependency. However, [d] assumes that layers are mutual-independenn and derives a layer-diagonal Hessian, leading to a layer-wise solution(Figure la). BRECQ[27] points that the dependency should not be ignored and the assumption of AdaRound is not accurate, they instead utilize the inter-layer dependency within each block to build biock-wise approach(Figure 1b). However, BRECQ[27] assumes that blocks are nutual-independeer argue that neither the approximations of AdaRound nor that of BRECQ are accurate enough, the lnter-layer dependency should be leveraged network-wise manner. To achieve this, we propose Network-Wii Quantization (NWQ, Figure Ic) scheme to fully leverage the inter-layer dependency. AdaRound and BRECQ separate a network into sub-nets and train them sequentially, this process makes the preceding layers unaware of succeeding layers. NWQ solves this problem via training the entire network end-to-end, where layers are aware of each other and they are able to cooperate for more optimal solution. However, quantizing the entire network end-to-end is combinatorial optimization problem where huge number of discrete parameters from all layers are combined together for optimization. This optimization problem will raise two major challenges that result in significant accuracy degradation.
hypothesize that the quantization errors of all layers in network
We
The first challenge is over-fitting, Extending the reconstruction unit from layer/block to entire network will increase parameters dramatically and thus raises the risk of over- over-fitting over typically small calibration set. To alleviate this issue, NWQ applies Activation Regularization (AR), a variant of L2 regularization, on intermediate representation to keep the quantized activations as close to the foating-point counterpart as possible. The benefits of AR are two folds. First, AR regularizes quantized activation flow to prevent over-fitting Second, works as a quantization loss function to encourage minimal intermediate quantization error. AR turns out to be both simple and effective.
Another challenge is discrete optimization. Based on AdaRound, NWQ learns the weight round policy and activation quantization parameters, both of which involve in discrete variables. However, the substantially increased depth and discrete variables make it more difficult converge compared to layer-wise[23, 32] and block-wise[27, 40] approach. The insufficient convergence will leave the discrete variables (weight round policy) in a state of continuity rather than discretization, leading to great performance drop. To solve this problem, we propose Annealing Sofmtax (ASoftmax) and Annealing Mixup (AMixup) to better optimize discrete weights and activations. Specifically, ASoftmax provides mechanism to guarantee the discritization of round policy learning via annealing temperature. AMixup resolves the train-test inconsistency problem indtroduced by QDROP[40] via gradually decreasing the doating-poin percentage in mixed activations.

---

Equipped with AR, ASoftmax and AMixup. we are able to build our NWQ framework that fully leverages inter-layer dependency to achieve higher accuracy than existing works.
To summarize, this paper makes the following contributions:
1. We propose a novel network-wise quantization framework for PTQ to fully leverage inter- layer dependency.
2. We analyze the combinatorial optimization problem introduced by network-wise quan- tization and propose AR, ASoftmax and AMixup to solve the over-fitting and discrete optimization problem.
3. We establish a new sfate-of-the-ar through significantly improving PTQ by large margin: 20.24% for MobileNetV2 with 2 bits and thus push the limit of extremely low-bit PTQ from feasibility to usability.
2 Related Work
Network quantization is commonly divided into QAT [7, 15, 22, 25, 48] and PTQ [27, 32, 33, 47J. QAT methods tend to minimize task error by training weights and quantization parameters jointly with the entire labeled dataset and thus require a large number of computing resources, whereas PTQ methods focus on finding optimal quantization parameters for off-the--shel well-trained networks with much less data and compute resources than QAT. PTQ can further be classified into statistic-based PTQ and searning-basedd PTQ. Statistic-based PTQ methods [33, 35, 47] first estimate the distribution of activation on calibration data, and then find the optimal quantization parameters that minimize local quantization errors heuristically. Learning-basee based PTQ methods[23, 27, 32, 40] dedicate to slightly weight tuning and quantization parameters learning. Our work follows the learning-base based
PTQ scheme.
Learning-based PTQ: [daRound[32] adopts the regime of learning the optimal round policy layer by layer. BRECQ [27] succeeds in extending AdaRound [32] to a block-wise reconstruction and concludes that block-wise quantization is superior to network-wise method. We argure that both layer-wise and block-wise approach is sub-optimal because the preceding layers/blocks are unaware of succeeding layers/block, thus the dependency they leveraged is unidirectional. Some works focus on the sequence of weight and activation quantization. AdaQuant [23] proposes to tune weights and activation quantization parameters separately for each layer to reduce over-fitting. while QDROP [40] finds that incorporating activation quantization into weight tuning can achieve higher accuracy.
Activiation Regularization: Large values in the learned dense representation can be sign of being overfit. To encourage small activations, [31] employs 1.2 regularization on hidden state of recurrent neural networks (RNN), while various normalizations [2, 24, 38, 42] focus on centralize the activation distribution to regularize large values. Some prior works attempt to learn sparse representation. Dropout and its variants [1, 10, 36, 37, 41] randomly drop activations from the neural network to encourage sparsity. ReLUs[16, 16, 20, 30, 43] dedicate to learning sparse representation via rectifying negative values. These techniques aim to regularize activations when training a neural network from scratch, which inapplicable for PTQ where the networks are off-the-sheif and well-trained, thus the activation distributions are deterministic and immutable. Instead of encouraging small and sparse activations, NWQ apply activation regularization on intermediate representation to encourage small offsets between quantized activations and the floating-poin counterpart. AdaRound and BRECQ [27] also employ quantization losses on intermediate activations but those losses works as quantization objectives rather than regularization, while the quantization objective of NWQ is to minimize the quantization error of the final network output and the regularization intermediate activations is to prevent optimization process from over-fitting.
Discrete Optimization for Quantization: Network quantization optimizes discrete variables via gradient backpropagation. However, the quantization process introduces a round function which is indifferentiable. Some works on QAT focus on approximating the gradient through the round function [3, 6, 13, 14, 17, 44]. For PTQ, AdaRound [32] introduces additional discrete parameters(the binary round policy) and optimize them via Rectified Sigmoid with an explicit Regularization (RSeR) loss function. However, RSeR is adversarial to quantization loss and thus difficult to convergence in network-wise guantization. QDROP [40] proposes to randomly drop quantized activations to improve
3

---

Layer 1
Layer N-1
Layer N Asoftmax
AMisup
AMixup
AMixup
ASoftmax
Asottmax
Layer 1
Layer N-1
Layer N
AR
NR
Lar
Lquant
loss
Figure 2: Overfiew NWQ. We use 1.2 loss both AR, and Iigantt The green is simulated quantization network and the orange is the floating-point counterpart. We freeze all weights and train weight round policy and activation quantization parameters only.
the flatness of optimization landscape, but introcues train-test inconsistency problem which results in performance degradation used in network-wise PTQ.
3 Approach
Network- Wise Quantization (NWQ) is simple end-to-end post-training quantization approach. The structure of NWQ illustrated Figure The optimization objective composed of output quantization loss and layer-wise AR loss. Meanwhile, the annealing approaches (Asoftmax and AMixup) are introduced in training process, which enable the quantized weights and activations to gently converge from continuous state to discrete state, lowering training complexity and improving accuracy.
3.1 Activation Regularization
NWQ leverages Activation Regularization (AR) to penalize quantized intermediate activations that substantially away from the floating-pon point counterparts, thereby encouraging the activations to remain unchanged.
Formally, we define AR by:
N Σ
Lor (G,
a arl
while G represents foating-poin network and G is the simulated quantized counterpart. az and T denote the maating-point and the simulated quantized activation of the I-th layer, respectively. N represents the number of total layers and ||-|| denotes the Frobenius norm.
Our network quantization error Liuant (T , aN) defined by:
Lquans GG a axl
then the final objective can be formulated as:
N
L(G, G) Lar + Lyuant
c
a1l
(2)
1=1
Practically, Tur is not restricted to be utilized layer by layer, it can also be performed block by block and stage by stage, as BRECQ [27] defined. We will discuss how the granularity affects the performance in Sec. 4.2. By default, employ layer-wise AR in experiments.

---

3.2 Annealing Softmax
For weight vector w, AdaRound [32] proposes to learn the round policy by:
W; clip (1 Wi
h(v,))
where [:] gives as output of the integral part and clip(-) clamps all values between q- and 4+- w; denotes the quantized weight for the i-th element of w. We will omit below for simplicity. s the quantizer step size. is a learnable continuous vector w.t. and h(v) is recitified sigmoid function (see (4)) which regularized to converge {0, 13.
1.2
h(v) clip( 1 exp( v)
0.1,0,1)
(4)
Based on (4n, AdaRound proposes Rectified Sigmoid with an explicit Regularization (RSeR) loss discrete h(v):
LASeR(h(v)) = -2h(() - 1
(5)
However, there are two limitations of RSeR. First, RSeR works only for binary round policy. Second, RSeR is adversarial to quantization loss. For example, for linear transformation network, the quantization loss is
Lg = || sw x|| || = =2 w
w
(6)
where x denotes the input and we ignore the clip function in Eqn. (3).
Eqn. (6) indicates that quantization loss will prevent LRSeR from convergence (i.e., LRSe R encour-
ages h(v) to converge to or while Ly prefers h(v) to remain w - [. Besides, this adversarial
become more severe when extending dayer/block-wis methods to network- wise quantization(see Sec.
4.2.2 and Table 4).
3.2.1 Formulation
To resolve the two problems of RSeR, we propose Annealing Softmax (ASoftmax). First, we extend the discrete optimization space of f(-)) from {0, 1} to K, where I. is integer range and defined by:
K. = [n, n + 1,..., m - 1, m,,,,, Zgo, m E Ezo,
where n and m are haper-parameters that specify the discrete optimization space. Then, ASoftmax can be formulated as:
w' clip (1 h' (V).
where V is continuous learnable matrix and represents the logits of probability over K. h²(V) is defined by:
ht hV( E E [
(9)
= KV,=K,VVrr') czp(V.//7)) S
(10)
where t represents for t-th iteration. r¹ is temperature that decays linearly transition h' hV(V) from continuity to discretization. For example, when r' is large. p(k/V, is continuous vector of probability and h (V) is the expectation of the discretization. When r¹ turns tiny, p(k|V, r²) becomes a one-hot vector and h (V) selects k by p(k VV r²) = 1. rf is simply decayed by:
7! TT 7
(11)
T
where 7 and TT are dyper-parameter initial and final temperature.
ASoftmax is general formation of learning wider range of discrete parameters than binary. It easily degenerates to round policy learning by setting n = 0 and m We use +? 0.01 and 70 by default.
5

---

3.2.2 Initialization
According to our observations, initialization has significant impacts on learning round policy. Given the weight quantizer step size s, which has been computed using statistic- cattstic-bbsed PTQ methods, we attempt to initialize V in order make s w as close to Doating-point as possible at the start. To achieve this goal, we first calculate the un-normalized probability by:
a' d'(V);
0 I
i
m n
(12)
K,
Then, we normalize the probability:
d'(V;; E oV())
o (V):
(13)
Finally, the initialization for V, is log(o((V))) It can be easily proved (see Appendix A) that the initialized 8 w equals w as long as the integer range K. is symmetric w.r.t. 0.5 (i.e. n = -m).
3.3 Annealing Mixup
NWQ considers activation quantization as discrete optimization problem and solves it via continuous relaxation. Different from ASoftmax which applied on element-wise level, AMixup works on eessor-wise level: we randomly mix up the floating-pont activations with the simulated quantized activations. To encourage continuous relaxation converge to discretization, NWQ progressively decreases the percentage of peating-point activations, so the mixed activations will become fully quantized activation gradually.
Formally, the percentage of floating p-int activations defined by:
P(() (Pe (P.-P.) P.:
(14)
where P(t) is the percentage of foating-poin activation in the mixed activation for the t-th iteration. P. and Pa denote the percentage at start and end respectively, and T is the number of iterations. We P. 0.5 and P. by default.
Mixxup similar to QDROP [40] in the mixup operation. The key difference, however, that AMixup introduces annealing process. This is very important for NWQ. QDROP trains with mixed activations while testing with fully quantized activations, thus introducing inconsistency bias. This bias accmulates network depth increases and leads great accuracy degradation in network- quantization. AMixup is able to eliminate the inconsistency bias via the annealing process.
4 Experiments
In this section, we conduct extensive experiments with various architectures to verify our NWQ. We first compare our method with previous state-of-the-art, and then we delve into our important designs and fully explore the scalability, effectiveness, and efficiency of our solution. We report top-1 classification accuracy on the ImageNet[9] validation set.
Implementation Details. Our code is based on open-source codes BRECQ and QDrop. We apply layer-wise AR. For ASoftmax, AMixup, we set default P; as 0.5 and P. as O. We randomly sample 1024 images from ImageNet train and employ Cutmix [45] and Mixup[46] data augmentation. The learning rates 0.01 for round policy and 0.0004 for activation quantizer step size. We train for 20000 iterations with a mini-batch size of 32 8 Tesla V100 GPUs, taking 30 minutes for ResNet18, which is on par with BRECQ and QDROP. Our experiments are conducted on 5 architectures, including ResNet1 8(Res 18), MobileNetV2(MNN3 and MnasNe((Mnas)) Other settings remain same as QDROP[40] if not specified.
set
m 1 and 5 is decayed from 1 to 0.01. For
BegNeet660MMFRRgg00000
BegNet-3.2GF(Reeg
4.1 Comprehensive Comparison
Table I compares our NWQ with several main-stream PTQ approaches. Overall, NWQ outperforms previous works across all neural network architectures and bit widths, and our performance improves more as the bit width decreases.

---

Bits(W/A)
Res18
MobileNetV2
Reg600MF 73.71
Reg3.2GF
MnasNet2.0
Full Prec
32/32
71.06
72.49
78.36
76.68
ACIQ-Mix* [35]
4/4
67.00
ZeroQ" [5]
4/4
21.71
26.24
28.54
12.24
3.89
LAPQ" [34]
4/4
60.30
49.70
57.71
55.89
65.32
AdaQuant" [25]
4/4
69.60
47.16
Bit-Split" [39]
4/4
67.56
AdaRound [62]
4/4
67.96
61.52
68.20
73.85
68.86
BRECQ [27]
4/4
69.60
66.57
68.33
74.21
73.56
QDROP" [40]
4/4
69.62
68.84
71.18
76.66
73.71
QDROPT [40]
4/4
69.88
69.15
71.45
77.02
74.09
NWQ(Ours)
4/4
69. 69.85=10.04
69. 69.14:1:.11 12
71.92:0.07
77.40=10.04
74.60=10.09
NWQ' (Ours)
4/4
69.95:00.07
69.60:10.03
72.03++0.09
77.42++0.03
74.86-10.03
AdaQuant* [23]]
3/3
60.09
2.23
AdaRound [32]
3/3
64.66
15,20
$1.01
56.79
47,89
BRECQ [27]
3/3
65.87
23.41
$5.16
$7.12
49.78
QDROP" [40]
3/3
66.75
57.98
65.54
72.51
66.81
QDROPT [40]
3/3
67.51
58.54
66.75
73.48
68.28
NWQ(Ours)
3/3
67.58:t0.14
61.24::0.21
67.38+t0. 13
74.79:10.09
68.85=10.18
NWQ' (Ours)
3/3
68.15:+00100
62.53+:0.05
68.31++0.09
75.44:10.08
70.77:+0.10
BRECQ" [27]
2/2
42.54
0.24
3.58
3.62
0.61
QDROP* [40]
2/2
54.72
13.05
41.47
55.11
28.77
QDROPI [40]
2/2
57.84
14.89
45.93
57.98
34.19
NWQ(Ours)
2/2
59.14=:0.05
26.42=1103
48.49:00.00
62.85++0.13
01.17=0.35
NWQ (Ours)
2/2
61.65:10.12
35.13::2.16
56.36:10.01
67.02:+0.01
53.55=10.20
Table Comprehensive comparison. report topl accuracy
ImageNet validation set. represents for reference from QDROP[40] and means scaling up the calibration set to 10240 images. QDROP results are produced by open-source codes from QDrop. We use n = 0, m = for NWQ and n -1, m = for NWQ We follow BRECQ[2]] to set the first and the last layer to 8-bit.
Classical Calibration Set. Following previous works, compare the performance small calibration set 1024 images. For start, we achieve 0.2% 0.9% improvement even compared with strong baselines W4A4, which rather close to full precision accuracy. For W3A3, NWQ yields 0.83% 3.26% and reduces the gap between W3A3 and W4A4. For the challenging W2A2, our method surpasses the previous state-of-the-an by large margin of 13.37% for MobileNetV2 and 12.4% for MnasNet, pushing the limit of extremely low-bit PTQ step further.
Scaled-up Calibration Set. To further explore the potential of NWQ, we scale up the calibration set by 10x and reproduce QDROP on the scaled-up calibration set, building a very strong baseline. Interestingly, NWQ outperforms QDROP even with one-tenth data. When equipping NWQ with the scaled-up calibration set, we observe significant improvement under all settings. Particularly, the accuracy gaps of W2A2 between NWQ and QDROP are enlarged even more the calibration set scales up. For example, the gap of MnasNet increases from 12.4% to 19.36%, and that of MobileNetV2 increases from 13.37% % to 20. 00.4%, showing the excellent scalability of method.
4.2 Ablation Study
18
AR Uk-train
ARval
w/o ARttrain
4.2.1 AR
web AR-val
loss000>
The Efffet of AR. We study the effect of AR via compar- ing the experiments with layer-wise AR to the experiments without AR on W2A2. Table shows that layer-wise AR improves the accuracy significantly. It's worth noting that laser-wise AR improves 5.22% MobileNetV2 and 10.42% for MetsNet. To understand how AR works, we analyze the loss curves during training in Figure First, AR experiment consistently converges lower loss
32
10
Iterattons10*
Figure zation loss curves on train set of 1024 images and valida- images.
MNV2 with W2A2. Quanti-

---

AR Granalizity
Resis
NNV2
RegooMM
Reg3.2G
Miass
AR
AA
Resi8
MNV2
RegA00M
Reg..2
Mans
byyer wise AR
59,14
26.12
45,19
62.85
41.17
57.51
21.20
46.82
59.82
30.75
block-wise
sage-wis
59.10
47.28
41.25
X
59.14
26.42
43.49
02.35
41.17
Wão AR
57.34
21.30
46.2
39.82
30.75
59.52
26.90
48.62
63.52
42,92
Table 2: AR vs. wlo AR on W2A2 with our NWQ appreach. Up: granularities Apply after each laye//block/suge).. Bottom: Experiments without AR.
Table 3: AR vx. Auto-ugment on W2A2, AutoAugment, we use the open-source codes from torchvision. short for AutoAug- ment.
Comparison
Method
RSRR
ASotima
1W
0.01
01
10
100
Initalization
BinsWWA)
KesNert8
MobuleNat
Res18
0.96
3449
34.75
46.36
44.54
05.83
59.14
random
2/2
55.33
22,00
MNV2
Table ASoftmax vs. RSeR on W2A2 with our NWQ approach. LW represents the regularization loss weight for AdaRound. We simply replace ASoftmax with RScR NWQ solution.
Table Comparison of different initialization methods. We compure random initialization with our method descrihed in Sec. 3.2.2.
both the train and test datasets, resulting in higher accuracy. Second, AR accelerates convergence significantly and thus allows NWQ to reach competitive accuracy at substantially lower computing cost. Third, even though AR does not totally eliminate over-fitting, effectively bridges the loss gap between train and test datasets.
The Effect of Granularity. Table 2 studies how AR granularity affects accuracy. AR granularity represents for applying AR after each layer/block/stage, while training(reconstructionn granularity represents for optimizing each layer/block/stage sequentially and independently. We discover that lightweight networks prefer coarse-grained AR, whereas heavyweight networks prefer fine-grained AR. MobileNetV2, RegNet-600MFF and MnasNet, for example, achieve the best performance with block-wise AR, whereas ResNet18 and RegNet-3.2GH achieve the maximum accuracy with layer-wise AR.
Comparison with Data Augmentation. Data augmentation is a classical strategy used to relieve over-fitting due its simplicity and effectiveness. We compare AR with strong augmentation (e.g. AutoAugmen((8) in Table 3. Notably, both AR and AutoAugment significantly improve accuracy, and AR contributes more than AutoAuument, partieularly for sparse networks like MobileNetV2 and MnasNet. Coupling these two techniques enables us to achieve better performance.
4.2.2 ASoftmax Study
The Effect of ASoftmax. Table compares ASoftmax to Rectified Sigmoid with explicit Regulariza- tion (RSeR) which used in AdaRound. We simply replace ASoftmax with RSeR in our NWQ and grid search regularization loss weight explore the potential of RSeR. We discover that RSeR requires a huge loss weight for NWQ, which 10 10 larger than that subnet-wise approaches. Even with exhausting hyper-paramenn parameter search, RSeR does not perform well as in the subnet-wise method. ASoftmax, on the other hand, outperforms RSeR by 12.58% and 21.566 1,56% on ResNet18 and MobileNetV2, respectively, without any additional hyper-parameter tuning.
The Effect Initialization. Table demonstrates the effectiveness of our initialization, which keeps the quantization model close the float-point at the start to steady the training. It is shown that our initialization method significantly outperforms random initialization.
The Robustness of r. We investigate the robustness of T in Table 6. starts with large number to encourage continuous relaxation and ends with a small value to favor one-hot distribution. The final -" determines the decaying speed of 7. It can be observed that 7" has robustness range of 10 10 making it casily adaptable a wider range of neural network architectures.
Rest8
MNV2
Reg6iKIM
Reg3.26
Mnas
Almages
n/m
Res18
MNV1
Rep500M
Reg3.26
Mnus
59.08
12.19
47.84
60.10
33.92
1024
0/1
59.14
36.42
45.49
62.85
41.17
-1/2
41.07
39.05
26.75
49.77
62.09
10240
0/1
61.61
29.34
53.83
67.02
51.15
16-4
58.98
24.74
49.42
62.06
41.32
102.0
-122
61.65
35.13
56.36
67.02
$3.55
Table 6: Robustness of for ASoftmax on W2A22 Experiments are conducted
Table Extensiveness of ASoftmax on W2A2, MNV2 short for MobileNer122.
1.0

---

Melood
Reshei18
MobucNetV2
Rests
MIN2
RegbRRMM
Kegs
Minas
ODROP-syle
22
30.47
3.15
59.47
25.34
50.53
63.06
40.35
58.12
15,09
50.05
40.66
AMaxup
22
59.12
26.42
0.5
59.14
26.42
48.49
62.85
41.17
0.4
59.06
08
25.79
83
61.73
40.63
Table 8: QDROP-slyle rx. w/o Drop v. AMixup on W2A2 with our NWQ appeouch. QDROP represents for droppin methods QDROP coupled with NWQ. w/o Drop represents for training without dropping mixup operation.
Table 9: Decay Policy of AMixup W2A2. use P. of the experiments.
for all
The Extensiveness of ASoftmax. As described in Sec.3.2.1 ASoftmax can easily extend the discrete parameter space from {0, 1 to a larger integer range, offering the opportunity for a better optimum. However, extending discrete range benefits MobileNetV2 and RegNet-600MP while degrading RegNet-3.2GF on small calibration set, as shown in Table upper group. We argue that the enlarged discrete range raises the risk of over-fitting for large networks. When scale up the calibration set, the performance no longer degrades and the performance improves even morc.
4.2.3 AMixup Study
The Effect of AMixup. QDROP presents good performance in block-wise reconstruction, but fails our network-wiss scheme. As illustrated in Table 8, we achieve 1.95% improvement for ResNet18 and 19. 19.96% for MobileNetV2 by simply removing QDROP (w/o Drop row). We hypothesize that randomly mixing activations up introduces inconsistency bias between the training and test phase, which is concealed by the improvement of flatness in block-wise reconstruction. However, as the reconstruction unit goes deeper (from block to network) and sparser, the problem becomes more pronounced, resulting in significant degeneration. Dver-parameterized ResNet18 is able to compensate for the side effect, but MobileNetV2 suffers a great performance drop. Remarkably, our AMixup is able to overcome the inconsistency while taking advantage of the flatness, contributing to higher accuracy.
The Policy of Decay. In Table 9, we run a grid search for P. to explores the influence of different decay policies. It's worth noting that MobileNetV2 and MnasNet prefer 0.5, whereas ResNet18 and RegNet prefer 1. The basic law is very similar to that of AR granularity, which states that heavyweight networks require more regularization, implying that AMixup and AR are both capable of regularizing accuracy.
4.3 Efficiency
We compare the efficiency of several approaches on ResNet18. According to Figure 4, NWQ out- performs QDROP and BRECQ under various it- erations. Specifically, NWQ achieve competitive results with only 10% computation cost, demon- strating the efficiency of NWQ. As the number iterations increases, NWQ continues to improve accuracy, whereas BRECQ and QDROP reach their saturation
60
56.04
57.76
58.13
58.92
59.14
53,24
52.89
53.74
54.33
54,28
Accuraryy%
49.72
.50
45
43/47
42,46
42,522
42.58
41.198
39.73
NWQ[Ours) QDROP
35,a1
35
BRECO
1.6 x 10 iterations.
2
8
12
16
20
horations/10
Discussion and Conclusion
Figure Accuracy wind iterations, experiments are con- ducted ResNet18 with W2A2.
Simple algorithms that scale well are the core of deep learning. Prior works tend to quantize net- works in layer-wise and block-wise manner. We prove that network-wist quantization is a simpler and more effective way than layer/block introduced by the network-wise regime and propose three key designs to address them. Based on that, we construct a simple and effective end-to-end network-wise PTQQ framework. Extensive experiments demonstrate that our solution is scalable, effective, and efficient. However, there is still a large accuracy gap between low-bit PTQ and full precision network, and we believe that there is still much room for further improvement for PTQ, which is key direction for future research.
solutions. accomplish this, we analyze the problems

---

Broder impacts. The proposed method uses GPU resources for training, thus resulting in a certain amount of energy consumption and carbon footprints. For example, takes about GPU-hours for our method to quantize ResNetl 8, which would consume around kWh and produce around lbs of CO2, accelerating climate change and global warming some degree.
References
[1] Jimmy Ba and Brendan Frey. Adaptive dropout for training deep neural networks. Advances in neural information processing systems, 26, 2013.
[2] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. 1xXiv:607.004555 2016.
arXiv preprint
[3] Yu Bai, Yu-Xiang Wang, and Fdo I. Liberty. Proxquant: Quantized neural networks via proximal operators. arXiv preprint arXiv: /ix:v:/8100/ /810.00861, 2018.
[4] Han Cai, Chuang Gan, Tianzhe Wang, Zhekai Zhang, and Song Han. fore-for-all: Train one network and specialize it for efficient deployment. In International Conference Learning Representations, 2019.
[5] Yaohui Cai, Zhewei Yao, Zhen Dong, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Zeroq: A novel zero shot quantization framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13169-13178, 2020.
[6] Shangyu Chen, Wenya Wang, and Sinno Jialin Pan. Metaquant: Learning quantize by learning to penetrate non non-differentiable quantization. Advances in Neural Information Processing Systems, 32, 2019.
[7] Matthieu Courbariaux, Yoshua Bengio, and Jean-Pierre David. Training deep neural networks with low precision multiplications. arXiv preprint arXiv: 1412.7024, 2014.
[8] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoe V Le. Autoaugment: Learning augmentation policies from data. arXiv preprint 18Xiv: 1805.09501, 2018.
[9] Deng. Wei Dong. Richard Socher, Li-Jia Li, Kai Li, Fei-Fei. Imagenet: large-scale hierarchical image database. In 2009 IEEE conference on computer vision and puttern recognition, pages 248-255. leee, 2009.
[10] Terrance DeVries and Graham Taylor. Improved regularization convolutional neural networks with 2017.
cutout. arXiv preprint
[11] Xuanyi Dong. Lu Liu, Katarzyna Musial, and Bogdan Gabrys. NATS-Bench: Benchmarking nas algorithms for architecture topology and size. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021. doi:10. 1109 APMMI 30548244
[12] Xuanyi Dong and Yi Yang. Network pruning via transformable architecture search. In The Conference on Neural Information Processing Systems (NeurlPS), pages 760-771, 2019.
[13] Zhen Dong, Zhewei Yao, Daiyaan Arfeen, Amir Gholami, Michael Mahoney, and Kurt Keutzer. Hawq- v2: Hessian aware weightee quantization of neural networks. Advances in neural information processing systems, 33:185188 8518-18522 2020.
[14] Zhen Dong, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Hawq: Hessian aware quantization of neural networks with mixed-precision. In Proceedings of the IEEF/CVE International Conference on Computer Vision, pages 293-302, 2019.
[15] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmendra Modha. Learned step size quantization. In International Conference on Learning Representations, 2019.
[16] Xavier Glorot, Antoine Bordes, Yoshua Bengio. Deep sparse rectifier neural networks. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 315-323. JMLR Workshop and Conference Proceedings. 2011.
[17] Ruihao Gong, Xianglong Liu, Shenghu Jiang, Tianxiang Li, Peng Hu, Jiazhen Lin, Fengwei Yu, and Junjie Yan. Differentiable soft quantizaaion: Bridging full-precision and low-bit neural networks. In Proceedings of the IEFF/CVF International Conference on Computer Vision, pages 4852-4861, 2019.
[18] Shaopeng Guo, Yujie Wang. Quanquan Li, and Junjie Yan. Dmep: Differenniablee markov channel pruning for neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
10

---

[19] Song Han, Huizi Mao, and William J Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding, International Conference Learning Representations (ICLR), 2016.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, In Proceedings of the IEEE international conference on computer vision, pages 1926-10344 2015.
[21] Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. In NIPS Deep Learning and Representation Learning Workshop. 2015.
[22] Iaa Hubara, Matthieu Courbariaux, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Quantized neural networks: Training neural networks with low precision weights and activations. The Journal of Machine Learning Research, 18(1 (11):6666-6888, 2017.
[23] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Improving post training neural quantization: Laycr-wisc calibration and integer programming. arXiv preprint arxiv:2006. 10518, 2020.
[24] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448 456. PMLR, 2015.
[25] Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer- onilh inference. In Proceedings of the IEEE conference on computer vision and puttern recognition, pages 2704 2713, 2018.
[26] Raghu Krishnamoorthi. QUANTIZATION IN PYTORCH. https://slidetodoo.. com/ quantization- in- pat-rrctrrrrhuuu--... kri shnamoorthi www.ooorrttt-cccbbbooo facebook quantization-n/ 2019. [On- line; accessed
[27] Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang. Fengwei Yu. Wei Wang, and Shi Gu. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint 1rXiv:2102.05426, 2021.
[28] Darryl Lin, Sachin Talathi, and Sreekanth Annapureddy. Fixed point quantization of deep convolutional networks. International conference on machine learning, pages 2849-2858. PMLR, 2016.
[29] Hanxiao Liu, Karen Simonyan, and Yiming Yang. Darts: Differentiable architecture search. International Conference on Learning Representations, 2018.
[30] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. Rectifier nonlincarities improve neural network acoustic models. In Proc. ieml, volume 30, page 3. Citeseer, 2013.
[31] Stephen Merity, Bryan McCann, and Richard Socher. Revisiting activation regularization for language rans. arXiv preprint uxXiv:1708001000 2017.
[32] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pages 7197- -206. PMLR, 2020.
[33] Markus Nagel, Mart van Baalen, Tijmen Blankevoort, and Max Welling, Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1325-1334, 2019.
[34] Yury Nahshan, Brian Chmiel, Chaim Baskin, Evgenii Zheltonozhskii, Ron Banner, Alex M Bronstein, and Avi Mendelson. Loss aware post-training quantization. Machine Learning, 110(11))3222 :)3245-3222, 2021.
[35] B Ron, J Perera, E Hoffer, and D Soudry. Aciq: Analytical clipping for integer quantization of neural networks. arXiv preprint arXiv: 1810.052,, 2018.
[36] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-195) 2014.
[37] Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann LeCun, and Christoph Bregler. Efficient object localization using convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 648-656, 2015.
11

---

[38] Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.00022, 2016.
[39] Peisong Wang. Qiang Chen, Xiangyu He, and Jian Cheng. Towards accurate post-training network quantization via hit-split and stitching. In International Conference Machine Learning, pages 9847- 9856. PMLR, 2020.
[40] Xiuying Wei, Ruihao Gong, Yuhang Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit bit post-training quantization. arXiv preprint arXiv:2203. 057440, 2022.
[41] Haibing Wu and Xiaodong Gu. Max-pooling dropout for regularization convolutional neural networks. In International Conference on Neural Information Processing, pages 46-54. Springer, 2015.
[42] Yuxin Wu and Kaiming He. Group normalization. In Proceedings of the European conference on computer vision (ECCV), pages 3-19, 2018.
[43] Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolu- tional network. arXiv preprint arXiv: 505,00053, 2015.
[44] Jiwei Yang, Xu Shen, Jun Xing, Xinmei Tian, Houqiang Li, Bing Deng, Jianqiang Huang, and sian-sheng Hua. Quantization networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 7308-7316, 2019.
[45] Sangdoo Yun, Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023 6032, 2019.
[46] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk 2017.
minimization. arXiv preprint
[47] Ritchie Zhao, Yuwei Hu, Jordan Dotzel, Chris De Sa, and Zhiru Zhang. Improving neural network quantization without retraining using outlier channel splitting. In International conference on machine learning, pages 7543-7552, PMLR, 2019.
[48] Shuchang Zhou, Yuxin Wu, Zekun Ni, Xinyu Zhou, He Wen, and Yuheng Zou. Dorefa-net: Training low bitwidth convolutional neural networks with low bitwidth gradients. arXiv preprint 1rXiv:1606.06 2016.
[49] Barret Zoph and Quoe V Le. Neural architecture search with reinforcement learning. arXiv preprint arXiv: 161166110000 1.0578,, 2016.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See broader impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yss]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [No] Our method is easy to implement based on the baseline codes specified in our paper, and the data is ImageNet KK which publicly available.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Sec. 4
12

---

(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes] We report error bars Table 1.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [No] We do not introduce new assets in our method.
(d) Did you discuss whether and how consent was obtained from people whose data you're
using/curating? [No]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [No] We use public datasets only.
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
A ASoftmax Initialization
First of all, We define the fractional part of x {z} [x]] and we use w to represent initially simulated quantization of w. According to equation (9),(1)) (12) and (13), we get:
h (V) E (.rtt) k
m n
Krj 00 (V):
1=0
m n
K₁ x 00 (V);
m- n (V); 1=
1=
1
1
K4 * (v(: + (1- Ki)) * (0(v) m n
-
( (V)
K₁ +
1- Kr
Kc;
- Kr
{
(1 2KC)){
}
w
8
(15)
W
K1, - K2- -
8
1 1-2K;
2K,
(1 - K;) - K;
5
13

---

1
1
+
===
8
W
K2
1
Kr
{ w
11
(V());
D n 00 (V),
we disregard the clip function, then
w
w 8
(1 + 110 (v)) 5
W
(1 W
w
+ 1)
(16)
8
w
w
14


Reviews:
[{"review_id": "gMP-QWJpBp4", "reviewer": "Reviewer_NNev", "summary": "### Overall:\nTo avoid the overfitting problem in quantized networks, the authors present several training strategies based on the previous Post-training Quantization (PTQ) methods, such as layer-wise activation fitting as proposed in BRECQ [1], Annealing Softmax (AS) to enlarge the searching space of AdaRound [2], and Annealing Mixup (AMixup) to smooth the training process of QDrop [3]. They conduct thorough ablation studies and experiments on several mainstream networks to fully verify the superiority of the proposed methods. \n\n### Reference:\n[1] Brecq: Pushing the limit of post-training quantization by block reconstruction. ICLR2021\n\n[2] Up or down? adaptive rounding for post-training quantization. ICMLR2020\n\n[3] Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. ICLR2022", "questions": "### Major concerns:\n- It seems that the proposed Activation Regularization (AR) shares the same idea with BRECQ [1]. The ablation study on layer-wise/block-wise/stage-wise fitting can also be found in [1]. Why do the authors list AR as the contribution of this paper? What is the definition of \"inter-layer dependency\"? What is the relationship between \"inter-layer dependency\" and the overfitting problem?\n- The proposed methods heavily rely on hyper-parameter tuning, as reported in Section 4.2. I do notice that the authors conduct ablation studies in most settings. However, I doubt the search space can be too large. Did the authors apply the same setting (i.e., same AR, $\\tau^T$, $n/m$, $P_s$) to all experiments reported in Table 1? Besides, did the authors conduct ablation studies using the validation set and also report the final results on the validation dataset in Table 1? \n- It is interesting to see that the $59.14, 26.42, 48.49, 62.85, 41.17$ widely exist in Tables 2-8. How to determine the best setting of AR, $\\tau^T$, $n/m$, and $P_s$ simultaneously?\n\n### Minor concerns:\n- It seems that the selection of $n/m$ is quite sensitive to the size of the calibration dataset. Besides, the authors may further explain the reason for using $0/1$ instead of $-1/2$.\n- I am not sure that it is reasonable to use 10240 calibration images in the circumstances of post-training quantization. It would be better to include a simple baseline utilizing naive Knowledge Distillation (KD) based on 10240 images to fine-tune a quantized network.\n- Table 5: I am curious why ASoftmax with an integer range of $\\\\{-1,0,+1,+2\\\\}$ outperforms RSeR [3] with $\\\\{-1, 0, +1\\\\}$ in 2-bit. Note that 2-bit weights only contain $\\\\{-2, -1, 0, +1\\\\}$. The results listed in Table 5 seems to indicate that ASoftmax turns some $-2$ and $-1$ into $0$ and $+1$, is that correct? It would be better to include $n/m=-1/+1$ to make a fair comparison with RSeR.\n- The QDrop [2] results reported in Table 4 seem inconsistent with Table 1 and Res18 AMixup with 59.12% Top-1 Acc. is slightly lower than Table 2.\n- The ablation study on the **Initialization** of ASoftmax seems missing.\n\n### Reference:\n[1] Brecq: Pushing the limit of post-training quantization by block reconstruction. ICLR2021\n\n[2] Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. ICLR2022\n\n[3] Up or down? adaptive rounding for post-training quantization. ICMLR2020", "limitations": "### Comments on limitations:\nAs claimed in the **Abstract**, \"This process pays little attention to the dependency across the sub-nets, hence is less optimal.\" However, the main idea has been rarely discussed in the current manuscript, which makes the submission serve more like a technical report (i.e., the descriptions and effect of different improvements). The authors are encouraged to solve the major concerns in the final version.", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n- The experimental results seem strong. The authors conduct extensive experiments on ImageNet with different bitwidth. Especially, the ablation studies are quite clear, which should be encouraged.\n- The technique part of this paper is well written. I personally think this paper is simple and easy to follow.\n- \"Annealing Softmax\" seems new in the community of model compression.\n\n### Weaknesses:\n- Please refer to *Questions* and *Limitations*. The main concern is the novelty of this paper. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "c32t1OI2a2-", "reviewer": "Reviewer_GPuz", "summary": "Based on the inter-layer dependency of quantization, this paper proposes a Network-Wise Quantization (NWQ) approach. NWQ arises overfitting and discrete optimization problems. \nIn this paper, activation regularization is introduced to handle the overfitting issue. To solve the combinatorial optimization problem,  the authors use Annealing Softmax and Anealing Mixup.\nThe experiments show that NWQ can achive significant improvements over previous works.\n", "questions": "According to paper of AdaRound and Brecq, the inferred layer-wise frobenius loss is based on the hypothesis that the Hessian matrix H is diagonal, which means that the optimization of each layer are independent. Moreover, Brecq has validated the finer granularity brought better quantization quality, which contradict with the idea of inter-layer dependency in this paper, can you explain more on that part?", "limitations": "Limited technical contribution. The proposed methods are largely based on previous papers. For the new techniques, no theoretical analysis is provided. For example, no theoretical analysis of ASoftmax regarding the better performance compared with RSeR.", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nPTQ is widely studied, and the methods proposed in this paper outperfome previous SOTA by a large margin. The paper is easy to follow and the authors provides several ablation study to show the effectiveness of proposed techniques.\n\nWeakness:\nThe paper shows some originality but is not novel enough. For example, Annealing Softmax and Annealing Mixup are largely based on AdaRound[32] and QDROP[41]. The authors clearly state the difference and demonstrate the effectiveness of modifications, but previous methods play a more important role in the discrete optimization problem.  Besides, the motivation and theoretical analysis of substituting RSeR with ASoftmax are absent.\n\nFrom finger 3, we can learn the efficiency of NWQ. However, the cost of each iteration is not clear. Does it require less or more computation/memory than previous methods in each iteration.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "HqLmHm7bZrO", "reviewer": "Reviewer_kWyx", "summary": "The paper studies post-training quantization, and propose to leverage inter-layer dependency to improve the post-training quantized network. The authors introduce a number of approaches: AR, ASoftmax, improved initialization and AMixup, and verify the effectiveness of each part accordingly. The experimental results demonstrate superiority over a number of previous strong PTQ baselines across various network architectures and quantization configurations. In summary, despite the paper is methodologically incremental compared with past literature, the experiments are thorough and solid.", "questions": "- While the paper is entitled with \"leveraging layer-wise dependency for PTQ\", layer-wise training still achieves the best on 2 out of 5 architectures according to Table 2. I wonder what is the effect to prolong the training iterations? Will it allow the network adjust itself to better leverage the inter-layer dependency, and alleviate the convergence issue (all variables are discretized instead of being left in continuity)?\n\n- The empirical effect of initialization (proposed in Section 3.2.2.) is missing?\n\n- According to Table 7, ASoftmax does not bring consistent improvement with 1024 images, and may even suffer from overfitting (L254). Is there a systematic way to determine the optimization space with the training size? Additionally, have you tried larger discrete optimization space given 10240 images?\n\n- Typo: in L159, $\\hat$ is placed improperly for $s\\cdot w^0$.\n\n- Equations should end up with \",\" or \".\".", "limitations": "The authors improves the performance of post-training quantized networks under low-bit configurations. Despite there is still a large gap with the full-precision model, the empirical improvement in this paper is still an important step for futural PTQ research.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\n\nStrengths:\n\n\n- The major strength for this paper should be the empirical results, which outperforms a number of strong baselines and pushes the limit of PTQ performance on low-bit quantized networks. The ablations are also thorough and complete.\n\n- The writing is clear and easy to follow.\n\n\nWeakness:\n\n- The paper is methodologically incremental as a large portion is built upon past efforts (BRECQ and QDROP). Activation regularization resembles knowledge distillation, a widely adopted approach in PTQ approaches utilizing calibration dataset. Asoftmax follows AdaRound and extends the range of discrete space in $h(v)$. Annealing mixup considers to add annealing process into the mixture of quantized and full-precision activations. \n ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "1SeAfwinrQs", "reviewer": "Reviewer_N5CW", "summary": "This paper proposes a post-training quantization (PTQ) technique that aims to search for a network-level quantization policy, compared to the previous works that only considers quantization policies on a layer-wise level. To tackle the challenges of solving a larger scale combinatorial optimization problem, and the issue of overfitting, the paper proposes two approaches: Activation Regularization, and Annealing Softmax and Annealing Mixup. Experiments show the proposed approach achieved higher performance especially when quantizing networks to extremely low bits. ", "questions": "When explaining the effect of AR, the author mentioned two roles: 1/ preventing overfitting and 2/ preventing the quantized values deviating from the floating point counter part. I can understand the effect of 2/, but 1/ is not clear to me. Are there more evidence supporting why overfitting is an issue? ", "limitations": "1/ Seems that the performance gain is not consistent. The performance gain of the proposed method is significant for some configurations, e.g., MobileNet with 2 & 3 bit quantization. However, the gain is quite small for others, such as MobileNetV2 with 4-bit quantization. Why is that? Is it simply because MobileNetV2 2&3 bit configurations do not have a strong baseline?\n\n2/ The proposed method seem to contain too many hyper-parameters, from the granularity of AR, to initialization scheme, to annealing schedules, etc. How difficult is it to find a good configuration for a new model and new target bits?\n\n", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1/ The insight of this paper is straight-forward but very sensible -- quantization errors can cancel each other, and optimizing with a higher granularity can achieve better performance. More importantly, based on the insight, the authors proposes effective approaches that achieved this.\n\n2/ The paper is clearly written. The derivation and introduction of the techniques are explained very clearly. \n\n3/ Nice ablation study shedding lights on the hyper-parameter sensitivity of the proposed method. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept