From Decoupling to Adaptive Transformation: a Wider Optimization Space for PTQ
Authors: Zhaojing Wen
Qiulin Zhang
Yuan Zhang
Rudan Chen
Xichao Yang
Di Xie
Jiang Zhu
Venue: ICLR 2025 Poster (2025)
Keywords: Post-Training Quantization
Primary area: optimization

Abstract: Post-Training low-bit Quantization (PTQ) is useful to accelerate DNNs due to its high efficiency, the current SOTAs of which mostly adopt feature reconstruction with self-distillation finetuning. However, when bitwidth goes to be extremely low, we find the current reconstruction optimization space is not optimal. Considering all possible parameters and the ignored fact that integer weight can be obtained early before actual inference, we thoroughly explore different optimization space by quant-step decoupling, where a wider PTQ optimization space, which consistently makes a better optimum, is found out. Based on these, we propose an Adaptive Quantization Transformation (AdaQTransform) for PTQ reconstruction,  which makes the quantized output feature better fit the FP32 counterpart with adaptive per-channel transformation, thus achieves lower feature reconstruction error.  In addition,  it incurs negligible extra finetuning cost and no extra inference cost. Based on AdaQTransform, for the first time, we build a general quantization setting paradigm subsuming current PTQs, QATs and other potential forms. Experiments demonstrate AdaQTransform expands the optimization space for PTQ and helps current PTQs find a better optimum over CNNs, ViTs, LLMs and image super-resolution networks,  e.g., it improves NWQ by 5.7% on ImageNet for W2A2-MobileNet-v2.

Published as a conference paper at ICLR 2025
FROM DECOUPLING TO ADAPTIVE TRANSFORMATION: A WIDER OPTIMIZATION SPACE FOR PTQ
Zhaojing Wen", Qiulin Zhang', Yuan Zhang, Rudan Chen, Xichao Yang, Di Xie, Jiang Zhu Hikvision Research Institute, Hangzhou, China (wenzhao ingg, zhangqiulins zhangyuan, chenrudan, yangxichao, xiedi, zhujiang haaaan...ri).i.. http://iiiiiiinncoooomo com
ABSTRACT
Post- Post-Training low-bit Quantization (PTQ) is useful to accelerate DNNs due to its high efficiency, the current SOTAs of which mostly adopt feature reconstruction with self-distillation finetuning. However, when bitwidth goes to be extremely low, find the current reconstruction optimization space is not optimal. Considering all possible parameters and the ignored fact that integer weight can be obtained early before actual inference, we thoroughly explore different optimization space by quant-step decoupling, where a wider PTQ optimization space, which consistently makes a better optimum, found out. Based on these, we propose an Adaptive Quantization TransformatonnaaaaCrratottooo (daQTrassform) for PTQ reconstruction, which makes the quantized output feature better the FP32 counterpart with adaptive per-channel transformation, thus achieves lower feature reconstruction error. In addition, it incurs negligible extra finetuning cost and no extra inference cost. Based on AdaQTransform, for the first time, we build general quantization setting paradigm subsuming current PTQs, QAT's and other potential forms. Experiments demonstrate AdaQTransform expands the optimization space for PTQ and helps current PTQs find better optimum over CNNs, ViTs, LLMs and image super- resolution networks, e.g., it improves NWQ by 5.7% ImageNet for W2A2- MobileNet-v2. Codes are available at ahttps//tttnu.coom/////////AAaaa
INTRODUCTION
Low-bit model quantization (quant) generally consists of Quantization-Aware Training (QAT) and Post-Traine Post-Training Quantization (PTQ). PTQ almost the first choice for fast model quantization since does not require the full training pipeline like labeled training data. Traditional PTQ (Krishnamoorthi 2018) searches quant parameters through Mean Squared Error(MSE). which suffers from severe accuracy drop in 4 or 2 bits. With the help of gradient descent through self-distillation, current PTQs gradually narrow the accuracy gap, such as AdaRound (Nagel et al. 2020) for weight quantization, and BRECQ (Li al. 2021) / NWQ (Wang al.) 2022a) for activation quantization. In quant params optimization, we can see they evolve closer and closer towards QAT, except tiny unlabeled calibration set and need for original FP32 training pipeline. Their process shown in (a) and
(b) of Figure I (i) freezes Conv's 's FP32 weight and bias; (ii) takes FP32 weight and FP32 activation as ground truth since PTQ owns no labeled dataset; (iii) iteratively searches an optimal weight's quant-step through the quant-error between FP32 weight and quantized weight, then freezes and (iv) PTQ reconstruction: optimize weight's 's AdaRound up-or-down parameter a and activation's 's quant-step s, per layer/blockknetworkk through quant- between FP32 and quant output activation.
For weight quantization AdaRound, it optimizes up-or-down parameter a with weight's quant-step frozen to ensure a fixed integer base thus it can make a stable up or down rounding learning. However, differing from that integer activation has to be computed online during inference, integer weight obtained early before inference and is fixed during inference, described in Formula With this ignored fact, still under fixed integer weight base, there exists a wider optimization space for the de- de-quantized FP32 weight, making it not limited to #1 quant-step optimization distance, thus
"Equal Contribution 1 "oorreponndig Author

---

Published as a conference paper at ICLR 2025
forwrd
of
Coverie Vodel [:] Cass 2.2: Ours on PTQ
(4) Case 1.1.1: Adalloond and BREEQ
[8) Case T.EE GUMEP and NWV
Casc 12:150
Figure 1: Different quantization settings with 'Case x.X.x' as detailed on S.2 W/X/sw/a/s: denote weight/input/weights ss guant-step/weight denote FP32/quantized values. The solid/dotted arrow line denote quantization forward/backward process. Note (c), we decouple the single Sw of (b) into quant-step Sw and de-quant-step s'y.
adaround param/X's 's quant-step. Blue/orange box
lower reconstruction error might be met. That is, as shown in Formula after computing the integer weight using 8w during fake quantization simulation (quant-process), the 8w used to convert integer weight into the FP32 counterpart (dequant-procee process) can be different. Or to say, the conventional weight's 's quant-step sw can be decoupled apart quant-step sw and dequant-step s'y, according to their different functions. The adaround learning process is still stable since the adaround learning base, integer weight, is still fixed. Although it is safe decouple, decoupling? Further, now that PTQ evolves close to QAT, what will we gain if we directly optimize weight's quant-step into PTQ reconstruction like QAT as (d) of Figure [ Considering all above, we thoroughly explore the setting of weight's quant-step into six possible cases through decoupling over various networks 3.2 where Case 1.1 the current PTQ BeaiiingAAdRRondBBRRCCCCC
obtain accuracy gain through
Case 1.2 is the current QAT's s weight quant-step setting like LSQ (Esser et al.] 2020) applied on PTQ reconstruction. We experimentally find new setting Case 2.2, where we decouple quant-step, freeze integer weight and jointly optimize dequant-step as Formula 6 consistently performs the best.
At deeper optimization side, as Figure 2 Case 2.2 achieves a wider optimization space and makes quant output better fit the FP32 counterpart. The decoupled de-quant step s'g of weight finally achieves adaptive per-channel transformation on the output feature. Indeed, through visual and theoretical analysis, Case 2.2 equals to an adaptive per-channels lincar transformation directly output feature towards its FP32 counterpart. It is different from the common per-channel quant-step or offset like LSQ LSQ+ (Bhalgat et al. 2020) applied on PTQ. Because they does not change the distribution of output feature, which only optimizes the current input while ignores current PTQ's s output feature reconstruction. It is also different from SmoothQuant (Xiao al.) 2023), which transforms quant difficulty from activation to weight while also does not change the distribution of output. In addition, current PTQs freeze Conv's bias away from PTQ reconstruction. We find finetuning bias helpful, which provides further translating on quant output to narrow quant-error gap.
Considering above, we find the decoupling Case 2.2 and learnable bias b can be derived into more general form with introducing a scaling, e, and a translating factor, n7, into current quantization settings, namely AdaQTrofform, which is an easy integration to current PTQs. It incurs negligible extra finetuning cost and no extra inference cost. As Sec 3.3.2 AdaQTransform also subsumes normalization layer as its special form, but ours can be applied networks/layers with or without normalization, e.g., low-level-vision-task nets like image super-resolution EDSR (Lim et al. 2017).
More importantly, AdaQTransform helps to build a general unified paradigm subsuming quantization settings of current PTQs, QATs, our new decoupling case and other cases. The detailed expansions from the unified paradigm to specific quant-settings can be seen in See 3.3 Our contributions are:
We explore different optimization space by quant-step decoupling for PTQ. Based on the explo- ration, we propose AdaQTransform, which directly transforms the quant output closer to the FP32 counterpart(groung truth), making wider optimization space and narrower quant-error gap.
Based on AdaQTransform, for the first time, we build a general unified paradigm subsuming quantization parameter settings for current PTTQs, QATs and other possible cases.
We evaluate AdaQTransform across CNNs, VITs, LLMs and image super-resolution network EDSR, which proves AdaoTransform can be easily integrated to current PTQs, and consistently helps them to find better optimum and better PTQ performance without extra inference cost.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Low bit model quantization is one of most effective technique in model compression (Zhang ct al. 2020
Zhu et al. 2021 Han et al. 2020). Here we simply revisit its two main parts: PTQ and QAT.
As one of the best weight PTQ. AdaRound (Nagel et al. 2020) proposed an adaptive rounding for weight rather than nounding-to-nearee operation. BRECQ (Li et al. 202T) found block-wise reconstruction behaves better than layer-wist wise ones. QDROP Wei et al. (022b) explored how activation quantization affected weight tuning, and proposed random activation quantization during weight adaround learning. NWQ (Wang et al. 2022a) proposed a network-wise PTQ with inter-layer dependency. MRECG (Ma al. [2023 /0123/Bit-She Bit-Shriin (Lin et al., 2023) tried to solve ascillation'sharpness problem. PD-Quant (Liu et al. (023a) proposed to consider global information based on prediction difference metric. PTQ4VIT (uan al. 2022). APQ-ViT (Ding al. 2022).
PTQ:
and RepQ-ViT (Li et al. 2023) tried to solve the PTQ for Vision Transformer( (ViT))
Differently, we find weight's 's quant-step can be safely decoupled, and experimentally find the best case for PTQ reconstruction among six possible cases. Based on this best case, we theoretically propose an AdaQTransform to provide adaptive per-channel linear transformation on output feature.
QAT: Jacob et al. (2018b) proposed a fake quantizer simulation into QAT using gradient descent with turaigh-tthrough estimator (STE). PACT (Choi et 2018) proposed parameterized clipping
activation to learn the quantization range. LSQ (Esser et al. 2020 proposed to learn the quantization
al. 2020) further proposed a learnable offset. Nagel et al. (2022) tried to sovle oscillations in QAT. Wang et al.
step directly, whose process applied on PTQ is shown as (d) of Figure LSQ+(Bhalgat
(202bb) learned lookup tables as quantizers. Liu et al. 2023b) proposed a data-free QAT for LLMs.
However, from Tab Case 1.2, we find it is not suitable to directly borrow QAT's quant-step update setting into PTQ. This is because PTQ owns only tiny unlabeled calibration set and no original FP32 training pipeline thus it need to freeze FP32 weight as ground truth, totally different from QAT.
3 METHOD
3.1
PRELIMINARIES AND CURRENT SOTA OF PTQ
As current PTQ SOTAs, we perform per-channel weight quantization and per-layer activation quan-
tization. A classic linear symmetric PTQ process is as Formula 3). sw/su, w///,/////r, is the quant-step, upper/lower bound of quantization levels of FP32 weight w and FP32 activation z. 1-7/1-1 indicates rounding/floor operation.
i are called quantized(integer de-quantized weight. h((x) is AdaRound (Nagel et al., 2020) parameter of weight. a(-) is Sigmoid function. They first initialize & through minimizing the MSE between FP32 and quantized weight as Formula (T. Then freezing Su and optimizing AdaRound h(aa) activation quantization as (2) through output feature reconstruction as (3). We can see FP32 weight w and FP32 bias b are freezed in current PTQs.
i = clip(1
1; wi w.) Sw
mingo
|| w 1|
8w
0 clip(
(a))
(a)== 2
clip( (u.) 18x
(2)
8r
PTQ Reconstruction:
E(( Sw
h(a))) Sw) (1 x sw)); min wn
9 ---yll
a.
Fake and Real Quantization. In order to better optimize PTQ parameters through gradient descent, the quantization function is simulated in FP32 format, denoted as the Fake Quant' bracket of Formula 4 During practical inference accelcration, the FP32 simulation is converted to be integer- arithmetic-only (Jacob et al. 2018a) as the 'Real Quant' bracket of Formula 44p.
y
w bbb
Σ LD a+b
Σ Wint
T 1+b
g.
PP32
Fake fuant,,fe for finetuning
Real Quant, inference
where
Sw
S
Wint
clip(1
[h(a)] ]: wi, w.)
Sw
Wint is integer weight, which can be obtained early before practical fixed-point inference as the lower part of Formula 4
3

---

Published as a conference paper at ICLR 2025
3.2 EMPIRICAL OBSERVATIONS: DECOUPLE QUANT-STEP OF WEIGHT
As the 'Raa Quanl' of Formula 4 the Wint and 8 are determined before deployment, thus they can be treated independent parameters. Given this property, for fake quantization, Formula S we propose to decouple the quant-step of weight into quant-step 8un which quantize FP32 weight to integer value, and dequant-step S'n which de-quantize integer weight back. Different from weight, quant-step of activation can not be decoupled since integer activation is different for different input.
clip(1 + h(a);; wt, ww)-sw
v clipl
+ h((x);; wi, w..)
(5)
8
Under the condition where the quant-step of weight can be decoupled, for the first time, we fully ex- plore different settings of weight's quant-step into six cases, based on whether quant-step is decoupled, and if decoupled, quant-step sw and de-quant step s'y are learnable or not after initialization.
Case 1: the original single quant-step Sw is not decoupled as convention.
@ Case 1.1: not participates joint PTQ reconstruction optimization, as (a.b) of Fig. I
O Case 1.1.1: Weight PTQ and activation PTQ are seperated, like AdaRound/BRECQ.
O Case 1.1.2: Consider Weight AdaRound into activation PTQ, like QDROP/NWQ.
C Case 1.2: participates joint optimization during feature reconstruction, as (d) of Fig 1
current QAT methods, like LSQ. QAT's updating setting is not the best for PTQ.
Case 2: the original single quant-step Sw is decoupled as two independent params and s'y.
Case 2.1: Only quant-step sw participates joint PTQ reconstruction optimization.
Case 2.2: Only dequant-step s',, participates PTQ reconstruction optimization, as(c) of Fig
frozen Sw: a fixed base for adaround learning; learnable s'y adaptive transformation. Case 2.3: 8u and '' both participate in joint PTQ reconstruction optimization.
To evaluate their efficiency, conduct experiments MobileNet-v2, ResNet- 18, and MnasNet2.0.
Table I: Acc@1 on ImageNet among different quant-step settings across various nets.
Methods
W/A
MobileNet-v2 ResNet-18 MnasNet2.00
Case 1.1.1 (AdaRound, last PTQ SOTAs)
3/2
0.32
41.65
1.07
Case 1.1.2 (NWQ, current PTQ SOTAs)
3/2
38.92
60.82
52.17
Case 1.2 (LSQ, QAT's SOTA on PTQ)
3/2
39.65
60.26
49.78
Case 2.1
3/2
38.77
59.90
48.40
Case 2.2
3/2
42.60
61.06
54.19
Case 2.3
3/2
41.42
60.86
49.33
We find out that Case 2.2, where we decouple the original quant-step 8u as 8w and 8'y, then make only dequant-step sy learnablely participates joint PTQ reconstruction, shown (c) of Figure [ consistently provides the best performance, which even does a better job than Case 2.3. This is because (i) there is only a tiny unlabeled calibration set in PTQ, frozen FP32 weight during finetuning makes the lowest quant-error. error. To further narrow quant-error gap. we need to apply AdaRound. and (ii) a stable AdaRound learning requires fixed integer base but Case 2.1 and 2.3 bring fluid integer base.
Table 2: Visualization of Decoupling Case 2.2 during PTQ Reconstruction.
5k
15k
20K
5k
15k
20K
1
0.544
0.544
0.544
0.544
Sw2 $/m
0.943
0.943
0.943
0.943
0.544
0.508
0.444
0.442
0.943
0.902
0.796
0.795
Loss of Case 1.1.2
107
59.3
55.2
50.7
Loss of Case 2.2
107
54.4
51.1
46.5
We visualize the learning process of the dequant-step S in Case 2.2 on W3A2 ResNet-18 as Tab. At iteration 0, s and s',, is decoupled from the the same value, then sw is frozen and s',, is learnable during PTQ reconstruction with weight's ' adaround param and activation quant-step. We can see Case 2.2's dequant-step s' is updated accordingly and provides lower loss than current PTQ Case 1.1.2.
4

---

Published as a conference paper at ICLR 2025
Channel Channel--1
per-channel transform
Channel-0
Channel-1
Scale 0.5 Translate=0.5
Scale = Translate=-0.2
-4
2
0
(a-1) PTQ's output feature
(a-2) AranTransformed output feature
Channel 0
Channal-1
minll
yll
4
min 0,88
0 yll
0,80
Higher reconstrucioon loss
Lower reconstutcctttttci.....oo. loss
(a-3) FP32 output feature (ground truth)
AdaQTransform
Figure 2: AdaQTransform: adaptive per-channel transformation on output feature.
3.3 FROM DECOUPLING TO ADAPTIVE QUANT TRANSFORMATION
Superficially, from empirical experiments in Sec 3.2 Case 2.2 consistently provides better accuracy and lower reconstruction loss. At deeper side, as Fig Case 2.2 provides a better fit for each output channel to its FP32 ground truth. That is, Case 2.2 achieves adaptive per-channel scaling on the output feature. In theory, PTQ reconstruction in Formula for Case 2.2 can be denoted as Formula where Wffloor w is frozen after initialization as Formula Recall that We perform per-channel weight quantization and per-tensor activation quantization. Thus s'y is vector with out-channels elements while &æ is a scalar with one element. Therefore, the decoupled-out s theoretically provides direct per-channel scaling output feature to further minimize the quant-error gap.
Σ (Woloor + h(a))
objective:
min 11 yl? a,
(6)
S
A deeper look at Formula 6 there a stable up or down rounding learning requires a fixed base. Bias b is frozen due to inherited PTQ tradition. However, out it is enough to finetune bias with self-distillation between FP32 and quant output, which helps narrow the quantization error caused by quantized weight and quantized input, making their output fit closer to its FP32 counterpart, shown in Fig[2 Therefore, in addition to per-channel scaling, our decoupling Case 2.2 can be further equipped with per-channed translating as
params, Wytoor and b, left frozen. ftlor is frozen since
know, current PTQ works step closer and closer towards QAT. We find
Σ (W h(a)) 1 + objective: min
y
0r, ".. .''
The PTQ reconstruction objective becomes the right of Formula In order be notation-consisten with previous methods as Formula 3 we introduce a per-channel scaling and translating factor, &, denote Formula as Formula & dubbed as Adaptive Quant Transformation (AdaQTransform)
((L 8w
w
h(a)) sw)
11 -8)) + b n, objective: min a,9e1E.7
|1ø
yl1? (8)
3x
3.3.1 No EXTRA INFERENCE COST IN ADAQTRANSFORM
Although AdaQTransform add extra parameters into PTQ finetuning process, they will be merged into existing params thus cause no extra cost in actual inference. That is, when PTQ finetuning reconstruction process finished, the finetuning Formula (8) can be inferred as Formula (9. which as original inference Formula (4) like prior hardware- hardware-friendly friendly PTQ works (Wang et al. 2022a).
ÿ = x.
Wiit *
I +
where == E 8w 8 1 b +1)
5
5

---

Published as a conference paper at ICLR 2025
3.3.2 DIFFENRENCE BETWEEN ADAQTRANSFORM AND NORMALIZATION
Our AdanTransform significantly different from normalization, such as Batch/GrouppLayer Nor- malizaion (BN/GN/LN), here we take BN as example, in three points.
First, BN used stabilize FP32 model training while AdaQTransform is used help quantized model better fit FP32 counterpart during self distillation. BN normalizes data with statistical mean and variance then accordingly scales and shifts them back. AdaQTransform directly learns a per-channel linear transformation from quantized feature to FP32 counterpart.
Second, current PTQ publications all fold BN into its proceeding Conv layer before quantization. Current QAT's do not fold BN but they choose to update both statistical mean, variance and learnable scale, shift. As far as we know, there no study exploring how BN influence PTQ up to now. For Conv-BN structures, if BN is not folded into its preceding Conv, and participates PTQ reconstruction, achieves finer-grained per-channel transformation by making BN's per-channel scaling and trans- lating parameter and B Iearnable as Formula (10) with frozen s and frozen mean and var o². Compare Formula
10) with Formula (8), , it can be seen as a special form our AdaQTransform.
Wint Sw
S
+
7/1
+6, objective: min
l1 0,5e:7.8
y|| (10)
V 0
5z
Vo=+e
Thirdly, Transronm both applicable to networks/layers with normalization or without normalization like image super- resputresolution networks, e.g., EDSR (Lim al.) (017). layers such as the latter Cony BNN-BB--Cov, two Convs in Cony-(ReLU) as demonstration in Sec
3.3.3 EASY INTEGRATION AND ENHIANCEMEN TO CURRENT REDISTRIBUTION METHOSS
The core process of current redistribution methods for quantization, like LSQ+ (Bhalgat al. 2020
with translating offset 22 or RepQ-Vi1 ViT (Li et al. 2023) with the scaling vector scale, is as follows.
2x
LSQ+: ÿ = v t +
Sw) * *1
Sq + Zx) +
(11)
82
scale
x /scale
RepQ- VIT: +
Sw) *
Sx) +
(12)
8z
We see their redistribution on output will be recovered. Thus they do no change the distribution of output feature. Differently, our AdasTransform directly performed output feature and changes the distribution of output features, which further narrows the quant-error gap between and j.and is more suitable for PTQ's output feature reconstruction. Thus, it an easy integration and enhancement to redistribution PTQs. Example for RepQ-ViT is as follows and experiments is as Tab At scale
1) sw)) #( 2/ mt/cale.
ÿ = E + E((
s. b
(13)
81
85
3.4 FROM ADAQTRANSFORM TO GENERAL QUANTIZATION PARADIGM
Formula also builds a general quantization paradigm expressing quantization settings from current PTQs current QAT's and PTQ-suitable AdaQTransform. We can induct each one follows,
From General Quant Paradigm to AdaRound/BRECQ h(a)=AdaRound, ç = 1, = 0, objective: min| ( - w//:: min| 9 yll then minlle L y
From General Quant Paradigm to QDROP/WQQ: M(a)=AdaRound, & = 1, 7 = 0, objective: min i w | then min 1 y/1 0,
sw
From General Quant Paradigm to LSQ on PTQ:
h(a)) Round( au 8 ). 1, 0, objective: min 11 yll?.
Sw.
From General Quant Paradigm to Our AdaQTransform: h(a)=AdaRound, &. learnable, objective: min i w|? then mén 110 - 211.. a.32.E.11
From General Quant Paradigm to QAT'S LSQ
h(())= Round( Ew
11 0. objective: min CrossEnntropyyLonii.ccoo label).
w. w.en.

---

Published as a conference paper at ICLR 2025
Algorithm 1: AdaQTransform PTQ
142
Input: Pretrained FP32 Model {W*) Ji=1' calib set;
of channel-0
Params Activation's quant-step sxi W's quant
of channel-1
param: Sw, AdaQTranform: 5,
of channel-0
of channel-1
0.2
1st Iterative MSE Optimization for 8. as
0.53
2nd: PTQ Reconstruction.
5c
10c
1.kk
20c
Ite" num
for j = 1 to T iterations do
for i to N layers do
Loss with Baseline
Loss with AdaQTransfomm
# Get output from FP32 and quantized
Wi fw.a W as Formula
Loss
86 x² Formula
y' = W * +
ÿ' & W # +0 + my. # as Formula
A lyy 1'
A = ΣA. Optimize 82, 0,5,yas
Iter
Output: Quantized model
Figure Visualization of AdaQTransform
4 EXPERIMENT
We evaluate AdaQTransform across various CNNs, ViTs, LLMs and image super-resolution networks using Pytorch (Paszke al. 2019). Experimental settings are kept the same each baselines. By convention, the first and last layer are quantized into bits. AdaQTransform as Formula is adopted. Integer inference with acceleration is performed with TVM practical hardware.
4.1 EXPERIMENTS ON IMAGENET AND MS COCO
Table 3: Acc@ I on ImageNet among current PTQ methods.
Methods
W/A
Mobile-v2
Res-18
Reg-600
Mnas2.0
Full Prec.
32/32
72.49
71.08
73.71
76.68
MaaRound(Nage) et al.
2020
4/4
64.33
69.36
Hubara
2021
4/4
47.16
69.60
BRECQ(Li et al. 2021
4/4
66.57
69.60
68.33
73.56
QDROP We1 al.
2022b
4/4
68.84
69.62
71.18
73.71
PD-Quant (Luu et al. 2023a
4/4
68.33
69.30
71.04
73.30
MRECG (Ma al.
2023
4/4
68.84
69.46
71.22
NWQ WWang et al.
2022a
4/4
69.14
69.85
71.92
74.60
dda@Traastormmsoo
(oours
(ours)
4/4
70.01
69.88
71.97
74.80
BRECQ et al. 2021)
3/3
23,41
65.87
55.16
49.78
QDROP (Wer al. 2022b
3/3
57.98
66.75
65.54
66.81
PD-Quant (Liu et al. 2023a
3/3
57.64
66.12
65.09
64.88
MRECG (Ma al. 2023
3/3
58.40
66.30
66.08
NWQ (Wang et al. 2022a
3/3
61.24
67.58
67.38
68.85
AraQTranstorm
3/3
63.44
67.73
67.81
69.52
BRECQ et al. 2021
2/2
0.24
42.54
3.58
0.61
QDROP Wei al. 2022b
2/2
13.05
54.72
41.47
28.77
PD-Quant (Liu et al. 2023a
2/2
13.67
53.14
40.92
28.03
MRECG (Ma al. 2023)
2/2
14.44
54.46
43.67
NWQ (Wang et al. 2022a
2/2
26.42
59.14
48.49
41.17
2/2
32.19
60.12
51.20
44.54

---

Published as a conference paper at ICLR 2025
We first experiment ImageNet classification task over various CNNs and vision transformers as shown in Tab. (44) The calibration set consists of 1024 unlabeled images randomly selected from the training set. We adopt Adam optimizer, the same learning rate (Wei et al. 2022b Ma et al.) 2023) and 20k iterations for network-wise PTQ reconstruction as (Wang et al. 2022a). 'hhe average experimental results over runs summarized in Tab In W4A4, our method provides about 0-1 % Acc@ improvement compared to the strong baseline including NWQ (Wang et al. (2022a). MRECG (Ma et al. 2023). In W3A3, our method improve MobileNet-v2 by 2.2% and MnasNet2.0 by 0.67%. In W2A2, where BRECQ shows nearly 0% Acc@ I on Mobile-va and Mnas2.0, our method still far outperforms NWQ by more than 3% on Mobile-v2, and Mnas2.0.
Table 4: Acc@1 on ImageNet for ViTs and DeïTs.
Methods
W/A
ViT-S
ViT-B
DeïT-S
DeîT-B
FP32
32/32
81.39
84.54
79.80
81.80
RepQ-ViT (Li et al.
2023
6/6
80.43
83.62
78.90
81.27
AdaQTrantorm ReppantoorrrQ-vV RePQ
6/6
80.59
83.89
79.12
81.53
PTQ4ViT Yuaan et al.
2022
4/4
42.57
30.69
34.08
64.39
APQ-VIT (Ding 2022a)
2022
4/4
47.95
41.41
43.55
67.48
NWQ (Wang el al.
4/4
57.79
56.87
65.76
76.06
Tra@Transtorim
4/4
58.12
57.24
66.34
76.20
RepQ-Vit ViT Li et al.
2023)
4/4
65.05
68.48
69.03
75.61
4/4
70.40
76.47
73.50
78.93
For vision transformers. experiments ViT (Dosovitskiy al. 2021 and DeiT (Touvron et al. 202I) as Tab Our AdaQTransform outperforms NWQ by 0.5%, and outperforms PTQ4ViT Yuan al. 2022) and APQ-ViT VPT (Ding et al. 2022) by a large margin, about 10%-32% better. Then apply AdaQTranform into RepQ-Vit VIT. We can see AdaQTransform helps RepQ- ViT improve about 4% W4A4 and 0.3% in W6A6. Thus demonstrates AdaQTranform helps narrow the quant-error gap between the quantized and FP32 activation.
Table 5: mAP on MS COCO for object detection.
Faster RCNN
RetinaNet
Methods
W/A
ResNet-50
ResNet-18
ResNet-50
MobileNet-v2
FP32
32/32
40.26
34.91
37.39
33.31
BRECQ (Li et 2021
4/4
37.19
33.41
34.67
29.81
QDROP Wei et al.
2022b
4/4
38.53
33.57
35.81
31.47
NWQ Wang et al.
2022a
4/4
38.54
33.63
35.98
31.81
AdaQTranstorm (ours)
4/4
38.62
33.87
35.96
31.93
QDROP Wei al. 2022b)
3/3
33.49
31.21
32.13
27.55
NWQ Wang et al. AdaQ Tee@Traanrrmwwoo
2022a
3/3
35.25
31,88
32.45
28.43
(ours)
3/3
35.72
32.25
32.48
28.86
QDROP (Wei et al.
2022b
2/2
21.05
21.95
20.27
12.01
NWQ (Wang et 2022a
2/2
25.01
23.92
22.95
16.21
Ada(Transtorm (ours)
2/2
27.79
26.10
24.13
18.10
For object detection, we experiments on one-stage RetinaNet (Lin et al., (017) and two-stage Faster RCNN (Ren et al. (2155) where Res- 18, Res-50 and Mobile-v2 are selected as backbones respectively. As (Wei et al. 2022b; Li et al. 202I), we quantize the input and output layers of the network to bits, do not quantize the head of the detection model, and quantize the neck (FPN). Results shown in Tab In W3A3 setting, AdaQTransform improves the mAP of Res-50-based Faster RCNN by 0.5% and Mobile-v's vibile----baas basbeed RetinaNet by 0.4%. In harder W2A2 setting, AdaQTransform achieves about 1% mAP improvement over the current best method across all four experimental networks, which obtains 2.78 7.78% improvement Res-50-based Faster RCNN.
8

---

Published as a conference paper at ICLR 2025
4.2 EXPERIMENTS ON IMAGE RESPUUEES....... (NETWORKS WITHOUT NORMALIZATION)
As shown on Sec. 3.3.2 layer normalization can be seen as a special form of our Ada To show the effectiveness AdaQTransform on networks without layer normalization, we experiment on image super-resolution networks, i.e., EDSR of scale 4 (Lim et al.) 2017). We borrow base code from AdaBM (Hong & Lee (024) and follow all the same settings except that we apply our AdaQTransform to AdaBM. The calibration dataset, 100 LR images, is randomly sampled from the DIV2K (Timofte et al. 2017) training dataset. The quantization range for activation initialized using MinMax and quantization step for weight is initialized by OMSE. Then we freeze the network weights and finetune the quantization parameters for 10 epochs using Adam optimizer. For evaluation metrics, we measure reconstruction accuracy using the peak signal-to-noise ratio (PSNR) and the structural similarity index (SSIM) on Set5/Set1 4////an 100/BSD10 (Huang et al.) 2013)) To compare the computational complexity of the quantized network, report the feature average bit-width (FAB) that is averaged throughout the images of the test dataset.
As shown in Tab. 6 where AdaQTrans denotes apply our AdaQTransform AdaBM (Hong & Lee 2024), we can see our AdaQTransfont consistently helps AdaBM improve the PSNR and SSIM on experimental test sets and 4/3/2-bit quantization settings. Tab. demonstrates our AdaQTransform differs from layer normalization and gains from a wider optimization space: it helps the quant output feature better the FP32 counterpart and achieves lower PTQ quantization error.
Table 6: PTQ for EDSR of scale 4.
Set5
Set14
BSD100
Urban
Model
W/A
FAB PSNR/SSIM FAB PSNR/SSIM FAB PSNR/SSIM FAB PSNR/SSIM
EDSR (x4)
32/32
32
32.10/0.893 32 28.57
32
27.56/0.736 32 26.02/ 0.784
AdaBM-
4/4MP
3.8
31.02 0.00
3.7 27.87 / 0.751
3.5
26.91 /0.700 3.7
25.11/0.736
AdaQTranst
4/4MP
3.7
31.17 / 0.870
3.5 27.99/ / 0.761
3.5
27.04/
0.713
3.7
25.03 /0.742
AdaBM
4/4
4
29.42 0.821
26.81 / 0.724
26.44 / 0.687
23.87 0.685
AdaQTranst
4/4
4
31.41 0.845
4
27.55 / 0.742
4
26.81/ 0.698
4 24.64 / 0.718
AdaBM
3/3
3
28.93 0.804
26.49 / 0.711
3
26.24 / 00679
23.53 0.667
AdaQTrans
3/3
3
29.01 0.810
3
26.55
0.716
3
26.27 0.683
3
23.56 / 0.672
AdaBM
2/2
2
28.76 0.791
26.38 / 0.702
2
26.17 0.673
2
23.45 / 0.657
AdaQTrans!
2/2
2
28.84/
0.802
2 26.44/ / 0.712
2
26.23 0.681
2
23.46 / 0.666
As shown in Fig 4 our AdaQTransform helps AdaBM produces visually better reconstructed images with more details, e.g., AdaBM is relatively slur on the arch curve while AdaQTransform is clearer.
21.96dB / 4bi
25dB/ / 4bi
GT(ing014)
AdaBM
AdaQTransform
Figure 4: Qualitative results on Urban 100 with 4-bit EDSR-based models.

---

Published as a conference paper at ICLR 2025
Table T:AdaQTransform on LLMs LAMBADA. Table 8: Exploration for BN and AdaQTransform
Method
W/A Opt-1.3B Opt-6.7B
Methods
W/AMobiil-v Res-- 18ge-600
FP32
32/32
72.0%
79.8%
NWQdBN-Floed))
3/3
61.24
67.58
67.38
BN-Not-Folded
3/3
63.26
67.67
67.65
Naive
8/8
69.1%
41.9%
Decoupling
3/3
63.17
67.64
67.42
SmoothQuant
8/8
70.8%
80.0%
AdaQTransform
3/3
63.44
67.73
67.81
AdaQTransform'
8/8
71.2% %
80.1 %
NWQ(BN-Flode)) 2/2
26.42
59.14
48,49
SmoothQuant
6/6
66.2%
75.4%
BN-Not-Folded
2/2
32.09
60.09
51.18
OS+
6/6
66.4%
75.5%
Decoupling
2/2
31.43
59.91
50.32
AdaQTransform'
6/6
67.5%
75.6%
AdaQTransform
2/2
32.19
60.12
51.20
4.3 EXPERIMENTS ON LARGE LANGUAGE MODELS (LLM)
As Tab 7, compare AdaQTransform with SmoothQuant (Xiao al. 2023 and OS+ (Wei et al. 2023) on LLM models 1pt-1.3B/6.771 by simple evaluation metric "Last Token Prediction Accuracy" on LAMBADA dataset's validation set. AdaQTransform' denotes AdaQTransform applied on SmoothQuant. We can see AdaQTransform brings SmoothQuant 1.3% gain on W6A6 Opt-1.3B.
4.4 ABLATION STUDY ON IMAGENET
4.4.1) AdaQTransform V.s. BN: For networks with CaammaaiiaaionBB/GNNLA as we know, there have not been an academic PTQ work exploring BN's folding or not. Thus explore it as Tab based on NWQ which adopts BN-Folded setting by default. For BN-Not-Folded, we jointly optimize BN's learnable params y and B. It provides almost the statistical params μ4, a or not. We see BN-Not-Folded provides better performance than BN-Folded (NWQ). AdaQTransform provides tiny better accuracy than BN-Not-Folded, since AdaQTransform can be applied on layers with BN (equals to BN-Not-Folded) and other layers without BN. Therefore, as Sec 3.3.2 AdaQ Transforrr subsumes BN-Not-Folded, and covers a wider application range.
accuracy whether update BN's
4.4.2) Visualization for AdaQTransform: As 3.g AdaQTransform achieves adaptive per- channel transformation with adaptive E, 17 for output channels, and converges lower loss.
4.5
TRAINING AND INFERENCE COST COMPARISON
For training, only integrate AdaQTransform into each PTQ baselines and keep other things unchanged. The training time and memory is almost the same as the baseline. For inference, We perform pure 4/8-bit integer inference with TVM hardware with code from HAWQ Yao et al. 2021). The first and last layer are in 8 bits. The middle layer convolution is 4bits. The average inference time per batch over 30 measurement, each with 50 inferences, as follows. We can AdaQTranform improves accuracy without extra inference cost, and almost without training cost.
Table W4A4 MobileNet-v2 training and inference cost
Net
Acc@1
Train-Time Infer- Time (Batch=8) Params Params Bit FLOPs
NWQ
69. 69.14%
46.5 min
1.28 ms
3.51 M
20.79 M
6.36 G
Ada@Transformwww
70.01 % 46.9 min
1.28
3.51 M
20.79 M
6.36 G
CONCLUSION
In this paper, we propose a novel PTQ approach, AdaQTransform, based on full exploration on weight's quantization step through decoupling. It can be easily integrated into current PTQ methods, expands their optimization space and helps the quantized output better fit the FP32 output, thus achieves lower PTQ feature reconstruction error. It incurs negligible extra finetuning cost and no extra inference cost. For the first time, AdaQTransform builds general paradigm in quantization parameter update settings from current PTQs to QATs. Experiments on CNNs, ViTs, LLMs, and image super-resolution EDSR demonstrate AdaQTransform sets up new PTQ SOTA.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was supported by the National Key Research and Development Program of China under Grant 2020AAA0109004.
REFERENCES
Yash Bhalgat, Jinwon Lee, Markus Nagel, Tijmen Blankevoort, and Nojun Kwak. Lsq+: Improving low-bit quantization through learnable offsets and better initialization. In CVPRW, June 2020.
Jungwook Choi, Zhuo Wang, Swagath Venkataramani, Pierce I-Jen Chuang. Vijayalakshmi Srinivasan, and Kailash Gopalakrishnan. Pact: Parameterized clipping activation for quantized neural networks. CoRR, 2018
Yifu Ding, Haotong Qin, Qinghua Yan, Zhenhua Chai, Junjie Liu, Xiaolin Wei, and Xianglong Liu. Towards accurate post-training quantization for vision transformer. In ACM-MM, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021.
Steven K. Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra S. Modha. Learned step size quantization. In ICLR, 2020.
Kai Han, Yunhe Wang, Qiulin Zhang, Wei Zhang, Chunjing XU, and Tong Zhang. Model rubik's cube: Twisting resolution, depth and width for tinynets. In NeurIPS, 2020.
Cheeun Hong and Kyoung Mu Lee. Adabm: On-the-fly adaptive bit mapping for image super- resolution. In CVPR. pp. 2641-265U1 June 2024.
Jia-Bin Huang, Abhishek Singh, and Narendra Ahuja. Single image super-resolution from transformed self-exemplars. In CVPR, pp. 5197-5206, 2015.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In ICMI., pp. 4466-4475, 2021.
Benoit Jacob, Skirmantas Kligys, Bo Chen, and Menglong Zhu. Quantization and training of neural networks for efficient integer-arihhmeti-onny inference. In CVPR, 2018a.
Benoit Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithneticoonly inference. CVPR, 2018b
Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint 1006. 2018
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang, Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. Breeq: Pushing the limit of post-training quantization by block reconstruction. 2021.
Zhikai Li, Junrui Xiao, Lianwei Yang, and Qingyi Gu. Repq-vit: Scale reparameteriizatoon for post-training quantization of vision transformers. In ICCV, pp. 17227-17236, 2023.
Bee Lim, Sanghyun Son, Heewon Kim, Seungjun Nah, and Kyoung Mu Lee. Enhanced deep residual networks for single image super-resolution. In CVPRW, July 2017.
Chen Lin, Bo Peng, Zheyang Li, Wenming Tan, Ye Ren, Jun Xiao, and Shiliang Pu. Bit-shrinking: Limiting instantaneous sharpness for improving post-training quantization. In CVPR, pp. 16196- 16205, June 2023.
Tsung- Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, pp. 2999-3007, 2017.
11

---

Published as a conference paper at ICLR 2025
Jiawei Liu, Lin Niu, Zhihang Yuan, Dawei Yang. Xinggang Wang, and Wenyu Liu. Pd-quant: Post-training quantization based on prediction difference metric. In CVPR, 2023a.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. 17888, 2023b.
Yuexiao Ma, Huixia Li, Xiawu Zheng, Xuefeng Xiao, Rui Wang. Shilei Wen, Xin Pan, Fei Chao, and Rongrong Ji. Solving oscillation problem in post-training quantization through a theoretical perspective. In CVPR, pp. 7950-7959, June 2023.
Markus Nagel. Rana Ali Amjad, Mart van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? adaptive rounding for post-training quantization. In ICML, 2020.
Markus Nagel, Marios Fournarakis, Yelysei Bondarenko, and Tijmen Blankevoort. Overcoming
oscillations in quantization-wware training. In ICMI., 2022.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer. James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, higg-ppeformance deep learning library. In NeurlPS, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. 28, 2015.
Senmao Tian, Ming Lu, Jiaming Liu, Yandong Guo, Yurong Chen, and Shunli Zhang. Cabm: Content-aware bit mapping for single image super-resolution network with large input. 2023.
Radu Timofte, Eirikur Agustsson, and et.al. Ntire 2017 challenge on single image super- resolution: Methods and results. CVPRW, pp. 1110-1121, 2017. URL https:// /api
semant caccocschhooooooooccooooo wt..cccc.coo..ooooooo.ooooooc.ce 0tt..orrr//oppus044442222
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Herve Jegou. Training data-efficient image transformers amp; distillation through attention. In ICML, 2021.
Changbao Wang, Dandan Zheng, Yuanliu Liu, and Liang Li. Leveraging inter-layer dependency for post -training quantization. In NeurIPS, 2022a.
Longguang Wang, Xiaoyu Dong, Yingqian Wang, Li Liu, Wei An, and Yulan Guo. Learnable lookup table for neural network quantization. In CVPR, pp. 12423- 12433, 2022b.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2022a.
Xiuying Wei. Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. Qdrop: Randomly dropping quantization for extremely low-bit post-training quantization. In ICLR, 2022b.
Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong Liu. Outlier suppression+: Accurate quantization of large language models by equivalent and effective shifting and scaling. In EMNLP, pp. 1648-1665, 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In ICML, 2023.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Lyyuan Wang, Qijing Huang, Yida Wang. Michael W. Mahoney, and Kurt Keutzer. Hawqv3: Dyadic neural network quantization. ICML, 2021.
Zhihang Yuan, Chenhao Xue, Yiqi Chen, Qiang Wu, and Guangyu Sun. Ptq4vit: Post-training quantization for vision transformers with twin uniform quantization. In ECCV, 2022.
Qiulin Zhang, Zhuqing Jiang, Qishuo Lu, Jia' nan Han, Zhengxin Zeng. Shanghua Gao, and Aidong Men. Split to be slim: An overlooked redundancy in vanilla convolution. In IJCAI-20, pp. 1195-3201, 2020.
Mingjian Zhu, Kai Han, Enhua Wu, Qiulin Zhang, Ying Nie, Zhenzhong Lan, and Yunhe Wang. Dynamic resolution network. In NeurIPS, 2021.
12

---

Published as a conference paper at ICLR 2025
A ADDITIONAL TRAINING OVERHEAD FOR ADAQTRANSFORM
Our AdaQTransform can be casily integrated into existing PTQ methods. It takes the same training iteration(2OK), the same optimizer(Adant (ddam) and learning rate their baseline. As Table TO the training time and memory cost for extra introduced & and 7) can be ignored compared to their baseline.
Table 10: W4A4 Training Time and Memory for AdaQTransform
ResNet-18 Train-Time
ResNet-18 Train-Mem
MobileNet-v2 Train-Time
MobileNet-v2 Train-Mem
Method
BRECQ (Li et al.
2021)
18.4 min
2665 M
43,9 min
5794 M
AdaQTranstorman
18.7 min
2667 M
44.8 min
5797 M
NWQ WWang et al.
2022a)
46.5 min
3267 M
66.6 min
7529 M
AraQTTanstorm
46.9 min
3272 M
67.3 min
7537 M
PD-Quant (Liu et al.
2023a)
97.9 min
3589
184.0 min
6759 M
AdaQTranstormpp. Quant
98.5 min
3592 M
186.6 min
6764 M
B QUANT-PARAM SETTING VS TRAINING PIPELINE FOR PTQ AND QAT
Quantization consists of two parts: training pipeline and quantization parameter update setting(quant- param setting). Their main differences over PTQ and QAT are as follows,
Training Pipeline:
Dataset:
PTQ: tiny unlabeled calibration set, e.g., 1024 unlabeled images randomly selected from ImageNet training set.
QAT: full labeled training set, e.g., 1280000 labeled images Training method:
PTQ: self-distillation to FP32 models, usually called finetuning. FP32 weight is usually frozen, or minor adjusted.
QAT: training with loss to labeled ground truth, FP32 weight is learnable with no limit.
Training time:
PTQ: usually in minutes level.
QAT: usually in days level.
Quant-param param setting:
PTQ: AdaRound (Nagel et al. 2020),
BRECQ (Li et al.
2021). QDROP (Wei et al.
2022a
NWQ (Wang et al. 2022a). AdaBM Hong & Lee
2024)
QAT: LSQ (Esser et al. 2020). LSQ+ (Bhalgat et al. 2020), CABM (Tian et al.
2023
Here we experiment AdaQTransform's 's quant-param setting on QAT's training pipeline, and classic
QAT's quant-param setting, LSQ, on PTQ's training pipeline as Table I1 for W2A2 MobileNet-v2 on ImageNet. We can see different quant-param setting suits different training pipeline.
Table 11: W2A2 MobileNet-v2: Quant-Paran Setting
Training Pipeline For PTQ and QAT
Ace @ 1/Training-Timm
PTQ training pipeline
QAT training pipeline
LSQ's quant-param setting
27.65% 66 min
47.96% 98 hours
AdaQTransform's 's maant-param setting
32.19% / 67 min
47.72% 98 hours
13

---

Published as a conference paper at ICLR 2025
C
ADAQTRANSFORM ON WEIGHT-ONLY PTQ
We conduct weight-only quantization experiments on CNNs, VIT' as Table T2
Table 12: AdaQTransform on Weight-Only PTQ over ViTs
Methods
W/A
VIT-S
ViT-B
DeïT-S
DeïT-B
FP32
32/32
81.39%
84.54%
79.80%
81.80%
RepQ-ViT (Li et al. 2023)
4/32
75.31%
78.34%
75.10%
78.48%
Ada@Transformkoyooa Vei
4/32
79.39%
82.57 %
78.12%
81.10%
RepQ-ViT (Li al. 2023)
6/32
81.01%
84.33%
79.50%
81.67%
Ada@Thransforrkggovvv
6/32
81.16% %
84.39 %
79.78%
81.78%
D ADAQTRANSFORM GRANULARITY
Except for per-channel AdaQTransfom, we also explored other granularity, e.g., per-layer, per-pixel and per-network. Given output feature shape Cout x x W.
Per-channel y) what our AdaQTransform adopts, are vectors, each with Cout inde- pendent values for output feature of each Conv/FC layer.
Per-layer E, 7) are two scalar, with only 2*1 independent value, for the whole output feature of each Conv/FC layer.
Per-pixel 1 set each scalar for each pixel, with matrix with Cout X H x W independent values for each Conv/FC layer.
Per-network only one ç. r for the whole network, here we put them on the final FC layer, as two vectors with shape Cout
Experiments on W2A2 MobileNet-v2 is as Table 13 where per-channel is the best among all settings which incur extra inference cost.
Table 13: AdaQTransform Granularity, Per-chamce/laycpppeeeeewwwwt on MobileNet-v2
Method
W/A
Ace@ for MobileNet-v2
No extra cost in inference
NWQ-baseline
2/2
26.42
per-channel (ours)
2/2
32.19
per-layer
2/2
26.92
per-pixel
2/2
32.36
per-network
2/2
26.44
14


Reviews:
[{"review_id": "aG3clGokGd", "reviewer": "Reviewer_A9Zs", "summary": "This paper proposes AdaQTransform, a new PTQ approach which provides adaptive per-channel quantization on the output feature and can reduce the PTQ feature reconstruction error. This method achieves a new PTQ SOTA on many network structures like CNNs, ViTs, LLMs, and image super-resolution networks. Experiments with TVM on hardware also demonstrate that this method does not introduce additional inference cost.", "questions": "1. This paper only provides the results under weight-activation quantization. How does the AdaQTransform method perform under the weight-only quantization of CNNs, ViTs, LLMs, and image super-resolution networks?\n\n2. What is the training cost of the additional parameters $\\xi$ and $\\eta$? How does the required number of iterations of the optimization process (to get the parameters $s_x$, $\\alpha$, $\\xi$ and $\\eta$) compared to the existing methods like BRECQ?\n\n3. Can the authors provide more models and quantization methods to compare the inference performance under the same hardware on TVM?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written and easy to follow, presenting a new PTQ method that can be integrated with existing methods into a general quantization framework.\n\n2. Comprehensive experiments across various tasks, models, and datasets demonstrate the method's effectiveness.", "weaknesses": "1. This paper does not provide an analysis of the additional training overhead caused by the training learnable parameters $\\xi$ and $\\eta$ compared to other methods.\n\n2. The hardware experiment only presents the results of the NWQ method on the Mobile-V2 model, which is not sufficient for the practical applications of relevant researchers and practitioners.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UVSIQBiYfC", "reviewer": "Reviewer_YY8F", "summary": "This paper presents a network quantization method for post-training quantization (PTQ). The proposed method incorporates learning per-channel scaling and translating by decoupling transformation before and after quantization. Experimental results present that the proposed method outperforms the compared methods in image classification and super-resolution.", "questions": "Why has the parameter s_w been fixed during the other quantization parameter learning?\nIs the parameter b not updated during PTQ?\nIs the parameter scale in Equation (13) learnable?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 1, "strengths": "The proposed method outperforms the compared methods in various backbone networks and tasks (image classification, super-resolution, and large language models).", "weaknesses": "The presentation of this paper is very confusing and difficult to follow.\nFor instance, Figure 1 does not describe “case1.1.1” and s_w’ which make it hard to understand at the beginning of this paper.\n\nThe proposed method of per-channel transformation is not clearly described and analyzed.\nFor instance, the manuscript does not define h(\\alpha) in Equation (2). Note that the manuscript should describe the exact operation of AdaRound used in the proposed method. Moreover, the experiments have not presented performance improvement through learning per-channel transformation after quantization and ablation studies of each proposed technique component.\n\nThe performance comparisons for image super-resolution in Table 6 incorporate wrong numbers.\nEDSR of scale 4 in FP32 can not perform 37.99 dB in Set 5, and AdaBM outperforms the reported scores with lower FAB in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hm7PLallS9", "reviewer": "Reviewer_b342", "summary": "In this paper, the authors propose an Adaptive Quantization Transformation (AdaQTransform) for post-training quantization (PTQ) reconstruction, which provides adaptive per-channel transformations on the quantized output features. They explore the quantization step for weights across six cases by decoupling and incorporating learnable parameters in PTQ, such as Batch Normalization (BN) and bias. Extensive experiments on both low-level and high-level tasks demonstrate the superiority of the proposed method.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method technically sounds and produces superior performance.", "weaknesses": "1. Figure 1 should be revised, as the presentation of decoupled PTQ is not sufficiently clear. The authors should also highlight the advantages of the decoupled design in the introduction.\n2. The RepQ-ViT model reparameterizes scales for hardware-friendly layer-wise quantization and log2 quantization during inference. In this paper, the authors introduce additional per-channel factors, which may not be hardware-friendly.\n3. Channel-wise scale factor can enhance the performance of PTQ, particularly in low-bit settings. Can the authors provide performance metrics for 2-bit PTQ on the ViT for classification tasks?\n4. For both ViT and CNN architectures, can optimizing scale factors in Batch Normalization and Layer Normalization during PTQ yield similar performance?\n5.  It would be beneficial for the authors to include an evaluation of inference speed on the hardware.\n6. The authors should cite recent works in the related literature, especially concerning Quantization-Aware Training (QAT), such as: \"Wang, Longguang, et al. 'Learnable Lookup Table for Neural Network Quantization.' Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tlitjlNSwS", "reviewer": "Reviewer_8Mvu", "summary": "This paper studies the problem of Post-training low-bit quantization (PTQ) and makes an interesting observation that wider optimization space can lead to better PTQ solutions. The key idea is to unify PTQ with Quantization-Aware Training (QAT) by introducing a scaling and a translating factor. The developed Adaptive Quantization Transformation(AdaQTransform) has been tested on several network architectures including CNNs, ViTs, and LLMs. The reported experimental results are unanimously promising and convincing.", "questions": "1) Did the authors forget to highlight the best-performing results in Table 3? Because they did use bold-type to mark the best results in Tables 4 and 5. Just curious about this apparent inconsistency.\n2) What is the model difference between the two settings (Lines 459 and 460)? Since the results show some notable difference, the two models must be different, I think.\n3) Any failure cases or the limitations of AdaQTransform? I think an improved understanding needs to cover both positive and negative findings. \n4) I am curious about the relationship between this work and Outlier Suppression+ (OS+) framework (EMNLP'2023). Both papers deal with scaling and translating. OS+ also developed a fast and stable scheme to calculate effective shifting and scaling values. It will be nice if the authors can compare their strategy for determining scaling/translation parameters.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "+ Originality: A notable contribution is the unified framework subsuming PTQs and QATs in the literature. Such a unified treatment is beneficial for improving our understanding of the existing works on network compression scattered in the literature. \n+ Quality: the overall quality of this paper is good. In particular, I appreciate the authors' empirical observations in Sec. 3.2, which helps the explanation of decoupling and the motivation behind AdaQTransform.\n+ Clarity: the paper is relatively easy to follow in most parts. The reported experimental results are comprehensive and convincing. Most figures and tables are self-contained.", "weaknesses": "- Significance: \"AdaQTransform improves the current best PTQ by 5.7% on W2A2-MobileNet-v2.\" This claim is vague because it does not mention 5.7% corresponds to which aspect of the performance. I fail to identify the corresponding discussion in Sec. 4. Accordingly, it is difficult to assess the significance of this work. \n- Experimental results. The benchmark methods used on MS COSO for object detection in Table 3 are all published before 2023. I doubt that those methods still represent the current SOTA given the rapid advance in the literature of PTQ. More recent benchmark methods (e.g., [1]-[5]) need to be included into the comparison. \n\n[1] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, and Christopher De Sa. Quip#: Even better\nllm quantization with hadamard incoherence and lattice codebooks. arXiv preprint arXiv:2402.04396,\n2024.\n[2] Haoxuan Wang, Yuzhang Shang, Zhihang Yuan, Junyi Wu, and Yan Yan. Quest: Low-bit diffusion model\nquantization via efficient selective finetuning. arXiv preprint arXiv:2402.03666, 2024.\n[3] Xiuying Wei, Yunchen Zhang, Yuhang Li, Xiangguo Zhang, Ruihao Gong, Jinyang Guo, and Xianglong\nLiu. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal\nshifting and scaling. arXiv preprint arXiv:2304.09145, 2023.\n[4] Junyi Wu, Haoxuan Wang, Yuzhang Shang, Mubarak Shah, and Yan Yan. Ptq4dit: Post-training quantization\nfor diffusion transformers. arXiv preprint arXiv:2405.16005, 2024.\n[5] Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant:\nAccurate and efficient post-training quantization for large language models. In International Conference\non Machine Learning, pages 38087–38099. PMLR, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)