GPTAQ: Efficient Finetuning-Free Quantization for Asymmetric Calibration
Authors: Yuhang Li
Ruokai Yin
Donghyun Lee
Shiting Xiao
Priyadarshini Panda
Venue: ICML 2025 poster (2025)
Keywords: Transformers
Quantization
Discrete Optimization
Primary area: optimization->discrete_and_combinatorial_optimization

Abstract: We introduce GPTAQ, a novel finetuning-free quantization method for compressing large-scale transformer architectures.
Unlike the previous GPTQ method, which independently calibrates each layer, we always match the quantized layer's output to the exact output in the full-precision model, resulting in a scheme that we call *asymmetric calibration*. Such a scheme can effectively reduce the quantization error accumulated in previous layers. We analyze this problem using optimal brain compression to derive a close-formed solution. The new solution explicitly minimizes the quantization error as well as the accumulated asymmetry error. Furthermore, we utilize various techniques to parallelize the solution calculation, including channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion. As a result, GPTAQ is easy to implement, simply using 20 more lines of code than GPTQ but improving its performance under low-bit quantization. Remarkably, on a single GPU, we quantize a 405B language transformer as well as EVA-02—the rank first vision transformer that achieves 90% pretraining Imagenet accuracy. Code is available at [Github](https://github.com/Intelligent-Computing-Lab-Yale/GPTAQ).

GPTAQ: Efficient Finetuning-Free -ree Quantization for Asymmetric Calibration
Yuhang Li" Ruokai Yin Donghyun Lee"
Shiting Xiao
Priyadarshini Panda
Abstract
min
w
wX|
|
We introduce GPTAQ, a novel finetuning-fres free quantization method for compressing large-scale transformer architectures. Unlike the previous GPTQ method, which independently calibrates each layer, we always match the quantized layer's s output to the exact output in the full-precision model, resulting in scheme that we call asym- metric calibration. Such scheme can effectively reduce the quantization error accumulated in pre- vious layers. We analyze this problem using opti- mal brain compression to derive a flose-formed solution. The new solution explicitly minimizes the quantization error well the accumulated asymmetry error. Furthermore, we utilize vari- ous techniques to parallelize the solution calcu- lation, including channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion. As result, GPTAQ is easy to implement, simply using 20 more lines of code than GPTQ but improving its performance under low-bit quantization. Remarkably, on a single GPU, we quantize a 405B language transformer as well as EVA-02- the rank first vision trans- former that achieves 90% pretraining Imagenet accuracy. Code available at Github.
Quantized
Calibrating
Unquantized
(a) Symmetric Calibration
Unquantized
Unquantized
Unquantized
W
w
X|I |
X
Quantized
Calibrating
Unquantized
Layer
Layer
Layer
(b) Asymmetric Calibration
Figure 1. Calibration pipeline in the symmetric way (GPTQ) and the asymmetric way (GPTAQ).
significant deployment challenges across all computing plat- forms, from servers to edge devices. Moreover, transformers enable multi-modi architectures, which combine multiple transformer architectures and make it challenging to operate under resource-constrained environments.
Quantization (Gholami et al., 2022) has emerged as promising approach to address these challenges by reducing the precision of weights and activations, thereby accelerat- ing both memory access and computation. However, current state-of-the-art quantization methods often rely on model fine-tuning. These approaches (Shao et al., 2023; Liu et al., 2024) update parameters through gradient descent, process that becomes increasingly challenging with model size. For instance, Malinovskii et al. (2024) requires 8 A A100 GPUs rumning for 8 days to fine-tune a 70B parameter language model. This computational burden becomes particularly problematic given the numerous possible quantization con- figurations, including bit formats, symmetry choices, etc., making fine-tuning-bass quantization less efficient.
1. Introduction
The emergence of transformer architectures (Vaswani, 2017) has led to unprecedented scaling in model sizes and com- putational demands in both vision and language domains. In computer vision, models like VIT-G/14 (Zhai et al., 2022) encompass 2 billion parameters and require 2860 GFLOPs 700 times that of ResNet-500 for processing a single image. Language models have scaled even further, with architectures like LLaMA-3-405B (Meta, 2024) reach- ing hundreds of billions of parameters. This scale poses
This work focuses on finetuning free quantization ap- proach that circumvents gradient-based optimization in fa- vor of forward-pat -aasrdd-ass calibration computation. Several meth- ods have emerged in recent literature (Lin et al., 2023; Dettmers et al., 2022; Nagel et al., 2019; Liu et al., 2021). Among these approaches, GPTQ (Frantar et al., 2022)! has distinguished itself through particularly effective com- bination of speed and accuracy. leading to its widespread adoption across the machine learning community, with over 5,321 quantized transformers now available on the Hugging-
Department of Electrical Engineering, Yale University. Corre- spondence to: Yuhang Li <yuhang.li@e yyl...uduu.
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 20255 by the author(s).
While the authors used the name "OPTQ" in the final camera- ready paper, we use "GPTQ" throughout this work it has become the more widely recognized name in the broader community.

---

GPTAQ
face Hub (Huggingface, 2025). The method's broad API support and straightforward implementation have made it the de facto standard for quantizing emerging transformer architectures, enabling rapid deployment.
and thus does not require backpropagation. Both our method and GPTQ can be combined with above mentioned methods.
Finetuning- Quantization. Fintuning-baseen -assed quan- tization costs more computation resources to compute for- ward and backward propagation. Full network finetun- ing (Malinovskii et al., 2024; Liu et al., 2024; Dettmers et al., 2023; Wang et al., 2023) can achieve good perfor- mance but they are restricted to smaller models. Local fine- tuning approaches can alleviate this problem (Nagel et al., 2020; Hubara et al., 2020; Li et al., 2021; Wei et al., 2022a; Shao et al., 2023; Tseng et al., 2024), nevertheless, the time and resources needed are still a lot more than finetuning-free approaches.
In this work, we identify a crucial problem in GPTQ, which we call symmetric calibration. The symmetric calibration problem arises from the fact that each layer optimizes the objective of || w2 where the input X is coming from the previous quantized layer's 's output. This imput poses a deviation from the original model's input activation X, as shown in Fig. 1. Our objective involves the input activation from the full precision model, which we call asymmetric calibration: In this work, we tackle asymmetric calibration by proposing GPTAQ, where stands for input activation asymmetry. Our contribution includes:
Optimal Brain Surgeon (OBS). It was originally applied to small networks with hundreds of weights (LeCun et al., 1989; Hassibi et al., 1993). Efforts have been made to re- duce the estimation complexity of Hessian on larger models, like Fisher approximation (Singh & Alistarh, 2020), K-FAC approximation (Dong et al., 2017). Our approach extends the Optimal Brain Compression (Frantar & Alistarh, 2022) which uses Gram layer input as the Hessian matrix to per- form layer-wise quantization. The key difference is the in- troduction of a correction term in the input for ground truth target, leading to an asymmetric calibration framework.
1. We analyze the asymmetric calibration and show that the optimal update to weights needs to account for quantiza- tion error, inverse Hessian, as well as input deviation.
2. We propose to execute the asymmetric calibration via several modifications, including efficiently parallelizing all output channels and decomposing the residual error into each channel dimension to leverage efficiency.
3. Using the above method, our proposed GPTAQ only re- quires 20 lines of more code than GPTQ and improve its performance on both vision and language transformers. GPTAQ effectively quantizes extremely large transform- ers, including LLaMA3. 1-00SB and EVA-02.
3. Background
3.1. Notations
2. Related Work
We adopt row-vector notation throughout this paper. Vectors and matrices are denoted by bold lowercase and uppercase letters respectively. For instance, a linear operation between weight vector and input activations is expressed as: y w where w E R Lxn represents row of weights (output channel) and X E Raxk denotes the input activation matrix, with n being the number of input neurons. Quantization is represented as w = quant(w). For indexing, we use subscripts to denote specific elements or subsets of vectors and matrices. Notably, negative indices indicate the removal of neuron (corresponding to row in X). For example, X_1 E (===1)x represents the input activation matrix with its first input neuron removed.
Finetuning-Free Quantization. Finetuning-free quantiza- tion gathers particular interest as it can immediately export quantized checkpoints given the massiveness of pre-trained large models. A classic finetuning-fres approach is to cor- rect the distribution using bias and scales, (Banner et al., 2019; Lin et al., 2023: Nagel et al., 2019). Moreover, Lin et al. (2023); Xiao et al. (2023); Wei et al. (2022b; 2023) have proposed to reduce the outlier problem in language transformers. Other approaches involve architecture modifi- cations like channel splitting and merging (Zhao et al., 2019; Liu et al., 2023a). Recently, incoherence processing (Chee et al., 2023) has been proposed to transform the weight distribution into a more crantization-friendll one. Tseng al. (2024) adopts Hadamard transforms to improve trans- form efficiency and Ashkboos et al. (2024) further extends it with an online operation to transform activations. Hetero- geneous treatment, e.g., Dettmers et al. (2022); Huang et al. (2024a;b), allocates more quantization spaces (higher preci- sion) for salient entries, which, however, requires complex hardware design.
3.2. OBQ & GPTQ
Converting floating-poin model parameters W E guxx from FP16 to integer representations W demands a calibra- tion process to preserve model behavior. This calibration is formalized in the Optimal Brain Quantization (OBQ) framework, which GPTQ implements efficiently for large models. At its core. OBQ calibration process minimizes the difference between the original and quantized layer outputs:
The precursor of our method, GPTQ (Frantar et al., 2022) optimizes the weight elements using closed-form solutions
min (w + Aw)x- - w s.t. Aw = -w - -.w. (1)
Aw
2

---

GPTAQ
(a) GPTQ
(b) GPTAQ
where the activation MAE loss across transformer blocks continues increase during GPTQ quantization, and thus reveals a fundamental bias in symmetric calibration. To address this challenge comprehensivell. our work advances two primary contributions: first, a rigorous theoretical frame- work that optimally handles asymmetric calibration, and second, an efficient GPU implementation strategy that crys- tallizes these insights into the GPTAQ framework. We also show the MAE loss surfaces using GPTAQ in Fig. 2(b).
6.7
0.7
04
8.4
Channdl Indez
Block Inlee
of index
glock
Figure 2. Visualization of input activation MAE loss (X XD
when calibrating LLaMA3-8B using GPTQ and GPTAQ.
4.1. Optimal Framework
We now develop solution to the asymmetric calibration problem defined in Eq. (4). Let us first consider opti- mization for single output channel (one row of weights), We introduce to represent the output activation residuals w wX, allowing us to reformulate the objective as:
Using OBS framework (Hassibi et al., 1993), OBQ solves this optimization problem iteratively. For each weight y, it computes the optimal quantized value and corresponding adjustment:
(Wg g
3 Hgg
(Wg Hgg
Wg)
min Aw
Awx rl s.t. Aw
W.
q arrggmi min
Aw
(H
qu
(2)
Our optimization strategy proceeds iteratively by:
where H = (XX represents the inverse Hessian
1. Selecting weight wa for quantization,
matrix. After quantizing each weight, the inverse Hessian matrix is updated efficiently using Gaussian Elimination to
2. Computing the optimal weight updates Aw to minimize the loss function.
exclude the quantized entry:
For the q-th weight quantization, we introduce the constraint Awe + W, w, = O, where e, represents a unit vector with all elements set to 0 except for a 1 in the q-th position. This leads to a nested optimization problem:
(X
(3)
Hgg
While OBQ provides a closed-form solution, its computa- tional cost becomes prohibitive for billion-paramtter trans- former models. GPTQ (Frantar et al., 2022) addresses this limitation by enhancing parallelization and improving merical stability through Cholesky Reformuation, making the approach practical and robust for models with hundreds of billions of parameters.
min
min Aww - r1} Aw
s.t. Awe, + W, - w = -].
(6)
To solve this constrained optimization problem, we formu- late its Lagrangian:
L = Awx r1 + X(Awe,) Wy -wy).
(7)
The optimal solutions are obtained by taking derivatives of the Lagrangian:
4. GPTAQ Methodology
We contend that key limitation of GPTQ and the original OBQ lies in their symmetric treatment of input activations, that fails to account for how quantized layers progressively transform the activation patterns. This oversight, which be- comes particularly acute as quantization proceeds through deeper layers, motivates our introduction of an asymmetric calibration framework designed to preserve the character- istics of the floating-poil model's layer inputs X. The optimization objective becomes:
aL 2AwH - 2rX + +ee
0Aw
(8)
ØL
Awey + +W, - W
0X
and setting these derivatives zero, which yields the opti- mal weight update Aw:
(w Wg)
Aw
(H
H
(9)
H 49
min [[ + Aw Aw
X
w
P. s.t. AW = = W- w, (4)
with the corresponding loss function:
In practice the difference X
X) can come from both activation quantization and weight quantization in previous layers. This deviation accumulates systematically through the network's depth-a phenomenon we show in Fig. 2(a),
Lg
(W.
FT
rX
Xr
H q9
(10)
2 W
Wg)
rX
TTT...
H Hgg
3

---

GPTAQ
where H according Eq. (3), represents the inverse Hessian matrix after applying Gaussian elimination out the q-th row and column. The complete derivation is provided in Appendix A.I. The optimal framework pro- ceeds iteratively by first computing q = arg min, Tup then quantizing the q-th weight to W.. and finally updating the remaining full-precision weights by Aw. This process con- tinues until all weight elements are quantized.
W
Fusion
Cholesky Reformulation
P
P
However, this direct implementation faces significant effi- ciency challenges when applied to large transformer models, that stem not only from the computational cost of evaluating L for each weight, but also from the way different optimal q orderings across output channels prevent effective GPU parallelization. The computational burden is particularly severe as the process involves multiple n X n matrix multi- plications for every single weight, becoming prohibitively expensive for large foundation models. Furthermore, each weight update necessitates re-estimation of the residual out- put r. To address these limitations, we propose four opti- mization steps that achieve efficiency comparable to GPTQ, which we detail in the following section.
AXX
MU
L
X
Unselected
0-valued
1-valued Real-valuud
Figure 3. Computing paradigm of the second term for residual output error in g iteration. The inverse Hessian matrix is fac- torized by Cholesky Decomposition, furthermore, A "HH is fused to the q-th row of matrix P. which can be computed in parallel. Dimensions are in the bottom left corner each matrix.
4.2. Efficient Solution
Step 1, Arbitrary orders of handling g. The massive parameter space large-scale foundation models suggests an important optimization: we can process weights an arbitrary order rather than strictly following the optimal quence of Lg. When quantizing individual weight elements, the extensive weight space provides sufficient flexibility to compensate for quantization errors. This insight is sup- ported by empirical evidence from GPTQ (Frantar et al., 2022), which demonstrates that following the optimal de- scending order of Ly offers only marginal improvements over arbitrary ordering.
These challenges should be addressed achieve practical efficiency for large-scale models.
Step 2, Efficient Residual Decomposition. Another cru- cial efficiency consideration in our algorithm is the repeated computation of residuals R. Although the term RX H effectively reduces the gap between quantized and full- precision model outputs, the residual output R. needs to be re-estimated each new iteration. Denoting the residual error in input activation AX is given by
X X, the update formula
Processing weights in an arbitrary order yields an additional practical advantage: it enables parallel processing of all rows, efficiently utilizing GPU computational capabilities. We therefore adopt GPTQ's order of processing columns, which is sequentially from the first column to the last. For each q = 1, 3,..3..... n, we compute the weight updates A W across all rows simultaneously using:
Raew = - AWAX
(12)
which incurs substantial computational cost, with com- plexity O(mnk), where l: is the product of token length and number of calibration samples. The dependence on & poses a particular challenge for large foundation models, where k typically exceeds m and n by a factor of 10 ~ 50 (for example on LLaMA2-7B the k 128 x 2048, which is 64x greater than nx(096)). leading prohibitive processing times.
W (W.-WW.. W
AW
(H RX H (11)
H
where R- WX WX represents the output residuals across all rows (output channels). This formulation allows us maintain single copy of H ] it remains constant across all rows for given y.
We propose an alternative approach that eliminates the need to recalculate residual outputs R. Our key insight is that residuals can be decomposed into n individual components from different neurons:
While this approach enables parallel processing of rows, two significant computational challenges remain: the need to estimate new residuals R at each iteration, and the com- putational burden of the matrix multiplication R.X H
R = WAX = Σ W AX
(13)
This decomposition enables single estimation of R before
4

---

GPTAQ
layer calibration. Hence, y-th iteration, with the first q - columns quantized, focus quantizing the q-th column while minimizing its associated residual error component:
Algorithm 1 GPTAQ quantization for layer
Input: FP weight W, calibration input X, FP input X, and Block size B
H Xx AXXT + XXx
min
AAW .q: X Pi:
W n. AX q
A W. 5
(14)
L = Inverse Inwerse.Cholesky Cwwleskylleeecccc (H X1)
s.t. AW C Wa i 0,
- (AXX @ Mu L
Similarly, the optimal weight update for the q-th iteration
Q 0mxns E + 0mxB
for i 0, B, 2B, ... do
becomes:
for j i,
i B 1 do Q., f quant W :.))
AW
:g Hga 44
H + AX H
Ag:
9
E.j-i - (W sj Q
(15) where H H_ (): is the inverse Hessian matrix elimi- nated by (q-1) iterations, therefore H = H Notably, the first term in Eq. (15) is the same with GPTQ, while the second term is accounted for minimizing the residual error in q-th neuron and does not need re-estimation of residual error since a is already updated by previous AW. Fig. gives example of the second term when q 2.
W Wi((tB + W (+((+BB E L (1
W.JPissttt) 33:(+B)
end for
W (.(++B): - W ((+BB): E L M
W,(0(B) P((+
end for
Furthermore, the residual decomposition yields significant computational advantages because we reduce the complex- ity of the second term by a factor of n, due to the fact that X
full matrix AXX RR*= The key challenge becomes efficiently utilizing the structure of L 1: 1
1++1::+1 g to enable parallel processing of each row in P on GPUs. The lower-triangular structure of Lgg1ig++1 1: presents an oppor- tunity for computational optimization through vectorization, as formalized in the following theorem:
AX,,X!! .qq H -q: E (x a row vector and W..g is a column vector, requiring only (mm)) complexity to com- pute the second term. We further optimize our algorithm by noting that AX ..f: H is independent to the update of weights, thus we can compute each row in advance and store it in a matrix P, such that the second term can be efficiently computed by W... W.. P q,:
Theorem 4.2. The matrix is equal to
P ()
(17)
in g-th iteration.
where MU E O, 1 is a strictlly upper-triangular masking matrix with ones above the diagonal and O denotes element- wise multiplication.
The final challenge lies in computing P efficiently while maintaining numer- ical stability. These requirements arise from the need to perform Gaussian Elimination on the inverse Hessian n times and compute matrix multiplications. As model size increases, repeated Gaussian Elimination operations accu- mulate numerical errors. We address this numerical insta- bility through Cholesky decomposition the inverse Hes- sian: H LLT where L represents the loiau-tria Cholesky factor. This enables efficient and stable computa- tion of H H-1 through the following lemma:
Step 3, Cholesky Reformulation.
Proof is provided in Appendix A.3. With the above formula, can compute P in one line code (See Fig. 3 bottom). Furthermore, the proposed method integrates seamlessly with GPTQ's existing implementation, as it also utilizes the Cholesky factor L for computing diagonal values and the q-th row vector H Our implementation extends the original GPTQ framework with minimal modifications.
Step 4, Lazy-Batch Updates. With matrix P computed in advance, we can easily obtain the second term. Moreover, just like GPTQ, the quantization results for column g are only affected by updates performed on this very column, and so updates to later columns are irrelevant at this point in the process. Thus we can lazily butch both terms to achieve better GPU utilization. Concretely, given block of columns Q, we first update the columns in the block with a sliced P and L, and then update the remaining weights outside the block with
Lemma 4.1. Given the Cholesky factor for the full in- verse Hessian matrix H-1, the inverse Hessian H-1 xT, 11 is equivalent to Ly+1:g+1: 1 19t1:t+1
The proof is provided in Appendix A.2. This formulation offers better numerical stability compared to Gaussian Elim- ination, requiring only matrix slicing and multiplication to obtain H -:- Consequently, we can reformulate P Pq.: as:
P AX q.: X L Lig L 9t1:
(16)
Here, we use X since X. H-| = x H To opti- mize the computation of P. we begin by computing the
(W.Q W.e) Le + W.oPoo (18)
AW QQ:
LQQ
5

---

GPTAQ
Table Quantization results of vision transformer (left) and language transformer (right). For vision models, we report ImageNet accuracy
(1) and for language models, we report Wikitex12 perplexity (J). denotes our implementation.
Precision
Method
FT-Free
DeiT-S
DeiT-B
Method
FT-Free
L3-8B
L3-70B
L2-7B
L2-13B
L2-70B
FPI6
Pretrained
79.8
81.3
Pretrained
6.14
2.85
5.47
4.88
3.32
PTQ4VIT
V
34.1
64.4
OmniQuant
14.3
12.3
41.1
APQ-VIT
/
43.6
67.5
QLLM
11.8
9.09
7.00
PD-Quant
X
64.9
60.1
DuQuant
8.06
6.08
5.33
3.76
W4A4
RepQ-ViT
69.0
75.6
QuaRot
9,91
46.6
7.98
5.93
GPTQ
71.9
77.7
QuaRot-GPTQ
7.80
9,44
6.00
5,30
3.71
GPTAQ
72.8
78.4
QuaRot+ GPTAQ
7.37
6.93
5.86
5.17
3.69
RepQ-ViTY VIT
0.23
0.30
QuaRot
6.0c5
3.9c4
7.7c3
5.6c3
1.5c3
W2A4
GPTQ
38.4
61.0
QuaRot-GPTQ
102
444
32.6
15.5
6.68
GPTAQ
46.8
62.2
QuaRot-GPTAQ
17.9
39.1
11.2
8.57
5.64
Comparing GPTQ with GPTAQ. Together, summa- rized our algorithm in Algorithm 1. The primary additions to GPTQ are highlighted in blue. The remaining implementa- tion follows the standard GPTQ procedure. for example, the Hessian diagonal is dampened by I % of the average values worthwhile to note that our method is not a simple extension of GPTQ. The second term accounting for residual output error is as important as the quantization error in the first term, as we will show in periments (subsubsection 1) Our optimization through neuron decomposition and Cholesky Reformulation ensures both terms can be handled under a unified framework.
finetuneing- approaches. On vision transformers, we act order, an option in GPTQ that sorts the columns based Hessian diagonal magnitude, which we found useful to improve the performance. The dampening ratio was set to 10% for improved generalization. We test with W2A4 and W4A4 quantization.
for better numerical stability.
We provide the results in Table left part. from which we observe that GPTQ and our GPTAQ outperform the existing quantization regime due to explicit optimization of weights accounting for quantization error minimization. Further- more, our proposed method outperforms GPTQ baseline by % on 4-bit DeiT-S and 0.7% on 4-bit Deit-B model. On W2A4, we run the existing baseline RepQ- ViT which has the best accuracy in W4A4, however, this method fails at this precision (0.23% accuracy for DeiT-S). GPTQ obtains 38.4% accuracy on DeiT-S and method further improves it to 46.8%.
5. Experiments
5.1. Setup
We implement GPTAQ using Hugging Face (Wolf, 2019) on top of the PyTorch framework (Paszke al., 2019). Unless specifically mentioned, we always use per-channel asym- metric quantization for weights and per-token asymmetric quantization for input activations. The input activation has a clipping ratio of 0.9 as suggested in Ashkboos et al. (2024) and the weight clipping range is searched by minimizing mean squared error (Frantar et al., 2022). We select 128 input samples calibration dataset, see detailed source in each model type section. For GPTQ implementation, we first quantize weights and then quantize activation following prior work (Ashkboos et al., 2024; Liu et al., 2024), while our GPTAQ quantizes activations in the first place and mini- mizes layer output residual error in weight quantization".
5.3. Results on Language Transformer
We also verify our experiments on large language trans- former architectures, including LLaMA2 and LLaMA33 We perform quantization in W4A4 and W2A4 scenarios as we did on the vision transformer. The language trans- former is more challenging due to the existence of out- liers. Therefore, we apply the incoherence processing with QuaRot (Ashkboos al., 2024), which is a finetuning-free transformation, to RTN, GPTQ and GPTAQ. We addition- ally compare several finetuning- based approaches including OmniQuant (Shao et al., 2023), QLLM (Liu et al., 2023a), DuQuant (Lin et al., 2024). Following standard setups, we select 128 2048-token training sequences from the Wiki- text2 training set as calibration dataset.
5.2. Results on Vision Transformer
We conduct our experiments on DeiT-S/B models (Tou- vron al., 2021). We select 128 samples from ImageNet training dataset as calibration data. The compared base- lines are PTQ4VIT (Liu al., 2021), APQ-VIT (Ding al., 2022), PD-Quant (Liu et al., 2023b), RepQ-ViT ViT (Li et al., 2023), and GPTQ (Frantar et al., 2022). Most of then are
Perplexity Evaluation. We first compare the perplexity performance in Table right part. We note that on LLaMA2 models, GPTQ on rotated model can achieve better per- formance than other existing methods. Our GPTAQ further improves the perplexity performance. On LLAMAA-7B, our method improves the perplexity from 6.0 to 5.85. For LLaMA3 models, quantization becomes more challenging.
AA detailed study of quantization order is in Sec. 5.5.2.
6

---

GPTAQ
Table 4-hit quantization results of I.L.M22/3 Models. We report GPU hours quantization (including processing time with
rotation, which is not modified by our method), Wikitex12 perplexity and zero- zero-shot reasoning tasks performance.
Model
Method
FT-Free
GPU Hours
Wikiz
PiQA
Arc
E
Are C
HellaSwag
Winogrande
BoolQ
Avg(
FPI6
6.44
80.7
77.7
53.7
79.1
73.2
81.1
74.3
QuaRni+GPTQ
0+0.2
7.80
75.0
70.5
43.5
73.9
66.3
73.2
67.1
L3-8B
QuaRot+GPTAQ
0+0.3
7.36
78.2
72.7
44.8
75.4
77.5
SpinQuant+GPTO
4.0+0.2
7.26
78.4
74.6
46.8
76.0
69.7
73.4
69.8
SPinQuant+GPPAQ
4.0+0.3
7.19
78.7
75.9
48.4
76.1
70.6
78.2
71.3
FP16
3.32
84.4
85.9
64.3
84,9
80.7
85.1
80.4
QuaRor+GPTO
0.1+18
9.44
74.6
65.0
38.7
64.2
62.7
69.3
62.4
L3-70B
GUTAO
0,1+2.7
6.94
78.2
73.2
49.0
72.6
68.3
73,3
69.1
SpinQuant+GPTO
28+1.8
6.14
79.8
76.6
$4.3
78.8
73.5
80.7
73.9
SpinQuant+GPTAQ
2842.7
5.00
82.1
81.4
58.2
82.5
76.8
85.1
77.7
FP16
5.47
79.0
74.6
46.5
76.0
68.9
77.7
70.5
OmniQuant
2.1
14.6
65.9
43.9
30.8
53.5
49.9
QLLM
11.8
67.7
44.4
30.9
58.5
56.6
51.6
L2-7B
DuQuant
6.08
75.7
37.5
69.7
61.0
QuaRot+GPTQ
0+0.2
6.00
77.2
70.4
42.6
73.0
65.7
74.5
67.2
QuaRot+GPTAQ
(+0.3
5.85
77.3
70.6
43.7
73.6
67.6
75.8
68.1
SpinQuant+GPPOO
3.3+0.2
5,90
76.7
70.2
43.3
73.1
65.2
73.7
67.1
SpinQuant+GPTAN
3,3+0.3
5,85
77.3
71.9
43.8
72.9
68.0
74,3
68.0
FPI6
4.88
80.5
77.5
49.2
79.4
72.4
80.6
73.3
UmmiQuant
3.2
12.3
69.8
47.2
33.8
39.3
55.5
53.1
QLLM
1.7
9.09
70.5
48.5
34.4
62.8
55.4
54,3
L2-13B
DuQuant
5.33
77.3
56.2
42.2
73.7
63.4
63.8
GuTR
5.30
78.1
46.4
76.5
70.4
78.3
QuaRot+GPTAQ
0+0.5
5.17
78.7
74.8
47.4
77.5
78.8
71.2
SpinQuant+GPTO
4.0+0.4
5.20
78.2
75.3
47.3
76.9
67.7
77.1
70.7
SpinQuant+GPTAQ
4,0+0.5
5,19
78.9
75.6
48.8
77.3
71.0
77.7
71.5
FP16
3.32
82.7
81.0
$7.3
83.8
78.0
83.8
77.8
OmniQuant
15
41.1
53.0
31.2
23.9
33,9
52.0
38.8
QLLM
9.3
7.00
74.3
50.6
37.2
71.6
59.4
58.6
L2-70B
DuQuant
15
3.76
79.8
59.8
46.8
79.4
74.1
73.1
68.8
QuaRot+GPTQ
0.1+1.8
3.71
81.6
79.7
55.6
81.8
76.6
81.5
76.1
QuaRot+GPTAQ
0.1+2.7
3.69
82.4
79.0
55.9
82.1
76.8
82.3
76.4
SpinQuant+GPPO
28+1.8
3.68
82.0
79.0
55.6
82.3
76.5
82.6
76.3
GPinQ
28+2.7
3.67
82.0
79.3
55.4
82.4
76.4
82.7
76.4
For example, GPTQ in 4-bit quantization increased the per- plexity LLaMA3-70B to 9.44, while our method reduced this to 6.93. To the best of our knowledge, we are the first to validate the W2A4 perplexity results in language transform- In this case, the quantized model easily crashed without delicate calibration, for example, applying QuaRot and RTN consistently yields > 1000 perplexity, significantly deteriorating the model performance. Applying GPTQ can recover the model but still degrades the full cell-precision perfor- mance by large margin. Remarkably, our method obtains great improvement over GPTQ, reducing the perplexity of GPTQ by 20%~90%
quired to run the algorithm. Although in practice, Spin- Quants runs on 8 A100 GPUs. The results are shown in Table 2. With QuaRot, GPTQ has 7.2% and 18% gap with the full precision 8B and 70B models, respectively. GP- TAQ can effectively reduce these gaps to 4.7% and 11%. Notably, QuaRot+GPTAQ can achieve similar average ac- curacy with SpinQuant+GPTQ. an approach that takes sig- nificantly more time than finetuning-fret quantization. For example, on LLaMA2-7B. QuaRot=GPTAQ achieves 5.85 perplexity and 68. % average accuracy, which is even 0.05 lower and 1% higher than the SpinQuant+GPTQ.
Since SpinQuant primarily focuses activation quanti- zation and our method can incorporate this part into our asymmetric calibration, we further test SpinQuan++GPTAO using the official checkpoints. GPTAQ can improve the per- formance of SpinQuant as well. For instance, on LLaMA3- 70B, our method improves the average accuracy from 73.9% to 77.7% (3.8% improvement).
Zero-Shot Accuracy Evaluation. We evaluate the quan- tized model's zero-shot performance on six downstream tasks: PiQA (Bisk et al., 2020), ARC easy chal- lenged (Clark et al., 2018), Hellaswag (Zellers et al., 2019), Winogrande (Sakaguchi al., 2021), and BoolQ (Clark al., 2019). In addition QuaRot, take the learned rota- tion matrices from SpinQuant (Liu al., 2024), finetuning- based approach, combine it with GPTAQ. To ensure comprehensive comparison in effectiveness and efficiency, we additionally report the GPU Hours (on one A A100) re-
Results Weight-Only Quantization. Given that the original GPTQ method mainly focuses on the weight-only quantization, we also evaluate our method under this case. To make a fair comparison, we use the original paper's
7

---

GPTAQ
Table 3-bit per-group weight symmetric quantization results of L.aMA2/33 Models. We report perplexity on Wikitext2 and C4 and the
Reasoning accuracy on 8 datasets. We use 128 examples from the C4 datasets (Raffel et al.. 2020) to calibrate the model.
Model
Method
Wiki2 (
C4(J)
PIQA
Are E
Are C
HS
WG
BoolQ
OBQA
SIQA
Avg(1)
FP16
7.21
11.39
81.2
79.3
55.0
79.2
73.7
84.1
43.2
32,9
66.1
AWQ
10.47
16.77
76.7
71.1
46.3
72.5
70.3
80.9
39.8
32,9
61.3
GPTQ
9.04
14.01
78.1
72.0
48.0
74.0
72.5
81.4
41.6
32.8
62.5
GPTAQ
8.86
13.84
78.7
77.2
50.6
74.9
71.6
83.2
41.6
32,9
63,8
FP16
5.47
7.26
79.1
74.5
46.3
76.0
69.0
77.7
44.2
32,9
62.5
AWQ
6.75
8.98
76.4
67.2
42.1
71.8
67.6
69.1
40.6
33.8
58.6
L2-7B
GPTQ
6.88
14.02
76.6
65.0
38.5
67.5
67.6
72.4
41.2
33.5
57.8
GPTAQ
6.59
8.39
78.0
68.6
42.2
72.9
65.7
72.2
40.2
33.1
59.1
FP16
4.88
6.72
80.5
77.5
49.2
79.4
72.2
45.2
33.2
64.7
AWQ
5.49
7.57
78.6
74.4
43.1
76.2
71.9
77.7
44.4
32.8
62.4
L2-13B
GPTQ
5.42
7.36
79.7
75.0
47.3
75.6
71.4
79.6
43.8
32.9
63.1
GPTAQ
5.41
7.33
79.6
76.1
48.5
76.3
72.0
81.8
43.2
33.1
63.8
Table Quantization results huge transformers.
Table Ablation study of AW on I.aMA3-8B.
Precision
Method Pretrained
FT-Free
EVA-02
L3.1-405B
Precision
Method
AW
Wiki2 ()
Avg(t)
FP16
90.05 (1)
1.44 (1)
RIN
9.91
65.5
RTN
85.72
17.4
W4A4
GPTQ
E. E.
7.80
67.1
W4A4
GPTQ
86.48
5.82
GPTAQ'
W..Pa.
7.97
69.0
GPTAQ
/
88.30
3.48
GPTAQ
E.g E.g., W P.P
7.36
69.6
primary settings: 3-bit per-group weight quantization. We use a symmetric format (no zero point), and the group size set to 128. We turn on act order operation that sorts the Hessian based on its diagonal values. Additionally, we test another weight-only quantization algorithm, AWQ (Lin al., 2023), for its layer-wise optimization. As shown in Table 3, GPTAQ is able to achieve the best performance among these layer-wise FT-free quantization algorithms. Remarkably, for the LLLMMA-8B-IInstruee we have 63.8% average accuracy, which is 2.4% higher than AWQ and 1.3% higher than GPTQ.
5.5. Ablation Study
5.5.1. WEIGHT UPDATE
The proposed algorithm comprises two different terms L and W p.. q:: for AW, which can be viewed as minimizing the quantization error from current layer, and minimizing the quantization error from the previous layer, respectively. Therefore, we test the performance of apply- ing these terms individually and observe how they con- tribute the final performance. We conduct experiments on W4A4 LLaMA3-2 with QuaRot transformations The results are demonstrated in Table 5, Note that if we do not empower any update to weights, the method will reduce to RTN; and GPTQ the case where we only apply the first term. Interestingly, we can find that solely applying the first term or second term can increase the quantization per- formance compared to RTN. While applying the first term (GPTQ) obtains a better perplexity score, the average accu- racy when applying the second term individually is much better, resulting in 2% higher performance. Combining both terms, which our GPTAQ, the quantization performance can further improved. This result suggests that quantiza- tion errors accumulated in previous layers should be taken into account during calibration.
W...P..
5.4. Results on Huge Transformers
To demonstrate the scalability of our method, we further test two huge transformers architectures, (1) EVA-02 (Fang al., 2024), the Rank 1st architecture in Pytorch Image Models benchmark that achieves 90.05% ImageNet top- I accuracy, and (2) LLaMA3.1-4055 that achieves 1.44 Wikitext2 perplexity. We perform GPTQ and GPTAQ on a single A100 GPU to quantize these two models into 4-bit weights and activations. The results are shown in Table 4. For EVA-02, the RTN and GPTQ method degrades the full precision accuracy by 4.3% and 3.5% respectively. Our GPTAQ, notably, reduces this gap to 1.7%, which is half of the GPTQ algorithm. For LLaMA3.1-44 1-05B, which contains 126 transformer blocks with an intermediate size of 8096, the RTN significantly degrades its perplexity performance to 17.4. Our GPTAQ achieves 4.32 perplexity. Compared to the GPTQ algorithm, our method effectively reduces perplexity by 2.3, demonstrating the scalability our method.
5.5.2. ACTIVATION QUANTIZATION ORDER
Conventionally, for GPTQ activation quantization is added after weight quantization done as did in QuaRot (Ashk- boos et al., 2024) and SpinQuant (Liu et al., 2024). For our GPTAQ, the activation quantization, however, is added
8

---

GPTAQ
Table 6. Comparison of activation/Weight quantization pipeline. All weights and activations quantized into 4-bit.
Model
Method
Q Order
(4)
PiQA
Are E
Arc
HellaSwag
Winogrande
BoolQ
Avg(1)
FPI6
6.44
80.7
77.7
53.7
79.1
73.2
81.1
74.3
QuRRot+GPTQ
W-A
7.80
75.0
70.5
43.5
73.9
66.3
73.2
67.1
L3-8B
QuaRot GPTAQ
W-A
7.68
78.2
73.8
45.8
74.3
68.4
74.6
69.2
GPTQ
A->W
7.78
76.3
72.3
45.3
73.8
67.7
75.8
68.6
QuaRot + GPTAQ
A-W
7.36
78.2
72.7
44.8
75.4
69.1
77.5
69.6
FP16
5.47
79.0
74.6
46.5
76.0
68.9
77.7
70.5
GPTQ
W 3A
6.00
77.2
70.4
42.6
73.0
65.7
74.5
67.2
1.2-7B
QuaRot GPTAQ
W-A
5.95
77.3
70.6
42.5
73.7
66.5
75.5
67.7
GPTQ
A-W
6.00
76.7
72.0
42.4
73.1
65.8
75.1
67.5
QuaRot + GPTAQ
A-W
5.85
77.3
70.6
43.7
73.6
67.6
75.8
68.1
(a) Computing P
cedure Algorithm We assume that AXX and are obtained previously, and test the latency on one A100 GPU with PyTorch 2.4. 2.4.1-cul 4.11-cu1..4 12.4. In Fig. 4(a), first compare the latency to compute P. Thanks to the highly optimized CUDA kernel, our parallel implementation takes less than Ims to finish, which is > 10 faster than the unparalleled implementation despite using the same number of opera- tions. Fig. 4(b) compares the speed of running GPTQ and GPTAQ, from which we observe that GPTAQ incurs less than 10% more latency when weight dimension is smaller than 4096. In these cases, the latency bottleneck is the quantization operation rather than computing A W. When dimension further expands, the bottleneck switches to com- puting A AW, and our GPTAQ increases the latency by a slight margin (30~40% latency).
1,024
Unparalle Parallel
#Neurons (m)
2,048
4.096
8,192
10
(b) Computing
Dimension X)))
1,024
GPTO GPTAQ
2,048
4,096
8,192
16,384
Latency (s)
Conclusion
Figure 4. Latency visualization of our algorithm under various
In this paper, introduce GPTAQ, efficient finetuning- free quantization to reduce the accumulated asymmetry error in quantization. Building upon the OBQ framework, our method introduces 4 steps that effectively parallelize and accelerate the computation of optimal weight update. As result, GPTAQ easy implement and can adapt to the previous GPTQ framework with minimum effort. Through both qualitative and quantitative verification, GPTAQ can ef- fectively reduce the asymmetry to improve the quantization performance without involving finetuning.
before weight quantization so that AX can capture this in- formation. We now demonstrate the cases where activation quantization is added before GPTQ or after GPTAQ.
We conduct experiments on 4-bit LLaMA3-8B and LLaMA2-7B. The results are shown in Table 6. From the ta- ble, can conclude that (1) For GPTQ calibration whether the activation quantization is enabled or not has no impact on the perplexity results. (2) However, activation quanti- zation may help GPTQ achieve higher downstream task accuracy. (3) For GPTAQ, enabling activation quantization can further aid the performance as it considers more infor- mation input activation error. (4) Even if we disable the activation guantization, GPTAQ still outperforms GPTQ regardless of quantization order.
Impact Statement
This paper presents work whose goal is to advance the field of model compression. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
5.6. Algorithm Efficiency
A Acknowledgment
We analyze the speed of running GPTAQ. (Memory anal- ysis is presented in Appendix C). First, we compare the latency of calculating P using unparalleled implementation (Eq. (16)) and paralleled implementation (Eq. (17)) under various size of n. Second, we compare the latency of GPTQ and GPTAQ on a layer with n x dimension, using the pro-
This work supported in part by CoCoSys, JUMP2.0 center sponsored by DARPA and SRC, the National Sci- ence Foundation (CAREER Award, Grant #23 #2312366, Grant #2318152), and the DoE MMICC center SEA-CROGS (Award #DE-SC0023198).

---

GPTAQ
References
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. Gptq: Accurate post-training quantization for generative pre- trained transformers. arXiv preprint arXiv:2210.17323, 2022.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B.,, Jaggi, M.. Alistarh, D., Hoefler, T., and Hensman, J. Quarot: Outlier-free 4-bit inference in rotated lims. arXiv preprint arXiv:2404.00555 2024.
Gholami, A., Kim, S., Dong, Z., Yao. Z., Mahoney, M. W., and Keutzer, K. A survey of quantization methods for effi- cient neural network inference. In Low-Power Computer Vision, pp. 291-326. Chapman and Hall/CRC, 2022.
Banner, R., Nahshan, Y., and Soudry, D. Post training 4-bit quantization of convolutional networks for rapid- deployment. Advances in Neural Information Processing Systems, 32, 2019.
Hassibi, B., Stork, D. G., and Wolff, G. Optimal brain surgeon and general network pruning. In IEEE interna- tional conference on neural networks, pp. 293-299. IEEE, 1993.
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. Piqa: Reasoning about physical commonsense natural language. In Pro- ceedings of the AAAI conference on artificial intelligence, volume 34, pp. 7432-7439, 2020.
Huang, W. Liu, Y., Qin, H., Li, Y., Zhang, S., Liu, X., Magno, M., and Qi, X. Billm: Pushing the limit of post-training quantization for llms. arXiv preprint arXiv:2402. 2024a.
Chee, J., Cai, Y., Kuleshov, V., and De Sa, C. Quip: 2-bit quantization of large language models with guarantees. arXiv preprint arXiv:2307.13304 2023.
Huang, W., Qin, H., Liu, Y., Li, Y., Liu, X., Benini, L., Magno, M., and Qi, X. Slim-Ilm: Salience-driven mixed- precision quantization for large language models. arXiv preprint arXiv:2405. /4917, 2024b.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1905: 2019.
Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Improving post training neural quantization: Layer- wise calibration and integer programming. arXiv preprint arXiv:2006.1055 2020.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv: 1rXxv:8803.05557 2018.
Huggingface. Huggingface models, 2025. URL https: / //huggingface wwwpco/ww//cccc//cooco./.c.o.lco co//noddllssaaacchpppppttt.
Dettmers, T., Lewis, M., Belkada, Y., and Zettlemoyer, L. Llm. int8 (): 8-bit matrix multiplication for transformers at scale. arXiv preprint wrXiv:2208.07339 2022.
LeCun, Y., Denker, J., and Solla, S. Optimal brain damage. Advances in neural information processing systems, 2, 1989.
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, L. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 143114 2023.
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q. Yu, F. Wang. W., and Gu, S. Brecq: Pushing the limit of post-training quantization by block reconstruction. arXiv preprint arXiv:2102.05422, 2021.
Ding, Y., Qin, H., Yan, Q., Chai, Z., Liu, J., Wei, X., and Liu, X. Towards accurate post-training quantization for vision transformer. In Proceedings of the 30th ACM international conference on multimedia, pp. 5380-5388, 2022.
Li, Z., Xiao, J., Yang, L., and Gu. Q. Repq-vit: Scale repa- rameterization for post-training quantization of vision transformers. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 17227-17236, 2023.
Dong, X., Chen, S., and Pan, S. Learning to prune deep neural networks via layer-wise optimal brain surgeon. Advances in neural information processing systems, 30, 2017.
Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, I.., Song, L., Sun, Z., and Wei, Y. Duquant: Distributing outliers via dual transformation makes stronger quantized 1lms. The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
Fang, Y.. Sun, Q., Wang, X., Huang, T., Wang, X., and Cao. Y. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 149:105171, 2024.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-traiming quantization and pruning. Advances in Neural Information Processing Systems, 35:4475 4488, 2022.
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., and Han, S. Awq: Activation-awan weight quantization for Ilm compression and acceleration. arXiv preprint MWXi:2336.00998. 2023.
10

---

GPTAQ
Liu, J., Gong, R., Wei, X., Dong, Z., Cai, J., and Zhuang, B. Qlim: Accurate and efficient low-bitwidth quan- tization for large language models. arXiv preprint arXiv:2310.08041 2023a.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L.., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. Omniquant: Ommidirectionally calibrated quantization for large lan- guage models. arXiv preprint arXiv:2308.133: 2023.
Liu, J., Niu, L., Yuan, Z., Yang, D., Wang, X., and Liu, W. Pd-quant: Post-training quantization based on prediction difference metric. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 24427-244377 2023b.
Singh, S. and Alistarh, D. Woodfisher: Efficient second-
order approximation for neural network compression. Ad- vances in Neural Information Processing Systems, 33: 18098 18 109, 2020.
Touvron, H.,, Cord, M., Douze, M., Massa, F., Sablayrolles, A., and Jégou, H. Training data-efficient image transform- ers & distillation through attention. In International con- ference on machine learning, pp. 10347-10357. PMLR, 2021.
Liu, Z., Wang, Y., Han, K., Zhang, W. Ma, S., and Gao, W. fost-training quantization for vision transformer. Ad- vances in Neural Information Processing Systems, 34: 28092-28103, 2021.
Liu, Z., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. Spinquant-IIm quantization with learned rotations. arXiv preprint arXiv:2405. 16406, 2024.
Tseng, A., Chee, J., Sun, Q. Kuleshov, V., and De Sa, C. Quipt:: Even better Ilm quantization with hadamard incoherence and lattice codebooks. arXiv preprint 2xXiv:242204399 2024.
Malinovskii, V., Mazur, D., Ilin, I., Kuznedelev, D., Burlachenko, K., Yi, K., Alistarh, D., and Richtarik, P. Pv-tuning: Beyond straight-through estimation for ex- treme Ilm compression. arXiv preprint 1484: 2024.
Vaswani, A. Attention all you need. Advances in Neural Information Processing Systems, 2017.
Wang. H., Ma, S., Dong, L., Huang, S., Wang, H., Ma, L., Yang, F., Wang. R., Wu, Y., and Wei, F, Bitnet: Scaling 1- bit transformers for large language models. arXiv preprint arXiv:2310.1 /453, 2023.
Meta. Introducing llama 3.1: Our most capable models to date, 2024. URL https: //tpss///aa.... meta. com/blog/ meta 1lamalama---- 3/1/.
Wei, X., Gong, R., Li, Y., Liu, X., and Yu, F. Qdrop: Ran- domly dropping quantization for extremely low-bit post- training quantization. arXiv preprint arXiv:2003.05740, 2022a.
Nagel, M., Baalen, M. v., Blankevoort, T., and Welling, M. Data-free quantization through weight equalization and bias correction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1325- 1334, 2019.
Wei, X., Zhang, Y., Zhang, X., Gong, R., Zhang, S., Zhang, Q. Yu, F., and Liu, X. Outlier suppression: Pushing the limit of low-bit transformer language models. Advances in Neural Information Processing Systems, 35:17402- 17414, 2022b.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C., and Blankevoort, T. Up down? adaptive rounding for post-training quantization. In International Conference on Machine Learning, pp. 7197-7206. PMLR, 2020.
Wei, X., Zhang, Li, Y., Zhang, X., Gong, R., Guo, J., and Liu, X. Outlier suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scaling. arXiv preprint wxXiv:2304.09145 2023.
Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019.
Wolf, T. Huggingface's transformers: Statc-of-the- art natural language processing. arXiv preprint arXiv: 1910.0371.,.,... 2019.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 21())55485-551, 2020.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099. PMLR, 2023.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 07830, 2019.

---

GPTAQ
Zhai, X., Kolesnikov, A., Houlsby, N., and Beyer, L. Scaling vision transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 2104-12113, 2022.
Zhao, R., Hu, Y., Dotzel, De Sa, C., and Zhang, Z. Im- proving neural network quantization without retraining using outlier channel splitting. In International confer- ence on machine learning. pp. 7543-7552. PMLR, 2019.
12

---

GPTAQ
A. Theoretical Derivation
A.1. Optimal Framework
We provide a detailed derivation the optimal framework. To compute the local minima of the Lagrangian form, we set the partial derivatives zeros:
aL
2AwH- 2rX + Aeg = 0
dAw
(19)
0L
-0 01 Awe, + W, - =0
Solving the first equation, have
ÁwH = +
2 e
(20)
Right multiplying the inverse Hessian on both sides, we get
A
Áw
2 c
H
H
(21a)
H +rx
H-1
(21b)
2
Substituting the equation into the second equation of Eq. (A.1), we have
H o +
+ H
eq
Wa
W.
- + +rX
H 2
H 1
+ Wg
W, =
(22a)
where we can easily compute À as
Wg
Wg
rX
=
(23)
Hgg
Hgg
Now substituting the above equation into Eq. (21b), we can compute Aw as
Aw
H
-x
H
(24a)
2
W₂
W.
rX
FrX
H
(24b)
Hg
H
W
W
rX
H
(24c)
Hgd
H
44
W Wa
rx
H
(24d)
H
The by is derived by replacing the optimal Aw into the loss function Aww -rli,, given by
[wg
wg)
by = rX H -
(H 1
(25)
H
Expanding this function, we obtain
+ (w, w²)
Ly =x H
Xr
+
22x H 1
(H - -- H
(26a)
+ W W
(w, wg))
xx H
(rX :4
(26b)
H
H
= H 1x + rr 2r H 1-1Xr T (W, w₂)
H
(26c)
(H
.9
(W. wg)
(xXT H ) - (W, wg)
+2
(rx (
(26d)
Hnp
H
13

---

GPTAQ
We start by simplifying H:
H
H
H
H
(27a)
Hqq
H. H 4: H
(27b)
Hqg
H T
9q Cgg H
I
H
(27c)
H 94
q
eq
(27d)
Hgg
Substitute this function back into Eq. (26d), we get
(W, wg)²
Lg
rx
H
e,
ey
H
Xr
Xr
+rr
-2rx H Xr
HIH
(28a)
Hgg
(Hgg) )2
2.
:9
+ Wg
Wg)
(rXT
[1-
1
H
(W. Wg)
(rX H
(28b)
Hqq
Hgq
H
(Wg - -wg)
rX
l Co H Xr
T
+
rX H Xr +
HH
(28c)
Hgg
(Hnd))
2
(w Wg.
(Hgg rX
H
og T e
(28d)
We note that the e, Eq. (28d) omitted. For last term, we can rewrite -2 (W, w.) rX H. due e H. H
H
all-zero vector since the q-th row of H eliminated. Therefore, the first term in
Hgy
We further simplify H 9.z
HH. by
:9
O. H
HH
(29a)
:9
4
e 19,H-be TeT
(29b)
HH1 07
(29c)
To this end, we can simplify the Ly to
(Wg )2)
(Wg
Wg) Hgg 09
Ly
H
rX H Xr
rX
H
(30)
Hgg q9
A.2. Proof of Lemma 4.1
Proof. Without loss of generality, we first prove the case of H -x_xx(, When performing the Cholesky decomposition H H we have H LL where L is lower-triangular matrix. Thus, rewrite the Cholesky factor as
L1
(31)
L
L2.2:
Now, substitute the LL T with above equation, we have
L
0
L
L
L
L L I
H
2.,1
(32)
2.22
L
L2.2:
L
L2.2.
L2
L
L
We thus construct a linear system from the above equation, given by
H 1
H L1 L2
(33)
H.l 2.2 L2 LT +
L
14

---

GPTAQ
is straightforward to solve the equations and obtain
L11
H
L2.1
2
Hil
(34)
2 1 =
H2.2
H
2:12
1,2:
H
H performs the Gaussian Elimination on the first row/column, which the H The third equation in Eq. (A.2) is essentially the H with the first row/column removed. Therefore, we have H L2. L2.2.
Recall that H1 Hi
The lemma can be derived by recursively removing the first row/column in the current inverse Hessian to get H 1g+ 1: 9+1:9+11"
A.3. Proof of Theorem 4.2
Proof. We start with computing each row of P. Recall that this formula is given by
p X b 1::41
(35)
For matrix L 10 the j-th column has non-zero values only if j i. therefore, elements in P can be written
Σ
O
Lug
if <
Paj
(36)
a=++1
ifi
where Oi,- = (AX,,) v X LL+155++1) is the element from the product of the first three terms. We again note that 1:44:
has non-zero values in a-th column only if a > i. Thus, the matrix O is computed as
Σ (AX, b=a
JoL
(AxX Li
if <
0 i
(37)
b=
if 2
Hence, O can be derived by masking out the lower-triangular area of AXX L,
0 = (AXX + Mu.
(38)
The fact that has zeros values on the lower-triangular area makes possible to directly multiply O and L get P. given by
(OL Ja
Σ O Oi ز,j
Σ 0x L
Σ a=i+1
OgaLaj +
Σ
Oia x 0
Σ
O IT
Pij
(39)
a=1-1
o=1+1
Thus, we have P ((axxTL) 1 Mu (I LT.
B. Additional Experiments
B.I. Weight-only Quantization with Rotation
We tested our method for weight-only quantization on LLaMA2 and LLaMA3 models. The weight is quantized to 2/3/4 bits in per-channel asymmetric format. The compared baseline includes AWQ (Lin al., 2023), OmniQuant (Shao et al.,
15

---

GPTAQ
Algorithm 2 GPTAQ quantization for entire transformer model
Input: Full-Precision Model, which contains blocks, model input X and X, AQ-True/False. for i 1,2,3, ...,, b-th block do Move block[i] to GPU memory Disable activation quantization, if any
# Only 1 block is loaded into GPU
X < block (X), store FP input for each layer
# update FP block imput data
if AQ is True then
Enable activation quantization, if any
Enable activation quantization during calibration # Compute H and AXX and delete FP inpul for this layer
end if
for -, ..... 1-th layer do
[lock[[]((] store imput for j-th layer Perform GPTAQ algorithm for j-th layer Quantize j-th layer's ' weight
end for
X c block (X)
# update quantized block input data
Move block[i] i]
to CPU memory
end for
2023). QuaRot (Ashkboos et al., 2024), and QuaRot+GPTQ (Frantar et al., 2022). As demonstrated in the following table, QuaRot+GPTAQ is able to enhance the performance of weight-only quantization, too. This result demonstrates that even weight-only guantization will introduce residual output error in the quantized model. In W2A16 cases, this will significantly impact the GPTQ performance and our method can reduce the perplexity by ~50%.
Table Weight quantization results of language transformer. We report Wikitext2 perplexity.
Precision
Method
FT-Free
L3-8B
L3-70B
L2-7B
L2-13B
L2-70B
FP16
Pretrained
6.14
2.85
5.47
4.88
3.32
AWQ
6.57
3.59
5.82
5.07
3.49
OmniQuant
5.74
5.02
3.47
W4A16
QuaRot
7.72
16.7
6.76
5.48
3.66
QuaRot+GPTQ
6.54
3.55
5.60
5.00
3.41
QUTRO++GPAAQ
6.46
3.35
5.54
4.96
3.40
AWQ
11.8
12.3
14.2
6.32
4.22
OmniQuant
X
6.58
5.58
3.92
W3A16
QuaRot
39.7
138
146
48.9
5.25
GPTO
7.59
5.37
6.08
5.37
3.72
QuaRO
7.25
4.70
5.86
5.20
3.66
AWQ
4.1c5
8.604
2.9c6
6.2c3
3,9c3
OmniQuant
37.4
17.2
7.81
W2A16
QuaRot
4.0e4
4.7e4
9.7e3
5.2e3
1.2e3
QuaRot+GPTQ
18.5
20.7
10.9
5.60
QuaRot+GPTAQ
13.4
10.7
9.02
7.72
5.18
C. Memory Analysis
We additionally analyze the GPU memory required to perform our algorithm (Algorithm 1). We focus our analysis on two key components in our method: X and P. We first make the observation that X is only involved in the computation of AXX Thus, the memory required for storing is temporary and can be safely released as soon as AXX is computed. P. on the other hand, needs to kept
GPU memory during the iterative quantization updates.
Though X only requires temporary GPU memory, due to the inherent large dimensionality of k, materializing X for the entire model can potentially cause the GPU to suffer from high peak memory usage. We thus utilize GPU-friendly strategy to manipulate the temporary storage of X. As shown in Algorithm 2, for each block that awaits calibration, only the that associates with the block will be materialized in the GPU memory. Assuming the block consists of / layers, the temporary memory required by is bounded by the complexity of O (nkk). As an example, on LLaMA2-7B, our profiling results show that X only temporarily induces ~ 12GB memory space. Note that since the complexity solcly depends on the block size, our storage strategy for X is scalable with the growing model size. In practice, we can further offload these data to
16

---

GPTAQ
CPU memory and only load them GPU when needed, which we empirically find brings negligible overhead to the overall latency but further improves the GPU memory efficiency.
As for P. thanks to the relatively smaller dimensionality of m. n, the memory storage requirement is very small. In Table 8, we detail the dimensionality of different matrices that are kept in GPU memory during the iterative updates (left part) and the one for each local update round within the column blocks. For concrete reference, Table provides the runtime GPU memory requirements for calibrating individual layers within a block LLaMA2-7B.
We would like strengthen and clarify the fact that throughout the entire quantization process, only one single copy of the model block exists inside the GPU memory. The X that contains the FP information of the block is first calculated by forwarding the input through the unquantized block and stored in the CPU memory. Then the same model block inside the GPU memory is quantized and calibrated by our proposed algorithm. As shown in Algorithm 2, the extra memory storage required by our method is solely brought by X and P which we have analyzed above.
Table 8. Dimensions of matrices used GPTOIGPTAQ
Method GPTQ
W
Q
E
P
Q
LQ.Q
Q.Q
E.Q
PQ. 0 0 B B
TA X n
T X TL
TAX X n
0x0
ML
X B
BxB
m B
m X B
GPTAQ
X
X
T T.
T.X
X
B
BxB
m B
X B
Table 9. Memory needed to perform calibration on LLaMA2-7B, B = 128 follows the standard setup.
Layer X
9-proj
k-proj
v-proj
o_proj
up-proj
gate-proj
down_pre
4096 4096
4096
4096
4096
4096
4096 x 4096
11008
4096
11008
4096
4096 11008
GPTQ GPTAQ
0.13GB
0.13GB
0.13GB
0.13GB
0.29GB
0,29GB
0.48GB
0.16GB
0.16GB
0.16GB
0.16GB
0.32GB
0.32GB
0.70GB
17


Reviews:
[{"review_id": "T2eP7Wi8ZR", "reviewer": "Reviewer_3TqR", "summary": "This paper proposed a modification to the widely-used GPTQ method. The main idea is that instead of minimizing the differences between quant(W)*A and W*A, authors proposed to minimize the differences between quant(W)*A with W*A_fp, i.e. its counterpart in the unquantized model. As in typical PTQ works, \"sequential\" quantization, i.e. assuming layer 0 to l-1 are quantized while quantizing layer l, is generally considered more effective, because later layers may have a chance to absorb some quantization errors accumulated from quantizing previous layers. Directly matching layer output to counterparts in unquantized model would be closer to the idea of distillation, which usually requires more iterations and data to achieve better results. Interestingly this work showed that for GPTQ, \"distillation style\" would be a better option than \"sequential PTQ style.\"", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive feedback on our manuscript. We appreciate your interpretation of GPTQ as a \"sequential PTQ style\" method versus the \"distillation style\" of our GPTQv2. Please see our responses to your specific concerns below:\n\n>1. The terms \"asymmetric calibration\" and \"symmetric calibration\" doesn't seem to be very intuitive, and maybe a bit confusing with the symmetric/asymmetric quantization. In fact, this set of terms is not used a lot in the manuscript. Maybe author can consider adding a few sentences around the definition of these terms to enhance the connection between the main proposed concept, so that the readers could grasp the main idea easier.\n\nWe agree that our terminology could be clearer. The terms \"symmetric\" and \"asymmetric\" specifically refer to the calibration objective: symmetric calibration uses the same input activation $\\mathbf{X}$ for both the quantized and full-precision weights, while asymmetric calibration accounts for the input activation discrepancy between quantized weights and full-precision weights. We will add clearer definitions of these terms in the introduction and highlight their conceptual differences. \n\n>2. Since this work is meant to be an improvement of the original GPTQ, it would be beneficial to start the discussion with a comparison to vanilla GPTQ, i.e. weight only, per-group quantization. Maybe author could consider moving Appendix B.3/Table 7 into main manuscript, with a few more examples from different Llama models.\n\nThank you for your suggestion. We agree that demonstrating improvements over vanilla GPTQ (weight-only, per-group quantization) is important since many existing libraries target this setup. In the revised version, we will move Table 7 to the main text and expand it with comprehensive comparisons across multiple LLaMA models. Here are additional results for LLaMA3-8B's perplexity:\n\n| Bitwidth | W4A16-G128 | W3A16-G128 | W2A16-G128 |\n|----------|------------|------------|------------|\n| GPTQ     | 6.71       | 7.91       | 25.24      |\n| GPTQv2   | 6.41       | 7.72       | 14.17      |\n\n\n\n>3. Even though (possibly) the majority of researchers still calls it GPTQ, the original author officially published their work at ICLR 2023 in the name of \"OPTQ\". I would not suggest the author of this work to change all the names/acronyms in the manuscript, but in respect of the choice of the original \"GPTQ\" authors, maybe include both names during first mention/citation and state that for only GPTQ will be used afterward for simplicity reason.\n\nThis is an essential problem that we were not aware of. We'll add a footnote in the introduction to clarify the naming issue. Thanks again for your suggestion.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes, Appendix A.1, seems fine.", "experimental_designs_or_analyses": "Yes, the choice of models include vision transformers and LLMs, model size ranged from <1B, 7-70B, and 405B is proper. The selection of metrics include ImageNet accuracy, wiki2 perplexity, and a range of 0-shot tasks is valid.", "relation_to_broader_scientific_literature": "This work is an improvement of a widely-used method, GPTQ, which is a weight-only quantization method that can address memory-bound issue of LLMs.", "essential_references_not_discussed": "Citations/references are sufficient.", "other_strengths_and_weaknesses": "Strength:\n1. Well written manuscript.\n2. Good amount of experimental data, including those additional results in appendix. Achieved meaningful improvement in accuracy/perplexity compared to original GPTQ.\n3. Considered implementation efficiency and provide improved formulas so that overall process time would be comparable to original GPTQ.\n\nWeakness:\noverall, very nice work. Just a few minor suggestions.\n1. The terms \"asymmetric calibration\" and \"symmetric calibration\" doesn't seem to be very intuitive, and maybe a bit confusing with the symmetric/asymmetric quantization. In fact, this set of terms is not used a lot in the manuscript. Maybe author can consider adding a few sentences around the definition of these terms to enhance the connection between the main proposed concept, so that the readers could grasp the main idea easier.\n\n2. Since this work is meant to be an improvement of the original GPTQ, it would be beneficial to start the discussion with a comparison to vanilla GPTQ, i.e. weight only, per-group quantization. Maybe author could consider moving Appendix B.3/Table 7 into main manuscript, with a few more examples from different Llama models.\n\n3. Even though (possibly) the majority of researchers still calls it GPTQ, the original author officially published their work at ICLR 2023 in the name of \"OPTQ\". I would not suggest the author of this work to change all the names/acronyms in the manuscript, but in respect of the choice of the original \"GPTQ\" authors, maybe include both names during first mention/citation and state that for only GPTQ will be used afterward for simplicity reason.", "other_comments_or_suggestions": "please see Weakness above", "questions_for_authors": "None", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wvBSs5gZgg", "reviewer": "Reviewer_vtou", "summary": "The authors introduce a new quantization method, GPTQv2. The key innovation here is the development of an asymmetric calibration approach, differing fundamentally from GPTQ, by explicitly aligning the quantized layer's outputs to the original, full-precision activations. They derive a closed-form solution using Optimal Brain Compression principles. Experiment evaluations show substantial improvements in model performance across vision and language tasks.\n\n## Update after rebuttal\nI maintain my original score. I am generally satisfied with the authors’ response.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your positive assessment of our theoretical foundation and experimental results. Regarding your concern about hardware-level deployment and overhead analyses, we would like to clarify that GPTQv2 maintains full compatibility with GPTQ's quantization format since we did not modify the `quant()` function in Algorithm 1. This means that our method can leverage all existing hardware-optimized kernels and infrastructure developed for GPTQ without additional overhead during inference.\nTake an example of the format of [`AutoGPTQ`](https://github.com/AutoGPTQ/AutoGPTQ) library, for every quantized layer, the variables are defined as\n\n```python\nclass QuantLinear(nn.Module): \n    def __init__(self, bits, group_size, in_features, out_features):\n        self.bits = bits\n        self.group_size = group_size\n        m, n = in_features, out_features\n        self.qweight = torch.zeros((n, m//32 * self.bits), dtype=torch.int32)\n        self.qzeros = torch.zeros((n//group_size, m//32 * self.bits), dtype=torch.int32)\n        self.scales = torch.zeros((n//group_size, m), dtype=torch.float16)\n        self.g_idx = torch.zeros(n, dtype=torch.int32)\n```\n\nThe same quantization format in GPTQv2 can immediately benefit from specialized kernels like [Marlin](https://github.com/IST-DASLab/marlin) and [ExLLaMA](https://github.com/turboderp-org/exllamav2) without requiring new hardware optimizations. Currently, we are integrating GPTQv2 into popular quantization libraries, and we will expand our hardware-specific analyses in the next version of the paper.", "claims_and_evidence": "The paper claims to achieve superior performance as compared to GPTQ and supports its claims via various experiments.", "methods_and_evaluation_criteria": "Chosen evaluation metrics (e.g., perplexity, accuracy on PiQA, HellaSwag, etc.) are appropriate and standard for the field.", "theoretical_claims": "Theoretical claims sound reasonable, addressing the shortcomings of the original GPTQ work.", "experimental_designs_or_analyses": "Experiments and ablation studies seem methodologically sound and thorough. However, actual hardware numbers would strengthen the work further.", "relation_to_broader_scientific_literature": "GPTQv2 is effectively contextualized against existing methods, clearly highlighting its innovative elements over the original GPTQ algorithm and other finetuning-free quantization methods.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "**Strengths**\n- Clear and thorough theoretical foundation with detailed mathematical derivations provided.\n- Efficient computational strategies that significantly reduce quantization overhead, making practical implementation feasible.\n- Extensive experiments demonstrating clear and consistent performance improvements across different transformer architectures and tasks.\n\n**Weaknesses**:\n\n- Lack of detailed hardware-level deployment and overhead analyses slightly limits practical applicability insights.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tjLl4EG0dW", "reviewer": "Reviewer_fFrA", "summary": "Authors propose a novel fine-tuning free quantization framework GPTQv2 for LLMs. Authors first  analyze the problem of previous \"symmetric calibration\"using optimal brain compression to derive a close-formed solution, and propose a novel \"asymmetric calibration\" to take  quantization error as well as the accumulated asymmetry error into consideration. Secondly, authors utilize various techniques to parallelize the solution calculation, including channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion. Extensive results on various LLMs and datasets reveal the effectiveness of the proposed methods.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your comments and thorough review. Please check our response to your questions. \n\n>1. What does the lambda in eq.10 represent for? Is it a hyper-parameter? If so, then why there is a gradient on it?\n\n$\\lambda$ is the Langrange multiplier, which is not a hyperparameter. By taking derivatives of the Lagrangian with respect to both $\\Delta\\mathbf{w}$ and $\\lambda$ and setting them to zero, we simultaneously: (1) ensure the quantization constraint is satisfied exactly ($\\partial L/\\partial \\lambda=0$ enforces $\\Delta\\mathbf{we}_q^\\top+\\mathbf{w}_q-\\hat{\\mathbf{w}}_q=0$), and (2) find the local minimum of the objective function. This approach follows standard constrained optimization techniques used in the original OBS and OBQ frameworks.\n\n>2. In table. 2, why the performance improvement is more significant on fine-tuning quantization methods than fine-tuning free ones, when compared with GPTQ? It would be better to deeply discuss about the phenomenon.\n\nWe think the reason why finetuning quantization performs worse is the need to handle the massive outliers in activations. This issue was addressed only with QuaRot (FT-free) and SpinQuant (FT-based) until recently. \n\n>3. In table. 4, why the ppl result of GPTQv2' is worse than GPTQ, while the zero-shot avg result is better than GPTQ, which is counter-intuitive.\n\nThis is an excellent observation. Perplexity and zero-shot accuracy measure different aspects of LLM capabilities. Perplexity primarily evaluates next-token prediction on the pre-training distribution (a form of memorization), while zero-shot accuracy tests the model's ability to generalize knowledge to new tasks.\nWhen applying only the second term of our method, we're optimizing for the residual output error from previous layers, which better preserves the model's generalization capabilities at the expense of exact next-token prediction. The full GPTQv2 balances both aspects by combining both terms. This suggests that different quantization objectives might be optimal depending on the downstream task priorities. \n\n\n>4. I curious about the performance improvement when activation quantization is added before weight quantizations. It seems like the improvement compared to GPTQ is more significant under this condition. In my perspective, the quantized $\\tilde{\\mathbf{X}}$ will introduce noise into $\\Delta \\mathbf{X}$, therefore, it should be worse.\n\nThanks for the question. We kindly refer you to our Algorithm 2 in Appendix C. If the activation quantization is enabled during calibration, we will disable it when computing and caching $\\tilde{\\mathbf{X}}$ to ensure we always use the FP model activation. This is why the improvement will be more significant. \n\n\n>5. Seems like the histograms in Figure 4(a) are missing.\n\nSorry for the missing histograms. This appears to be a browser-specific rendering problem. If you download our paper, the histograms should be visible when viewing the PDF in Chrome or in standard PDF readers. We will ensure the figure is displayed correctly in all formats in the revised version of the paper.", "claims_and_evidence": "All claims are well-explained.", "methods_and_evaluation_criteria": "I've checked all theoretical and qualitative analysis and claims in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the methodology and equation derivation.", "theoretical_claims": "I've checked all theoretical and qualitative analysis and claims in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the methodology and equation derivation.", "experimental_designs_or_analyses": "I've checked all experimental settings, comparison and results in this paper. See \"Other Strengths And Weaknesses\" part of this review for my major & minor concerns about the experimental part.", "relation_to_broader_scientific_literature": "All contributions are technical and all datasets used for experiments are open-sourced. Thus no key contributions of this paper related to the broader scientific literature.", "essential_references_not_discussed": "All necessary references are discussed.", "other_strengths_and_weaknesses": "## Major weakness\n1. What does the lambda in eq.10 represent for? Is it a hyper-parameter? If so, then why there is a gradient on it?\n2. In table. 2, why the performance improvement is more significant on fine-tuning quantization methods than fine-tuning free ones, when compared with GPTQ? It would be better to deeply discuss about the phenomenon. \n3. In table. 4, why the ppl result of GPTQv2' is worse than GPTQ, while the zero-shot avg result is better than GPTQ, which is counter-intuitive. \n4. I curious about the performance improvement when activation quantization is added before weight quantizations. It seems like the improvement compared to GPTQ is more significant under this condition. In my perspective, the quantized  $\\\\tilde{X}$ will introduce noise into $\\\\Delta X$, therefore, it should be worse. \n\n## Minor weakness\n1. Seems like the histograms in Figure 4(a) are missing.", "other_comments_or_suggestions": "See above.", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ANzQQyKTDB", "reviewer": "Reviewer_aj2m", "summary": "Following the widely used quantization framework GPTQ, this work identifies the problem in GPTQ named symmetric calibration that emerges from the per-layer optimization scheme. To tackle these challenges, this work proposes a unique calibration pipeline based on asymmetric calibration, which fully considers the quantization error and deviation in the output when updating the weights. Concretely, channel parallelization, neuron decomposition, and Cholesky reformulation for matrix fusion are utilized to parallelize the solution. The proposed GPTQv2 is extensively verified across various LLMs and ViTs on multiple tasks, demonstrating remarkable efficiency and effectiveness. It can be a plugin for QuaRot/SpinQuant and improve the performance with minimal overhead.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your comments and positive feedback. Please check our response below. \n\n>1. Bitwidth settings are limited to W4A4/W2A4. Considering the need for near-lossless quantization of LLMs/ViTs in some application scenarios, W6A6/W8A8 results would be a plus.\n\nThank you for this suggestion. We have conducted additional experiments with W6A6 bitwidth on LLaMA2-7B and LLaMA3-8B to evaluate GPTQv2 in near-lossless quantization scenarios. The results are presented below (reporting WikiText-2 perplexity):\n\n| Model             | LLaMA2-7B | LLaMA3-8B |\n|-------------------|-----------|-----------|\n| Pretrained (FP16) | 5.47      | 6.14      |\n| OmniQuant         | 5.87      | 7.24      |\n| QLLM              | 5.91      | -         |\n| DuQuant           | 5.53      | 6.27      |\n| QuaRot+GPTQ       | 5.50      | 6.24      |\n| QuaRot+GPTQv2     | 5.49      | 6.21      |\n\n\nAs expected, the improvements in the W6A6 setting are more modest compared to those in lower bitwidth scenarios, since higher precision quantization already preserves most of the model's capabilities, leaving less room for enhancement. Nevertheless, GPTQv2 still consistently outperforms GPTQ across both models. \n\n>2. Missing baselines for ViT quantization. Rotation-based methods, such as QuIP, can also be applied in ViT quantization, and FrameQuant[1] is omitted. \n\nThanks for letting us know about the FrameQuant paper. As far as we can tell, QuIP and FrameQuant are performing weight-only quantization on ViTs, which would be less effective than LLM decoding, as the ViT inference is compute-bounded instead of I/O-bounded.\n\nNevertheless, we test DeiT-S with 2-bit per-channel quantization with or without QuIP incoherence processing. The results are shown below, with * indicating our implementation. \n\n| Method             | ImageNet accuracy |\n|--------------------|-------------------|\n| FrameQuant (r=1.0) | 66.35             |\n| QuIP               | 65.70             |\n| GPTQ*              | 57.11             |\n| GPTQv2*            | 60.58             |\n| QuIP + GPTQ*       | 65.45             |\n| QuIP + GPTQv2*     | 68.02             |\n\n\n>3. Figure 4 (a) seems to be missing bars for latency comparison. The latency overhead of GPTQv2 compared to GPTQ is non-negligible, especially on higher dimensions.\n\nSorry for the missing histograms. This appears to be a browser-specific rendering problem. If you download our paper, the histograms should be visible when viewing the PDF in Chrome or in standard PDF readers. We will ensure the figure is displayed correctly in all formats in the revised version of the paper.\n\nFor the GPTQv2 vs GPTQ latency comparison, we can give a theoretical upper bound here. According to Algorithm 1, the operations of v2 are at most 2x of v1. With sufficiently high dimensions, we can expect the latency may approach 2x. However, in practice, we observe 30%-50% more time in typical LLMs (7B to 405B). \n\n>4. For SpinQuant, I think it should be categorized as FT-free since it only involves optimizing the rotation but not updating the weights. In addition, the author of SpinQuant mentioned that optimizing rotation is highly efficient, 0.5h for L3-8B in their paper. It could be the difference in hardware system, please double check it.\n\nWe maintain that SpinQuant should be categorized as finetuning-based rather than finetuning-free for the following reasons:\n\n+ From a computational perspective, optimizing rotation matrices or weights involves the same core operations - both require backpropagation through the network, use of the Straight-Through Estimator (STE), and gradient-based optimization. The fact that the optimization target is a rotation matrix rather than weight values does not reduce the computational requirements.\n\n+ Conceptually, SpinQuant modifies the effective weight representation through rotation optimization. Whether directly updating weights or optimizing transformations applied to weights, both approaches adjust the model's parameters that will be quantized. \n\nRegarding the finetuning time, it’s due to the SpinQuant authors using 8 A100 GPUs to finetune the rotation matrix. Therefore, in Section 5.3, we explained that “we additionally report the GPU Hours (on one A100) required to run the algorithm. Although in practice, SpinQuants runs on 8 A100 GPUs ” \n\n>5. It would be interesting to see if the proposed GPTQv2 could be applied to Diffusion Transformers such as Q-DiT [2]. \n\nThanks for the reference on transformer-based diffusion models. We noticed that Q-DiT pointed out an important observation that activation distribution undergoes continuous changes across timesteps. In this case, the activation asymmetry may accumulate not just through layers but through time steps as well. We expect GPTQv2 will have better performance if $\\Delta \\mathbf{X}$ can capture more information as we did in the experiments of quantization order (Appendix B.1). We will discuss this problem of diffusion models in our paper related work.", "claims_and_evidence": "Most claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria make sense for the problem. Algorithm1 gives a clear introduction to how GPTQv2 differs from GPTQ. The proposed techniques are well-motivated.", "theoretical_claims": "check the correctness of proofs for theoretical claims in the main text.", "experimental_designs_or_analyses": "I have checked the soundness/validity of experimental designs and analyses. Some issues include:\n- Bitwidth settings are limited to W4A4/W2A4. Considering the need for near-lossless quantization of LLMs/ViTs in some application scenarios, W6A6/W8A8 results would be a plus.\n- Missing baselines for ViT quantization. Rotation-based methods such as QuIP can also be applied in ViT quantization, and FrameQuant[1] is omitted.\n- Figure 4 (a) seems to be missing bars for latency comparison. The latency overhead of GPTQv2 compared to GPTQ is non-negligible, especially on higher dimensions.\n- For SpinQuant, I think it should be categorized as FT-free since it only involves optimizing the rotation but not updating the weights. In addition, the author of SpinQuant mentioned that optimizing rotation is highly efficient, 0.5h for L3-8B in their paper. It could be the difference in hardware system, please double check it.\n- It would be interesting to see if the proposed GPTQv2 could be applied to Diffusion Transformers such as Q-DiT [2]. (No need to perform additional experiments, discussion on it would be enough)\n\n[1] FrameQuant: Flexible Low-Bit Quantization for Transformers, ICML 2024\n\n[2] Q-DiT: Accurate Post-Training Quantization for Diffusion Transformers, arxiv 2023/CVPR 2025", "relation_to_broader_scientific_literature": "This work is built on GPTQ, but the core contribution is novel.", "essential_references_not_discussed": "Existing work related to quantization and Optimal Brain Surgeon (OBS) are cited.", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Please see the points above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper enhances the widely used GPTQ approach, with its primary novelty being the development of an asymmetric calibration method to align the quantized layer's outputs with full-precision activations. The work achieves significant improvements across benchmarks and provides extensive evaluation. AC agrees with all reviewers in recommending acceptance of this paper.
Decision: Accept (poster)