BoA: Attention-aware Post-training Quantization without Backpropagation
Authors: Junhan Kim
Ho-young Kim
Eulrang Cho
Chungman Lee
Joonyoung Kim
Yongkweon Jeon
Venue: ICML 2025 poster (2025)
Keywords: Quantization
Hyper-scale LLMs
Attention
Hessian
Primary area: deep_learning->everything_else
TL;DR: We propose a novel backpropagation-free post-training quantization algorithm that optimizes integer weights by considering inter-layer dependencies inside the attention module.

Abstract: Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices. 
Early methods developed for small-scale networks, such as ResNet, rely on gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters.
While recently proposed backpropagation-free or transformation-based methods alleviate this issue, they ignore inter-layer interactions or use the naive nearest-rounding-based quantized weight assignment to save the heavy computational cost of weight optimization.
In this paper, we introduce a novel backpropagation-free PTQ algorithm that optimizes quantized weights by considering inter-layer dependencies. 
The key innovation is the development of attention-aware Hessian matrices that capture inter-layer interactions within the attention module. 
Extensive experiments demonstrate that our approach not only outperforms existing weight quantization methods but also shows good synergy with conventional methods to suppress activation outliers, leading to state-of-the-art weight-activation quantization performance.
The code will be available at https://github.com/SamsungLabs/BoA.

BOA: Attention-aware Post-training Quantization without Backpropagation
Junhan Kim '1
Ho-young Kim
Eulrang Cho
Chungman Lee"
Joonyoung Kim Yongkweon Jeon
Abstract
that primarily support fixed-poine arithmetic.
Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices. Early methods developed for small-scale networks, such as ResNet, rely gradient-based optimization, which becomes impractical for hyper-scale LLMs with billions of parameters. While recently pro- posed backpropagation-fri or transformation- based methods alleviate this issue, they ignore inter-layer interactions or use the naive nearest- rounding-basee quantized weight assignment save the heavy computational cost of weight op- timization. In this paper, we introduce novel backpropagation--fre PTQ algorithm that opti- mizes quantized weights by considering inter- layer dependencies. The key innovation the development of attention-aware Hessian matri- ces that capture inter-layer interactions within the attention module. Extensive experiments demon- strate that our approach not only outperforms existing weight quantization methods but also shows good synergy with conventional methods to suppress activation outliers, leading to state- of-the-art weight-activation quantization perfor- mance. The code will be available https: //github. com/ Swtp///aaannnaaaaaaaaaaaaaaaaaaa
Two main categories of quantization approaches have been proposed to preserve the performance of original models: quantization-aware training (QAT) (Jin et al., 2021; Liu et al., 2023) and post-training quantization (PTQ) (Nagel et al., 2020; Kim et al., 2024). Although QAT can potentially outperform PTQ, its practicality diminishes when handling hyper-scale LLMs featuring billions of parameters. Con- sequently, recent quantization efforts have been directed toward PTQ.
Although classic PTQ methods have successfully quan- tized small models such as ResNet (Nagel et al., 2020; Li et al., 2021; Jeon et al., 2022), they rely on time-consuming gradient-baser optimization, so their efficacy decreases when the complexity of LLMs increases. Accordingly, re- cent efforts have shifted to develop (a) backpropagation-fii methods that optimize quantized weights based on Hes- sian (Frantar et al., 2023) or (b) transformation-base meth- ods that convert a model to be robust to quantization by applying smoothing, rotation, and permutation (Shao et al., 2023; Ashkboos et al., 2024; Liu et al., 2024; Lin et al., 2024). However, their weight quantization performance limited as they ignore inter-layer dependencies use naive nearest rounding for weight quantization.
In this paper, we propose novel backpropagation--ie -free PTQ algorithm that considers inter-layer dependencies in opti- mizing quantized weights. Our contributions are as follows:
1. Introduction
We propose novel PTQ algorithm called BoA! The key contribution is to exploit the attention reconstruc- tion error, not the layer-wise reconstruction error, in approximating the Hessian to consider inter-layer de- pendencies within the attention module (Section 3.2). To our knowledge, BoA is the first method optimize quantized weights by considering inter-layer dependen- cies without relying gradient-based optimization.
The explosive growth in complexity (parameters) of large language models (LLMs) (Touvron et al., 2023a; Zhang et al., 2022) has resulted in a proportional increase in com- putational costs, which has prompted an urgent need for effi- cient model processing and compression strategies. Quanti- zation has emerged a pivotal solution and essential step for deploying Al models resource-constraine devices
While the proposed Hessian facilitates the considera- tion of inter-layer dependencies, it requires additional memory and computations. To mitigate the overhead, we propose several techniques, including Hessian re- laxation, efficient computation of inverse Hessians, and
'Equal contribution 'Samsung Research, Seoul, Repub- lie of Korea. Correspondence to: Yongkweon Jeon <drag- won.jeon @ hamsung.com..
Proceedings of the 4dnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
BBackpropaaatinnffres optimization for Attention-aware PTQ

---

BoA: Attention-aware Post-training Quantization without Backpropagation
head-wise simultaneous quantization (Section 3.3).
(b) transformation-basse methods that convert model to be more robust to quantization by applying smoothing (e.g., SmoothQuant (Xiao et al., 2023), OmniQuant (Shao et al., 2023), and AffineQuant (Ma et al., 2024)), rotation (e.g., QuaRot (Ashkboos et al., 2024) and SpinQuant (Liu et al., 2024)), or permutation (e.g. DuQuant (Lin et al., 2024)).
We evaluate BoA on publicly available LLMs. From extensive experiments, we show that BoA outper- forms existing backpropagation-fre weight quanti- zation methods by a significant margin, particularly for low-bit precision (e.g., INT2) (Section 4.2). Fur- thermore, when combined with existing methods to suppress outliers (Ashkboos et al., 2024; Liu et al., 2024), BoA achieves the state-of-the-art performance for both weight-only and weight- activation quantiza- tion (Sections 4.2 and 4.3).
The proposed BoA belongs to the class (a) in that it opti- mizes quantized weights using the Hessian without relying backpropagation. Furthermore, similar to GPTQ, BoA can be combined with transformation as QuaRot and SpinQuant (see Section 4). The primary difference over GPTQ lies in our optimization objective: while GPTQ assumes layer-wise independence and focuses on layer-wise reconstruction (which leads to performance degradation), our approach explicitly aims to preserve the attention output, enabling us account for inter-layer de- pendencies within the attention module.
methods such
2. Related Works
When calibration data are available, PTQ aims to minimize the increase in task loss incurred by quantization. Assuming the convergence of network and the independence between layers, the problem of quantizing weights to minimize task loss degradation can be formulated as the following layer- wise reconstruction problem (Nagel al., 2020):
Other algorithms exploiting different strategies have also been proposed. For example, SpQR (Dettmers et al., 2023), SquezzeLLM (Kim et al., 2023), and OAC (Edalati et al., 2024) proposed a mixed precision approach that assigns a large bit-width to quantization-sensiivve weights or retains them in full full-precision. Compared to the standard uniform quantization, these algorithms require additional processing and memory costs in the inference and need special hard- ware and dedicated kernels without which accelerating the inference may not be easy. Furthermore, unlike server-grade GPUs, on-device NPUs (e.g. Qualcomm Hexagon) lack sup- port for the mixed precision format, and customizing ker- nels for desired functionalities is very challenging. Thus, we exclude these algorithms in our comparison and focus on the more universally supported uniform quantization for- mat. We also exclude recent vector quantization approaches (e.g. QulP# (Tseng et al., 2024) and AQLM (Egiazarian et al., 2024)) because they need additional memory (bits) for storing codebooks, which are required to perform the dequantization during the inference.
[[e(w())- w -w())
x [.
min AWO
(1)
F
where ((-1) is the input to the b-th layer parameterized by (( and Q is a quantization function. For a uniform quantization, if the nearest quantization bin is assigned to each weight Q is defined as
((x) = 8 (clamp (1=1 + +2,0, - -1,,,, -)),
where 8, z, n are the scale, zero-point, and bit-width, respec- tively, and [-] represents the round-off.
Early works aimed to optimize the weight-rounding mecha- nism (Nagel et al., 2020; Hubara et al., 2021; Li et al., 2021; Jeon et al., 2023a). Instead of allocating the nearest inte- ger, these studies attempted to assign integer weights that minimize the reconstruction error. One popular approach is AdaRound, which learns quantized weights satisfying (1) via backpropagation (Nagel et al., 2020). This algorithm has been extended to BRECQ where the block-wise reconstruc- tion error has been used, instead of the layer-wise reconstruc- tion error, to consider the inter-layer dependencies (Iii et al., 2021). Although AdaRound and BRECQ have successfully quantized small-sized models, they rely on time-consuming gradient-based optimizations, which renders their applica- tion LLMs with billions of parameters challenging. Con- sequently, recent efforts have shifted towards the develop- ment of cost-effective quantization methods for L.Mss.
3. Method
3.1. Overview of Proposed BoA
The proposed BoA quantizes weights by repeating quanti- zation and weight-update steps; once BoA quantizes one weight, it updates the remaining (not-yet-quaanizedd) weights to compensate for the task loss degradation caused by the quantization. The update formula to compensate for the quantization of the g-th weight w. is formulated as (Frantar et al., 2023) dw ((w)y) Wig [U],.. where U = Chol(H (2) [
These efforts can be classified into two orthogonal classes: (a) backpropagation-fri -free Hessian-bace integer weight op- timization methods (e.g., GPTQ (Frantar et al., 2023)) and
where H is the Hessian and Chol(.) denotes a Cholesky de- composition (ie., U is an upper triangular matrix satisfying
2

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 1. Approximated Hessians in GPTQ and the proposed BoA
Hessian for query When quantizing the query projection weights W q. W wutth and Vn remain unchanged, but the attention weights A change. Using the first-order Taylor polynomial, the perturbation in Ab can be approximated as
Method
Layer
H How
Has
GPTQ
W (Q.K.V)
2Xx*
Wa.
2XX"
K
WKA
2XX
Q Q
QQ + AQ,)
QQ.K%
BoA
A =0
Wy.
A
W
o W
Woath
Vdh
ddh
W
2X
X
@
I
AQ ddh
X vdn
K.
For We, and & embedding, see (14) and (15).
K. of models exploiting rotary position
J
Q.4
(5)
where Ja is the Jacobian matrix the softmax function a. Thus, the attention reconstruction error is expressed as
[AMHA(X)]
|| our, (AA, V
The key difference over GPTQ lies in the approximation of the Hessian H. In GPTQ, the layer-wise reconstruction error x wise independence has been assumed) approximate H which yields the following Hessian equation":
W A J. M.J.K, K vd,
in (1) has been used (i.., the layer-
AW Q Q
(6) F
which yields the following Hessian for W We, (see Foot- note 2):
((W = 2X &
(3)
out out V J.
W di
H (w..) -2X
where w(( is the flattened representation of W H(w!!) is the Hessian for the ε-th layer, denotes the Kronecker product operation, and I is the identity matrix. The approxi- mated Hessian H (w(") in GPTQ relies solely on the input, which means that GPTQ cannot consider the influence of other layers. In other words, GPTQ neglects inter-layer dependencies within the attention module, a crucial aspect of Transformers, which results in limited performance (Jeon et al., 2023b). To overcome this, we develop Hessians that incorporate inter-layer dependencies and then use them in- stead of the conventional Hessian (3).
Hessian for key As in the quantization of the query pro- jection weights, the attention weight A changes when quantizing the key projection weights W WK,.. By following the steps for (5), A, can be approximated as
QR JJ Q, X.I
AA, ~
ddn
vdh
and then the attention reconstruction error is expressed as
Q₂
||АHHА(]]]
AW K. XJ Vn W
Vd,
out
In Table 1, we summarize the proposed Hessians; the de- tailed derivation is provided in the following subsections. It can be observed that the proposed Hessians not only contain the term related to the input X, but also involve the terms related to other layers (e.g., K K,, for Wo).
Thus, we obtain the following Hessian for WK
Q Q dp
W o V J H H Aut,
X
(8)
Hessian for value When quantizing the weights W v.h. Wy.k the value projection, only Vn changes. Thus, we have
of
3.2. Proposed Attention-aware Hessian
A MHA(X)] = || (A,AV,) P
To consider the inter-layer dependencies within the attention module, we exploit the attention reconstruction error rather than the layer-wise reconstruction error when approximating the Hessian. For an input sequence X E RdxL, the output of the multi-head attention (MHA) is expressed as
W ou A W V.A V.A XA 1b:
which yields the following Hessian for WV.k:
H (ww,w) = 2XA X W A h
W Wouth
(9)
H
Q. Vdr.
Hessian for out When the out-projection weights W Wouth are quantized, the attention reconstruction error is
MHA(X)
W oo (A, VA)7, ",, A,=o
(4)
hml
a MHA(X) = l am A (A,V))
where Qh. Kn, VA RLxda are query, key, value for the h-th attention head, dn is the head dimension, a is the row- wise softmax function, and H is the total number of heads.
Thus, the corresponding Hessian is H =W A A VA @ I=2X out X
but I,
(10)
The second-order derivative of || A WM : with respect to A Awix is M & M M, (see Appendix A for the proof).
where Xouth = (A, V.)..
3

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Algorithm 1 BOA
Input: weights W (Q.K.V) E B _dxxd and inputs X of the Transformer layer
I: for W E (Wq, WK, Wr) do
2: Initialize quantized output: Q +- 0Hxd xd
3: Initialize crw-wise) quantization errors: E OHxd
4: Compute attention-aware Hessians: H,, = A @ Howw,
D. see Table
5: Set step size (scale) S: ming tr W W
6: Compute inverse Hessians H and H
7: Compute a = Chol(H ool, and U = Chol(H
8:
for O,... ................................ d, 1 do
9:
Construct W E R #xd by stacking the j-th rows [Walis:
10:
Quantize W (Q E) GPTQ(W6), S)
D Appendix E Proposition 3.1
[1 Alyis BB,, Uon,
11:
Update remaining rows: [Wll..: WWalp.
12:
end for
13:
end for
Output: quantized weights Q
3.3. Efficient Implementation of BoA
We note that if rotary position embedding (RoPE) (Su et al., 2023) is used, the MHA output is different from that in (4), and thus the corresponding attention-aware Hessians would also be different. Specifically, since the attention weight A for models exploiting RoPE is expressed as
While inter-layer dependencies within the attention module can be considered by exploiting the proposed Hessians, they are significantly more complex than the conventional Hes- sian in (3), which may incur high computational costs. For example, computing the proposed Hessians in (7) and (8) would be more expensive than computing the conventional one in (3). In this subsection, present techniques miti- gate the computational overheads incurred by the proposed attention-awae Hessians.
Q Vih
A
Q Th--RoPE(QN)) (RPPOE((K))
(13)
the surrogate || |K, l7 used to develop the attention-
aware Hessian in (11) changes to A 1| f.- As result, we obtain different Hessians for models exploiting RoPE (see Appendix B for the detailed derivation):
Hessian relaxation The largest overhead related to the computation of the proposed Hessians is the Jacobian matrix J, in (7) and (8). For an input sequence of length T., the shape J. is H x L L x L, which requires large amount of memory and high computational cost (e.g., more than 400 GB even for the OPT-1 15M model when I. = 2048).
H (wQ.a) 2XX"
R
K
(14)
L
To mitigate such overhead, we establish a relaxed Hessian that does not require computing J,, To this end, we build the following upper bound for the attention reconstruction error in (6), which will be used its surrogate:
H (wK.)) = 2Xx
& I
R
Q
Q
(15)
where R& be the rotary matrix for the f-th token (see eq. (15)
in (Su al., 2023)).
W V Vdh
| MHA(X)II
K.AW W
We summarize the relaxed Hessians in Table 1.
Noting that the constant term W A h. Ja Il does not affect quantization," 3 we use the term ||K A W | as a surrogate of the attention reconstruction error when deriving the Hessian for W q.h. which yields the following Hessian:
Efficient computation of inverse Hessians After com- puting Hessians, their inverse matrices need to be computed to update the remaining weights after each quantization (see (2)). Owing to the size of the proposed Hessians be- ing dd,, x ddg, the complexity of the computation of the inverse Hessian would be O(da dg) in approach. This is considerably more expensive than the complexity O(d)) of GPTQ, where the inverse of only XXT E Rddxd in (3) is needed (Frantar et al., 2023).
H (wo n) = 2XX" K.
(11)
Similarly, we can establish relaxed Hessian for W K.A.
(wk,)) 2
Q6 Q
(12)
'hhe update 8w in (2) is affected by the constant multiple [cU])
For the efficient inverse computation, we exploit the use- ful properties of the Kronecker product (see (17a)-(17c) in
H because |cU]../ [cU] j.qq = [U].:/ [U], g. for any constant c.

---

BoA: Attention-aware Post-training Quantization without Backpropagation
How 2XX7
His 2XX"
Table 2. Processing time (hour) of BoA with and without simulta- ncous quantization of different heads
Waa
Wi.1
independence
head wise
Simultameous Quantization
LLaMA Model Size
7B
13B
30B
27.75
51.66
135.4
Kuki
Wow
Wart
0
0.961
1,553
3.295
(a) Hessians the conventional GPTQ (lef)) and the proposed BoA (right)
where cool, Chol(H cot,)) Uiow, = Chol (H now ST. and e E RRidd is the quantization error of the j-th row defined as e₁ ([Wn)s
2XX"
2XXx
[Wes]
Meal.
update
Houe K,KK
W
W
quantize stack of
update other
quantize stack of Ind rows
Proof. See Appendix C.
(b) Quantizaaion procedire in BoA
Figure 1. Illustration of BoA for the query projection Wo-
The update formula in (16) implies that do not need to compute and store the Cholesky decomposition U,- = c @ Unw,A of the full Hessian H,, for updating weights; only Usll, and U mwwh are sufficient for updating weights, and thus we can save the memory cost caused by the high- dimensional Kronecker product operation. Proposition 3.1 also implies that the conventional GPTQ cannot compen- sate quantization error of certain row by updating other rows because How = I (see (3)) and thus U = I and [oW].. = O for all i + j.
Appendix A). Specifically, let H = Hes & Hoo where
Haal E Rddd and Hoow E RR. xds, then we obtain
H = (Hool 8 Howw) = H col H Tow
This implies that the inverse Hessian H can be obtained by computing H and HoT (line 5 in Algorithm 1) whose complexity (dd²) + O(dR) (= O(d)), not (ddd))) Similarly, we can efficiently perform the Cholesky decom- position (i.e., Chol(H ) in (2)) with the same order of complexity as in GPTQ. Specifically, if L, = (Hol(H and L2 = Chol(H tow ) H can be expressed as
col
While the proposed BoA can compensate for the quanti- zation error of each row, the rows must be quantized se- quentially (not simultaneously). For example, the second row can be quantized after being updated to compensate for the quantization error of the first row. To accelerate the quantization process, we assume independence between dif- ferent attention heads (see Figure I(a)), under which rows related to different heads are independent and can thus be quantized together. For a better understanding, we consider the query projection Wq Q as an example (see Figure ((b)). In the quantization step, we stack the j-th rows [W Q.nis of all different heads, constructing the sub-weight matrix w() E #xd (line 8 in Algorithm 1). Because the rows of Q
H L, @ L2L,- = (I, @ (L, & Lz)
Subsequently. noting that the Kronecker product of lower triangular matrices is also lower triangular, we obtain
Chol(H = L, @ L2 = Chol(H col @ Chol(H ).
Thus, we can obtain Chol(H 1) by computing (hol(H ) col
and (m))) (line 6 in Algorithm 1). Consequently, the computational complexity of the Choleksy decomposition would be (dd), not ())
w(O) are mutually independent, all the rows of W be quantized simultaneously as in GPTQ (line 9 in Algo- rithm 1). Following the quantization of j-th rows, we com- pensate for the quantization error by updating the remaining rows. In this update step, we use the refined weight-update formula in (16) (line 10 in Algorithm 1).
Q can
Simultaneous quantization of different heads Because the proposed Hessians model the dependency between dif- ferent rows (e.g., Haw- = K K, for Wq.:: see (11)), we can compensate for the quantization error of a certain row by updating other rows. Specifically. the row-update for- mula is formulated in the following proposition for given Hessian H- Heol Ø Hoow
We measure quantization processing times of BoA with and without simultaneous quantization to evaluate how much the head-wise joint quantization can accelerate the quantization process. From Table 2, we observe that BoA requires a significantly long processing time without the simultaneous quantization (more than one day even for the 7B model), This is because all rows need to be quantized sequentially (e.g., 4096 rows are quantized sequentially for LLaMA- -B)) and thus the massive compute capabilities of modern GPUs cannot be utilized properly. As evident, we can significantly
Proposition 3.1. Let Wr be a da d matrix whose Hessian is Hr, = H & Hoow, Suppose Wt. is quantized via (2). If the j-th row of Wi has been quantized, then the update formula to compensate for the quantization is expressed as
[UT Wij e. Ucot,
@Wnle
(16)
5

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table Weight-only quantization performance LLaMA2 and I.LaMA3 models without transformation
Model
Precision
Method
Wiki2 PPI.(L) (L)
sbot Accuracy (t)
Arc-c
Arc-c
BQ
HS
LAMB
OBQA
PIQA
WG
Average
FP16
Baseline
5.473
45,90
74.66
77.92
75.94
70.86
44,00
78.89
68.90
67.13
RTN
7,8c3
26.45
26.18
39.30
25.99
0.00
24.20
49.40
49.96
30.19
INT2
GPTQ
42.52
23.95
28,20
63.22
54,70
42.50
LLaMA2-7B
BOA
12.76
28.33
47.73
64.71
45.46
31.33
29.60
64,85
54,46
45.81
RTN
342.4
22.53
35.06
44.13
28.87
1.63
26.40
58.11
48.70
33. $3.18
INT3
GPTQ
6.719
36.95
59.72
65.54
66.68
58.
39.20
74.65
66.06
58.
BOA
6.007
40.27
72.00
66.60
41.20
78.35
67.64
63.70
FP16
Baseline
4.885
49.06
77.65
80.49
79.38
73:41
45.80
$0.69
72.22
69. 00.84
RIN
5.763
27.47
26.89
37.95
25.97
0.00
25,20
49.08
48.38
30.12
INT2
GPTQ
35.08
21.76
35.31
61.96
35.22
19.41
28.80
57.29
52.88
39.08
LLaMA2-13B
BOA
18.33
29.78
46,30
62.23
49,30
29.33
27.60
63.33
52.64
45,06
RIN
227.2
23.98
28.75
49.24
28.51
4.46
24.40
53.26
50.75
32,92
INT3
GPTQ
9,790
67.06
55.57
47.61
37.00
73.18
61.33
54.84
BOA
5,833
43.52
69.28
78.93
74.71
65.22
35,20
77.42
62.51
63.35
FP16
Baseline
6.137
53.67
77.61
81.19
79.15
72.23
45.00
81.01
73.24
70.39
RIN
6.6c4
25.51
26.34
0.00
29.00
51.52
50.20
32.79
INT2
GPTQ
24.54
23.29
32.58
53.19
39.17
7.10
27.00
53.32
$2.25
36.01
LLaMA3-8B
BOA
21.70
26.62
44.87
60.52
43.34
16.89
29.40
59.85
55,80
42.16
RTN
129.1
23.21
33.88
55.60
34.83
4.60
25.20
58.22
$2.57
36.01
INT3
GPTQ
8.226
41,47
63.01
75.47
70.05
59.53
40.60
73.78
69.85
61.72
BOA
7.782
45,14
72.77
78.69
72.66
62,41
42,60
77.31
71.35
65.37
FP16
Baseline
13.15
38.14
63.26
69.51
60.78
54.38
34.60
74.37
59.51
56.82
RTN
6,304
26.96
25.59
41.53
26.05
0.01
26.40
51.52
50.59
31.08
IN12
GPTQ
538.9
25.26
26.64
37.83
26.41
0.22
27.60
51.41
48.46
30.48
LLaMA3-2-IB
BOA
312.2
25.09
26.85
40.06
27.17
1.42
27.00
51.96
51.07
31.33
RIN
0.58
52.18
49.17
33.19
INT3
GPTQ
112.0
24.06
39.48
53.85
31.07
13.85
27.80
60.07
49.33
37. 37.44
BOA
59.97
48.33
31.95
29.60
66.65
53.99
47.05
FP16
Baseline
11.04
46.16
67.80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
RIN
2.0g4
26.79
26.52
37.89
25.93
0.00
30,80
50.92
49.09
30.99
INT2
GPTQ
27.78
33.83
4.38
28.60
52.23
51.14
34.39
LLaMA3.2-3B
BOA
54.64
25.77
35.48
57.52
35.63
14.42
29.00
56.96
53.43
38.53
RTN
882.6
26.37
27.86
45.87
37.04
1.44
26.00
53.92
48.46
33.37
INT3
GPTQ
46.14
28.92
37.63
44.01
39.27
18.76
28.60
61.64
54,70
39. 39.19
BOA
13.64
42.32
66.12
77.52
64.46
54.18
35.20
72.69
62.51
59.38
Resulls high bit widhhs and resalts on ILaMM1 and OPT models are provided in Appendix
page limitation.
reduce the processing time by applying the head-wise joint quantization (more than 40 times reduction on the 30B model).
the perplexity (PPL) score on the WikiText-2 test dataset and accuracy on eight zero-shot commonsense reasoning tasks (ARC-challenge (Are-c) and ARC-easy (Arc-e) (Clark et al., 2018), BoolQ (BQ) (Clark et al., 2019), HellaSwag (HS) (Zellers et al., 2019), LAMBADA (LAMB) (Paperno et al., 2016), OpenbookQA (OBQA) (Mihaylov et al., 2018), 8), PIQA (Bisk et al., 2020), and WinoGrande (WG) (Sak- aguchi et al., 2021)). All experiments were conducted using a single NVIDIA H100 GPU (80 GB).
4. Experiments
In this section, we evaluate the weight-only quantization performance of the proposed BoA (Section 4.2) and synergy with transformation-base methods in terms of weight-only and weight- activation quantization (Sections 4.2 and 4.3).
When determining a quantization order in BOA, the heuris- tic introduced by GPTQ can be used; the column/row cor- responding to the largest diag(Ha.l) most quantization-sensiivv column/row) is first quantized for better compensation. Empirically, we observed that this heuristic could occasionally enhance the performance, yet at other times, it may result in inferior performance. For GPTQ and the proposed BoA, we conduct experiments with and without this heuristic and report the better results.
4.1. Experimental Setup
(How) (i.e., the
We conduct experiments on OPT (Zhang et al., 2022), LLaMA (Touvron al., 2023a)). LLaMA2 (Touvron al., 2023b), and LLaMA3. As in previous studies (Shao al., 2023; Ma et al., 2024; Lin al., 2024; Ashkboos et al., 2024; Liu et al., 2024), we construct calibration dataset by sam- pling 128 random sequences of length 2048 from WikiText- 2 (Merity et al., 2016). As a performance metric, we use
6

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 4. Memory costs of GPTQ and the proposed BoA
Table 5. Processing time (hour) of different quantization methods
Method
Memory Cost (GB)
Wiki2 PPL (J)
Method
Inter-ayer Dependoncy
Optimization
LLaMA Model Size
7B
138
30B
7B
13B
308
Type
7B
138
308
GPTQ
4.426
5.872
8:271
19.63
34.63
9.770
GPTQ
one-shot
0.12
0.20
BoA
9.550
16.56
32.79
10.28
8.309
6.669
OmniQuaant
gradient
1.83
3.32
7.66
Relaxed BOA"
6,446
8.397
11.55
10:53
8.700
7.007
AffineQuant
4.32
8.41
21.4
DuQuamt-LWC
-based
1.22
2.08
4.55
Attention- aware lessians have been upplied to query and key, but not to value.
BoA
0
one-shot
0.96
1.55
3.30
4.2. Weight-Only Quantization Results
error of each row (see Figure 1(b)). Clearly, there exists trade-off between quantization speed and accuracy. In real cases, when one needs to consider inter-layer depen- dencies to preserve the performance of the original model as much as possible, the proposed BOA would be an in- triguing solution when compared to existing gradient-base approaches (see Table 5 and Table 6). When faster quan- tization is required, one possible solution is to reduce the number of sequential quantizations by quantizing multiple rows in each head simultancously, which will be considered in our future studies.
Comparison with GPTQ We first compare the proposed BoA with GPTQ to demonstrate the efficacy of the proposed attention aware Hessians. In this experiment, we do not ap- ply any model transformation method (e.g. QuaRot (Ashk- boos et al., 2024) and SpinQuant (Liu et al., 2024)) to solely evaluate the effectiveness of the proposed Hessians.
In Table and Tables 8-10 (see Appendix D.I), we summa- rize the PPL and zero-shot task performances of BoA and GPTQ. We also include the performance of the rounding- to-nearest (RTN) method which naively assigns the near- est quantization bin. We observe BoA and GPTQ exhibit reasonable PPL even for INT2 quantization where RTN collapses significantly (i.e., PPI. is almost 10*). This is because BoA and GPTQ minimize the task loss degrada- tion rather than the weight quantization error AW. As evident, BOA significantly surpasses GPTQ on all models in both PPL and zero-shot accuracy. For example, BoA achieves 10%p accuracy improvement on the 3-bit quan- tized LLaMA2-13B and LLaMA3.2-1B models. In addition, BoA shows 20%p higher accuracy for the 3-bit quantized LLaMA3.2-3B model. The key factor leading to such an outstanding performance is that BoA considers inter-layer dependencies within the attention module by exploiting the proposed attention- aware Hessians while GPTQ assumes the independence of layers.
Comparison with transformation- based methods To improve the quantization performance, recent studies have transformed models by applying smoothing, rotation, or permutation (Shao et al., 2023; Ma al., 2024; Ashkboos et al., 2024; Liu et al., 2024; Lin et al., 2024). In this experi- ment, we evaluate BoA under those model transformations (see Table 6 and Tables 1 11-12 in Appendix D.2). For con- ventional methods, we ran the official codes provided by the authors and reported the obtained results; details for implementing conventional methods are provided in Ap- pendix D.2. When measuring the performance of BoA, we also transformed the model for a fair comparison. We ap- plied QuaRot for the transformation because QuaRot does not require any training and incurs no extra costs during the actual inference (Ashkboos et al., 2024).
In Table 4, we summarize the memory costs of BoA and GPTQ. We observe that BoA requires larger memory be- cause BoA additionally uses outputs of other layers to con- sider inter-layer dependencies. It is worth mentioning that when the memory resource is limited, BoA can still be used with slight relaxation. Noting that the large memory cost of BoA is attributable to the Hessian for the value projection (Hoot, A X in (9)) whose shape is H d x d, we can greatly reduce the memory cost by exploiting the standard Hessian in (3) for the value projection and apply- ing the proposed attention-awar aware Hessians only for query and key projections. In doing so, BoA needs slightly more memory (e.g. 3 GB for 30B; see Table 4) yet still performs much better than GPTQ
We observe that the performance of the proposed BoA is boosted significantly when applying the transformation. For example, the PPLs of the bt quantized LLaMA2- 13B LLaMA3-8B models improve from 18.33 / 21.70 to 8.237 / 15.24, respectively (see Tables 3 and 6). For INT3 quantization, BoA almost preserves the performance of the original full- null-precision model: the accuracy drop is 2.3%p for L.LMMA-8B and 1.3%p for LLaMA2-133B (see Table 11). For all models, we achieve at least 5%p improvement in the zero-shot accuracy (in particular 1 1% improvement for the 2-bit quantized LLaMA2-13B model). We note that the performance gap between BoA and GPTQ remains significant after the model transformation; for example, the zero-shot accuracy of BoA on the 2- quantized LLaMA2- 7B model is 12%p larger than that obtained by QuaRot- GPTQ.
We now compare the processing times of BoA and GPTQ. As evident from Table 5, BoA requires a longer processing time than that needed by GPTQ. This is because GPTQ quantizes all the rows simultancously, while BoA sequen- tially quantizes them to compensate for the quantization
Finally, we emphasize that BoA not only surpasses existing
7

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 6. Weight-only quantization performance on transformed LLaMA2 and M.M. models
Zeno shot Accuracy (1)
Model
Precision
Method
Wiki2 PPI. (4)
Arc-c
Arc-c
BQ
HS
LAMB
OBQA
PIQA
WG
Avernge
FP
Baseline
5.473
45,90
74.66
77.92
75.94
70.86
44.00
78.89
68.90
67.13
OnniQuant
21.85
25.17
36.91
61.80
38.38
5.80
28.40
57.40
49.49
37.92
AffineQuant
91.19
23.46
29.17
40.03
27.4
51.78
15.05
QuaRot-RTN
1.4
27.22
26.05
37.83
26.19
0.01
25.60
50.54
48.70
30.27
LLaMA2-7B
INT2
QuaRot-GPTQ
22.05
22.27
36.24
61.99
33.54
20.63
28.60
56.75
51.78
38,98
DuQuant
1.6c4
26.62
25.72
39.08
26.06
0.00
22.20
49.73
50.12
29.94
DuQuant LWC
46.27
23.12
28.96
44,04
29.60
2.41
27.40
53.70
49.17
32.30
BoA
10.42
29.69
54.84
64.37
52.03
46.54
33.60
67.52
59.43
51.00
FP
Baseline
4,885
49.06
77.65
80.49
79.38
73.41
45.80
80.69
72.22
69.84
OmniQuant
12.92
26.71
46.00
63.64
51.19
16.93
31.40
64.64
52.64
44.14
AffinsQuant
9.415
26.96
49.83
63.15
52.40
26.58
33.60
64.47
53.83
46.35
QuaRot-KIN
7.9c3
27.22
26.26
37.83
25.74
0.00
24.00
49.02
49.17
29.91
LLMMA2-13B
INT2
QuaRot GPTQ
9,593
31.66
56.52
63.15
48.81
36.93
32.60
66.05
60.38
49.51
DuQuant
1.4p4
28.33
26.64
44.43
26.12
0.00
25.00
50,16
49.57
31.28
DuQuant + LWC
10:40
28,50
46.51
63,70
52.79
25.24
30.80
65.
54.14
45.85
BoA
8.237
35.49
63.97
71.28
58.36
56.25
35.40
71.82
62.75
56.92
FP
Baseline
6.137
53.67
77.61
81.19
79.15
72.23
45.00
81.01
73.24
70.39
QnniQuant
955.8
22.78
28.24
37.83
26.18
0.00
27.20
52.83
49.01
30.51
AllinQuuan"
1.1e3
23.46
27.31
37.86
26.04
0.00
26.80
52.12
51.22
30.60
QuaRoi RTN
3.5e5
26.19
25.21
39.02
26.86
0.00
28.60
50.38
49.41
30.71
LLaMA3-8B
INT2
QunRot-GPT
18:28
28.33
46.68
64.10
45.04
29.49
29.20
62.08
55.25
45,02
DuQuant
1,466
25,00
25.34
47.37
26.68
0.00
29.80
$1,96
31.89
DuQuant+ + LWC
2.6c4
24.40
26.68
38.26
25.30
0.00
29.20
49.67
52,01
30.69
BoA
15.24
30.38
54.42
68.17
49.17
39.89
34,20
65.94
60. 14
50,29
FP
Baseline
13.15
38.14
63.26
69.51
60.78
54.38
34.60
74.37
59.51
56.82
OmniQuant
302.3
22.01
31.69
37.95
27 64
0,14
25.60 25.60
54,95
50,12
31.26
AffineQuant"
268.3
22.44
31.61
37.83
27.75
0.12
26.00
54,30
49.88
31.24
QuaRoi- RIN
2.6e5
26.62
25.84
38.75
26.75
0.00
28.20
51,09
51.07
31.04
LLaMA3.2-1B
INT2
QuaRot-GPTQ
54.28
22.18
34.01
56.39
31.90
14.65
24.80
56,91
50.59
36.43
DuQuant
4,9c4
25.51
25.97
39.94
26.17
48.97
30.75
DuQuant + LWC
9.3e3
28.16
25.38
37.89
25.27
0.00
26.60
50.49
49.57
30.42
BoA
40.86
24.06
39.77
58.20
34.62
16.79
26.20
57.07
52.64
38.67
FP
Baseline
11.04
46.16
67,80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
OmniQuantt
273.4
23.38
31.82
56.42
29.35
0.28
27.80
56.15
51.07
34.53
AllinQuamt"
282.2
22.35
32.87
47.28
29:44
0.25
28.20
56.26
48.38
33.13
QuaRot RTN
2.3c4
26.19
24.49
47.09
26.55
0.00
30.40
51.74
48.30
31.85
LLaMA3.2-3B
INT2
QunRot-GPTQ
52.18
23.98
36.95
60.92
34.68
19.24
27.40
57.67
52.49
39.17
DuQuant
7.764
25.17
25.55
26.01
0.00
51.47
DuQuant LWC
770.9
23.63
26.64
38,20
26.32
0.03
26.20
51.74
51.70
30.56
BoA
33.40
27.30
45.66
65.87
40.86
24.57
29.00
61.37
56.27
43.86
BoA has been applied after transforming the model vin QuaRot.
The learnable equivalent transformation (LET) option has been deactivated because this option does not support models exploiting grouped-query attention (GQA). Results for high bit-widths and results LLaMAl models provided in Appendix D.2 due the page limitation.
wansformation-basee methods exploiting the naive nearest rounding (i.e., OmniQuant, AffineQuant, and DuQuant + LWC) but also is much faster than those methods relying on the gradient- based optimization (see Table 5). For example, on the I.I L.L.MA3--. A3-8B model, BoA achieves 20%p accuracy improvement for INT2 (see Table 6) and 13%p improvement for INT3 (see Table 11) over OmniQuant, AffineQuant, and DuQuant + LWC, yet reduces the quantization processing time of OmniQuant and AffineQuant more than twofold (see Table 5). Moreover, the degree reduction increases with the model size; BoA requires 7 times shorter processing time than that needed by AffineQuant LLaMA-30B.
in previous studies (Shao et al., 2023; Ma et al., 2024; Lin et al., 2024; Ashkboos et al., 2024; Liu et al., 2024), we quantize imput activations to all linear layers and KV caches with the Min-Max nearest-rounding quantizer where quan- tization parameters token. We use the notation 'WxAyKVz' to denote the x-bit weight quantization, y-bit input activation quantization, and z-bit KV cache quantization. In this experiment, when mea- suring the performance of RTN, GPTQ, and the proposed BoA, we use SpinQuant (instead QuaRot) for the model transformation. It is worth noting that while QuaRot uses Hadamard matrices (that independent of data) for the rotation (Ashkboos et al., 2024), SpinQuant optimizes rota- tion matrices that make models more robust to the activation quantization and thus could outperform QuaRot (Liu et al., 2024). In our experiments, rotation matrices have been op-
determined dynamically for each
4.3. Weight-Activation Quantization Results
We now evaluate the neight-activation quantization perfor- mance (see Table 7 and Tables 13-15 in Appendix D.3). As
8

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 7. Weight-activation quantization performance on transformed L.aMM22 and ILMMA3 models
sbet Accuracy (1)
Model
Precision
Method
Wiki2 PP).(U)
Arc-c
Arc-c
BQ
HS
LAMB
OBQA
PIQA
WG
Average
FP16
Baseline
5.473
45,90
74.66
77.92
75.94
70.86
44,00
78.89
68.90
67.13
Omnii
1.0c5
26.88
26.30
39.02
25.57
0.00
25.60
48.42
50.51
30.29
QufineQuant
NaN
NaN
NaN
LLaMA2-7B
SpinQuant-RTT RTN
23,23
25.51
34.01
62.20
32.09
14,48
26.00
55.17
50.91
37.55
W2A4KV4
SpinQuant-GPT
24.29
22.95
36.74
59.88
32.83
13.81
26.20
56.64
51.30
37.54
DuQuant
6.4e3
28.41
26.73
38.01
25.65
0.00
26.00
48.48
51.62
30.61
DuQuant LWC
16.35
24.91
37.33
61.99
41.46
10.54
28.20
58.71
53.28
39.55
BoA
11.80
26.79
49.20
63.09
48.05
37.76
30,80
63.55
57.85
47.14
FP16
Baseline
4.885
49.06
77.65
80.49
79.38
73.41
45,80
80.69
72.22
69.
OmniQuant
3,8c3
26.88
26.30
37.83
25.48
0.00
23.60
48. 2/
49.96
29.
AffioeQuant
1,2c3
26.62
27.19
37.83
26.37
0.00
24.60
49.17
30.07
LLaMA2-13B
MonnQuant-KNN
11.71
26.96
40.61
63.12
44.54
29.64
29.20
60.88
53.67
43.58
W2A4KV4
SpinQuant-GP GPTQ
15.54
23.55
41.29
62.17
35.76
16.50
29.80
59.52
51.54
40.02
DuQuant
6.4c3
28.41
26.73
38.01
25.65
0.00
26.00
48.48
51.62
30.61
DuQuant + LWC
16,35
24.91
37.33
61.99
41,46
10.54
28.20
58.71
53.28
39.55
BoA
8.974
31.74
55.98
64.80
56.22
49.18
34.40
68.44
59.27
$2.50
FP16
Baseline
6.137
53.67
77.61
81,19
79.15
72.23
45.00
81.01
73.24
70.39
OmniQuant
3.2e5
25.85
25.46
38.26
25.35
0.00
26.80
$1.58
49.09
30.30
AllineQuami"
NaN
NaN
NaN
NaNN
NaN
NaN
NaN
NaN
NaN
NaN
LLaMA3-8B
SpinQuant RTN
31,27
24.40
33.50
62.14
36.01
19.99
27.40
56.69
52,80
39.12
W2A4KV4
SpinQuant-GPTQ
29.60
22.18
34.89
58.23
35.07
13.42
27.20
55.88
$1.22
37.26
DuQuant
5.4c5
27.30
24.62
50.67
0.00
28.20
50.82
50.59
32.34
DuQuant + LWC
4.1c4
25.60
26.09
37.86
25.55
0.00
25.60
$1.36
49.25
30.16
BoA
18.23
28.92
48. 86
62.54
44.62
29.38
28.80
62,79
54.22
45.02
FP16
Baseline
13.15
38.14
63.26
69.51
60.78
$4.38
34.60
74.37
$9.51
$6.82
OmniQuant"
7.0c3
24.83
26.26
37.98
25.59
0.00
27.60
49.84
50.75
30.36
AffineQuant"
NaN
NaN
NaN
NaNN
NaN
NaN
NaN
NaN
NaN
NaN
LLaMA3.2-1B
SpinQuant-KIX KIN
110.6
22.61
32.07
52.14
28.59
4.08
28.20
52.83
49.49
33.75
W2A4KV4
SpinQuant-GPTO
143.8
22.35
31.23
51.96
28.91
2.91
26.20
53.32
$1.85
33.59
DuQuant
DuQuant LWC
8.5e.
28.07
26.01
38.26
25.84
0.00
28.80
$0.11
$0.91
31.00
BoA
77.05
22.61
36.66
57.22
32.13
8.07
25.80
55.55
50.99
36.13
FP16
Baseline
11.04
46.16
67.80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
OmniQuantt
6.6c3
27.82
25.00
38.35
25.43
0.00
28.00
$2.18
49.80
30.82
AllineQuant"
6.0e3
25.94
26.89
38.10
25.66
0.01
28.40
51.58
49.80
30.80
LLMMA3.2-3B
SpinQuant RTN
51.19
24.83
31.48
45.32
29.90
11.05
27.80
55,17
50.20
34.47
W2A4KV4
SpinQuant-GPTQ
65.09
23.21
33.54
38.26
31.16
7.18
25,80
55.98
$1.38
33.31
DuQuant
29.00
DuQuant + LWC
1.7e3
24.40
26.09
37.83
25.91
0.00
27.80
50.49
$2.49
30.63
BoA
37.12
27.99
37.67
59.30
37.91
15.81
27.40
57.73
52,41
39.53
BOA has been applied after transforming the model via SpinQuant.
'Th LET option has been deactivated bocause this option does not support models exploiting GQA.
""NaN" means that loss diverges the guantization process.
... Results for other configurations are provided in Appendix D.3 due to the page limitation.
timized with respect activation-only quantized networks (i.e., weights are fixed with full- precision) in (Liu al., 2024).
IB, and LLaMA3.2-3B, respectively.
5. Conclusion
From Table 7 and Tables 13-15 in Appendix D.3, we observe that the outstanding weight- qutittt-uuaizztionnn performance of the proposed BoA leads to the state-of-the-nn performance for the weight-activation quantization. In particular, the perfor- mance gain is noticeable for low-bit (e.g., W2A4KV V4 and W2A4KV16). For example, compared to Saanuuuunt-PPOO BoA achieves 10%p accuracy improvement LLaMA2- 7B and 12.5%p improvement on LLaMA2-13B (see Ta- ble 7). Compared to the conventional gradient-basse meth- ods (i.e. OmniQuant, AffineQuant, and DuQuant), BOA achieves 8%p. 13%p, 13%p. 5%p, and 7%p improvement on LLaMA2-7B 7B, LLaMA2- 13B, LLaMA3-80 8B, LLaMA3.2-
In this paper, we proposed novel backpropagation-free PTQ algorithm called BoA. To consider the inter-layer de- pendencies within the attention module, we approximated the Hessian matrices by exploiting the attention reconstruc- tion rather than the layer-wise reconstruction error. To mitigate the computational overhead incurred by the pro- posed attention- aware Hessians, we also incorporated sev- eral techniques. including Hessian relaxation, efficient com- putation of inverse and Cholesky decomposition of attention- aware Hessians, and simultaneous quantization different attention heads. Finally, from extensive experiments, we demonstrated the efficacy of the proposed BoA.

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Acknowledgment
Egiazarian, V., Panferov, A., Kuznedelev, D., Frantar, E., Babenko, A., and Alistarh, D. Extreme compres- sion of large language models via additive quantization. arXiv:2401.06/16 2024.
We would like thanks to Dachyun Kim, Ph.D., Hyeonmok Ko, Ph.D., and Hyemi Jang, Ph.D. for their helpful discus- sion.
Frantar, E. and Alistarh, D. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 35:4475 4488, 2022.
Impact Statement
BoA is a highly efficient and accurate quantization algo- rithm that minimizes accuracy loss while significantly re- ducing the time required for model quantization and deploy- ment. By synergizing with other methods, BoA facilitates the efficient operation of LLMs on resource-constrained hardware using only integer arithmetic. This breakthrough is particularly impactful for on-device AI, allowing real- time model inference on mobile and edge devices without the need for server access, paving the way for scalable, decentralized AI solutions.
Frantar, E., Ashkboos, S., Hoefler, T., and Alistarh, D. OPTQ: Accurate quantization for generative pre-trained Transformers. The Eleventh International Conference on Learning Representations, 2023.
Hubara, I., Nahshan, Y., Hanani, Y., Banner, R., and Soudry, D. Accurate post training quantization with small cal- ibration sets. In International Conference on Machine Learning, pp. 4466-4475, PMLR, 2021.
References
Jeon, Y., Lee, C., Cho, E., and Ro, Y. Mr.BiQ: Post-training non-uniform quantization based on minimizing the recon- struction error. In Proceedings of the IEEE/CVF Confer- ence Computer Vision and Pattern Recognition, pp. 12329-12338, 2022.
Ashkboos, S., Mohtashami, A., Croci, M. L., Li, B.,
Cameron, P., Jaggi, M., Alistarh, D., Hoefler, T., and
Hensman, J. QuaRot: Outlier-free 4-bit inference in rotated LLMs. 40X1v:2404.00456, 2024.
Jeon, Y., Lee, C., and Kim, H.-y. GENIE: data for quantization. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 12064-12073, 2023a.
show me the
Bisk, Y., Zellers, R., Gao, J., Choi, Y., et al. PIQA: Reason- ing about physical commonsense in natural language. In Proceedings of the AAAI conference on artificial intelli- 2020.
gence, volume 34, pp.
Jeon, Y., Lee, C., Park, K., and Kim, H.-y. A frustratingly easy post-training quantization scheme for LLMs. In Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Prucessing, pp. 14446-14461, 2023b.
Botev, A., Ritter, H., and Barber, D. Practical Gauss-Newtor optimisation for deep learning. In International Confer- ence on Machine Learning, pp. 557-565. PMLR, 2017.
Clark, C., Lee, K., Chang, M.-W., Kwiatkowski, T., Collins, M., and Toutanova, K. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv:1905.10044, 2019.
Jin, J., Liang, C.. Wu, T., Zou, L., and Gan, Z. KDLSQ- BERT: A quantized BERT combining knowledge distilla- tion with learned step size quantization. arXiv preprint M:Xiv:2001.0990 2021.
Clark, P., Cowhey, I., Etzioni, O., Khot, T., Sabharwal, A., Schoenick, C., and Tafjord, O. Think you have solved question answering? Try ARC, the A12 reasoning chal- lenge. 17Xhn:1803.05457v.. 2018.
Kim, J., Lee, C., Cho, E.,, Park, K.,, Kim. H.-y., Kim, J., and Jeon, Y. Towards next-level post-training quantization of hyper-scale transformers. In Advances in Neural In- formation Processing Systems (NeurIPS), volume 37, pp. 94292-94326, 2024.
Dettmers, T., Svirschevski, R., Egiazarian, V., Kuznedelev, D., Frantar, E., Ashkboos, S., Borzumov, A., Hoefler, T., and Alistarh, D. SpQR: sparse-quantized rep- resentation for near-lossless I1m weight compression. wrXiv:2306.03078. 2023.
Kim, S., Hooper, C., Gholami, A., Dong, Z., Li, X., Shen, S., Mahoney, M. W., and Keutzer, K. SqueezeLLM: Dense-and-sparse quantization. 1rXiv:2306.07622 2023.
Li, Y., Gong, R., Tan, X., Yang, Y., Hu, P., Zhang, Q. Yu, F., Wang, W., and Gu, S. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
Edalati, A., Ghaffari, A., Asgharian, M., Hou, L., Chen, B., and Nia, P. OAC: Oupput-adaptive calibration for accurate post-training quantization. arXiv:2405.15023 2024.
10

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Lin, H., Xu, H., Wu, Y., Cui, J., Zhang, Y., Mou, L., Song. L.,, Sun, Z.., and Wei, Y. DuQuant: Distributing outliers via dual transformation makes stronger quantized LLMs. In The Thirty-eighth Annual Conference on Neural Infor- mation Processing Systems, 2024.
Touvron, H., Martin, L.,, Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv:2307.09928. 2023b.
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., and De Sa, C. QulP#: Even better LLM quantization with Hadamard incoherence and lattice codebooks. arXiv:2402.04:5 04396. 2024.
Liu, Z., Oguz, B., Zhao, C., Chang, E., Stock, P., Mchdad, Y., Shi, Y., Krishnamoorthi, R., and Chandra, V. LIm-qat: Data-free quantization aware training for large language models, 2023.
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., and Han, S. SmoothQuant: Accurate and efficient post- training quantization for large language models. In In- ternational Conference on Machine Learning, pp. 38087- 38099, PMLR, 2023.
Liu, 7., Zhao, C., Fedorov, I., Soran, B., Choudhary, D., Kr- ishnamoorthi, R., Chandra, V., Tian, Y., and Blankevoort, T. SpinQuant: LLM quantization with learned rotations. arXiv:2405. 16406, 2024.
Ma. Y., Li, H., Zheng, X., Ling, F., Xiao, X., Wang, Wen, S., Chao, F. and Ji, R. AffineQuant: Affine transformation quantization for large language models. arXiv:2403. /2544, 2024.
Zellers, R., Holtzman, Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4791-4800 2019.
Merity, S., Xiong, C.,, Bradbury, J., and Socher, R. Pointer sentinel mixture models. 1XX:::6099008433 2016.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. OPT: Open pre-trained Transformer language models. arXiv:2205.01065 2022.
Mihaylov, T., Clark, P., Khot, T., and Sabharwal, A. Can suit of armor conduct electricity? a new dataset for open book question answering. 1XXXv:18090027899 2018.
Nagel, M., Amjad, R. A., Van Baalen, M., Louizos, C.,, and Blankevoort, T. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICMI.,, pp. 7197-7206, 2020.
Paperno, D., Kruszewski, G., Lazaridou, A., Pham, Q. N., Bemardi, R., Pezzelle, S., Baroni, M., Boleda, G., and Fernández, R. The LAMBADA dataset: Word prediction requiring broad discourse context. arXiv:1606.06031, 2016.
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. WinoGrande: adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Shao, W., Chen, M., Zhang, Z., Xu, P., Zhao, L., Li, Z., Zhang, K., Gao, P., Qiao, Y., and Luo, P. OmniQuant: Omnidirectionally calibrated quantization for large lan- guage models. arXiv:2308. 13137, 2023.
Su, J. Lu, Y., Pan, S., Murtadha, A., Wen, B., and Liu, Y. RoFormer: Enhanced Transformer with rotary position embedding, S.Kiv:210400966. 2023.
Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al. LLaMA: Open and efficient foundation language models. arXiv:2302.. 13971, 2023a. 2023a.

---

BoA: Attention-aware Post-training Quantization without Backpropagation
A. Proof of Footnote 2
In our proof, we use the following useful properties of the Kronecker product:
vec (2))) = (M² @ Mi) vec (M)),
(17a)
(M1 @ M2) T MÍ @ M
(17b)
MM1 & M2) (Mg @ M.) M, & M,M,;
(17c)
where vco(-) denotes the vectorization operation.
Using (17a), we have
M [I & M1) Aw Aw" (M² & Mi)² (M² @ M,) Aw,
where Aw vec A W. In addition, by (17b) and (17c), we have
Aw" (M² M.) T (M² M1) Aw Aw" (M, Mf) (M² & M1) Aw
= Aw" (M2M² & Mf Aw.
Finally, by exploiting the fact that
dxxAAx Ax 8x
A + AT, we obtain
02 M,WMM
M @ M + (MMM & MFM))
dAw2
M & M M, + (M2M3)* @ (MFM)) T
= 2M,M & M,M,, M2,
where (a) follows from (17b). This completes the proof.
B. Attention-aware Hessians for Models Exploiting RoPE
As mentioned, when RoPE is applied, the proposed surrogate K2 l used to develop the attention-awart Hessian H (w..)) in (11) changes to |K, A AQ% I|;, where K, RoPE((K)) and Q = RoPE(Q,). Let Re be the rotary matrix for the ε-th token (see eq. (15) in (Su et al., 2023)) and Q [ ... L.. then the objective can be expressed as
L
L
Kr II
Σ
K, 1
AR.
Σ
R. AW
which yields the following Hessian equation (see Footnote 2):
H (wo.N) = Σ (2xxx & R K. Kr R.).
Finally, take the factorized approximation for the summation of Kronecker products (i.e., EM &M) [MM eq. (20) in (Botev et al., 2017)):
L
H (W
Σ
2x
1 I
Σ K
2Xx & I Σ R K.K.K.R.
Σ
By taking similar steps, the attention-awarc Hessian for the key projection weight W K. with RoPE can be established as
1
(w..) = 2 &
R
R QQQQR.
L
Em1
12

---

BoA: Attention-aware Post-training Quantization without Backpropagation
C. Refined Weight-update Formula
recall that the Hessian-based weight-update formula given by (Frantar Alistarh, 2022; Frantar et al., 2023)
ow Wy ((w)g)
[U]g where U = Chol(H
[U] 4.4
For the proposed ditention-aware Hessians in Table have
A Cool, @ U Urowhh
where U = Chol(H and row = Chol H tow,h ! (see Section 3.3). Therefore, the weight- formula can be recast as
Wlg (Wwg)
8w,
[Ueot U Uoww,hlg..
[U col.1 & U Uoow,, Alg.9
Quant. of 1st row
Update of 2nd row
Ucol. At
U row. At
a
b
a co
col.
cUcol
Quant. of 2nd row
Figure 2. Illustration of the Hessian information when doo = 2 and dea =
For simplicity, suppose we quantize the first (0-th) row. When the weight [WnJos [w(")) (n,;) quantized, the upight-update of the i-th row is simplified as (see Figure 2 for the ease of understanding)
in the j-th column
Walo. - (([Wlo.s) [Urow,llo. (
[Uw A/o.0 (Uro,,lo.00
[Waoo,j - @(Walos) [Unow,)oo, (eat,
[Urow
Thus, after the quantization of all weights in the first row, the total amount of the weight-update for the i-th row can be expressed as
W ((WWloos)
Jow U
&W A.total 6.s
[Ucnt,
fow..oooo
U (Urw,,All Jo,
dea
WAlos @(WNloo))
[Uoot
U mm 0.0
[UentBlas
Furthermore, by noting that (see line 5 in Algorithm 2)
W Q(Wwlo..)
ECPTOlAJ
[Ueo
13

---

BoA: Attention-aware Post-training Quantization without Backpropagation
obtain
U Unow.hlo.
[ Unw,hlo, 0,4
@W to,al,
Eurrolh.j [Uoot,
Earrola,
U 1oo,olo..
[ row, 10000
As result, the weight-update matrix to compensate for the quantization error of the first row is given by
UUT
row Alo.o EErrOOh, Ucal,
8& lo.
(18)
[loo,
By taking similar steps as above, we can easily generalize (18) for the j-th row as follows:
U
W j::
(19)
[U Jow..
14

---

BoA: Attention-aware Post-training Quantization without Backpropagation
D. Additional Experimental Results
D.I. Performance of Weight-Round Rounding Optimization
We evaluate the weight-only quantization performance the proposed BoA. To solely compare the performance of the weight-rounding optimization, we do not apply any transform (e.g., smoothing. rotation, and permutation) to the models. In this appendix, we provide the results for the INT4 weight quantization of LLaMA2 and LLaMA3 models (see Table 8) and the results on LLaMA1 models (see Table 9) and OPT models (see Table 10).
Table 8, INT4 wnlght-only quantization performance on LLaMA2 and LLaMA3 models without transformation
Model
Procision
Method
Wiki (P)
Arp-p
Aro-p
BQ
HS
7ero-shee Accuncy (1)
LAMB
CIRQA
PRQA
WG
Average
Haseline
54433
45,00
14.66
75.94
44.00
68.90
67.13
LLaMA2-7B
RTN GPTQ
6.998
41.64
67.05
72.72
71.36
59,78
42.60
77.04
65.75
62.24
42.05
70.44
43.00
68.11
43.94
72.26
74.53
68.96
78.40
69.14
79.38
90.69
72:22
11aMA2.138
KIN
6.319
43,00
66.88
66.14
62.25
38.20
77.04
63.85
61,95
GPTQ
47.18
74.37
75.00
70.89
78,40
69.69
BoA
5,096
47.95
75,80
78.93
77.96
70.61
41.60
79.87
71.35
68.01
Baseline
6.137
53.67
77.61
81.19
79.15
72,23
45.00
81.01
73.24
70.39
LLaMA3-SB
7,881
78.29
INT4
GPTQ
7,960
49.05
73:44
73,70
69.30
60.19
45.00
79.54
73.56
65.47
BOA
6.561
50.17
71.74
80.31
77.42
70.20
44.40
80.14
73.88
69.28
Raveline
13.15
38.14
63.26
69.51
50.78
54.38
34.60
74.37
59.51
56.82
LLaMA322-1B
RTN
18.81
34,30
57.41
65,20
32,72
33.00
69.15
56.35
50.51
INT4
GPTQ
16.20
35.41
58.63
64.46
$5.69
38.08
32.3 20
69.04
56.27
51.22
BOA
14.29
36.18
60.82
67.22
59.03
51.65
34.00
71.87
58.09
54.86
FP
Hascline
11.04
46.16
67.80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
LLAMA322-3R
KIN
13:86
43:25
61.24
73.67
68:27
54.18
38. 40
71.76
64.01
59.35
INT4
GPTQ
12.41
44.28
63.51
77.00
68.44
57.50
36
74.16
67.01
61.05
BOA
11.74
44.11
36.20
66.85
62.25
Table 9. Weight-only quantization performance LLaMA models without transformation
Model
Precision
Methud
Wiki2 PPL (1)
Are
Arc
IQQ
HB
Zero thot Accuracy (f)
LAMB
OBQA
PIQA
WG
Averae
FP
Baseline
5,677
44.71
72.94
75.05
76.21
70.66
44.40
79.15
70.01
66.64
26.37
26.00
$1.80
IN12
GPTQ
19:03
25.34
41.58
36.27
37.21
26.24
30.20
61.70
55.56
41.77
BoA
10.28
28.75
$2.89
64.19
49.34
41.48
32.60
66.54
59.19
49.35
LLaMA-7B
RIN
61.31
26.45
50.46
59.76
40.07
14.64
33.00
67.08
$5.88
43.42
IN13
GPTQ
18.44
33.55
61.35
64.40
56.22
44.91
35.40
72.09
64.33
54,03
BOA
71.01
RIN
7.112
41.47
70.29
72.81
73.16
63.81
41.60
77.86
69.27
63.66
INT4
72.18
75.54
74.75
69.60
78.62
66.08
74.75
79.07
73.66
2.263
26.01
39.48
26.00
0.00
25:20
49.46
30,04
GPTQ
56.87
33.11
BOA
8,309
32.76
39.51
69.57
59.52
51.79
35.20
71.22
63.19
55.60
LLMMA-13B
RTN
134.5
25,00
49.24
36.91
11.41
29.30
64.20
54.62
40.87
INTI
GPTQ
7.006
42.24
69.74
70.75
73.40
62.99
50.30
77.09
68.11
62.94
BOÅ
5.461
45.82
71,95
75.96
75.71
70.62
43.40
78.35
70,24
66.50
RTN
7.743
45.22
69.90
72.42
75.67
65.65
41.60
78.78
70.56
64,99
INT4
GPTQ
6.147
45.82
72.26
74.10
76.22
70.15
43.80
80.20
71.90
66.81
76.00
73.19
Baseline
4.301
$2.90
78.96
82.69
82.03
15.47
48:20
82.26
75.77
RTN
7.163
25,60
26.52
40.67
27.20
0.11
24.60
50.92
49.64
30.66
INT2
64.77
70.35
37.71
73.00
64.77
61.28
39.00
73.34
L1aMA-30B
$4,10
42.94
INT3
GPTQ
7,061
48.12
71.66
74,89
76.73
66.39
43.20
78.35
70.80
66.24
BoA
4.575
49.57
74.75
$1.67
79.21
73.77
46.20
79.38
74,19
69.87
RUN
5,802
49.60
24.16
78.07
79.48
66.67
44.00
79.87
73.16
68.10
INTV
GPTQ
5.188
$1.45
75.67
80.18
79.69
70.38
45.40
79.43
73.66
69.18
BOA
4.318
53.16
78.41
82.72
81.95
75.96
49.40
81.88
75.06
72.32
15

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 10. Weight-only quantization performance OPT models without transformation
Zero-shol Accuracy (1)
Model
Precision
Method
Wiki2 PPL. (
Arc-c
Arc-e
BQ
HS
LAMB
OBQA
PIQA
WG
Average
FP
Bascline
27.65
22.78
39.98
55.44
31.35
33.44
28.00
62.02
50.28
40.41
1.0c4
23.89
27.31
25.96
27.00
50.49
51.14
30.45
INT2
GPTQ
411.3
22.61
31.52
27.16
27.20
52.18
51.70
31.38
OPT-125M
BoA
85.63
21.42
32.15
48.20
28.50
26.60
56.47
49.17
33.67
RIN
233.9
22.01
33.33
52.94
28.52
2.81
28.40
59.19
51.14
34.79
INT3
GPTQ
50.75
21.67
33.84
58.38
29.41
16,96
25,40
58.98
51.14
36,97
BoA
31.95
22.95
36.83
59.66
30.67
27.10
28.80
60.34
50.83
39.65
FP
Baseline
22.00
23.89
40.36
57.68
36.67
40.47
28:20
64.74
$2.33
43.04
RIN
4.6c3
24.74
27.65
37.83
26.87
0.00
23,40
52.83
52.09
30.68
INT2
GPTQ
61.93
22.53
33.46
57.28
29.66
10.70
23.80
56.75 75
51,22
35.68
OPT:350M
BOA
46.53
22.01
35,90
62.32
30.43
11.85
26.60
58.98
51.62
37.46
RTN
35.91
23.38
36.99
52.14
34.00
30.37
27.00
62.02
$1.62
39.69
INT3
GPTQ
25.25
22,95
38.47
60.58
35.25
41.38
27.60
62.46
$1.22
42.49
BoA
23.96
22.87
38.76
61.22
35,30
38.08
28.40
62.57
$3.28
42.56
FP
Baseline
14.62
29.52
50.97
57.83
53.70
55.19
72.47
59.51
$1.57
RTN
1.6c4
24.23
24.75
51.19
26.51
0.00
29.00
52.29
50.04
32,25
INT2
GPTQ
177.4
22.78
31.78
57.06
28.13
6.37
24.80
53.75
48.70
34.17
OFT-1.3B
BoA
31.65
22.70
37.67
62.17
34.65
18.02
26.80
61.48
51.78
39.41
RIN
754.8
25.34
34,72
52.17
36.31
10.25
27:20
60.07
51.78
37.23
INT3
GPTQ
19.35
27.13
43.43
54.25
47.23
36.30
29.40
68.01
54.30
45.01
15.77
27.39
48.74
58.99
50.09
51,50
30.60
69.37
$7.30
49.25
FP
Baseline
12.47
31.31
54.38
60.40
60.62
59.78
35.20
74.81 81
61.01
54.69
RTN
2.7c5
26.11
26.73
62.17
26.77
0.00
29.20
50.87
$1.07
34.12
INT2
GPTQ
135.5
24.66
33.71
60.28
33.62
17.62
27.00
57.73
50.59
38.15
OPT-2.7B
BOA
22.30
25.77
41.62
62.26
42.00
33.76
28.80
62.08
$2.96
43.66
RTN
170.5
25.26
38.34
56.51
36.64
14.61
29.80
66.27
53.04
40.06
INT3
GPTQ
15.64
29.27
49.07
$1.79
42,92
33.00
80.80
$7.54
47.41
BoA
13.63
$2.90
55.86
60.06
53.27
FP
Bascline
10.86
34.73
60.02
66.06
67.16
65.50
37.40
76.50
65.43
59.10
RTN
2.9c4
25.00
25.38
37.83
26.66
0.00
29.20
50.00
$1.46
30.69
INT2
GPTQ
95.95
23.21
33.75
61.77
32.09
16.25
25.80
57.02
$2.01
37.74
OPT-6.7B
BOA
20.57
28.24
47.31
61.80
47.00
34.09
33.20
66.87
$7.54
47.01
RTN
53.76
27.73
40.87
50.95
48.67
28.32
31,20
68. 17
53.12
43.63
INT3
GPTQ
12.05
32.51
56.40
54.53
62.49
55,99
36.60
74.21
60.85
54,20
BoA
11.25
32.76
57.32
66.85
64.53
63.64
37.60
75.57
64.09
$7.80
FP
Baseline
10.13
35.75
61.87
65.84
69.87
65.98
39.00
76.88
65.11
60.04
4.764
26.79
26.64
26.25
27.40
$0.04
30.91
INT2
GPTQ
37.04
41.60
29.17
27.60
$2.17
42.13
OPT-13B
15.34
29.27
48.95
62.78
53.63
44.71
32.60
$8.64
49.89
RTN
40.46
27.47
38.72
62.54
51.21
29.44
26.80
63.38
49.88
43,68
INT3
GPTQ
11.11
34.56
59.51
62.45
64.81
58.57
36.00
74.92
63.54
56.80
BoA
10.39
34.30
59.93
69.94
66.95
66.61
37.60
75.95
66.93
59.78
FP
Buseline
9.557
38.05
65.36
70.46
72.27
67.85
40.20
78.07
68.43
62,59
RTN
4.2c4
26.79
25.55
37.83
0.00
28.60
50.60
$1.22
30.81
INT2
GPTQ
21.19
27.99
41.96
62.45
46.90
39.48
30.40
65.72
$4,22
46.14
OPT-30B
BOA
11.15
30.55
55.81
62.81
60.19
60.39
34.80
72.47
62.67
54.96
RTN
86.63
24.32
35.06
57.98
38.07
12.17
26.00
$9.90
$2.09
38.20
INT3
GPTQ
10.14
34,90
61.70
66.97
69.34
63,70
40.40
77.26
67.01
60.16
BOA
9.744
36.26
62.63
70.28
70.52
68.86
40.20
77.75
67.17
61.71
16

---

BoA: Attention-aware Post-training Quantization without Backpropagation
D.2. Comparison with Transformation-basel Methods
We compare the weight-only quantization performances of the proposed BoA and existing transformation-basse methods. For conventional methods, OmniQuant (Shao et al., 2023) and AffineQuant (Ma al., 2024), activated both learnable equivalent transformation (LET) and learnable weight clipping (LWC) options. For A AffineQuant, we did not activate the use-In-mate rix option because this adds extra affine transformation between the normalization layer and linear layers, which incurs additional processing time during the inference. QuaRot-RTT - and QuaRot-GPTQ mean that RTN and GPTQ have been applicd for the weight quantization after transforming models via QuaRot (Ashkboos et al., 2024). For DuQuant, we report the results obtained with and without activating the LWC option as in the original paper (Lin et al., 2024). When measuring the performance of the proposed BoA, we also transformed the model for fair comparison. We applied QuaRot for the transformation because QuaRot does not require training and incurs no extra costs during the actual inference (Ashkboos et al., 2024).
the official codes provided by the authors and reported the obtained results. For
In this appendix, we provide the results for the INT3 weight quantization of L.aMA2 and LLaMA3 models (see Table 11) and the results on LLaMAl models (see Table 12).
Table 11. INT3 weight-only quantization performance on transformed L.aMA2 and I.MaMA3 models
shol Accuracy (1) ORQA
Model
Precision
Method
Wiki2 (P).
Arc-c
Arc-e
BQ
HS
L.M
PIQA
WG
Avernge
FP16
Baseline
5.473
45.90
74.66
77.92
75.94
70.86
44.00
78.89
68.90
67.13
OmniQuant
6.640
39.59
65.66
69.82
70.09
57.58
38.40
76.01
64.88
60.25
Affineuunnt
6.468
70,86
70.32
61.87
65.27
61.43
QuaRot-RTN
129.2
23.46
30.60
37.83
29.30
4.75
24.40
53.10
50.20
31.71
LLaMA2-7B
INT3
QuaRot-GPTQ
6.122
40.96
70.16
73.30
71.44
67.63
41.40
77.42
67.48
63.72
DuQuant
6.831
41.13
67.59
69.36
70.52
62.25
41.40
76.01
65.35
61.70
DuQuant + LWC
6.226
40.70
69.15
74.77
70.97
64.06
42.40
77.04
66.85
63.24
BoA
5,874
42.06
71.17
73.73
72.29
69.85
41.60
77.37
67.48
64.44
FP16
Baseline
4,885
49.06
77.65
80,49
79.38
73.41
45.80
80.69
72.22
69.84
OmniQuant
5,593
44,80
71.72
75.29
75.63
64.94
43.20
78.67
69.30
65.44
AllineQunnt
5.526
73.74
77.34
74.79
77.15
67.01
65.54
QuaRot-RTN
48.06
22.01
34.72
62.14
33.44
7,80
26.60
57.67
50.51
36.86
1LAMA2-13B
INT3
DuaRot-GPTO
5.382
47.01
75.42
78.84
75.70
72.14
44.60
78.56
70.01
67.79
DaQuant
5.749
44.88
72.10
79.91
75.37
68.23
43.00
77,80
70.72
66.50
DuQuant + LWC
5.414
46.67
74.62
77.92
75.93
68.63
43.80
79.90
68. 75
17.04
BoA
5,202
48.29
76.39
79.45
76.23
73.19
45.20
78.67
70.96
68.55
HP16
Baseline
6.137
53.67
77.61
81.19
79.15
72.23
45.00
81.01
73.24
70.39
OmniQuant
11.75
37.37
60,94
65.57
66.81
33.02
34,80
71.49
$9.59
$3,70
AffineQuant
37.97
64.92
67.42
33.95
36.20
71.98
60.30
54.55
QuaRot-RTNN
38.64
23.72
38:22
51.28
47.53
23.63
33.60
62.35
62.04
12,80
LLaMA3-8B
INT3
QuaRot-GPTQ
7.490
47.87
74.49
78.44
74.67
67.43
40.00
78.07
72.38
66.67
DaQuant
11.35
35.84
53.37
64.56
65.56
49.52
36.20
72.25
65,90
55,40
DuQuant + LWC
10.78
31.66
46.59
64.65
65.25
41.29
38.00
70.13
61.80
52.42
BoA
7.145
49.06
77.40
80.49
74.54
69.20
43.00
78.51
72.53
68.09
FP16
Baseline
13:15
38.14
63.26
69.51
60.78
54.38
34.60
74.37
59.51
56.82
OmniQuant
26.61
31.31
$1.39
63.67
49.18
18.83
29.60
67.57
55.49
45.9 88
AffineQuant
26.43
32.00
52.48
64.31
48.71
20.67
26.20
67.57
52.88
45.60
RNNN
98.24
24.06
36.03
61.96
35.31
7.68
28.60
59.14
51.62
38.05
LLMMA3.2-IB
INT3
QuaRot GPTQ
16.56
32.42
$7.95
64.10
52.48
43.79
32.20
68.99
57.14
51.13
DaQuant
2.1
25.26
25.67
41.35
26.68
0.00
27.80
49.51
48.78
30.63
DuQuani + LWC
2.7e4
25.68
26.73
40.73
25.34
0.00
29.00
49.40
$2.09
31.12
B
15.73
32.42
57.95
66.24
53.86
48.74
33.20
78.18
57.06
52.46
FP16
Baseline
11.04
46.16
67.80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
OmniQuant
16.97
36.01
57.58
71.47
59.57
39.17
34.00
69.31
60.22
53.42
AllineQuant
16.79
36.43
56.57
72.97
59.19
40:07
35.20
70.13
60.00
53.83
QuaRot RTN
89.54
22.01
32.79
59.08
36.18
4.82
25.00
54.35
53.67
35,99
LLaMA3.2-3B
INT3
QuaRot-GPTQ
13.58
38.31
58.96
75.72
64.59
54.59
35.80
70. 18
64.96
$7.89
DuQuant
18.92
30.55
48,32
71.71
60.34
40.54
32.40
66.54
60.38
51.35
DuQuant= + LWC
15.18
37.20
58.33
72.02
61.22
44.58
33.40
70.02
59.98
$4.59
BoA!
12.97
42,49
65.74
78.90
65.59
58.40
34.80
73.07
63.46
60.31
BOA has heen applied after transforning the model via QuaRot.
Thee LET option been deactivated bocause this option does not support models exploiting GQA.
17

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 12, Weight-only quantization performance transformed LLaMA models
Model
Precision
Method
Wiki2 PPL.
Accuracy (t)
Arc-c
Arc-c
RQ
HS
LAMB
OBQA
PIQA
WG
Average
FP
Baseline
5.677
44.71
72.94
75.05
76.21
70.66
44,40
79.16
70.01
66.64
QuaniQuant
15.74
26.28
45.71
61.93
42.66
17.43
29.00
62.40
52.17
42.20
AffineQuant
27.22
32.20
QuaRot-RTN
1.0c4
27.90
26.39
37.83
26.03
0.00
21.60
49.24
49,49
29.81
INT2
QuaRet-GPTQ
11.53
26.45
45.75
62.48
48.23
37.37
32.80
65.67
57.46
47.03
DuQuant
9.4c3
28.33
26.89
46.42
26.24
0.00
21.20
50.71
51.62
31.43
DuQuant LWC
12.39
27.73
49.37
63.03
49.88
23.65
31,80
64.96
56.75
45.90
LLaMA-7B
BOA
9.812
30.63
54.88
63.55
52.20
46.50
33.00
68.72
61.01
51.31
OmniQuant
6,463
38.74
66.96
72.35
69.84
62.79
40.00
77.20
66.69
61.82
AftinoQuant
39.68
QuaRet RTN
24.03
29.86
53.11
61.13
$0.30
24.53
31.40
69.37
57.30
47.13
QuaRor-GPTI
43.09
72.18
DuQuant
6.976
37.37
64.23
71.13
70.81
60.45
39,20
76.39
65.98
60.70
DuQuant LWC
6.328
38.74
66.20
73.73
71.24
63.78
42.40
77.31
68.51
62.74
71.77
72.26
69.32
63.73
FP
Baseline
5,090
47.70
74.75
77.95
79.07
73.66
44,80
80.14
72.77
68.86
OmniQuant
13.29
29.69
53.79
62.29
55.31
19.52
31.60
70.29
57, 14
47.45
AffineQuant
QuaRot-RTN
4.1c3
28.16
26
37.83
25.72
0.00
22.60
$1.25
48.46
30.06
INT2
QuaRot-GPTQ
9.331
31.06
58.55 $8.59
63.79
54,96
45.54
35,20
69.97
63.30
52.80
DuQuant
6.2c3
25.94
25.93
39.66
26.70
0.00
23.60
50.87
51.62
30.54
DuQuunt LWC
8,770
31.74
58.38
66.27
60.98
44.46
37.00
72.20
62.19
54.15
LLaMA-13B
BoA
8.341
33.19
63.
71.44
60.75
56.30
37.00
71.93
65.90
57.52
OrnniQuant
5.671
45.05
71.13
75.44
75.35
66.82
44.40
78.56
69.46
65.78
AffinQuunt
5.615
43.00
70.79
74.83
74.99
68.95
43,40
79.76
69.38
65.64
QuaRor-RTX
7.082
37.37
60.14
73:18
67.93
56.42
35.60
76.22
66.
59.13
INT3
QuaRot GPTQ
DuQuant
5.923
43.94
70.12
74.40
75.76
68.33
42,00
78.56
72.06
65.65
DuQuant LWC
5.554
45.31
71.97
72.78
75.85
69.81
45.00
79.49
70.64
66.36
5.411
72.81
75.60
75.95
45.20
79.27
67.27
Baseline
4.101
52,90
78.96
82.69
82.63
75.47
48.20
82.26
75.77
72.36
OrnniQuant
8.598
33.45
57.37
62.84
$9.21
39.36
37.20
70.02
60.14
$2.45
Affineuuant
7,267
38.14
65.00
73.33
67.61
53.75
38.60
74.21
64.96
59.46
QuaRet-RTN
3.8c3
25.77
26.52
37.83
25.80
0.01
23.80
$1.69
47.83
29.91
INT2
QuaRet-GPTO
7,283
DuQuant
5,763
27.13
26.98
38.75
26.76
0.00
26.20
49.88 89
49.17
30.61
DuQuant LWC
7,706
38.48
64.60
70.00
67.56
48.08
36.40
72.96
65.11
$7.90
LLaMA-30B
BOA
OnnniQuant
4.766
50.09
76.18
79.91
79.58
70.79
45.20
79.92
74.43
69.51
AflinQuunt
4,729
50.60
76.81
79.85
79.60
72.83
44.60
80.52
74.74
69.94
QuaRet RTN
6.355
37.80
65.32
67.98
64.79
47.46
38.80
72.80
67.17
57.77
INT3
QuaRor-GPTQ
4.759
49.23
77.19
81.22
79.97
75.98
45,20
80.41
75.61
70.60
DuQuant
5.066
49.49
74.71
79.97
79.87
72.78
46.40
79.54
74.51
69.66
DuQuant LWC
75.17
BOA'
4.602
52.73
77.23
80.83
80.04
75.91
45.60
80.79
76.16
71.16
ROA applied transforming QuaRot.
18

---

BoA: Attention-aware Post-training Quantization without Backpropagation
D.3. Weight-Activation Quantization Results
We evaluate the weight-activation quantization performances of the proposed BoA. As in prior works (Shao al., 2023; Ma al., 2024; Lin al., 2024; Ashkboos et al., 2024; Liu et al., 2024), apply per-token nearest-rounding quantization for imput activations and KV caches. We use the notation 'WxAyKVz' to denote the x-bit weight quantization, y-bit input activation quantization, and z-bit KV cache quantization. When measuring the performance of RTN, GPTQ, and the proposed BOA, we use SpinQuant for the model transformation as SpinQuant outperforms QuaRot by optimizing the transformation for the activation quantization (Liu et al., 2024). When optimizing the rotation matrix in SpinQuant, we quantize both weights and activations for SpinQuant-RFi RTN and quantize only activations for SpinQuant-GPTQ and the proposed BoA, as in the original paper (Liu al., 2024). this appendix, we provide the results on LLaMA3 models for various quantization configurations.
Table 13. Weight-activation quantization performance on the transformed LLaMA3-8B
Precision
Mathod
WikiZ PPL (U)
Zav-shot Accuracy (t)
HIS
LAMB
WG
FP16
Baseline
6.137
33.67
71.61
81.19
79:15
72.23
45.00
81.01
73.24
70.39
OmniQQuat
2,965
25.94
26.22
38.50
24.86
0.00
28.00
50.44
49.33
30.41
AftineQuant
3.2c5
26.
52.57
SpinnQuantRCC
28.38
25.77
35.52
62.11
37.55
18.89
24.20
56.42
$1.85
39.04
W2A4KV16
GPTQ
26.35
23.38
38.30
60.43
37.34
16.44
26.20
58.71
53.51
39.29
DuQuant
DuQuant + LWC
4,4e4
26.28
27.10
38.38
25.81
0.00
28.00
50.49
50.12
30.77
BoA
OmniQquat
26.05
0.00
26.20
50.16
$0.36
AffineQuant
3.064
26.45
25.25
38.62
25.20
0.00
30,00
50.76
49.41
30.71
SpinQrant-REE
W3A3KV16
SpinQuanz-GiPQ
20.49
26.06
40,40
54.77
50.47
26.13
31.80
62.13
53.35
43.26
DuQuant
280.8
23.29
29.42
30.84
1.37
26.40
51.09
$1.22
31,88
DuQuant + LWC
783.1
22.35
27.57
38.07
28.05
0.18
27.00
51.74
50.59
30.69
BOA
17.56
32.56
OmniQqant
1424
35.02
33.69
2.46
AffineQuant
144.5
25.43
32.70
45.17
33.72
2.83
25.20
56.58
-9. 96
33.95
SpinQsant-BB RTN
8:229
45.14
72,10
75.44
74.07
59.53
40.80
76,44
67.88
63.93
W4AЛКI16
SpinQuant-GPTI GPTQ
7,636
16.33
72.31
75.17
72.69
64.27
42,80
76.44
68.27
64.79
DuQuant
7,793
45.90
71.46
72,60
74.36
66.06
42.40
78.02
69.61
65.05
DuQuant BOA
7,8%6
47.10
72.05
74.98
74.22
66.29
42.00
76.33
69.53
65.31
OmniQuant
188.2
22.78
32,32
43.39
31.69
2.01
25:40
$1,79
$0,20
32.82
AffineQuant
31.06
1.53
SpinQuant-Be RTN
8.503
12.06
69.40
72.78
72.69
60.50
38.40
74,21
65.19
61.90
W4A4K44
SpinQuantt-GPT GPTQ
66.59
DuQuant
8.000
45.09
70.88
74.01
73.67
64.32
41.00
76.55
68.90
64.42
DuQuant + LWC
8,402
44.50
70.29
73.82
73.69
38.61
39.00
75,08
66.77
62.76
BoA
7.705
45.22
74.54
72.81
74.09
65.72
43.60
77.53
66.69
68.03
BoA has been applied after transforming the model via SpinQuant.
The LET option has been deactivated for OmniQuant and AffineQuant
option does support models exploiiig GQA.
19

---

BoA: Attention-aware Post-training Quantization without Backpropagation
Table 14, Weight-activation quantization performance on the transformed L.aMA3.2-18
(P).(U)
Accuracy (f)
Arc-c
Arc-c
BQQ
HS
LAMB
OBQA
PIQA
WO
Average
Baseline
60.78
74.37
26.01
AllineQuant
5.1c3
25.00
26.73
37.89
25.
0.00
26.80
52.18
$1.36
30.5%
SpinQuant-RTN
93.90
23.63
$4.59
28.71
7.51
24.60
53.43
34.48
WZ44KV16
SpinQunnt-GPTC
104.4
21.59
32.28
50.83
30.11
7.55
26.20
53.75
49.96
34,03
DuQuant
1.0e5
76.28
25.04
46.61
26,
0.00
29.40
49.13
47.51
$1.30
DeQuant
1.0e4
26.88
25.78
28.00
50.98
BoA
59.95
23.98
37.04
54,68
32.02
9.33
27.00
56.04
52.41
36.56
OmniQQuant
7.1c3
26.79
26.60
38.20
25.24
0.00
29.20
51.03
48.93
30.75
AffineQuaat
3.6e3
27.39
26.77
25.59
49.89
SpinQqann-RIN
57.45
24.23
33.75
53.76
32.59
12.11
25.40
53.86
49.72
35.68
W3A3KV16
SpinQuant GPTQ
55.82
Duquuntt
2.9c4
25.09
26.
40.00
25.94
0.01
28.40
50.49
50.75
30.85
DuQuant LWC
1.2e4
25.17
25.21
39.33
25.65
0.00
27.20
51.85
51,22
30.70
BoA
48.47
23.81
38,97
53.21
35.55
13,90
26.80
51.84
$2.19
37.45
OmniiQunt
156.0
24.23
32.66
$7.89
31.15
0.84
27.20
54.84
50.12
33.62
AftineQuant
155.8
23.63
47.86
30.61
54.52
48.30
33.78
SpinQqunttKK KTN
17.66
32.25
54.90
63.30
53.38
40.69
30,80
68.93
53.75
49.74
W4M4VI6
SpinQuant: GPTQ
16.58
33.79
55.56
64.77
55.08
41.35
33.40
68.28
S1L85
50.89
DuQuani
2,3e4
26.37
26:47
40.28
26.40
0.01
30.00
48.59
48.38
30.81
DuQuant + LWC
1.9c4
26.19
26.81
40.31
25.48
0.00
26.00
50.76
47.12
30.33
BoA
16.25
33.62
58.12
65.72
$5.06
44.28
31,80
69.80
55.64
51.76
OmniQuant
219.2
23.55
31.82
45,47
29:46
0.79
28.00
51.52
48.22
32.35
AllineQuant
222.0
23.46
32,74
48.53
29.17
0.61
30.20
52.88
52.61
33.78
SpinQuant-Re
61.71
50.74
WIAIKYI
SpinQuant: GPTQ
18.31
30.72
55.18
61.90
52.91
39.13
31,80
67.30
51.93
48.86
DuQuant
25.97
DuQuant L.C.
1.704
27.05
26.01
38.96
26.07
0.01
28.40
49.73
49.41
50.71
BOA
17,83
RoA applied after transforming model via SpinQuant.
The LET option has been deactivated for OmniQuant and AffineQunnt hecomse this option does not support models exploiting GQA.
Table 15. Weight-activation quantization performance the transformed LLaMA3.2-3B
Zero-shut sho-shotddd (f)
Precision
Wiki2
Arc-c
Arc-c
BQ
HS
LAMB
OBQA
PIQA
Average
FP16
Baseline
11.04
46.16
67.80
78.62
70.44
62.15
36.00
75.52
67.40
63.01
OmniQQant
5.8c3
25.94
26.81
37.89
25.83
0.00
28.80
50.41
49.80
30.69
Allinoquant
23.89
26.64
38.41
25.74
0.00
26.00
49.56
49.49
29.97
46.31
34.06
31.14
51.87
31.41
12,70
26.60
55.39
50.99
35.52
SpinQuans-Co
23.12
Duiquatt
1,2c5
25.94
24.75
30.91
26.77
0.00
29.20
51.63
49.80
$1.001
DuQuant LWC
1.2c3
24.91
25.63
37.83
26.16
0.00
24.40
52.72
48.07
30.47
BoA
34.25
24.57
36.28
60.24
38.57
17.38
29.40
57,56
$2.49
39.56
OmniQuant
1,2e4
28.00
26.56
$7.95
25.84
0.00
27.20
49.73
51.14
30.45
AflineQuant
1.0e4
SpinQQann-RR
36.37
24.23
35.52
55.66
36.04
15.19
26.80
$7.78
49.72
$7.62
W3A3KV16
SpinQuuns GPTQ
27.23
28.33
40.70
60.73
44.15
21.53
28.20
58.16
$4.06
41.98
DuQuant
564.5
22.01
27.02
43.21
29.68
0.94
27.00
50.60
$2.80
31.66
DuQuant e LWC
$2.65
23.04
31.61
48:75
33.24
3.01
28.80
53.32
53.43
34,40
BoA
23.76
28.75
41.53
60.98
46.08
29.32
28.40
60.07
51..2
44.04
OmniQuant
131.8
24.57
34.55
48.75
36.10
3.21
27.00
56.37
$1.07
35.20
AffineQcant
151.8
23.55
31.60
47.65
35.80
3.01
27.80
56.26
49.57
34.78
SpinQuant-RIT
12.42
38.23
72.69
71.22
W4A4KV16
SpinQuans-Grr GPTQ
11.87
39.51
63.05
74,31
66.42
56.57
35,80
71.22
62.88
58.71
DuQuant
70.73
.2
DuiQuant L.C
13.32
38.23
60:73
75.29
65.93
51.5%
36,40
71.87
63.38
$7.95
BoA
OmniQQant
54.24
AllineCsant
197.1
21.12
32.87
42.20
31.70
1.25
27.00
$5.22
$0.75
32,80
SpiorQuant-RTN
39.16
48
$6.03
W1AKKY4
SpinQuant- GPTQ
12.24
39.33
61.91
72.32
65.56
54.57
35.20
70.13
61.33
57.54
DuQuamt
14,65
39.51
60.02
72.08
65.76
52.54
34,00
69.53
62.19
$6.95
DueQunt
13.84
38.99
59.68
72.05
64.76
49.
70.40
56.55
BoA
11.98
39.08
63.64
74.22
65.60
55.51
38,80
71.22
63.14
$8.90
BoA been applaed after transforming model via SpinQuant,
option has besn deactivated for OmniQuant and AffineQuant hocomse this option not support exploiting GQA.
20

---

BoA: Attention-aware Post-training Quantization without Backpropagation
E. Pseudocode for GPTQ
this appendix, we provide the pseudocode of the conventional GPTQ (Frantar et al., 2023), which omitted in the main manuscript due the page limitation.
Algorithm 2 GPTQ
Input: weights W, Hessian information Ueel, and pre-determined step size S
1: Initialize quantized output: Q +- Xdon
2: Initialize quantization errors: E t 0 dan Xd
3: for O,... deet 1 do
4: Quantize the j-th column: Q., + quant W :..
5: Estimate quantization error: E.j f- (W., Q..)/(Umllss
6: Update weights: Waj: + W ..:: - E. [Uaaljs:
7: end for
Output: quantized weights Q, quantization error
21


Reviews:
[{"review_id": "887DjuETzl", "reviewer": "Reviewer_DruQ", "summary": "This paper introduces a novel backpropagation-free PTQ method called BoA, which improves the conventional Hessian-based compensation strategy by comsidering layer-wise dependencies. Extensive experiments show its superiorty on both weight-only and weight-activation quantization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Memory comparison with GPTQ**\n - In Table I, we summarized memory costs of GPTQ and BoA. BoA indeed needs larger memory since BoA additionally uses outputs of other layers to consider inter-layer dependencies, which leads to better performance.\n - When the memory resource is limited, BoA can be used with a slight relaxation. We note that large memory cost of BoA is attributable to the row-wise Hessian for the value ($XA_{h}^{T}A_{h}X^{T}$ in (8)) whose shape is $[H,d,d]$ ($H$ is number of heads and $d$ is hidden size). In memory-limited cases, one solution is to apply the conventional Hessian ($XX^{T}$) for the value and consider inter-layer dependencies only for query and key. In doing so, BoA needs slightly more memory (e.g., 3GB for 30B) yet still performs much better than GPTQ (see Table I).\n\n<Table I. Comparison with GPTQ>\n\n(a) Memory (GB)\n|Method|7B|13B|30B|\n|-|-|-|-|\n|GPTQ|4.43|5.87|8.27|\n|BoA|9.55|16.6|32.8|\n|Relaxed BoA|6.45|8.40|11.6|\n\n(b) INT2 Wiki2 PPL\n|Method|7B|13B|30B|\n|-|-|-|-|\n|GPTQ|19.6|34.6|9.77|\n|BoA|10.3|8.31|6.67|\n|Relaxed BoA|10.5|8.70|7.01|\n\n**2. Time comparison with GPTQ**\n - For this issue, please see our response 1 to the reviewer dtMA due to the character limit. We note that QuIP and QuaRot require longer processing times than GPTQ since they conduct incoherent processing or rotation before applying GPTQ.\n\n**3. Individual effect of three tricks**\n - The ablation for the simultaneous quantization has been reported in Table 2. As evident, the simultaneous quantization accelerates the process significantly (e.g., 45 times on 30B).\n - The main goal of the other tricks is to reduce the memory usage, which allows us to quantize LLMs with a single GPU.\n     - Without the relaxation on Hessians, we need to compute and store the Jacobian $J$ for the softmax (see (6), (7)). Noting that the shape of $J$ is $[H,L,L,L]$ where $H$ is number of heads and $L$ is sequence length, storing $J$ needs 400 GB memory even for OPT-125M ($H=12, L=2048$), which is not possible with a single GPU of 80 GB memory.\n     - Without the proposed computation of inverse Hessians, we need to compute and store the Kronecker product $H_{col} \\otimes H_{row}$ before computing the inverse where shapes of $H_{col}$ and $H_{row}$ are $[d,d]$ and $[d_{h},d_{h}]$, respectively ($d_{h}$ is the head dimension and $d=Hd_{h}$).\n    Noting that the shape of the Kronecker product is $[d_{h}d,d_{h}d]$ for each head, we need 100 GB memory even for OPT-125M, which cannot be done with one GPU.\n - In the future version, we will elucidate the benefits of each trick.\n\n**4. Sequential processing of BoA for models with GQA**\n - For models with GQA, key and value have a few heads, so the gain achieved by the simultaneous quantization is reduced (see Table II(a)). In particular, if multi-query attention is used, key and value have only one head and thus cannot enjoy the benefits. \n - For those models, if one wants to speed up the process, the multiple-row selection, i.e., quantizing multiple $N$ rows simultaneously where $N$ rows do not participate in the error compensation of each others (see our second response), could be an intriguing solution. From Table II(b), we observe that such relaxation brings significant reduction in the processing time with marginal degradation.\n\n<Table II. Processing time of different strategies on LLaMA3-8B>\n\n(a) Time (sec) for one layer\n|Layer|n_heads|Sequential|Simultaneous|\n|-|-|-|-|\n|Query|32|1440|47.2 (-97%)|\n|Key|8|359|47.1 (-87%)|\n\n(b) Process time (sec) and INT2 Wiki2 PPL\n|Method|N|Time|PPL|\n|-|-|-|-|\n|BoA|1|4047|21.7|\n|Relaxed BoA|8|1772|21.9|\n||16|1393|22.3|\n\n**5. Block-wise update for row-wise compensation**\n - The block-wise update in GPTQ is based on the following fact: \"The final decision for the $i$-th column is affected only by updates performed on this very column, so updates to later columns are irrelevant [1].\"\n - Obviously, this can also be applied to the rows; rows outside the block are updated all at once after all rows inside the block are quantized. Under this strategy, the update formula in Prop. 3.1 is modified to $[\\delta W] _{j+B:,:} = -[ U _{row}^{T}] _{j+B:,j:j+B} E$, where $B$ is the block size and $E$ is the $[B,d]$ matrix whose $i$-th row is defined as $e _{i} U _{col} / [U _{row}] _{i, i}$. We will discuss this issue and provide the detailed proof in the future version.\n\n[1] E. Frantar et al., GPTQ\n\n**6. Misleading expressions**\n - We feel sorry for misleading expressions. We will globally modify \"integer weights\" to \"quantized weights\".\n - While BoA uses RTN as the reviewer noted, BoA updates weights during quantization, and thus the results become different. We also note that the key difference over conventional methods exploiting the naive nearest rounding is that BoA minimizes the attention reconstruction error (rather than the weight perturbation $\\Delta W$), which has been considered as a good approximation of the task loss degradation. We will clarify this point.", "claims_and_evidence": "Yes, the claims made in the submission are supported by evidence.", "methods_and_evaluation_criteria": "Yes, the methods in this paper make sense for the problem.", "theoretical_claims": "Yes, I have checked the proof.", "experimental_designs_or_analyses": "Yes, I have checked experimental designs and analyses in this paper.", "relation_to_broader_scientific_literature": "This paper is related to model compression.", "essential_references_not_discussed": "No missing references.", "other_strengths_and_weaknesses": "Strength:\n1. In addition to refining the existing dynamic weight compensation scheme, the authors also recognize the issue of increased computational complexity it introduced and propose corresponding solutions to address it.\n2. The theoretical derivation is thorough and detailed.\n\nWeakness:\n1. In abstract and introduction, the authors claim that the proposed BoA \"optimizes integer weights\", which leads to siginificant ambiguity. This makes me (or future readers) to initially believe that the paper achieves the integer format real-quantized weights without backpropagation, only to later discover that this is not the case.\n2. Still in abstract and introduction, the authors claim the conventional methods use naive nearest rounding, which cannot be seen as their shortages. Because the quantizer in BoA is still RTN except for AdaRound.\n3. Due to much complex Hessian calculation, the memory usage maybe unnegligible. But this paper misses the memory usage comparison between BoA and conventional layer-independent Hessian method GPTQ.  Especially on larger LLMs such as 70B, where the memory usage gap maybe much larger.\n4. In Table 5, the optimization type included into comparison is insufficient. Because in recent PTQ research, backpropagation-free methods are regarded as mainstream, such as GPTQ, QuIP, QuaRot. So it is crucial to include these backpropagation-free methods into comparison.\n5. In order to highlight the individual effectiveness of the three proposed tricks for efficient Hessian implementation, the authors should provide a ablation study on processing time reduction of each trick.", "other_comments_or_suggestions": "None", "questions_for_authors": "1. Can block-wise update strategy in GPTQ be used into row-wise compensation? \n2. In line 248-255, the authors assume different attention heads are indepent so that they can be quantized simultaneously. But what about V, O and K in GQA? In my opinion they can only be quantized sequentially.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "c9vxmxn39P", "reviewer": "Reviewer_QB2D", "summary": "This paper proposes a post-training quantization method based on the construction of attention-aware Hessian matrices to capture inter-layer interactions. The method generalizes GPTQ and also requires no back-propagation. Extensive experiments demonstrate the effectiveness of the proposed approach.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Processing time comparison**\n - The processing times of block-wise reconstruction techniques (e.g., OmniQuant and AffineQuant) have been summarized in Table 5. As evident, the proposed BoA completes quantization much faster than block-wise reconstruction techniques (e.g., for 30B, about 2.5 times faster than OmniQuant and 7 times faster than AffineQuant), yet outperforms those methods by a large margin (e.g., 20%p accuracy improvement for the 2-bit quantized LLaMA3-8B; see Table 4).\n\n - As suggested, we have newly compared the processing times of GPTQ and BoA (see Table I). We observe that BoA requires a longer processing time than that needed by GPTQ. This is because GPTQ quantizes all the rows simultaneously, while BoA sequentially quantizes them (see Fig. 1(b)) to consider the inter-layer dependencies within the attention module, which leads to better performance than GPTQ. Clearly, there is a trade-off between quantization speed and accuracy. In real cases, when one needs to consider inter-layer dependencies to preserve the performance of the original model as much as possible, the proposed BoA would be an intriguing solution when compared to existing gradient-based methods in terms of both quantization speed (see Table 5) and accuracy (see Tables 3, 4, and 6). \n\n - When faster quantization is required, BoA can still be used with a slight relaxation. Recalling that a longer processing time of BoA is attributable to the one-by-one quantization of rows, we can reduce the processing time by quantizing multiple, say $N$, rows simultaneously. For better understanding, let $N=2$ and $w_{i}$ be the $i$-th row. In the original BoA, the second row $w_{2}$ is quantized after compensating for the quantization error of the first row $w_{1}$. In the accelerated version, $w_{2}$ does not participate in the error compensation of $w_{1}$ but is quantized together, as if $w_{2}$ is independent of $w_{1}$. Our experimental results demonstrate that a significant reduction in the processing time can be achieved via this multiple-row selection strategy (see Table I(a)), yet this relaxation incurs marginal performance degradation (see Table I(b)).\n\n - In the future version, we will add a processing time comparison with GPTQ and discuss how to accelerate BoA in time-limited cases.\n\n<Table I. Performance comparison with backpropagation-free layer-wise PTQ methods>\n\n(a) Quantization processing time (sec)\n|Method|N|7B|13B|30B|\n|-|-|-|-|-|\n|GPTQ||427|731|1560|\n|QuantEase||5160|8530|17900|\n|BoA|1|3460|5590|11900|\n|Relaxed BoA|8|1620|3250|8880|\n||16|1230|2610|7410|\n\n(b) INT2 PPL ($\\downarrow$) on Wiki-2\n|Method|N|7B|13B|30B|\n|-|-|-|-|-|\n|GPTQ||19.6|34.6|9.77|\n|QuantEase||14.3|16.7|9.51|\n|BoA|1|10.3|8.31|6.67|\n|Relaxed BoA|8|10.4|8.37|6.78|\n||16|10.5|8.39|6.80|\n\n**2. Not discussed references**\n - We thank the reviewer for pointing out interesting references. The key difference of the proposed BoA over COMQ and QuantEase is that BoA considers inter-layer dependencies while COMQ and QuantEase assume layer-wise independence as in GPTQ. In Table I, we summarize the processing time and the PPL performance of QuantEase; results for COMQ are not included because its official github code does not support quantization of LLMs. As evident, BoA outperforms QuantEase yet completes quantization faster. In the future version, we will cite these works and mention the differences in the related work section.\n\n**3. Clarification of notations**\n - We appreciate the reviewer's constructive suggestion. The weight denoted by a capital letter (e.g., $W$) means the matrix, and the weight denoted by a small letter (e.g., $w$) means its vectorized representation. In the future version, we will add a paragraph that summarizes the notations to enhance the clarity of the paper.\n\n**4. Typo**\n - We appreciate the reviewer's careful reading. As the reviewer pointed out, \"$\\approx$\" is correct. We will fix this typo in a future version.", "claims_and_evidence": "Yes.", "methods_and_evaluation_criteria": "Yes.", "theoretical_claims": "The theoretical results are correct.", "experimental_designs_or_analyses": "The experimental evaluation is adequate.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "In addition to GPTQ, there are also some other backpropagation free PTQ methods that are based on the coordinate descent algorithm, e.g., [1,2]\n\n[1] Zhang et al., COMQ: A Backpropagation-Free Algorithm for Post-Training Quantization.\n[2] Behdin et al., QuantEase: Optimization-based Quantization for Language Models.", "other_strengths_and_weaknesses": "Strengths:\n\nThe paper is well written. The proposed method is original, and the quantization results are impressive when compared to SOTA. \n\nWeaknesses:\n\nTo improve the clarity,  it should be stated clearly the difference between the weights denoted by a small letter and the ones denoted by a capital letter, such as $\\Delta w^{(l)}$ and $\\Delta W^{(l)}$ in Eqn (3). It would be beneficial to include a paragraph to clarify the notations.", "other_comments_or_suggestions": "I recommend that the authors compare the quantization times of BoA with other methods including GPTQ and block-wise reconstruction techniques like OmniQuant.", "questions_for_authors": "Should the second \"=\" in Eqn (5) be \"$\\approx$\"?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cmk8lAR6BD", "reviewer": "Reviewer_dtMA", "summary": "This paper proposes a backpropagation-free PTQ algorithm, named BOA, to quantize LLMs. Authors consider inter-layer dependencies to optimize the weights being quantized, and in this case they use attention-aware Hessian matrices. As the computational cost of hessian matrices is too expansive, BOA simplifies the hessian calculation by involving several approximations. Experiments are benchmarked against existing PTQ approaches on LLMs, showing improved performance on perplexity and zero-shot task accuracy.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Processing time comparison with GPTQ**\n - We appreciate the reviewer's constructive suggestion. The main goal of Table 5 is to compare the processing times of PTQ algorithms _**considering inter-layer dependencies**_. Since GPTQ assumes layer-wise independence, we did not include it in Table 5.\n\n - As suggested, we have compared the processing times of GPTQ and the proposed BoA (see Table I). We observe that BoA requires a longer processing time than GPTQ. This is because GPTQ quantizes all the rows simultaneously, while BoA sequentially quantizes them (see Fig. 1(b)) to consider the inter-layer dependencies within the attention module, which leads to better performance than GPTQ. Clearly, there is a trade-off between quantization speed and accuracy. In real cases, when one needs to consider inter-layer dependencies to preserve the performance of the original model as much as possible, the proposed BoA would be an intriguing solution when compared to existing gradient-based methods in terms of both quantization speed (see Table 5) and accuracy (see Tables 3, 4, and 6). \n\n - When faster quantization is required, BoA can still be used with a slight relaxation. Recalling that a longer processing time of BoA is attributable to the one-by-one quantization of rows, we can accelerate the process by quantizing multiple, say $N$, rows simultaneously. Let $N=2$ and $w_{i}$ be the $i$-th row. In the original BoA, the second row $w_{2}$ is quantized after compensating for the quantization error of the first row $w_{1}$. In the accelerated version, $w_{2}$ does not participate in the error compensation of $w_{1}$ but is quantized together, as if $w_{2}$ is independent of $w_{1}$. Our experimental results demonstrate that a significant reduction in the processing time can be achieved via this multiple-row selection strategy (see Table I(a)), yet this relaxation incurs marginal performance degradation (see Table I(b)).\n\n - In the future version, we will add a processing time comparison with GPTQ and discuss how to accelerate BoA in time-limited cases.\n\n<Table I. Comparison with GPTQ>\n\n(a) Processing time (sec)\n|Method|N|7B|13B|30B|\n|-|-|-|-|-|\n|GPTQ||427|731|1560|\n|BoA|1|3460|5590|11900|\n|Relaxed BoA|8|1620|3250|8880|\n||16|1230|2610|7410|\n\n(b) INT2 PPL ($\\downarrow$) on Wiki-2\n|Method|N|7B|13B|30B|\n|-|-|-|-|-|\n|GPTQ||19.6|34.6|9.77|\n|BoA|1|10.3|8.31|6.67|\n|Relaxed BoA|8|10.4|8.37|6.78|\n||16|10.5|8.39|6.80|\n\n**2. Limited INT3 comparison and substantial performance degradation for INT2**\n - Most of INT3 quantization results have been reported in Appendix C (see Tables 7-11) due to the page limitation. In Table 10, we compare the performance of BoA with various transformation-based methods such as OmniQuant, AffineQuant, QuaRot, and DuQuant. Other transformation-based methods that the reviewer mentioned (AWQ and SmoothQuant) are not included since they perform worse than OmniQuant and AffineQuant [1], [2]. For INT3 quantization, BoA still performs the best and almost preserves the performance of the original full-precision model; the accuracy drop is 2.3%p for LLaMA3-8B and 1.3%p for LLaMA2-13B. \n\n - While it may not be easy to use 2-bit quantized models practically, we kindly note that low-bit results are commonly used as benchmarks to identify which algorithm performs the best [1]-[4]. One main reason is that it is often challenging to observe significant performance differences between algorithms at high precision (e.g., 4-bit) because the underlying models are robust enough to be quantized into 4-bit. In fact, even a naive rounding-to-nearest (RTN) quantizer usually does not incur severe performance degradation for INT4 (see Tables 7 and 8). On large-scale models (e.g., LLaMA2-13B), it is difficult to distinguish the performances of different algorithms even for INT3 (see Table 10). Thus, we believe that INT2 results could be a good metric to compare the performances of different algorithms.\n\n[1] W. Shao et al, \"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models,\" ICLR 2024.\n\n[2] Y. Ma et al, \"AffineQuant: Affine Transformation Quantization for Large Language Models,\" ICLR 2024.\n\n[3] Y. Jeon et al, \"A Frustratingly Easy Post-Training Quantization Scheme for LLMs,\" EMNLP 2023.\n\n[4] J. Chee et al, \"QuIP: 2-Bit Quantization of Large Language Models With Guarantees,\" NeurIPS 2023.\n\n**3. Better presentation of weight-only quantization results**\n - We appreciate the reviewer's constructive suggestion. While both Tables 3 and 4 present weight-only quantization results, the key difference of Table 4 over Table 3 is that the weight quantization has been performed after transforming models to be robust to quantization. As suggested, in the future version, we will make a single table by combining these results and enhance the clarity and readability.", "claims_and_evidence": "The claims made in the submission are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods make sense for the problem. The evaluation criteria is commonly used in the field.", "theoretical_claims": "The proofs of BOA are correct.", "experimental_designs_or_analyses": "Limited INT3 comparison: The INT3 experiments in table.3 only compare against GPTQ. A robust evaluation should include comparisons to a wider range of relevant quantization methods, such as AWQ, SmoothQuant.\n\nSignificant INT2 degradation: While the INT2 experiments include comparisons to several existing methods, the performance drop compared to FP16 is significant. The degradation suggests that the INT2 approach is not practical for real-world deployment.", "relation_to_broader_scientific_literature": "The topic of this paper is of significant importance and represents one of the most active and rapidly evolving research areas in the field.", "essential_references_not_discussed": "No missing citations of essential, directly relevant prior work.", "other_strengths_and_weaknesses": "strengths: \n - The mathematical derivation of the attention-aware Hessian approach provides a strong theoretical foundation for the proposed method. The steps are clearly explained, and the assumptions are explicitly stated.\n\nweaknesses:\n - The presentation of weight-only quantization results could be improved. Tables 3 and 4 both present results for weight-only quantization. Combining these results into a single table, or at least presenting them contiguously with a clear explanation of their differences, would enhance the clarity and readability of the paper. \n - A significant portion of the main experimental results (Tables 3, 4, and 6) focuses on INT2 quantization. However, the performance degradation is substantial. This level of degradation raises concerns about the practical applicability of the proposed method.", "other_comments_or_suggestions": "no", "questions_for_authors": "Table 5 presents processing times for the proposed method (BOA). However, a direct comparison with GPTQ, a closely related method, is missing.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FrLFSWTqso", "reviewer": "Reviewer_UfK3", "summary": "The paper proposes a novel Hessian-based backprop-free method for quantizing the attention weights of transformer models. The proposed method utilizes the computation pattern between different model weights within the attention modules to compute the Hessian with respect to quantization residual. Compared to previous work, which considers each weight independently, the resulting Hessian for each weight matrix incorporates information of other weight matrices based on the attention computation. For example, the Hessian with respect to the quantization residual of key matrix involves terms from the query matrix due to the multiplication operation between the two matrices for the attention computation. The paper also shows that the computation of the Hessian can be made efficient.\n\nThe paper conducts extensive experiments on Llama and OPT models with different sizes, and show that their method improves consistently over other backprop-free baselines and some baselines that involve backprop to compute the gradients.\n\n\n===Post rebuttal===\n\nThank the authors for addressing my concerns regarding the compatibility with RoPE. I think this is a very important part of the algorithm, which is missing from the current draft. Please do include these details if the paper is accepted.\n\nI still think the paper would benefit from including more comparisons with gradient-based methods, especially the light-weight ones. But I don't think this is a reason to reject since the paper mainly focus on gradient-free methods.\n\nHence I would recommend the paper for acceptance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**1. Hessians for models with RoPE**\n - When RoPE is applied, the proposed objective $|| K \\Delta Q ^{T} || _{F}^{2} = || K \\Delta W _{Q} X || _{F}^{2}$ used to develop the attention-aware Hessian for the query projection $W _{Q}$ (see line 196) is converted to $|| \\tilde{K} \\Delta \\tilde{Q} ^{T} || _{F}^{2}$, where $\\tilde{K} = RoPE(K)$ and $\\tilde{Q} = RoPE(Q)$.\n\n - Let $R _{l}$ be the rotary matrix for the $l$-th token (see [1, eq.(15)]) and $\\tilde{Q} ^{T} = [\\tilde{q} _{1} \\ldots \\tilde{q} _{L}]$, then the objective can be expressed as\n$$|| \\tilde{K} \\Delta \\tilde{Q} ^{T} || _{F}^{2} = \\sum _{l=1}^{L} || \\tilde{K} \\Delta \\tilde{q} _{l} || _{2}^{2} = \\sum _{l=1}^{L} || \\tilde{K} \\Delta (R _{l} W _{Q} x _{l}) || _{2}^{2} = \\sum _{l=1}^{L} || \\tilde{K} R _{l} \\Delta W _{Q} x _{l} || _{2}^{2}.$$\n\n - Recalling that the second-order derivative of $|| M _{1} \\Delta W M _{2} || _{F}^{2}$ with respect to $\\Delta w$ is $2M _{2} M _{2}^{T} \\otimes M _{1}^{T} M _{1}$ (see Footnote 2), the Hessian for $W _{Q}$ is given as\n$$H ^{(w _{Q})} = \\sum _{l=1}^{L} ( 2x _{l} x _{l}^{T} \\otimes R _{l}^{T} \\tilde{K} ^{T} \\tilde{K} R _{l} ).$$\n\n - Finally, we take the factorized approximation for the summation of Kronecker products [2, eq.(20)] (i.e., $\\mathbb{E} [M _{1} \\otimes M _{2}] \\approx \\mathbb{E} [M _{1}] \\otimes \\mathbb{E} [M _{2}]$):\n$$H ^{(w _{Q})} \\approx \\sum _{l=1}^{L} 2x _{l} x _{l}^{T} \\otimes \\frac{1}{L} \\sum _{l=1}^{L} R _{l}^{T} \\tilde{K} ^{T} \\tilde{K} R _{l} = 2X X ^{T} \\otimes \\frac{1}{L} \\sum _{l=1}^{L} R _{l}^{T} \\tilde{K} ^{T} \\tilde{K} R _{l}.$$\nBy taking similar steps, the attention-aware Hessian for the key projection $W _{K}$ with RoPE can be established as \n$$H ^{(w _{K})} = 2X X ^{T} \\otimes \\frac{1}{L} \\sum _{l=1}^{L} R _{l}^{T} \\tilde{Q} ^{T} \\tilde{Q} R _{l}.$$\n\n - We note that our results for LLaMA models are based on the above Hessians.\nIn the future version, we will clarify the attention-aware Hessians for models exploiting RoPE.\n\n[1] J. Su et al, \"RoFormer: Enhanced Transformer with Rotary Position Embedding,\" arXiv: 2104.09864. \n\n[2] A. Botev et al, \"Practical Gauss-Newton Optimisation for Deep Learning,'' ICML 2017.\n\n**2. Comparison with more current gradient-based methods, e.g., QuIP# and AQLM**\n - We sincerely thank the reviewer for pointing out interesting references. To say the conclusion first, it is not easy to compare the performance of the proposed BoA with the methods that the reviewer mentioned due to the following reasons:\n   - QuIP# and AQLM are vector quantization algorithms. Compared to BoA targeting scalar uniform quantization, those methods need additional memory (bits) for storing codebooks, which are required to perform the dequantization during the inference. In short, it is hard to perform a comparison under the same number of bits.\n   - QuIP# requires additional processing time to undo the incoherent processing during the actual inference [1, Algorithm 2], which hinders us from performing an apple-to-apple comparison.\n   - Reported results for QuIP# and AQLM are based on different numbers of calibration data; QuIP# and AQLM have used 6144 [1, Appendix F.2] and 2048 samples [2, Appendix C], respectively, while we utilized 128 samples.\n\n - We believe that the fine-tuning strategy adopted in QuIP# and AQLM is orthogonal to our approach and can be integrated with the proposed BoA in the following way: after quantizing weights inside the Transformer block via BoA, unquantized parameters (e.g., weights of normalization layers) are fine-tuned to preserve the output of the Transformer block, compensating for the quantization error of linear layers. Such integration will lead to performance improvement but may incur a longer quantization processing time. Indeed, it has been reported that AQLM requires 36-48 GPU hours for quantizing and fine-tuning a 7B model [2, Appendix D] while BoA completes the quantization within an hour. We leave this integration as a future research direction and will mention it in the conclusion section. We will also mention other works that the reviewer noted in the related work section.\n\n[1] A. Tseng et al, \"QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks,\" ICML 2024.\n\n[2] V. Egiazarian et al, \"Extreme Compression of Large Language Models via Additive Quantization,\" ICML 2024.\n\n**3. Better presentation to enhance the clarity**\n - We appreciate the reviewer's careful reading and constructive suggestions. In the future version, we will add the other weights inside the attention module as inputs for Algorithm 1 to clarify that the proposed BoA uses other weights to consider inter-layer dependencies. We will also clarify that SpinQuant, which has been integrated with BoA for the activation quantization and shows great synergy, uses gradient-based optimization to optimize rotation matrices that can well suppress activation outliers.", "claims_and_evidence": "The algorithm is motivated by a concrete insight on the computation within attention module and solid mathematical derivations on the Hessian with respect to the quantization residual. The insights are then backed up by extensive experiments on different models and datasets. The claims are solid to the best of my knowledge.", "methods_and_evaluation_criteria": "The evaluations are performed on Llama and OPT models with standard benchmarking datasets. The paper considers different quantization settings (different numbers of bits for weights, activations, and KV cache), which are standard practice in the quantization literature.", "theoretical_claims": "The paper is mainly algorithmic and experimental. The only claim is Proposition 3.1, which seems correct to me.", "experimental_designs_or_analyses": "The experiments cover a good set of state-of-the-art quantization methods. However, there are still a few missing, especially if the authors want to claim their advantage over methods that involve gradient computation (backprop). See papers in `Essential References Not Discussed`. In particular, a set of the works, e.g., [Tseng et al 2024a] and [Egiazarian et al 2024] perform gradient-based finetuning for PTQ which captures inter-layer interaction to some degree, which is the same intuition used in this paper. While these methods involves gradient computation, they are relatively lightweight, and the quantization time might be comparable. It would be nice if the authors could perform a comparison to some of these more current gradient based methods, in terms of both the performance and efficiency.\n\nI want to point out that some of the setups for BoA uses SpinQuant, which involves gradient computation to learn the transformation as well.", "relation_to_broader_scientific_literature": "The paper relates to a broad literature of Hessian-based quantization methods for LLM. The introduced computation of attention-aware Hessian that relates the different weight matrices within an attention module is novel to the best of my knowledge. See `Experimental Designs Or Analyses` for discussion on other methods.", "essential_references_not_discussed": "[Chee et al 2023] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, Christopher De Sa QuIP: 2-Bit Quantization of Large Language Models With Guarantees\n\n[Tseng et al 2024a] Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks\n\n[Tseng et al 2024b] Albert Tseng, Qingyao Sun, David Hou, Christopher De SaQTIP: Quantization with Trellises and Incoherence Processing\n\n[Chee et al 2024] Jerry Chee, Arturs Backurs, Rainie Heck, Li Zhang, Janardhan Kulkarni, Thomas Rothvoss, Sivakanth Gopi DiscQuant: A Quantization Method for Neural Networks Inspired by Discrepancy Theory\n\n[Egiazarian et al 2024] Vage Egiazarian, Andrei Panferov, Denis Kuznedelev, Elias Frantar, Artem Babenko, Dan Alistarh Extreme Compression of Large Language Models via Additive Quantization", "other_strengths_and_weaknesses": "One question I have is that the Hessian computation is based on the MHA computation between line 142-144, which doesn't involve RoPE embedding. With RoPE embedding, the Hessian will be different. How is the position embedding handled in the algorithm?", "other_comments_or_suggestions": "Input to Algorithm 1: make other matrices as input explicitly since the quantization of W depends on other weight matrices as well.", "questions_for_authors": "See my questions in previous sections. I am happy to increase my score if the questions could be addressed.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a novel backpropagation-free post-training quantization (PTQ) algorithm named BoA, which optimizes quantized weights by capturing inter-layer dependencies using attention-aware Hessian matrices. The main innovation lies in developing an efficient computation scheme for Hessian matrices that consider interactions within the attention modules of transformer architectures.

The reviewers largely appreciate the originality, strong theoretical foundation, and comprehensive experimental validation of the proposed approach  advancing existing block-wise quantization techniques.

However, some minor concerns were raised regarding clarity, the need for better integration of key results (particularly INT3 quantization results), and comparisons with gradient-based methods. The authors adequately addressed these concerns during the rebuttal phase:

- Provided additional analyses, particularly addressing the memory and processing time comparisons with GPTQ and other relevant methods.
- Demonstrated the practical applicability and competitive performance at INT3 precision levels, responding effectively to reviewer requests. Reviewer `dtMA` recommended moving the INT3 results from the appendix to the main text if you can manage space.

The authors are encouraged to integrate the additional clarifications provided during the rebuttal, especially regarding RoPE compatibility and INT3 quantization performance, into the final manuscript.
Decision: Accept (poster)