Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Authors: Junhan Kim
Chungman Lee
Eulrang Cho
Kyungphil Park
Ho-young Kim
Joonyoung Kim
Yongkweon Jeon
Venue: NeurIPS 2024 poster (2024)
Keywords: Hyper-scale
Compression
Quantization
Transformers
LLM
Primary area: natural_language_processing
TL;DR: We have proposed a next-level post-training quantization scheme for Hyper-scale Transformer models

Abstract: With the increasing complexity of generative AI models, post-training quantization (PTQ) has emerged as a promising solution for deploying hyper-scale models on edge devices such as mobile and TVs.
Existing PTQ schemes, however, consume considerable time and resources, which could be a bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required.
As a cost-effective alternative, learning-free PTQ schemes have been proposed. 
However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is a significant feature of Transformers.
In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency.
The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency.
Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa is accurate and efficient in quantizing Transformer models. The code will be available at https: //github.com/SamsungLabs/aespa.

Towards Next-Level Post-Training Quantization of Hyper-Scale Transformers
Junhan Kim," Chungman Lee; Eulrang Cho, Kyungphil Park,
Ho-young Kim,
Joonyoung Kim, Yongkweon Jeon
Samsung Research [jun_one. kim, chungman. lee,
Abstract
With the increasing complexity of generative Al models, post-training quantization (PTQ) has emerged promising solution for deploying hyper-scale models edge devices such as mobile and TVs. Existing PTQ schemes, however, consume considerable time and resources, which could be bottleneck in real situations where frequent model updates and multiple hyperparameter tunings are required. As cost-effective alternative, learning-free PTQ schemes have been proposed. However, the performance is somewhat limited because they cannot consider the inter-layer dependency within the attention module, which is significant feature of Transformers. In this paper, we thus propose a novel PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wiss wise for efficiency while targeting attention- wise reconstruction consider the cross-layer dependency. Through extensive experiments on various language models and complexity analysis, we demonstrate that aespa accurate and efficient quantizing Transformer models. The code will be available at fthhtpss://g thub. com/ SamsungLabs aespa
Introduction
Model size has been gradually growing, resulting in deep generative models such diffusion 25 and large-scale language models (LLMs) [29 35) becoming more mainstream; the trend of Al is transitioning from discriminative models generative models with numerous parameters in trillions. With the explosive growth in model complexity (parameters), the performance of AI models has been advancing and now approaching or even exceeding human intelligence levels. However, this growth in scale has resulted in a corresponding increase in computational costs, which necessitates the efficient processing and compression of AI models. Interestingly, one attempts to expand the complexity of AI models to scale up performance, whereas the other aims to compress models to reduce cost.
Quantization a promising solution and indispensable procedure facilitating the efficient deployment of Al models devices that mainly support fixed-point arithmetic. By reducing the precision of weights, the memory bandwidth requirements can be relieved, and the embarrassing parallelism of quantized models can be SIMDified using highly efficient vector processing units such as NPU. To minimize the inevitable performance degradation caused by quantization, we can choose one of two approaches: quantization-awar training (QAT) 5 [ and post-training quantization (PTQ) [2 [8]. Considering the model complexity and required resources (e.g., training costs and available datasets), QAT is not practical for compressing models with billions of parameters. Consequently. recent quantization studies on hyper-scake scale Transformer
B1 models have focused more on PTQ.
* Equual Contribution, t *Coresponding Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Although existing PTQ schemes have successfully quantized relatively small-scale models (e.g., ResNet) [23
I0 18 IT) they have difficulty handling large-scale models because of their time and space complexity. As cost-effective alternative, learning-free algorithms have been proposed recently 7 13 19,, but their performance somewhat limited because they not consider the inter-layer dependency and are reliant on the nearest rounding. There is an accuracy-efficien trade-off; thus, we aim to bridge the gap toward next-level quantization of hyper-scale Transformer models.
In this paper, we propose a novel PTQ algorithm, called aespa that pursues both accuracy and efficiency. The key idea of aespa is to perform quantization layer-wist for efficiency while targeting the attention-win wise reconstruction consider the cross-layer dependency.
Our contributions are summarized as follows:
We propose a new quantization strategy that balances accuracy and efficiency. Our scheme aims to reconstruct the attention output to consider the cross-layer dependency while quantizing models
layer-wise to pursue efficiency.
To accelerate the quantization process, we propose refined quantization objectives for the attention module. Through a complexity analysis. we demonstrate that quantization that is approximately 10 times faster than existing block-wise approaches can be achieved by exploiting the proposed objectives.
From extensive experiments language models, we demonstrate that our approach outperforms conventional schemes by significant margin, particularly for low-bit precision (INT2).
2
Background
2.1
Classic PTQ methods
Recent studies PTQ have mostly attempted minimize the increase in the task loss incurred by quantization rather than the quantization error itself (AW). Consider a pre-trained neural network parameterized by weights W. If we assume the well-convergence of the network, the problem of
quantizing weights W to minimize the loss degradation can be formulated as [16] 23
min Aw
E AwT.
H
Am
where (e)) is the Hessian related to the flattened weight w. Because computing and storing ((w) is infeasible, further assumptions have been made simplify (.. In (33, for example, layer-wise independence has been assumed, relaxing I into the layer-wise reconstruction problem:
min E [llloo AW
W x
where ( denotes the weights of the 6-th layer, X is the input, and Q is quantization function. For a uniform quantization, if the nearest-rounding is used to assign integer weights, Q is defined as
(xx) (clamp (L 2, -1) -2)
where s, and n are the scale, zero-point, and bit-width, respectively, and [-] represents the round-off.
Early studies on PTQ focused on optimizing the weight-rounding policy 23
10 18
These
studies have attempted assign each weight a "proper" grid (instead of adjacent grid), such
that the loss degradation could be minimized. In [23]. a dearning-based weight-rounding optimization algorithm, called AdaRound, has been proposed to solve the layer-wise reconstruction problem in 2 In 18 AdaRound has been extended to the following blick-wise reconstruction problem:
min E $[1,00 ( (Q(W(9)) f(( (w(t), (.0.)))l >)] (w
(4)
where ()( denotes the weights of the l-th block ((() (e.g., ResNet or Transformer block). By considering the dependency between layers inside the block, this algorithm, termed BRECQ, not only performs better than AdaRound, but also exhibits robust performance for a low bit-width (e.g., INT2).
aespa: attention-centric efficient and scalable post- post-training quantization algorithm

---

aespa
Phicaey
Divide wnd.-nnddonaaooo Enn--------rrrr Compier
Inner
Liner
Linwor
Linear Mig
(quanttrable)
Bigg
PAMPP
repetitively
Linear
Linear
fteed (fullpreecsion)
pre-computed
consddered
Kay Lasss
Proposed
Figure I: Overview aespa. Each weight is quantized separately to reconstruct the attention output.
2.2 PTQ for LLMs
Although AdaRound and BRECQ have been successful in quantizing small-scale networks (e.g., ResNet), scaling those learning-bann based schemes to LLMSs with billions of parameters is challenging. In fact, BRECQ requires more than 20 GPU hours quantize relatively small-sized language models (e.g., OPT-2.7B; see Appendix K which would not be suitable for the real-world deployment of LLMs where models be deployed are frequently updated.
Owing to the excessive time and memory costs of classic PTQ schemes, recent studies have focused developing cost-effective alternatives for quantizing LLMs. In OPTQ [7]. one-shot PTQ scheme that optimizes a bright-rounding policy without relying on learning, has been proposed. In addition, PTQ schemes that enhance the performance of the nearest-rounding, rather than optimizing the weight-rounding policy, have been proposed. These schemes use additional "foldable" parameters
suppress activation outliers quantize weights more precisely [33 19
13
27 20
Although previous studies have mitigated the computational overhead of classic PTQ methods, they often sacrifice the low-bit quantization performance or suffer from unstable quantization process. The main reason for this unsatisfactory performance is that all the schemes mentioned above, except OPTQ, rely on nearest-rounding and do not optimize the weight-rounding policy, Moreover, most of them target layer-wise reconstruction in 2 not block-wise reconstruction in (4). thus ignoring the cross-layer dependency within the attention module. Although [27] [00 target block-wise reconstruction via learning. they need to approximate gradients for con-differentiable quantization function, which results in unstable training process (see Table in Section 19]
Thus, we propose a novel PTQ scheme that balances accuracy and efficiency. In contrast to conven- tional LLM quantization methods, our scheme optimizes weight-rounding policy while targeting block-wise reconstruction to consider the cross-layer dependency. The key difference over clas- sic block-wise weight-rounding optimization is that quantize models layer-wise for scalability, whereas layers are jointly quantized the existing methods. Furthermore, we present an efficient pre- computation-basee based method for the computation of the block-wise objective in (4), which significantly reduces the computational overhead caused by repeated attention operations.
3 Method
3.1 Motivation
To gain insight into approach, we first consider the objective of the layer-wise reconstruction
in 2 Let (o ((W(()) (()), then the reconstruction error can be expressed as
E |AWx
|2] =E TTAAAX =tr (AW E XXX AWT).
(5)
By foldable parameters, we mean the parameters that can be merged into other layers within the Transformer block (e.g. LayerNorm), thereby imposing extra computational cost during the inference [13].

---

X
W
W
W
W
L
L
||z
(a) Layer Wise, Layer Output
(b) Block Wise, Block Output
(c) Layer Wise, Block Output (Proposed)
Figure 2: Quantization strategies (simplified)
Consequently, the layer-wise quantization problem can be recast follows:
min (AW- X AW"). AW
(6)
The new form of the quantization objective in (6) implies that if E X is pre-computed and stored before quantization, we can measure the reconstruction error over the entire calibration dataset with a single matrix multiplication and element wise multiplication El This is in contrast to the original formulation
which requires the computation of Q(W) X or AWW for every input X.
A natural question that arises from this finding is "Can we also measure the block reconstruction error efficiently based on such a pre-computation?" In the following subsections, we describe our main strategy to simplify block-wise quantization and then present refined objective for the attention module, where the objective can be computed efficiently with certain pre-computed values.
3.2 Quantization strategy of aespa
When quantizing the attention module using conventional block-wise reconstruction methods (Fig- ure a)), the query, key, and value projections have been jointly optimized such that
SA K - SA v
min
E
AWQ
AAW
where the output of attention module SA(Q, K, V) is defined as
QK
SA(Q, K, V) softmax
V V
(8)
In such case, we need to compute SA K, V) for every batch sequence in each iteration, which computationally heavy and consuming (see Section 3.5 for details on complexity).
To overcome this computational overhead, we quantize each projection separately divide-and- conquer manner. For example, when quantizing the query projection Wo. we fix WK and Wy with full-precision (Figure (())), which facilitates the factoring out of common terms affected by WK k and Wv (see Section 3.3 for details). We emphasize that this strategy differs from conventional layer-wise quantization schemes (e.g., AdaRound and OPTQ) in that we aim to minimize the reconstruction егror for the attention module, not the reconstruction error for each layer.
We conduct experiments to demonstrate the importance targeting attention wise reconstruction and validity of the proposed quantization strategy. In experiments, set the loss function for each projection as the attention reconstruction error in 77 but quantize each projection separately (see Figure (c). Table in Appendix B summarizes the performance of AdaRound, BRECQ, and approach. As evident, our approach uniformly outperforms AdaRound for all bit-widths, although both methods quantize models layer-wise. This because we can consider cross-layer dependency (i.e., relationship between the query, key, and value) by targeting attention-wise reconstruction, which different from AdaRound wherein layers are considered independent. Furthermore, once target attention-wii wise reconstruction, separate layer-wiss wise quantization does not incur severe performance degradation compared the joint quantization method (BRECQ). In fact, our approach causes only a marginal performance degradation for 2-bit and exhibits comparable performance for 3-bit and 4-bit. For further discussion on the proposed strategy, see Appendix
3.3 Refined quantization objectives for aespa
One might ask whether our strategy incurs more computational cost than that required by the joint quantization because update only one layer at time (see Figure [b)). This is in contrast
'We note that the computation of tr(ABCT) can be implemented torch. sunch.B O C), where O denotes the coment-wise product operation. They are mathematically equivalent.

---

to existing methods, in which the layers inside the attention module are updated simultancously (Figure Ta)). To reduce this additional cost, we refine the quantization objective for each projection.
Value projection When quantizing the value projection Wv, the query and key projections fixed with full-precision. In this case, by factoring out the common term influenced by Q and K, we can simplify the attention reconstruction error ASAv as follows:
ASAv R
[|| AV
-vv-A A = [IAAVI]]] = AWWWAA
|
(9)
Thus, the problem to quantize Wv to minimize the attention reconstruction error can be recast as
min AW,
E
[IAW>xXA]|]]
(10)
Query projection When the key and value projections are fixed with full-precision, the attention reconstruction error ASAq caused by AWQ is expressed as
ASAq = E SA(Q, K, V- SA(Q, K, v/ = =E [AAVII?]
(11)
where AA- softmax ( /Va) softmax(QQe /Vd). To avoid the computational overhead of repetitive softmax operations, we approximate AA with its first-order Taylor series as
AQK
A A
.JT sofemax:
(12)
Vd
where J softmax is the Jacobian of the softmax function. By combining [1] and 12
we obtain
ASAQ ~ 1 [E|IAQRO |I softmax VI
1 _ [ softmax K KAW.
(13)
Although we can circumvent conducting attention operations using the modified form in (3). large amount of memory is required to store the Jacobian Jaoftmax (e.g.. more than 100 GB of memory for OPT-125M) As cost-effective alternative, we build an upper bound of (3 and then employ it as a surrogate of ASAQ when quantizing Wa Specifically, by noting that
|| J softmaa KA KAWQ X
| Jeoftmax
|KWW
(14)
and the term VV softmax 1| is fixed in the quantization process, minimize | FWWW XI with the hope that ASAQ also decreases. In other words, our quantization objective for Wa is
E ||IKAW X AWO
min
(15)
Key projection By taking similar steps, the quantization objective for the key projection WK can be formulated as (see Appendix for the detailed derivation)
min AWK
E B|1QAW>>X1
(16)
3.4 Algorithm description
The proposed aespa consists of two main steps. Specifically, aespa first determines the quantization parameters (i.e., scale and zero-point) and then optimizes integer weight W... for each weight.
Note that only used the definition of the attention operation when developing the refined objectives in (IO), (5).. and (16). Thus, our objectives can be integrated into any layer-wise quantization scheme without effort. For example, we can compute the quantization parameters by combining existing parameter initialization algorithms (e.g., AWQ [19] and Z-FOLD (33 with the proposed objectives. We can also optimize a weight-
policy using conventional methods (e.g., AdaRound 23
Note that shape Jsoftmax is [T., I., T.] (T the input sequence length) each attention head because J softmax (az) diag(az) at E R LxL for each row of A.
5

---

together with our objectives (see Appendix for details). In the proposed aespa, we use Z-FOLD in computing the quantization parameters and employ AdaRound in optimizing a weight-rounding policy. In Algorithm (see Appendix A we summarize the proposed aespa
To acceleratee the weight-rounding learning process, we further modify the objective functions such that the value can be computed efficiently via pre-computation, as in (5).
Modified objective for (10 The proposed objective for the value projection can be recast as
E
|| = (AW, XA" AX AW;)
(17)
The modified objective allows us to perform each iteration of the rounding learning efficiently, Specifically, by computing [[X ["XA"AA A"AX"" before quantization and reusing it in the quantization pro-
cess]) we can avoid the overhead of computing loss with one simple matrix multiplication and single element-wise multiplication (see Footnote 4
XAT
2 for every input X and compute the
Another intriguing feature of this modification is that facilitates a more reliable update of A AW than the original objective in (10). Specifically, because E| A" A"AAA"X"" AX"] is pre-computed using all calibration data, the loss computed with (17) considers the entire calibration dataset (i.e., the batch size is the total number of data). Thus, a better estimate of the true gradient can be obtained without any memory issues, which could lead to more consistent updates AW and faster convergence [28].
The modified objective in 17 also implies that the Hessian Hy for each row of Wy is
Hy 22 A A
(18)
We note that the proposed Hessian Hy differs from H 2 XXX which has been commonly
used as an approximated Hessian in conventional methods (6, 7 13 31. The key reason for the difference is that we consider the dependency between We. WK. and Wy by targeting attention- wise reconstruction, whereas the previous methods assumed independence. To observe the effect of considering the cross-layer dependency, language models and then compare the performance of the quantized models (see Appendix D). Evidently, the quantization performance is much better when the proposed Hessian Hy is used, which demonstrates the importance of considering the cross-layer dependency.
different Hessians (i.e., Hy and H) when quantizing
Modified objectives for (15) and (16) If we denote the vectorized representation of A AWQ Awg.
the proposed objective in 115 can be expressed as (see Appendix for the derivation)
E KAW x
= E [
K)
Auq-
(19)
where @ is the Kronecker product operation. To reduce the memory cost of storing the Kronecker product term E Xx" & KT K we approximate it as 2
x
K
K
E
&
E
K
(20)
By combining (19 and
20
we obtain
E
KA
Aw
EE
XX"
E K" K))
AWe
K
AWOE
XX
AW?),
(21)
where the proof of (a) is provided in Appendix projection be recast as
By taking similar steps, the objective for the key
E
|QQAw<X|
= tr (E Q AW xx" AW
(22)
The modified objectives in (2T and 22
imply that the loss over the total calibration dataset can be
calculated efficiently by computing K
T K), E Q). and E X X in advance.
'We use layer-wise objective in for the weights other than query, key, and value projections (i.e.. out-projection and weights inside the feed-forward network). 'The term | A" AxT] is not affected by AWv and thus fixed in the quantization process.

---

3.5 Complexity analysis for aespa
We discuss the computational complexity of aespa. Specifically, we analyze the number of floating- point operations (flops) required to perform one iteration for weight-rounding optimization (line 6 in Algorithm D. For each projection, the required number of flops is summarized as follows.
By reusing the pre-computed E A" AX*], J. the loss value in can be computed with matrix multiplication and one element-wise nultiplicationnaadditon (see Footnote 4 The associated cost is 2d,,d² + d, - flops, where d is the hidden size and d, is the input dimension for each attention head.
Value:
Query/key: Once Ej K K]. E[Q" and [[Xx") have been computed in advance, the loss values in 21) and 22 be computed by performing two matrix multiplications and one element-wise multiplication/addiiin
This requires 2d + 2d flops for each projection.
To summarize, the total number of flops required in each iteration of the proposed aespa is
Cuespa = 6d + 4df. d + d,d dd)
(23)
We emphasize that regardless of the amount calibration data, the number of flops to compute the loss considering the entire dataset is fixed as Cuespa.
We now compare the complexities of aespa and conventional block-wise quantization methods. can be easily verified that the existing methods require the following number of flops for handling B input sequences of length L (see Appendix G):
Cezistt ((6d,,L + 4d, + 21 - - O(Bd, max LJ).
(24)
Table 7 in Appendix G summarizes the computational costs for different sizes of OPT models. For the conventional methods, we report the cost of using four sequences in each iteration (B 4). We observe that the computational cost of aespa is considerably lower than that of conventional methods. In particular, for small-scale models, aespa performs ten times fewer number of flops. It can be observed that the gap between Cuespa and Cezist decreases as the model size iecrasss. This is because the hidden size d exceeds the sequence length (which fixed for all models) for large models. Nevertheless, aespa still incurs a lower computational cost, and the gap increases if conventional methods use larger batch sizes.
4 Experimental results
4.1 Experimental setup
We quantize publicly available LLMs (e.g., OPT (35), BLOOM [26], LLaMA 29, and LLaMA2 30)
using the proposed aespa. When implementing aespa, we compute the quantization parameters with Z-FOLLD
13] and optimize a weight-rounding policy via AdaRound 23). where the proposed row- wise Hessians and loss functions (see Table in Appendix A are utilized instead of the existing ones. When computing the quantization parameters, we follow the stopping criterion introduced by (13). Before optimizing a weight-rounding policy, we update the full- full-precision weights via OPTQ [7]. which empirically reduces the number of iterations required for weight-rounding optimization. When optimizing a weight-rounding policy, we set the number of iterations, learning rate, and weight of the rounding loss (see in 28 to 2,000, 0.015, and 1.5, respectively.
When constructing the calibration dataset, we randomly sample 128 segments consisting of 2048 tokens from the C4 dataset [24] as in 7 3 In our experiments, quantize only weights and retain activations in full-precision because activations are not a significant bottleneck for L.L.s 7 and the inference of L.LMs can be accelerated sufficiently by reducing memory movement through weight quantization [15]. We evaluate the performance of the quantized models using benchmark datasets (e.g., WikiText-2 [22), C4 (24), and PTB (21]) and zero-shot tasks. Except for the experiments on the LLaMA2 models, which were performed using an NVIDIA H100 GPU, we conducted all experiments using single NVIDIA A100 GPU (80 GB).
4.2 Comparison with prior arts
Comparison with block-wise PTQ schemes We compare the proposed aespa with conventional wisc PTQ methods, among which BRECQ is classic weight-rounding optimization method,
7

---

Table 1: Performance (PPL .) of the proposed aespa and conventional block-wise PTQ methods.
(a) WikiText-2
OPT
LLaMA
LLaMA2
Precision
Method
125M
1,38
2.78
6.78
7B
138
30B
7B
138
FP16
Baseline
27.65
14.63
12.47
10.86
5.677
5.091
4,101
5.472
4.884
BRECQ
[B
33.25
16.09
13.37
OOM
OOM
OOM
0OM
OOM
OOM
OmniQuan
27
39.14
17.59
14.87
12.87
6.716
5.798
4,963
6.798
5.751
INT3
Affineuuant
36.15
17.26
14.25
12.30
6.712
5.820
4,951
6.795
5.757
aespa
32.71
15.79
13.14
6.579
5.611
4.688
6.241
5.462
BRECQ
IB
60.38
56.25
113.6
OOM
OOM
OOM
OOM
OOM
OOM
OmniQuant
27
NaN
399.6
1.6c3
4.9c3
18.18
NaN
10.15
35.40
20.19
INT2
AftineQuant
20
143.9
56.45
35.16
25.32
18.83
11.08
NaN
NaN
18.49
aespa
71.18
24.26
22.22
15.71
11.94
10.30
7.845
13.99
12.14
(b)
OPT
LLaMA
LLaMA2
Precision
Method
125M
1.3B
2.7B
6.7B
7B
13B
30B
7B
13B
HP16
Baseline
26.56
16.07
14.34
12,71
7.344
6.79%
6,131
7.264
6.727
BRECQ
18
29.74
17.46
15.39
OOM
0OM
OOM
OmniQuant
34,92
18.83
16.80
14.21
8.605
7.604
6.822
9.085
7,821
INT3
Aftineuuant
20
32.78
18.27
16.11
13,80
8.631
7.609
6.803
9.059
7.732
acspa
29.51
17.10
15.27
13.15
8,465
7.399
6.634
8.225
7.392
BRECQ
LB
47.85
41.05
83.32
OOM
OOM
OOM
OOM
OOM
OOM
OmniQuant
x
NaN
239.1
1.1e3
18.59
NaN
14.74
26.27
18.93
INT2
AftineQuant
43.66
29.75
16.87
NaN
15.20
aespa
56.88
23.54
22.53
17.28
13.63
11.46
10.35
14.36
13,59
'NaN' means that loss diverges in the quantization process.
'OOM' means that out-of-memory issues occur when quantizing models with single A GPU.
Results for high bit-widths are provided in Appendix due to the page limitation.
and OmniQuant and AffineQuant are LLM quantization methods that mitigate the computational overhead of BRECQ by learning only a few quantization and foldable parameters 227 20 For OmniQuant and AffineQuant, we ran the official codes provided by the authors. For both methods, we activated the learnable equivalent transformation (LET) and learnable weight clipping (LWC) options and reported the obtained results. When implementing BRECQ, employed the hyperparameter settings provided in [8]. In this comparison, the BLOOM models and OPT-350M were excluded because they not supported by OmniQuant and AffineQuant.
As Table shows, aespa uniformly outperforms OmniQuant/ Sataantt//finuuaatttt particular, the perfor- mance gap is significant for 2-bit; while Onniiuan//AffineQQR diverges) or collapse (perplexity (PPL) > 10²), aespa exhibits reasonable PPL. The outstanding performance is attributed to the fact that aespa optimizes weight-rounding policy after determining the quantization parameters (lines 5-8 Algorithm []. whereas Omni ManniQuant/Aiit muanttAffiieeuum rely on naive nearest rounding and approximate gradients for the con-differentiable quantization function.
suffer from instability (i.e., loss
Although BRECQ performs best for the 2-bit quantization of OPT-125M, it lacks scalability: BRECQ requires approximately 20 GPU hours for a relatively small-scale OPT-2.7B (see Table 14] in Ap- pendix K. Even for O25, BRECQ requires approximately GPU hours, whereas the proposed aespa completes quantization in 5 minutes. One might wonder why the performance of BRECQ wors- ens as the model size increases. We assume that this is attributable to the choice of hyperparameters (e.g., learning rate and weight of rounding loss). In fact, the hyperparameters presented in [18] have been optimized for ImageNet, but not for LLMs. It is expected that we can obtain better performance for BRECQ via deliberate hyperparameter tuning; however, this would not be feasible for real-world deployment because it requires considerable time (see Table 14 in Appendix K).
Comparison with layer-wise PTQ schemes We compare the proposed aespa with conventional layer-wise PTQ schemes, among which RTN the method that naively assigns the nearest grid, OPTQ is backpropagation-free weight-rounding optimization algorithm 77. and Z-FOLD is the
httpss//gituub.coo//peeccc Laabeemiiaa//m http:://iitubb.oommyycdaeeeee/e WWe note that our results are different from those reported in [27 20 where different calibration dataset (WikiText-2) used; see Appendix [ for more discussion this issue.

---

Table 2: Performance (PPL 1) of aespa and existing layer- wise PTQ methods on BLOOM models.
WikiText-2
C4
Precision
Method
560M
1.1B
1.78
3B
7.1B
560M
1.1B
1,7B
3B
7.1B
FP16
Baseline
22.42
17.69
15.39
13:48
11.37
26.60
22.05
19.49
17.49
15,20
RTN
56.74
49.85
63.37
39.07
17.35
66,99
60.41
113.6
79.84
22.54
OPTQ
31.55
23.84
20.06
17.13
13.56
34.62
27.62
23.87
20.96
17.43
INT3
Z-FOLD
26.52
20.99
17.39
15.11
12.26
29.97
24.43
21.52
19.01
16.12
aespa
25.39
19.81
16.95
14.68
12,00
29.10
23.80
20.93
18.55
15.91
RTN
7.8e5
9.8c5
3.5e5
1.4c5
2.1e5
1.4e6
2.1c6
2.7c5
9.2c4
1.3e5
OPTQ
1.7c3
1.9c3
1.4e3
796.5
194.2
533,4
538.0
562.9
351.6
112.8
INT2
Z-FOLD
65.45
44,50
35.69
27.40
18.87
64.11
42.96
37.26
32.64
22.46
aespa
34.12
27.67
21.65
16.31
45.04
35.12
29.95
25.04
20.00
Results for high bit-widths and other language models (e.g. OPT, LLaMA, and LLaMA2) are provided in Appendix
method exploiting additional foldable parameters to quantize weights more elaborately and Tables 9 12 (see Appendix summarize the results for the OPT, BLOOM, LLaMA, and LLaMA2 models of various sizes. Evidently, aespa uniformly outperforms conventional schemes, regardless of the size and type of LLMs. In particular, for 2-bit, there exists significant performance gap between aespa and existing methods; the PPL obtained by aespa is twice as low as those of conventional methods for small-scale models (e.g., OPT-125M) The key factors leading to such an outstanding performance are: 1) the consideration of the cross-layer dependency achieved by targeting attention- wise reconstruction. and 2) efficient weight-rounding optimization based pre-computations.
Table
Zero-shot task performance We evaluate the reasoning performance of quantized models using
zero-shot tasks (e.g., ARC [4) HellaSwag [34], and MMLU ( ). We note that the zero-shot setting
was ensured in our experiments because we used excerpts from randomly crawled websites (not
task-specific data) as calibration dataset. From the zero-shot results in Table andd Table [3 (see
Appendix . we observe that the proposed aespa performs the best in almost all cases, and the performance gap between aespa and the existing methods is large for 2-bit.
Time cost We summarize the processing times of the different quantization algorithms in Appendix K We note that the processing time of aespa includes the time required for pre-computations (lines 2-4 in Algorithm []] As expected, aespa completes quantization much faster than BRECQ For example. while BRECQ requires more than 10 GPU hours for OPT-1.3B, aespa completes quantization in I 1.24 hours, which demonstrates the effectiveness of the proposed pre-computation-baseee based loss computation strategy. Although other block-wise methods (OmniQuant/ Caamiiuuan/AAiiieQeee AftfiAffffuunn perform quantization faster than aespa for hyper-scale models, they suffer from unstable training processes or exhibit poor PPL performance (e.g., PPL of OmniQuant is larger than 10x for OPT-6.7B see Table D.. In addition, we observe that OPTQ performs quantization quickly, but its 2-bit performance collapses regardless of the model size (see Table in Appendix D.. Except for aespa, F.FFOL is the only method that shows both reasonable performance and processing time.
Discussion In real situations, when one needs to preserve the performance of the original model as much as possible, the proposed aespa would be an intriguing solution. In particular, when deploying LLMs on resource-constrained platforms where up to 7B models are commonly employed (e.g., mobile devices), aespa would be good fit. Even when fast quantization of hyper-scale models is required, aespa can be used with a slight modification. Specifically, in time-limited cases, one can skip the weight-rounding optimization (lines 5-8 in Algorithm [ and simply determine the quantization parameters using the proposed Hessian that considers the cross-layer dependency (line 4 in Algorithm []. In doing so, we can not only save the time required to optimize weight- rounding mechanism, but also save the memory required to store pre-computed values (F[K? K] and E EjQ" Q). Indeed, when performing only quantization parameter computation, we achieved a significant reduction in the processing time (see Table T5jin Appendix K) while still exhibiting better performance than conventional methods (see Table 6 in Appendix D
5 Conclusion
We proposed a next-level PTQ scheme for Transformers, called aespa. By targeting the attention- wise reconstruction while quantizing Transformers layer-wise, we could consider dependency within the attention module and complete the quantization much faster than the existing
the cross-layer

---

Table 3: INT2 zero-shot performance (aceuracy 1) of aespa and existing methods.
Model
Method
ARC-e
ARC-e
HellaSwag
MMLU
Average
FP16
44.62
72.85
76.18
32.19
56.46
28.67
25.00
26.43
25.72
26.46
OPTQ
29.18
24.04
LLaMA-7B
F-FOLD
30.63
52.44
53.55
23.27
39,97
OmniQuant
27
27.22
49.20
50.65
23.74
37.70
Affineuuant
20
27.90
49.58
51.85
24.15
38.37
aespa
33,36
55.64
58.31
23.12
42.61
47.87
74.75
43.46
61.29
RTN
28.16
27.15
26.09
25.53
26.73
OPTQ
7
27.22
25.76
25.67
25.05
25.93
LLaMA-13B
Z-FOLD
32.68
58.08
57.89
26.44
43.77
OmniQuan Quant
27
NaN
NaN
NaN
NaN
NaN
AffineQuant
20
32.17
56,36
60.29
25.22
43.51
aespa
34.73
FP16
52.90
78.96
82.63
54.66
67.29
27.05
26.39
25.87
25.48
26.20
OPTQ
27.13
26.60
26.12
23.56
25.85
LLaMA-30B
Z- FOLD
39.93
65.07
65.89
30.85
50.44
OmniQuant
27
34.22
$8.50
64.83
25.91
45.87
AffineQuant
20
NaN
NaN
NaN
NaN
NaN
aespa
41.13
67.00
67.90
35.67
$2.93
FP16
46.16
74.49
75.99
41.87
59.63
RTN
28.33
26.01
25.88
23.02
25.81
OPTQ
7
26.37
26.09
25.11
25.10
25.67
LLaMA2-7B
FOLDD
R
26.62
42.68
44.71
22.88
34,22
OmniQuunt x7
25.00
42.97
23.03
32.45
AffineQuant
NaN
NaN
aespa
30.29
$1.47
56.75
25.59
41.03
FP16
49.06
77.44
79.39
$2.10
64.50
RTN
27.22
25.04
25.58
24.69
25.63
OPTQ
26.71
27.19
25.42
23.74
25.77
LLaMA2-13B
Z- FOLD
P
28.41
48.32
$1.59
23.98
38.08
OmniQuant
A7
27.13
47.98
53.27
23.81
38.05
AffineQuant
20
30.80
52.90
57.74
24.45
41.47
aespa
31.91
55.18
55.49
29.97
43.14
'NaN' that loss diverges quantization process.
Results for high bit-widths are provided in Appendix
J due puge limitation.
approach for block-wise reconstruction (i.e., BRECQ), Extensive experiments on language models have demonstrated the outstanding performance of aexpa.
Limitations and future work While focused on the attention output, the output of the entire Transformer block (containing fully connected layers) can be used to consider the dependencies between more layers. However, in this case, the objective functions would be more complicated than those in and (25) due to nonlinear activation functions (e.g., SiLU for LLaMA models), normalization layers, and weights of larger dimensions. Enhancing the quantization performance by developing an efficient form of the reconstruction eтror for the Transformer block would be an interesting future work. Furthermore, while we focused on weight-only quantization, activations may need to be quantized to deploy AI models on integer-only arithmetic hardware (e.g.. NPU). Extending the proposed aespa for weight-activation quantization by integrating existing techniques to suppress activation outliers [33 I is also interesting research direction. Finally, while we verified the performance of aespa with LLMs, we believe that aespa can also used for the quantization of diffusion models. To that end, we may need to incorporate some diffusion-speen speccffcc quantization strategies to overcome output distribution discrepancies over different time steps (e.g., grouping of time-step steps with similar distributions [32], temporal feature preservation 19), and separate quantization for shortcuts U-Net [77), which will be considered in our future studies.
10

---

References
[1] Saleh Ashkboos, Amirkeivan Mohtashami, Maximilian L Croci, Bo Li, Martin Jaggi, Dan Alistarh, Torsten Hoefler, and James Hensman. QuaRot: Outlier-free 4-bit inference in rotated LLMs. 2xXiv:2404.004556 2024.
[2] Aleksandar Botev, Hippolyt Ritter, and David Barber. Practical Gauss-Newton optimisation for deep learning. In International Conference on Machine Learning, pages 557-565. PMLR,
2017.
[3] Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. QulP: 2-bit quantization of large language models with guarantees. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[4] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the A12 reasoning challenge. arXiv:1803.05457vl 2018.
[5] Steven K Esser, Jeffrey L McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dhar- mendra S Modha. Learned step size quantization. In International Conference on Learning Representations (ICLR), 2019.
[6] Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post- training quantization and pruning. Advances Neural Information Processing Systems, 35:4475-- 4488, 2022.
[7] Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. OPTQ: Accurate quantization for generative pre-trained Transformers. In The Eleventh International Conference on Learning Representations, 2023.
[8] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2020.
[9] Yushi Huang, Ruihao Gong, Jing Liu, Tianlong Chen, and Xianglong Liu. TFMQ-DM: Temporal feature maintenance quantization for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7362-7371, 2024.
[10] Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. Accurate post training quantization with small calibration sets. In International Conference on Machine Learning, pages 4466-4475. PMLR, 2021.
[11] Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. Mr. BiQ: Post-training non- uniform quantization based minimizing the reconstruction error. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 12329-12338, 2022.
[12] Yongkweon Jeon, Chungman Lee, and Ho-young Kim. GENIE: show me the data for quantiza- tion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12064-12073, 2023.
[13] Yongkweon Jeon, Chungman Lee, Kyungphil Park, and Ho-young Kim. A frustratingly easy post-training quantization scheme for 1lms. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 14446-14461, 2023.
[14] Sangil Jung, Changyong Son, Seohyung Lee, Jinwoo Son, Jae-Joon Han, Youngjun Kwak, Sung Ju Hwang, and Changkyu Choi. Learning to quantize deep networks by optimizing quantization intervals with task loss. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 4350-4359, 2019.
[15] Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squee7el.LMM: Dense-and-sparse quantization. arXiv:2306. 2306.06629, 2023.
11

---

[16] Yann LeCun, John S Denker, Sara A Solla, Richard E Howard, and Lawrence D Jackel. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), volume 2, pages 598-605, 1989.
[17] Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-Diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages
2023.
[18) Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu, Wei Wang, and Shi Gu. BRECQ: Pushing the limit of post-training quantization by block reconstruction. In International Conference on Learning Representations (ICLR), 2021.
[19] Lin, Jiaming Tang, Haotian Tang. Shang Yang. Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. AWQ: Activation-awan weight quantization for Ilm compression and acceleration. In MLSys, 2024.
[20] Yuexiao Ma, Huixia Li, Xiawu Zheng, Feng Ling, Xuefeng Xiao, Rui Wang, Shilei Wen, Fei Chao, and Rongrong Ji. AffineQuant: Affine transformation quantization for large language models. arXiv:2403. /2544, 2024.
[21] Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. 1993.
[22] Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, arXiv: 1609.07845 2016.
[23] Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. Up or down? Adaptive rounding for post-training quantization. In International Conference on Machine Learning (ICML), pages 7197-7206, 2020.
[24] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, (1))::5885555555 2020.
[25] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[26] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. BLOOM: A 1 176B-parameter open-access multilingual language model. arXiv:2211 122221..0 1.05 2022.
[27] Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. OmniQuant: Omnidirectionally calibrated quantiza- tion for large language models. ar0iv: 13137, 2023.
[28] Samuel Smith, Pieter-Jan Kindermans, Chris Ying. and Quoe Le. Don't decay the learning rate, increase the batch size. In International Conference on Learning Representations, 2018.
[29] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv:2302. 2023,
[30] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv::307. 000880 2023.
[31] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[32] Changyuan Wang, Ziwei Wang, Xiuwei Xu, Yansong Tang, Jie Zhou, and Jiwen Lu. Towards accurate post-training quantization for diffusion models. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pages 16026-16035, 2024.
12

---

[33] Guangxuan Xiao, Ji Lin. Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pages 38087-38099. PMLR, 2023.
[34] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a machine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791-48000 2019.
[35] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. OPT: Open pre-trained transformer language models. arXiv:2205.0106 2022.
13

---

Appendices
A Pseudo-code for the proposed aespa
In this appendix, we provide the pseudo-code for the proposed aespa excluded in the main text due to the page limitation.
Algorithm 1 Quantization
1: def QUANTIZATION W .X)
2: Approximate the Hessian H
See Table > Table
3: Estimate TK[K" K] BIQ" (Q"QQ Q) for We, Wx
the step size S mins (AWHAW")
5: repeat
Compute the Loss C
Table
7: Optimize S or Wint w.r.t C by certain algorithm
until converged return S and W
9:
step size and integer weight
Table 4: Row-wise Hessian H and quantization loss L for each layer
Layer
H
We
E
X
tr (E K K] AWHAW
WK
E
XX"
tr
Q" Q
AWHAW"
Wv
R
XATAX
( (WWHAWT)
tr
Others
[XX"]
(AWHAWT)
As mentioned, the proposed aespa consists of two main steps; aespa first determines the quantization parameters (i.e., scale and zero-point in 33) together with foldable parameters, in [19 13 27
20 (see line Algorithm|] l]. and then optimizes an integer weight Wint for each weight (see lines 5-8 in Algorithm [p. We emphasize that each iteration for the integer weight optimization can be performed efficiently based on pre-computed values (i.e., E X ] E| A AX " E[K K]. and E[QT Q] in Table 4). We also note that while we have used Z-FOLD in computing the quantization parameters and used AdaRound in optimizing integer weights, refined objectives in (77), (2I). and (22) can be integrated into any layer-wise quantization scheme without effort because we only used the definition of the attention operation in derivation.
14

---

B
Validity of the proposed separate quantization strategy
Table 5: Performance (PPL 4) of OPT-125M quantized with different strategies.
Quantization
Reconstruction
W2A16
W3A16
W4A16
Method
Grunularity
Target
Wiki-2
C4
Wiki-2
C4
Wiki-2
C4
AdaRound
Layer-wise
Layer Output
160.7
95.63
35.44
31.86
29.51
27.78
BRECQ
Block-wise
Attention Ourput
60.38
47.85
33.25
29.74
28.86
27.43
Proposed
Layer-wise
Attention Output
69.23
51.92
32.89
29.75
28.98
27.42
We conduct experiments to demonstrate the importance of targeting attention- wise reconstruction and the validity of the proposed separate quantization strategy. In our experiments, we learn weight- rounding policy using conventional AdaRound, but we set the loss function for each projection as the attention reconstruction error in 7 (not the layer reconstruction error; see Figure((c)).
Table summarizes the quantization performance of AdaRound, BRECQ, and approach the OPT-125M model. As evident, our approach uniformly outperforms AdaRound for all bit- widths, although both methods quantize models layer-wise. This because we can consider the cross-layer dependency (i.e., the relationship between the query, key, and value) by targeting attention- wise reconstruction, which differs from AdaRound wherein layers are considered independent. Furthermore, once we target attention- coset reconstruction, separate layer-wise quantization does not incur severe performance degradation compared the joint quantization method (BRECQ). Indeed, our approach causes only a marginal performance degradation for 2-bit and exhibits comparable performance for 3-bit and 4-bit.
One might wonder about the strategy of quantizing more than one layer jointly while maintaining remaining weights with full-precision, e.g. simultaneous quantization of the query and key projections while fixing the value projection with full-precision. To say the conclusion first, in this case, we cannot use the proposed pre-computation--ae -based loss computation strategy (see Section 3.4). resulting in a much longer quantization processing time. Specifically, when quantizing Wq and WK simultaneously, the attention reconstruction error expressed as
ASA K
||AQQ, V) SA(ASAAQQAUU K, V) [ F
=E [IAAVI]]]
where
QK'
QK"
AA softmax
softmax
vd
vd
Then, by taking similar steps in Section 3.3 (i.e., approximating AA with its first-order Taylor
series and constructing an upper bound of ASA
Q.k), we can obtain the following objective:
| QK
min AW,
E
AQK + QAKKT + AQAKT|
min
E
AW
min
E
x" AWWK" +QAWXX+X* XW
AW,.WW
Obviously, the objective becomes much more complex than the proposed ones in 15] and (16 and it would be difficult to simplify and accelerate the loss computation by exploiting pre-computed values as in aespa. In fact, without the proposed pre-computation based loss computation, the simultaneous quantization of WQ and WK requires 3.5 hours for the quantization of 1P5-125M, which is about 44 times longer than the proposed aespa and even 1.9 times longer than BRECQ
15

---

C Refined quantization objective (16) for the key projection
When quantizing the key projection WK. fix the query and value projections with full-precision. In this case, the attention reconstruction error ASAK can be expressed as
ASAK = E |SA[Q. K, V) - SA(Q, V) | = E [IAVI??]
where
QK
QKT vd
AA = softmax
softmax
Vd
To avoid the computational overhead of repetitive softmax operation, we approximate A AA with its first-order Taylor series, which leads
ASAK Y |leAkT softmax r v12 - - [ K XJT
softmax 11/
(25)
Furthermore, to reduce the huge memory cost required store the Jacobian softmax having 3 elements (see Footnote we establish an upper bound of (23) and then use it as surrogate of ASA K- Specifically, we separate the term [QQWWK XJ"
softmmx into two components as follows:
QAWKXJI softmax
2 = [IQAWKXI
J softmax F
| softmax quantization process, we minimize proposed objective in (16).
Noting that the term
is not affected by the quantization of W and thus fixed in the QQAWK X enforce ASAk be small, which leads the
F
16

---

D Effectiveness of the proposed Hessian in (18)
We recall from Section 3.4 that the proposed quantization objective for the value projection is
A AWWE XA AX" AW?).
which implies that the Hessian Hv for each of Wy is
Hv 2 A AX*).
We note that the proposed Hessian Hy differs from
H 2E X
which has been commonly used approximated Hessian in existing methods 6 The key reason for the difference is that we consider the dependency between the query, key, and value projections by targeting iitention-wise reconstruction, whereas the previous methods assumed independence.
S
To observe the effect of considering the cross-layer dependency, we different Hessians (i.e., Hv and H) when quantizing language models via Z.- Z.FOLD and then compare the performance of the quantized models. As T the quantization performance is much better when the proposed Hessian Hy is used, which demonstrates the importance of considering the cross-layer dependency.
Table 6: Quantization performance (PPL. 4) of Z.-OLD under different Hessians.
(a) WikiText-2
OPT
LLaMA
Hessian
Precision
T25M
350M
1.38
2.78
6.78
7B
13B
308
E X
X E X
d 7 L
39.59
25.97
16.10
13.54
11.65
6.756
5.708
4,931
INT3
AX
(ours)
35.05
24.81
16.25
13.40
11.43
6.529
5.669
4.693
EXX
6
3 13
190.1
102.5
33.97
27.10
18.07
14.93
13.03
9.250
INT2
EXA
AX
(ours)
146.4
68.30
31.43
25.17
17.92
14.20
13.15
8.138
(b) PTB
OPT
ILaMA
Hessian
Precision
125M
350M
1,38
2.78
6.7B
133
30B
EXX" 0 XAT
L6
53.08
39.23
22.73
20.18
16.64
11.73
10.09
8,979
INT3
X
T
(ours)
37.62
22.66
19.78
16.55
11.39
10.48
8.657
X
d
5 13
331.6
130.7
53,80
46.08
26.79
26.87
19.37
15.23
INT2
XAT
AXX
(ours)
212.8
100.1
53.64
42.93
26.09
24.88
18.01
12.99
(c) C4
OPT
LLaMA
Hessian
Precision
125M
350M
1.3B
2.7B
6.7B
13B
XXX"
6
1
33.67
26.45
17.33
15.50
13.28
8.719
7.554
6.912
INT3
XXATAA AX
(ours)
31.27
25.51
17.27
15.42
13.22
8.313
7.437
6.638
XX"
36
125.3
71.37
31.67
25.99
19.79
16.88
14.61
11.90
INT2
X
A
(ours)
112.6
56.48
30.06
25.34
19.32
16.87
13.46
10.32
17

---

E Proof of 119 and (21
Note that E || KAW |
E
|| vec (KAWQ Q 11 where vec(.) denotes the vectorization
operation. Then, by exploiting the following properties of Kronecker product
vec (ABC) (c" A vec B);
(A B) = A @ BT
(A & (C D) AC 8 BD,
we have
1
2
E KAW
X
E
X
& K
AwO
2
E Aw
X
K)
(x7
8 K) Awo
E Awy
X 2 k")
(XT @k) AwO
E L
XXT KT K)
AwO)
(26)
Aub E
XXT KTK]
AwQ.
which is the desired result in 19
We now prove 21
By combining
and (0), we have
E IAWW
X
F Awō E X T
8 [K(K]]
AwQ
Note that since E x and E K K] are symmetric, there exist Gx and GK such that
E E G G E = G G..
Then, by following the steps used to derive 26 in the reverse order, we have
E KAWo
Aw? (G & GK Awo
IGKAWoGXIEE
=u(GAAGGAGCAWWCC)
-u -u(GGG AWO- GXG, AW3)
(E K AWOE Xx" AW3).
which completes the proof of 21
18

---

Integration of proposed loss functions into existing PTQ schemes
We recall that we only utilized the definition of the attention operation when developing the proposed loss functions for the attention output reconstruction. Therefore, our loss functions can be integrated into any PTQ schemes based on layer- wise reconstruction and used to enhance their performance. In this section, we describe how to combine our loss functions with existing quantization schemes by taking AdaRound as an example.
In short, AdaRound learns weight-rounding mechanism by solving the following optimization
problem 23
arg min B
WX
W + A
-11).
(27)
where is the continuous variable to be learned, is the rectified sigmoid function, and W the soft-quantized weights defined as
(1
W +
+k(B).n.p)
W = s clamp
One can see that the loss function of AdaRound consists of two components, layer-wise reconstruction error and weight-rounding loss.
To consider the cross-layer dependency between WQ. WK. and Wv in learning process, we integrate the proposed loss functions developed for the attention output reconstruction into (7)) In
other words, we replace the layer-wise reconstruction error in (27) with our loss functions in (17 21
and (22). For example, when learning the rounding policy for the query projection matrix W Q. the objective of the proposed aespa is expressed as
arg min tr T K] AWQR Xx AW?) + + +xE(1- Bg
(1 (Q,..) -11"), (28)
6
where AWQ Wq WQ.
19

---

G Complexity analysis for conventional block-wise quantization schemes
Recall from (7) that conventional block-wise quantization schemes require compute SA(Q, K V) in each iteration. This means that for each input sequence, one needs to perform
forward pass for Q K, and V: 32d 1) flops
matrix multiplications for computing QK" and AV: 4d, L2 d₁ L - L flops
QK'
softmax operation with additional scaling (i.e., softmax
)) 3L + dn,L - L flops vd.
final computation of reconstruction error: 3d, flops
If B imput sequences are used in each quantization iteration, then the total number of flops required in conventional methods is
Cexist = dL + 4d, 21, L - O( max (d, Lt).
Comparison of Carypa and Cenist We now compare the complexities of aespa and conventional block-wise quantization methods in terms the number of flops. Table 7 summarizes the compu- tational costs required to quantize different sizes of OPT models. For conventional methods, report the cost of using four sequences in each iteration (B = 4). We observe that the computational cost of aespa is considerably lower than that of conventional methods. In particular, for small-scale models (e.g., OPT- 1,,M, OPT-350M, and OPT-1.3B), aespa performs ten times fewer number of flops. One can notice that the gap between Cuespa and Cerist decreases the model size increases. This is because the hidden size exceeds the sequence length L (which is fixed for all models) as the model size increases. Nevertheless, aespa still incurs lower computational cost, and the gap increases if conventional methods use larger batch sizes.
Table 7: Cost of aespa and conventional methods (GFLOPS)
125M
350M
1.3B
2.7B
6.7B
13B
Gexist
6.7
7.5
11
15
34
41
Caesp
0.24
0.42
1.6
3.2
13
20
20

---

H
Comparison with biock-wise PTQ schemes
We provide experimental results excluded from the main due page limitations.
Table 8: Performance (PPL 4) of the proposed aespa and conventional block wise PTQ methods.
(a) INT4 performance WikiText-2 and C4
LLaMA2
Dataset
Method
125M
L3B
2.7B
6.7B
7B
13B
30B
13B
HP16
14.63
5.677
4.101
4.884
18
28.86
14.83
12.71
0OM
0OM
OOM
0OM
OOM
OOM
WikiText-2
OmniQuunt
2M
30.42
15.15
12.89
11.20
5.907
5,256
4.263
5.850
5.064
AflineQuant
20
29.81
15.09
12.72
11.12
5,905
5.256
4,269
5.782
5.062
aespa
28.87
14.81
12.36
10.95
5.890
5.226
4.254
5.684
5.031
FP16
26.56
16.07
14.34
12.71
7.344
6.798
6.131
7.264
6.727
BRECO
18
27.43
16.42
14.61
OOM
OOM
OOM
OOM
OOM
OOM
C4
OmniQuant
16.68
7.656
6.269
AffineQuant
28.04
16.58
14.74
12.92
7.654
6.270
7.644
6.927
aespa
27.24
16.31
14.55
12.82
7.633
6.945
6.256
7.508
6.891
(b) Performance PTB
OPT
LLaMA
Precision
Method
125M
1.3B
6.7B
13B
FP16
Baseline
38.99
17.97
15.77
BRECQ
[
41.04
20.97
18.41
OOM
OOM
OOM
OOM
OmniQuant
27
42.34
21.32
18,70
16.04
10.57
9.330
8.354
INT4
AffineQuant
20
42.99
21.26
18.49
16.02
10.53
9.325
8.355
aespa
40.50
20.78
18.30
15.84
10.43
9.277
8.283
BRECQ
46.93
23.41
19.82
OOM
OOM
OOM
OOM
INT3
OmniQuant
56.88
25.11
22.59
18.33
11.98
10.24
9.065
Affineuuant
20
51.47
24.38
21.03
17.40
11.92
10.24
8.998
aespa
44.96
22.35
19.48
16.28
11.45
9.818
8.684
BRECQ
OOM
OmniQuant
NaN
377.9
2.0c3
7.7c3
33,51
NaN
17.38
INT2
AffineQuant 20
177.8
75.25
47.07
37,90
29.33
18.58
NaN
aespa
99.12
37.19
32.57
22,80
19.83
15.65
12.98
'Na'' means that loss diverges the quantization process.
'OMM' means our-of-memory issues occur when quantizing models with a single A100 GPU.
21

---

I
Comparison with layer-wise PTQ schemes
We provide experimental results excluded from the main due page limitations.
1.1
Results OPT models
Table 9; Performance (PPL L) of aespa and existing layer-wise PTQ methods on OPT models.
(a) WikiText-2
Precision
Method
125M
350M
1.3B
2.7B
6.7B
13B
30B
FP16
Baseline
27.65
22.00
14.63
12.47
10.86
10.13
9.56
RTN
37.28
25.94
48.20
16.92
12.10
11.32
10.98
OPTQ
32.49
23.68
15.50
12.85
11.12
10.33
9,670
INT4
Z-FOLD
31.03
23.08
15.00
12.47
11.01
10.21
9.537
aespa
28.87
22.55
14.81
12.36
10.95
10.18
9,511
RTN
1.3e3
64.57
1.3e4
1.6e4
5.8e3
3.4e3
1.6e3
OPTQ
52.95
33.29
20.36
16.94
13.01
11.65
10.44
INT3
Z-FOLD
39.59
25.97
16.10
13.54
11.65
10.62
9.902
aespa
32.71
24.45
15.79
13.14
11.23
10.52
9.760
RTN
5.5e3
2.8e4
1.1e5
9.5e3
2.8e4
1.9e5
1.7e5
OPTQ
4.1e3
1.1e4
8.3e3
9.3e3
2.0e3
539.8
56.63
INT2
Z-FOLD
190.1
102.5
33.97
27.10
18.07
33.48
13.48
aespa
71.18
54.89
24.26
22.22
15.71
15.27
11.91
(b) PTB
Precision
Method
125M
350M
1.3B
2.7B
6.7B
30B
FP16
Bascline
38.99
31.08
20.29
17.97
15.77
14.52
14.04
RTN
53.88
36.79
75.37
32,41
18.86
16.41
15.44
OPTQ
46.54
33.27
21.74
19.04
16.42
14.88
14.21
INT4
Z-FOLD
44.17
33.51
20.96
18.45
15.98
14.65
14.11
aespa
40.50
32.17
20.78
18.30
15.84
14.65
14.09
RTN
1.4e3
87.21
1.5e4
1.4e4
5.3e3
2.2e3
1.5e3
OPTQ
74.07
46.10
29.76
25.06
19.22
16.42
15.08
INT3
Z-FOLD
53.08
39.23
22.73
20.18
16.64
15.23
14.60
aespa
44.96
36.15
22.35
19.48
16.28
15.06
14.43
RTN
4.3e3
2.8e4
1.1e4
6.8e3
1.8c4
1.2e5
1.7e5
OPTQ
3.5e3
1.2e4
6.6e3
8.0e3
2.5e3
458.4
83.81
INT2
Z-FOLD
331.6
130.7
53.80
46.08
26.79
79.69
20.39
aespa
99.12
79.86
37.19
32.57
22.80
23.93
17.51
(c)
Precision
Method
125M
350M
1.3B
2.7B
6.7B
13B
30B
FP16
Baseline
26.56
22.59
16.07
14.34
12.71
12.06
11.44
RTN
33.88
26.21
27.50
18.83
14.37
13.32
13.55
OPTQ
29.64
24.15
16.75
14.86
13.00
12.24
11.56
INT4
Z-FOLD
28.92
23.71
16.38
14.60
12.85
12.14
11.49
aespa
27.24
23.15
16.31
14.55
12.82
12.13
11.47
RTN
834.4
55.15
6.6e3
1.2e4
5.0e3
2.8e3
1.8e3
OPTQ
42.88
30.60
20.53
17.66
14.61
13.19
12.15
INT3
Z-FOLD
33.67
26.45
17.33
15.50
13.28
12.45
11.73
aespa
29.51
24.96
17.10
15.27
13.15
12.39
11.68
RTN
3.7e3
1.6e4
7.7e3
7.7e3
1.4e4
9.7e4
5.8e4
OPTQ
2.1e3
4.4e3
3.0e3
3.7e3
290.9
157.7
29.40
INT2
Z-FOLD
125.3
71.37
31.67
25.98
19.79
47.10
14.51
aespa
56,88
46.36
23.54
22.53
17.28
16.30
13.32
22

---

1.2 Results on BLOOM models
Table 10: Performance (PPL. -) of aespa and existing layer-wise PTQ methods on BLOOM models.
(a) INT4 performance WikiText-2 and C4
WikiText-2
Precision
Method
560M
1.7B
38
1.18
7.1B
Baseline
22.42
15.39
13.48
22.05
17.49
15.20
RTN
25.82
19.98
16.96
14.75
12.09
29.80
24,42
21.24
18.75
16.05
OPTQ
23.83
18.74
16.16
14.01
11.72
27.74
23.05
20.26
18.00
15.54
INT4
Z-FoLD
23.60
18.44
15.87
13.90
11.59
27.36
22.66
20.00
17.87
15.42
aespa
23.21
18.28
15.76
13.81
11.56
27.20
22.49
19.86
17.76
15.38
(b) Performance on PTB
Precision
Method
560M
1.IB
1.7B
3B
7.1B
FP16
Baseline
43.69
57.96
30.00
25.34
20.83
50.96
66.79
33.52
27.65
22.40
OPTQ
46.83
31.63
26.72
21.52
INT4
Z-FOLD
45.77
31.26
26.27
21.35
aespa
44.73
60.41
31.05
26.01
21.17
RTN
124.8
184.0
105.5
66.24
34.94
OFTQ
64.43
82.91
40.27
33.13
25.94
INT3
Z-FOLD
53,01
69.93
35.12
28.41
22.83
aespa
RTN
7,4e5
1,1c6
2,5e5
1.2c5
2.2e5
OPTQ
4.1e3
2,4c3
1.4e3
1.4e3
428.4
INT2
Z-FOLD
194.9
174.9
74.03
69.49
38.50
aespa
91.14
120.7
57.48
46.40
31.28
23

---

1,3
Results on LLaMA models
Table
Performance (PPL. .) of aespa and existing layer-wise PTQ methods on LLaMA models.
WikiTex-22
FTB
Precision
Method
7B
13B
30B
13B
30B
7B
13B
30B
FP16
Baseline
5.677
5.091
4.101
10.12
9,081
8.159
7.344
6.798
6.131
RTN
6.291
5,525
4.536
11.25
9.775
8.653
8.121
7.232
6.537
OPTQ
6.167
5,365
4.452
11.51
9.526
8.426
7.792
7.082
6.399
INT4
Z-FOLD
6,069
5,278
11.45
9.335
8.410
7.797
6,984
6,318
aespa
5,226
4.254
9.277
8.283
6.256
RTN
25.61
11.78
14.87
98.89
28.94
28.79
30.86
14.46
30,04
OPTQ
8.290
6.729
5.705
16.11
11.91
9.964
10.51
8.832
7.977
INT3
Z-FOLD
6,756
5,708
4,931
11.73
10.09
8.979
8,719
7.554
6.912
aespa
6.579
5.611
4,688
11.45
9.818
8.684
8.465
7.399
6.634
RTN
1.1e5
5.7e4
2.7e4
9.9e4
8.1e4
3.3e4
1.1e5
5.9e4
2. 8e4
OPTQ
1.0c4
3.7e3
1.5e3
1.1e4
8.5e3
1.0c3
872.7
809.7
304.4
INT2
Z-FOLD
14.93
13.03
9.250
26.87
19.37
15.23
16.88
14.61
11.90
aespa
11.94
10.30
7.845
19.83
15.65
12.98
13.63
11.46
10.35
1.4
Results LLaMA2 models
Table 12: Performance (PPL 4) of aespa and existing layer-wise PTQ methods on LLaMA2 models.
WikiText-2
C4
Precision
Method
7B
FP16
Buseline
5.472
4.884
7.264
6.727
RTN
6.116
5,205
8.165
7.142
OPTQ
6,060
5.158
7,838
7.030
INT4
Z-FOLD
5.815
5.099
7.602
6.996
aespa
5.684
5.031
7.508
6.891
RTN
542.0
10.69
527.2
13.87
OPTQ
8.664
6.554
11.24
8.761
INT3
Z-FOLD
6.606
5,710
8.666
7.692
aespa
6.241
5.462
8.225
7.392
5.1e4
OPTQ
7,5e3
2.1c3
1,7e3
560.7
INT2
Z-FOLD
20.79
15.55
21.98
16.90
aespa
13.99
12.14
14.36
13.59
24

---

J Results for zero-shot tasks
We provide INT3 zero-shot performance results that are excluded from the main text due to page limitations.
Table 13: INT3 zero-shot performance (accuracy 1) of aespa and existing methods.
Model
Method
ARC-c
ARC-e
HellaSwag
MMLU
Average
FP16
44.62
72.85
76.18
32.19
56.46
RTN
27.47
45.45
45.46
24.94
35.83
OPTQ
36.95
62.63
68.33
25.51
48.36
LLaMA-7B
FOLD
66.92
72.50
52.38
OmniQuant
38.99
67.30
70.31
29.33
51.48
AflineQuant
20
39.25
65.61
70.56
29.68
51.28
aespa
40.87
69.15
71.54
30.57
53.03
FP16
47.87
74.75
79.08
43.46
61,29
RTN
36.09
56.23
62.03
26,20
45,14
OPTQ
43,00
67. 67.89
72.45
28.62
52.99
LLaMA-13B
Z- FOLD
1B
44.88
71.00
75.66
36.88
57.11
OmniQuant
2
44.03
69.70
75.15
35.89
56.19
AltineQuant
20
43.60
70.24
75.10
32.67
55,40
aespa
45.82
71.80
75.87
38.63
58.03
FP16
52,90
78,96
82.63
54.66
67.29
RIN
27.90
43.64
31.42
23.34
31.58
OPTQ
71.55
77.17
59.01
Z- FOLD
50.34
75.84
$1,00
64.22
OmniQuant
49.49
76.52
79.76
50.68
64.11
Affineuuant
20
49.66
77.10
79.49
50.37
64.16
aespa
50.34
77.53
79.79
50,55
64.55
FP16
74.49
59.63
RTN
25.94
35.48
35.39
23.14
29.99
OPTQ
8
37.46
63.01
64.85
28.79
48.53
L.aMA2-78
Z-FoLD
1K]
40.10
64.65
69.92
33.69
52.09
DuniQuant
40.36
67.30
71.00
31.26
52.48
AffineQuant
40
40.78
67.21
70.75
30.93
52.42
aespa
69.11
55.11
FP16
49.06
77.44
79.39
52.10
64,50
RTN
$5.98
43.86
OPTQ
43,09
70.45
72.02
39.37
56.23
1LaMA2-138
Z-FOLD
1k
46.42
72.77
74.79
47.91
60.47
OmniQuant
27
45.65
74.33
74.77
43.92
59.67
AffineQuant
20
47.18
75.42
75.28
45.61
60.87
aespa
46.84
75.25
75.78
47.09
61.24
25

---

K
Time and memory cost comparison
Table 14: Time and memory cost of aespa and existing methods
(a) INT2 quantization processing time
OPT
LLaMA
Target
Method
125M
1.3
2.7B
6.7B
7B
138
30B
layer-wise
OPTQ
0.66 min
0.08 hr
0.14 hr
0.29 hr
0.25
0.45
1.08
reconstruction
Z- FOLD
R
1.09 min
0.27 hr
0.61 hr
2.58 hr
1.13 hr
2.48 hr
10.51 hr
BRECQ
108.2 min
10.71
19.15
OOM
0OM
OOM
OOM
attention-wise
OmniQuant
16.20 min
1.02 hr
1.63 hr
2.93 hr
2.37
4.20
9.84
reconstruction
Afaineuuant
min
2.57 hr
hr
9.85
10.09 hr
18.76
hr
aespa
4.78 min
hr
2.83 hr
10.24
6.84
15.89 hr
$3.69 hr
(h) Memory (GB)
OPT
LLaMA
Target
Method
125M
1.3B
2.7B
6.7B
7B
13B
30B
layer-wise
OPTQ 7
1.39
4.49
6.43
13.07
8.76
12.34
18.59
reconstruction
Z-FOLD
1
1.39
4.49
6.43
13.07
8.76
12.34
18.59
BRECQ
18
3.39
16.60
27.79
OOM
OOM
0OM
OOM
attention-wiss wise
Omni Quant
27
1.94
5.87
7.09
11.68
12.61
17.02
24.53
reconstruction
AffineQuant
20
3.47
9.96
12.25
20.08
24.28
27.10
38.59
aespa
1.68
5.47
6.84
12.26
21.69
29.27
43.00
'OOM' that out-of-memory occur when quantizing models with single NVIDIA A100 GPU.
Table 14 summarizes the processing time and memory cost of different quantization algorithms. We note that the processing time of the proposed aespa includes the time required for pre-computations (lines 2-4 in Algorithm D.
As expected, aespa completes quantization much faster than BRECQ. For example, while BRECQ requires than 10 hours to quantize OPT-1.3B, aespa completes quantization in 1.24 hours, which demonstrates the effectiveness of the proposed objectives and pre-computation-beee based loss computation strategy. Although other block-wiss wise PTQ methods quantization faster than aespa for hyper-scale models, they suffer from unstable training process exhibit poor PPL performance (e.g., PPL of OmniQuant is larger than 10 for OPT-6.7B; see Table []. We also observe that OPTQ performs quantization very fast, but its PPL performance collapses completely regardless of the model size (see Table (9). Except aespa, Z-FOLD is the only method that shows both reasonable performance and processing time.
perform
In real situations, when one needs to preserve the performance of the original model as much as possible, the proposed aespa would an intriguing solution. In particular, when deploying LLMs on coource-constrained platforms where up 7B models commonly employed (e.g.. mobile devices), aespa would be a good fit. Even when fast quantization of hyper-scale models is needed, aespa can be used with slight modification. Specifically, in limited cases, can skip weight-rounding optimization (lines 5-8 in Algorithm [ and simply perform the quantization parameter computation (line 4 in Algorithm I using the proposed Hessian that considers the cross-layer dependency (see 18)). In doing so, we can not only save the time required to perform weight-rounding learning, but also save the memory required store pre-computed values (E K] and E[ Indeed, when performing only quantization parameter computation, we achieved a significant reduction in the processing time (see Table beloow) while still exhibiting better performance than conventional methods (see Table 6 in Appendix D
Table 15: INT2 quantization processing time of aespa without weight- optimization
OPT
LLaMA
125M
L.3
2.7B
6.7B
13B
30B
1.29 min
0.35
0.74
2.92 hr
1.47
3.26
12.50
26

---

L Experimental results for different calibration datasets
One might wonder why the PPL performances of OmniQuant summarized in Table are much worse than those reported in the original paper [27]: INT2 PPL. performances of quantized LLaMA models are 18.18, NaN, and 10.15 for WikiText-2 in Table which are worse than the values (15.47, 13.21, and 8.71) reported in [27] This is because we used a different calibration dataset for quantization. Specifically,, we used C4 when constructing a calibration dataset, while (2 used WikiText-2.
Additionally, we evaluate the performance the proposed aespa using WikiText-2 as a calibration dataset. From Table [6 observe that when calibration data sampled from WikiText-2, results for OmniQuant are comparable with those reported in the original paper [27). While it has been reported that the performance variance of OmniQuant across different calibration datasets is low for INT3 and INT4 (see [27] Table A10)), such low variance does not hold for INT2. Furthermore, we observe that the proposed aespa outperforms OmniQuant regardless of the type of the calibration dataset.
Table 16: INT2 performances (PPL 1) aespu and OmniQuant for different calibration datasets
Calibration
LLaMA
Method
Dataset
7B
13B
30B
OmniQuant
18.18
NaN
10.15
C4
aespa
11.94
10.30
7.845
OmniQuant
15.59
13.76
9.230
WikiText-2
aespa
8.818
7.423
6.232
Test dataset: WikiText-2
M Quantization performance of aespa for high bit-widths
While previous results demonstrate that the proposed aespa is very competitive for low-bit quanti- zation (e.g., INT2 and INT3), one might wonder whether aespa can preserve the performance of the original full-precision model at high bit-widths. We thus evaluate INTT and INT6 quantization performances of aespa with ILLMMA models. From Table 17 observe that aespa almost preserves the performance of the original full-precision model for the INT6 quantization. Even for the INT4 quantization, the performance degradation is very marginal (e.g., less than 1 1% degradation for 13B and 30B models).
Table 17: INT4 and INT6 quantization performances of the proposed aespa (calibration data: C4)
Perplexity (L)
Zero-shot Accuracy (t)
Model
Precision
Wiki-2
C4
ARC-c
ARC-e
HellaSwag
MMLU
Average
FP16
5.677
7.344
44.62
72.85
76.18
32.19
56.46
LLaMA-7B
INT4
5.896
7.602
43.77
71.51
74.90
31.33
55.38
INT6
5.694
7.360
44.62
72.35
75.96
32.27
56.30
FP16
5.091
6.798
47.87
74.75
79.08
43.46
61.29
LLaMA-13B
INT4
5.232
6.938
47.53
73.74
78.35
43.49
60.78
INT6
5.096
6.809
48.04
74.96
78.98
43.24
61.31
FP16
4.101
6.131
52,90
78.96
82.63
54.66
67.29
LLAMA-30B
INT4
4.260
6.254
$2.99
78.16
82.28
53.62
66.76
INT6
4.110
6.139
53.07
78.96
82.60
54.61
67.31
27

---

Experimental results for different seeds
We recall that when constructing calibration dataset, we randomly draw 128 sequences from the C4 dataset [24] By changing the seed for the sampling, different calibration datasets can be constructed, which leads to different quantization results. In this appendix, we report the corresponding results and overall statistics.
Table 18: Quantization performance (PPL L) of aespa on OPT models for different seeds.
(a) WikiText-2
Precision
Seed
125M
350M
2.7B
6.7B
28.87
22.55
14.81
12.36
10.95
INT4
10
28.60
22.55
14.91
12.31
10.83
100
28:75
22.85
14.94
12.35
10.90
32.71
24.45
15.79
13.14
11.23
INT3
10
32.95
24.57
16.10
13.21
11.11
100
33.38
24.45
15.70
13.27
11.24
0
71.18
54.89
24.26
22.22
15.71
INT2
10
74.41
50.84
24.38
22.36
15.06
100
77.03
53.12
25.93
22.39
15.66
(b) PTB
Precision
Seed
125M
350M
2.7B
6.7B
40.50
32.17
20.78
18.30
15.84
INT4
10
40.62
32.33
20.56
18.21
15.91
100
40.11
32.60
20.55
18.20
15.86
0
44.96
36.15
22.35
19.48
16.28
INT3
10
46.26
36.19
22.06
19.46
16.32
100
47.54
35.61
22.10
19.66
16.39
0
99.12
79.86
37.19
32.57
22.80
INT2
10
110.0
73.98
35.94
32.25
21.51
100
106.0
79.09
37.33
31.90
21.86
(c) C4
Precision
Seed
125M
350M
1,3B
2,7B
6.7B
27.24
23.15
16.31
14.55
12.82
INT4
10
27.23
23.13
16.32
14.54
12.81
100
27.29
23.15
16.34
14.54
12.81
0
29.51
24.96
17.10
15.27
13.15
INT3
10
29.59
24.98
17.06
15:29
13.15
100
29.58
25.00 25.00
17.09
15.37
13.15
0
56.88
46.36
23.54
22.53
17.28
INT2
10
56.23
44.02
23.91
22.56
16.91
100
56.78
45.21
24.41
22.42
17.30
Table 19: Quantization performance statistics (PPL 1) of aespa OPT models.
Precision
Dataset
125M
350M
1.3B
2.7B
6.7B
Wiki-2
28.74 + 0.139
22.65 + 0.172
14.89 + 0.066
12.34 + 0.023
10.89 : 0.058
INT4
PTB
40.41 + 0.264
32.36 + 0.217
20.63 + 0.128
18.24 + 0.057
15.87 +: 0.034
C4
27.25 +: 0.036
23.14 + 0.014
16.33 + 0.016
14.55 + 0.005
12.81 +: 0.002
Wiki-2
33.01 0.340
24.49 = 0.068
15.87 - 0.209
13.21 = 0.064
11.19 + 0.068
INT3
PTB
46.26 = 1.287
35.98 = 0.321
22.17 0.159
19.54 + 0.109
16.33 0.058
29.56 0.043
24.98 + 0.024
17.08 + 0.021
15.31 + 0.050
13.15 0.004
Wiki-2
74.20 2.931
52.95 + 2.029
24.86 0.930
22.32 + 0.088
15.48 0.363
INT2
PTB
105.0 5.495
77.64 3.195
36.82 0.766
32.24 + 0.335
22.06 0.667
C4
56.63 0.350
45.20 1 1.171
23.95 0.438
22.50 1 0.076
17.17 0.219
28

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The abstract and introduction clearly state our main contribution (proposal of a next-level quantization algorithms that balance accuracy and efficiency for the quantization of hyper-scale Transformer models).
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We have discussed the limitations together with some future research directions in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
29

---

Answer: [Yes]
Justification: We have included detailed derivations and complete proofs of the proposed
quantization loss functions in Sections 3.3 3.4 3.5 and Appendices C
G
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: We have included detailed instructions, e.g., pseudo-code for the proposed algorithm (see Algorithm D. hyperparameter settings, and stopping criterion (see Section to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
30

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The code will be available at httttts://gittuu.comm/ htt.a//w...coom///sssnnaaaaaaaap
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.ca
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (ht
wttp://biiiiiioooooooooooooooooo for more details. / htcc//ppblccggiieeeeooeeeeeeeeee
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have specified all the training details including hyperparameter settings and stopping criterion in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We have reported the results accompanied by statistics (e.g., mean and standard deviation) in Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
31

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have mentioned that we used single NVIDIA A100 GPU (80 GB) in our experiments (see the last sentence in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics httpps///eerrrss..cco/ooooccttee
Answer: [Yes]
Justification: Our research have been conducted the paper conform, every respect, with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have discussed potential positive societal impacts our work in Section Section 4 and Section 5) As evident from our experiments, the proposed algorithm can serve as useful quantization solution that pursues both efficiency and accuracy when deploying LLMs cosstrce- devices.
32

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considcrations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning conrectly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses risks for misuse (e.g., pretrained language models, image generators, or scraped datasets).
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-uss should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have cited the original paper and the codes provided by the authors of the original paper as Footnote
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
33

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paper swi thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
34

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
35


Reviews:
[{"review_id": "tzhyShH9dJ", "reviewer": "Reviewer_vuzn", "summary": "This paper presents a post-training quantization approach for transformers. The proposed algorithm finds a better balance between accuracy and efficiency. The method is to perform quantization in a layer-wise way for efficiency and the optimization objective is to maintain the reconstruction of the quantized model to obtain good accuracy. Experimental results on 3 datasets over 2 transformers (OPT and LLaMA) are reported at the low bit widths (3 Bits and 2 Bits). The proposed approach outperforms other baselines at the low bit widths.", "questions": "- The idea of reducing reconstruction error for quantization is not new - several prior works have already proposed this idea and adopted it in their works. The optimization method (iterative optimization - updating one variable and fixing others) is not new. \n\n- Actually, this paper only considers the reconstruction of the current quantization layer, which is sub-optimal. A better way is to optimize the reconstruction of the last layer (the reconstruction of the whole network). Because output errors could accumulate, small output errors at the first layers may change to large ones when going to the last layers. Only the reconstruction of the last layer's output can reflect the impact on the network.\n\n- This paper only reports results at low bit widths (4, 3, and 2 Bits). Although it outperforms the three baselines (BRECQ, OmniQuant, AffineQuant), the accuracy loss is noticeable at low bit widths. How about the results at higher bit widths (6 bits or 8 bits)? Is that possible to maintain the accuracy at higher bit widths? People want to see quantization to a certain number of bits without losing accuracy.\n\n- One important contribution claimed by this paper is the capability to handle situations where frequent model updates and multiple hyperparameter tunings are required. However, we haven't found practical results about the analysis and comparison of such situations. Although the authors gave the time complexity of the algorithm and provided the time to run the algorithm, I think this is different with the real situation when weights are frequently updated. Moreover, some baselines also show faster speed (OPTQ, Z-FOLD) than the proposed method.\n\n- I suggest that the authors cite the latest related post-training quantization works for generative models or diffusion models [1][2][3][4][5], discuss the differences with them, and compare with them if possible.  \n\n[1] Towards Accurate Post-training Quantization for Diffusion Models\n\n[2] TFMQ-DM: Temporal Feature Maintenance Quantization for Diffusion Models\n\n[3] Post-training Quantization on Diffusion Models\n\n[4] Q-Diffusion: Quantizing Diffusion Models\n\n[5] PTQD: Accurate Post-Training Quantization for Diffusion Models", "limitations": "N.A.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- This paper is well written and organized in general.\n\n- The mathematical formulation and derivation are clear and easy to follow.\n\n- Results on low bit widths outperform several baselines including BRECQ, OmniQuant, and AffineQuant.\n\n- Experimental results on two sota LLMs (OPT and LLaMA) are reported.", "weaknesses": "- The novelty in this paper is limited, as similar formulation and optimization methods have been widely used in prior papers. \n\n- Experiments are not solid. Some important results and comparisons are missing.\n\n- The paper lacks analysis and results to support the claims of the contributions.\n\nPlease check the questions section for details.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's invaluable comments.\n\n**1. Novelty (Question 1)**\n\n - Our contribution does NOT lie in the idea of reducing reconstruction error. Our primary contribution is to mitigate the computational overhead of the conventional block-wise reconstruction error minimization method (BRECQ) through a novel quantization strategy coupled with a time- and memory-efficient error computation method. Please take a look at our global response for this comment.\n\n**2. Reconstruction of the last layer's output (Question 2)**\n\n - While network-wise reconstruction can prevent the error accumulation, inference of the whole network is needed for every input in each quantization step to measure the reconstruction error of the last layer's output. Obviously, this process is time-consuming and not feasible for LLMs with billions of parameters. Indeed, it has been reported that network-wise reconstruction takes 30 minutes even for ResNet18 having 11M parameters [6], while our method quantizes the 125M model in 5 minutes (see Table 11(a) in Appendix K).\n\n - Moreover, multiple GPUs are indispensable for network-wise reconstruction because forward and backward pass information of all layers in the whole network needs to be saved for backpropagation. Even for block-wise reconstruction (BRECQ) that considers a single Transformer block, we encountered out-of-memory (OOM) issue when quantizing 13B models with a single GPU (see Table 11(b)). Whereas, our method conducts quantization in a layer-wise manner (yet still targeting the reconstruction of the attention output), and thus backpropagation information of only one layer needs to be stored at a time, which facilitates the quantization of 30B models using a single GPU.\n\n - We note that such overhead of the block- and network-wise reconstruction can be mitigated via recent approaches that learn only a small number of quantization parameters, instead of optimizing a weight-rounding policy (e.g., OmniQuant and AffineQuant). However, due to the gradient approximation involved in the quantization parameter learning, they often suffer from an unstable low-bit performance and perform worse than our method (see Table 1).\n\n - In the final version, we will discuss why we focus on the output of the attention module, not the output of the whole network. We hope for the reviewer's kind evaluation on our effort to balance accuracy and quantization efficiency.\n\n   [6] C. Wang et. al., \"Leveraging inter-layer dependency for post-training quantization,\" NeurIPS 2022.\n\n**3. Results for higher bit-widths (Question 3)**\n\n -  In Table II (see the PDF attached to the global response), we summarize INT4 and INT6 quantization performances of the proposed method. We observe that our method almost preserves the performance of the original full-precision model for the INT6 quantization. Even for the INT4 quantization, the performance degradation is very marginal (e.g., less than 1% degradation for 13B and 30B models). \n\n**4. Why the proposed method is suitable when models are updated frequently (Question 4)**\n\n - In real situations, the deployment process of LLMs usually consists of three steps: training, quantization, and evaluation on target devices such as mobile. From the perspective of quantization, it is important to 1) preserve the performance of original full-precision models as much as possible and 2) complete the quantization in a reasonable time to frequently verify whether updated pre-trained models satisfy the target performance on target devices.\n\n - In this context, the conventional BRECQ would not be a practical solution because it requires too much processing time (e.g., almost 20 hours for OPT-2.7B; see Table 11(a)) and needs multiple searches of hyperparameters (e.g., learning rate, rounding loss weight ($\\lambda$ in Eq. (27)), and calibration dataset choice) without which BRECQ performs worse than our method (see Table 1).\n\n - While other methods (OPTQ, Z-Fold, OmniQuant, and AffineQuant) are faster, they suffer from unstable or unsatisfactory low-bit performance (see Tables 1 and 7). We emphasize that our method would be an intriguing solution in real situations where the performance of original models needs to be preserved as much as possible (see Tables 1 and 7). Even when faster quantization is needed, our method can be efficiently used by skipping the weight-rounding optimization step (lines 5-8 in Algorithm 1) and just performing the quantization parameter computation step (line 4 in Algorithm 1). In doing so, we can greatly reduce the processing time and memory costs required for the quantization (see Table 12), yet the proposed method still exhibits better performance than conventional approaches (see Table 4).\n\n**5. Comparison with PTQ methods for diffusion (Question 5)**\n\n - Thank you for pointing out interesting references. In contrast to our approach that focuses on mitigating the computational overhead of the conventional BRECQ, all the recommended methods have tried to incorporate diffusion-specific quantization strategies into BRECQ to overcome output distribution discrepancies over different time steps (e.g., grouping of time-steps with similar distributions [1], temporal feature preservation [2], separate quantization for shortcuts in U-Net [4], and time step-aware mixed-precision [5]). In the final version, we will cite the recommended works and discuss the differences.\n\n - We recall that we have not assumed any specific network architecture and just used the definition of the attention operation when developing our quantization objectives (see Section 3). Thus, we believe that the proposed method can be integrated with those diffusion-specific strategies without any modification, thereby accelerating the quantization processing time of diffusion models. Because the quantization of diffusion models is not our area of expertise, we leave this integration as a future work and mention it in the conclusion and limitation section.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "I3cIx7DQen", "reviewer": "Reviewer_Juv9", "summary": "This paper propose a new quantization strategy that balances accuracy and efficiency that reconstruct the attention output to consider cross-layer dependency. To accelerate the quantization process, this paper proposes refined quantization objectives for the attention module. This approach gets better results in low-bit precision.", "questions": "1 In Figure 2, what is Layer, what is Block?\n\n2 Following the method in paper, will the training time longer than ordinary PTQ?", "limitations": "I did not identify any issues related to limitations and potential negative societal impact.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 This article does a lot of experiments, which are also in the appendix section.\n\n2 This paper obtains higher accuracy at ultra-low bit than any other work.", "weaknesses": "1 Formula derivation is too redundant.\n\n2 Paper does not make clear the trade-off between accuracy and efficiency in introduction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's positive evaluation and invaluable comments on our work.\nOur point-to-point responses are as follows.\n\n**1. Redundant formula derivation (Weakness 1)**\n\n - We appreciate the reviewer's comment. In the final version, we will simplify the technical derivations and instead enrich the experiment section by including more results in the main text.\n\n**2. In Figure 2, what is \"Layer\" and what is \"Block\"? (Question 1)**\n\n - Figure 2 is a simplified illustration of our quantization strategy that performs quantization in a layer-wise manner while targeting block-wise reconstruction. In the context of Transformers, \"block\" means the attention module, and \"layer'' denotes each of layers inside the attention module (i.e., query, key, and value projections). In the final version, we will add this explanation to the caption of Figure 2 for a better understanding of our quantization strategy.\n\n**3. Following the method in paper, will the training time be longer than ordinary PTQ? Paper does not make clear the trade-off between accuracy and efficiency in introduction (Weakness 2 and Question 2)**\n\n - We appreciate the reviewer's comment. The processing times of the proposed method and conventional PTQ approaches have been reported in Appendix K (see Table 11(a)). The proposed method completes quantization much faster than the conventional BRECQ which we aim to simplify in this work; e.g., the proposed method completes quantization of OPT-1.3B in 1.24 hours while BRECQ requires more than 10 hours.\n\n - While other methods proposed for the quantization of LLMs (OPTQ, Z-Fold, OmniQuant, and AffineQuant) are faster than the proposed method, they suffer from unstable or unsatisfactory low-bit performance (see Table 1 and Table 7). In the final version, we will elaborate on the trade-off between low-bit quantization performance and processing time in the introduction.\n\n - Finally, we note that the proposed method would be an intriguing solution in real situations where the performance of original full-precision models needs to be preserved as much as possible (see Table 1 and Table 7). Even when faster quantization is needed, the proposed method can be efficiently used by skipping the weight-rounding optimization step (lines 5-8 in Algorithm 1) and just performing the quantization parameter computation step (line 4 in Algorithm 1). In doing so, we can greatly reduce the processing time and memory costs required for the quantization (see Table 12), yet the proposed method still exhibits better performance than conventional approaches (see Table 4). In the final version, we will incorporate this discussion into the main text.\n\nAgain, we express our sincere thanks to the reviewer for the acknowledgment of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gtK2zHXec3", "reviewer": "Reviewer_E1kR", "summary": "As a cost-effective alternative, learning-free PTQ schemes have been proposed for LLMs. Still, the performance is somewhat limited because they cannot consider inter-layer dependency within the attention module, a significant feature of Transformers. This paper propose a PTQ algorithm that balances accuracy and efficiency. The key idea of the proposed algorithm called aespa is to perform quantization layer-wise for efficiency while targeting attention-wise reconstruction to consider the cross-layer dependency. Through extensive  experiments on various language models and complexity analysis, it demonstrates that aespa is accurate and efficient in quantizing Transformer models.", "questions": "See the weakness.", "limitations": "N/A", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The scheme aims to reconstruct the attention output to consider cross-layer dependency while quantizing models layer-wise to pursue efficiency. It is different from previous layer-wise error minimization. \n\nFrom extensive experiments on language models, it demonstrates that the approach outperforms conventional schemes by a significant margin, particularly for low-bit precision (INT2).  \n\nTo accelerate the quantization process, it proposes refined quantization objectives for the attention module. Through a complexity analysis, it demonstrates that about 10 times faster quantization than existing block-wise approaches can be achieved by exploiting the proposed objectives.", "weaknesses": "I am a bit confused about the results of baselines. For OmniQuant, in the original paper, the perplexity on wikitext2 for 2bit weight quantization on llama-7B, 13B, and 30B are 15.47, 13.21, 8.71, respectively. However, in this paper, they become 24.28, 22.85,13.04. The differences are too large. Though the authors mention that the difference is due to different calibration dataset, the ablation study in OmniQuant demonstrates that different calibration datasets (wikitext2, C4, and pile) do not lead to significant perplexity difference with very small variance, see table A10 of OmniQuant.  So the performance of OmniQuant using another calibration dataset such as C4, should also achieve a similar performance close to the original paper. However, the results in this paper are significantly different from the original paper. It is better to provide more discussions about this problem.\n\n\nThe OmniQuant and AffineQuant methods demonstrate the performance of quantization in groups. Smaller groupsize leads to better performance. It is better to include some ablation study with different groupsizes. \n\n\nLLaMA3 is published. It is better to have some results for recent models such as LLaMA2.\n\n\nMost of the results are shown in the appendix. It is better to organize the experimental part better to include the main results in the main paper.  Currently there are only block-wise perplexity comparison in the main paper. It may be better to put zero-shot evaluation in the main paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's invaluable comments and constructive suggestions.\nOur point-to-point responses are as follows.\n\n**1. Results for OmniQuant (Weakness 1)**\n\n -  We appreciate the reviewer's invaluable comment and careful reading. We mention that we used the official GitHub code without any modification when evaluating the performance of OmniQuant.\n\n -  To check the performance variance of OmniQuant across different calibration datasets, we have changed a calibration dataset (i.e. C4 $\\rightarrow$ Wikitext-2 as in the original paper) and measured the performance of OmniQuant. As Table IV (see the PDF attached to the global response) shows, the low variance of OmniQuant does NOT hold for INT2 weight quantization. In fact, the low variance has NOT been verified for INT2 weight quantization in the original paper and has been shown only for INT3 and INT4 (see [1, Table A10]).\n\n - Furthermore, we observe from Table IV that regardless of the type of the calibration dataset, the proposed method outperforms OmniQuant, which demonstrates the superiority of the proposed method. In the final version, we will incorporate this result (Table IV) and discuss the performance variance issue.\n\n  [1] W. Shao et. al., \"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models,\" ICLR 2024.\n\n**2. Group-wise quantization (Weakness 2)**\n\n - We appreciate the reviewer's constructive suggestion. In this work, we did not consider group-wise quantization because it requires more memory costs and processing time for the inference than per-channel quantization [2, Tables 3 and 4].\n\n - As suggested by the reviewer, we have newly evaluated the group-wise quantization performance of the proposed method (see Table III in the attached PDF). From the results, we observe that regardless of quantization algorithms, smaller groupsize leads to better performance, as mentioned by the reviewer. As evident, the proposed method outperforms conventional methods also for group-wise quantization.\n\n - In the final version, we will add this result to demonstrate the group-wise quantization performance of the proposed method. \n\n  [2] H. Shen et. al., \"Efficient LLM Inference on CPUs,\" arXiv 2311.00502.\n\n**3. Results for recent models such as LLaMA2 (Weakness 3)**\n\n - We appreciate the reviewer's constructive suggestion. Table I (in the attached PDF) summarizes the performances of the proposed method and conventional approaches on LLaMA2 models. As evident, the proposed method outperforms conventional approaches with respect to both perplexity and zero-shot accuracy performance. We will add this result to the main text in the final version.\n\n**4. Rearrangement of contents: most of the results are shown in the appendix. It is better to organize the experimental part better to include the main results in the main paper. Currently, there are only block-wise perplexity comparisons in the main paper. It may be better to put zero-shot evaluation in the main paper. (Weakness 4)**\n\n - We appreciate the reviewer's constructive suggestion. In the final version, we will enrich the experiment section by simplifying the technical derivations and including experimental results, such as zero-shot evaluation results, results on LLaMA2 models, and group-wise quantization results, in the main text.\n\nAgain, we express our sincere thanks to the reviewer for the invaluable comments and constructive suggestions, which will greatly assist in enhancing the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8wEtj0HdGQ", "reviewer": "Reviewer_tsw3", "summary": "The paper introduces 'aespa,' a novel post-training quantization (PTQ) algorithm designed to balance accuracy and efficiency in quantizing hyper-scale Transformer models. This method quantizes layer-wise for efficiency and employs attention-wise reconstruction to account for cross-layer dependencies. Extensive experiments and complexity analysis demonstrate that aespa outperforms existing PTQ methods, especially in low-bit quantization scenarios.", "questions": "please refer to Weakness.", "limitations": "A more in-depth discussion on implementation details would be beneficial.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Aespa, which combines layer-wise quantization with attention-wise reconstruction, is a significant advancement in the field of model quantization.\n2. This paper emphasizes a pre-computation-based loss computation strategy, demonstrating its significant speed advantage over traditional block-wise PTQ methods like BRECQ.\n3. Extensive experiments on diverse language models and datasets provide a comprehensive evaluation of aespa's performance.\n4. Enhancing the credibility of this method would benefit from conducting thorough ablation studies.", "weaknesses": "1. Further experiments on LLMs are crucial, particularly with the LlaMa 2 & 3 family.\n2. How were other layers updated within a transformer block, apart from the attention module? Were they done layer-wise or block-wise?\n3. The implementation details require clarification. Specifically, in the attention module, Q, K, and V are updated separately. How many iterations are necessary for each layer? Does the order of updating Q, K, and V impact performance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's acknowledgment of our work and constructive suggestions. Our point-to-point responses are as follows.\n\n**1. Further experiments on LLMs such as LLaMA2 and LLaMA3 (Weakness 1)**\n\n - We appreciate the reviewer's constructive suggestion. Table I (see the PDF attached to the global response) summarizes the performances of the proposed method and conventional approaches on LLaMA2 models. As evident, the proposed method outperforms conventional approaches with respect to both perplexity and zero-shot accuracy performance. We will add this result to the main text in the final version.\n\n - It requires more effort to measure the quantization performance on LLaMA3 models because LLaMA3 models adopted a different type of attention operation (grouped query attention (GQA) instead of the standard multi-head attention) and most of the conventional algorithms used for comparison do not support LLaMA3 models yet. For LLaMA3 models, we will try to update the results in the final version.\n\n**2. How were other layers updated within a transformer block, apart from the attention module? Were they done layer-wise or block-wise? (Weakness 2)**\n\n - As mentioned in Footnote 5 and Table 2 (see Appendix A), other layers apart from the attention module are quantized with the layer-wise objective in Eq. (6). In the final version, we will incorporate this explanation into the main text.\n\n - It is worth noting that if we use the reconstruction error for the entire Transformer block (rather than the attention module), then other layers apart from the attention module could also be quantized by targeting the block-wise reconstruction, which would lead to further enhancement. However, due to nonlinear activation functions (e.g., SiLU for LLaMA), normalization layers, and weights of higher dimensions, such a method would result in complex objective functions, significantly more complicated than those developed for the attention module (see Eqs. (13) and (25)). In the final version, we will discuss this trade-off between accuracy and efficiency.\n\n**3. The implementation details require clarification. Specifically, in the attention module, Q, K, and V are updated separately. How many iterations are necessary for each layer? Does the order of updating Q, K, and V impact performance? (Weakness 3)**\n\n - We appreciate the reviewer's comment. In our experiments, we set the number of iterations to 2,000 for all layers, as reported in Section 4.1.\n\n - The performance of the proposed method is NOT affected by the quantization order of $\\mathbf{Q}$, $\\mathbf{K}$, and $\\mathbf{V}$. Before quantizing each Transformer block, we retain all layers inside the Transformer block in full-precision and then compute the values required for an efficient weight-rounding optimization (i.e., $\\mathbb{E} [\\mathbf{X} \\mathbf{X} ^{T}]$ for each layer, $\\mathbb{E} [\\mathbf{K} ^{T} \\mathbf{K}]$ for the query projection, $\\mathbb{E}[\\mathbf{Q} ^{T} \\mathbf{Q}]$ for the key projection, and $\\mathbb{E}[\\mathbf{X} \\mathbf{A} ^{T} \\mathbf{A} \\mathbf{X} ^{T}]$ for the value projection; see Table 2 in Appendix A). Because the quantization of each layer depends only on these pre-computed values obtained from the full-precision model, each layer is NOT affected by the quantization of other layers. In the final version, we will clarify this issue in the experimental setup section (Section 4.1).\n\nAgain, we express our sincere thanks to the reviewer for the invaluable comments and constructive suggestions, which will greatly assist in improving our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MBfo5hSMtv", "reviewer": "Reviewer_QBHg", "summary": "This manuscript proposed a layer-wise post-training quantization (PTQ) algorithm called aespa aiming at quantizing large transformer models, and refined quantization objectives for the attention layer which can accelerate the quantization process by pre-computation.\nThe authors gave clearly descriptions for the formulas which are used as quantization objectives and showed analysis results that aespa has a lower computational cost. The authors also showed stable experiment results comparing aespa with other quantization methods. Overall, I would suggest an Accept.", "questions": "1. When quantizing a transformer model with the proposed algorithm, will all weights and activations stay the same quantization precision? Or only the weight will be quantized?\n2. Will it be possible to discuss more on the accuracy of aespa compared to other methods?", "limitations": "The authors dicusse dthe limitation of aespa: this algorithm only focused on the attention module as a block.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. This PTQ algorithm is proved by mathematical methods, so it is easy to be applied to other transformer models.\n2. The proposed algorithm considers the cross-layer dependency by targeting attention modules. In detail, it quantizes query, key, and value in an attention module separately: when quantizing one projection, the other 2 projections are fixed with full precision, decreasing the heavy computation of block-wise quantization.\n3. Compared with conventional block-wise quantization and layer-wise PTQ schemes, the perplexity of aespa showed good results", "weaknesses": "1. This quantization algorithm only focuses on the attention module, targeting the cross-layer dependency within an attention module, not cross-layer dependency within the full model.\n2. In Experiments and Appendices, most tables compare the performance (perplexity), time, and memory cost. There is only one table (Table 10) showing the accuracy of zero-shot performance. There is no explicit result showing how aespa is more accurate than other methods. How much is the decrease in the accuracy compared with the original floating point models?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the reviewer's positive evaluation and invaluable comments on our work. Our point-to-point responses are as follows.\n\n**1. Consideration of the cross-layer dependencies within the full model (Weakness 1)**\n\n - We appreciate the reviewer's invaluable comments. By considering the reconstruction error for the entire Transformer block or the whole network, rather than the attention module, the dependencies between more layers could be considered, which would result in further enhancement.\n\n - However, due to nonlinear activation functions (e.g., SiLU for LLaMA), normalization layers, and weights of higher dimensions, considering the reconstruction error for the larger block (e.g., Transformer block) would result in complex objective functions, significantly more complicated than those developed for the attention module (see Eqs. (13) and (25)):\n$$\\Delta \\text{SA} _{Q} = \\frac{1}{d} \\mathbb{E} \\left [ || \\mathbf{V} ^{T} \\mathbf{J} _{\\text{softmax}} \\mathbf{K} \\Delta \\mathbf{W} _{Q} \\mathbf{X} || _{F}^{2} \\right ], \\Delta \\text{SA} _{K} = \\frac{1}{d} \\mathbb{E} \\left [ || \\mathbf{Q} \\Delta \\mathbf{W} _{K} \\mathbf{X}  \\mathbf{J} _{\\text{softmax}}^{T} \\mathbf{V} || _{F}^{2} \\right ].$$\nIn fact, although we established exact reconstruction errors for the attention module as above, we employed relaxed versions in Eqs. (15) and (16) for efficient quantization (i.e., reasonable processing time and quantization with a single GPU). Therefore, when trying to consider cross-layer dependencies between more layers, simplifying the reconstruction error as in our approach would not be a good option, and in this case, just exploiting the original reconstruction error would be better.\n\n - It is worth noting that recent approaches have tried to consider all layers using the reconstruction error for the entire Transformer block (e.g., OmniQuant [1] and AffineQuant [2]). However, to alleviate the computational overhead, they rely on a naive rounding-to-nearest weight-rounding method and the gradient approximation. As a result, although considering dependencies between more layers, they suffer from an unstable low-bit quantization process and exhibit inferior performance than the proposed method targeting the attention module (see Table 1).\n\n - In the final version, we will discuss why we concentrate on the attention module, not the entire Transformer block. We hope for the reviewer's kind evaluation and acknowledgment of our effort to balance accuracy and efficiency for the real-world deployment of LLMs.\n\n[1] W. Shao et. al., \"OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models,\" ICLR 2024.\n\n[2] Y. Ma et. al., \"AffineQuant: Affine Transformation Quantization for Large Language Models,\" ICLR 2024.\n\n**2. More accuracy results: There is only one table (Table 10) showing the accuracy of zero-shot performance. There is no explicit result showing how aespa is more accurate than other methods. How much is the decrease in the accuracy compared with the original floating point models? (Weakness 2 and Question 2)**\n\n - We appreciate the reviewer's constructive suggestion. As suggested, we have newly evaluated the zero-shot performance on LLaMA2 models (see Table I in the PDF attached to the global response) and the zero-shot performances of the proposed method for higher bit-widths (see Table II in the attached PDF).\n\n - From Table I, we observe that the proposed method outperforms the conventional approaches also on LLaMA2 models with respect to the zero-shot performance. \n    \n - From Table II, we observe that the proposed method almost preserves the performance of the original full-precision model for the INT6 quantization. Even for the INT4 quantization, the performance difference between the full-precision and quantized models is very marginal (e.g., less than 1% degradation for 13B and 30B models).\n\n - In the final version, we will incorporate these results to enrich the zero-shot performance comparison.\n\n**3. Quantization setting: When quantizing a transformer model with the proposed algorithm, will all weights and activations stay the same quantization precision? Or only the weight will be quantized? (Question 1)**\n\n -  We appreciate the reviewer's comment. As mentioned in Section 4.1, we quantized only weights and retained activations in full-precision as in the existing works [3]-[6] because activations do not pose a significant bottleneck and the inference of LLMs can be sufficiently accelerated by reducing memory movement via weight quantization [6], [7]. \n    \n - Nevertheless, when the activations need to be quantized, we can combine existing orthogonal techniques for suppressing activation outliers [8], [9] with the proposed method, which will be considered in our future studies.\n\n[3] J. Chee et. al., \"QuIP: 2-Bit Quantization of Large Language Models With Guarantees,'' NeurIPS 2023.\n\n[4] J. Lin et. al., \"AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration,'' MLSys 2024.\n\n[5] Y. Jeon et. al., \"A Frustratingly Easy Post-Training Quantization Scheme for LLMs,'' EMNLP 2023.\n\n[6] E. Frantar et. al., \"GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers,\" ICLR 2023.\n\n[7] S. Kim et. al., \"SqueezeLLM: Dense-and-Sparse Quantization,'' ICML 2024.\n\n[8] G. Xiao et. al., \"SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models,'' ICML 2023.\n\n[9] S. Ashkboos et. al., \"QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs,'' arXiv 2404.00456.\n\nAgain, we express our sincere thanks to the reviewer for the invaluable comments and constructive suggestions, which will be very useful in improving the quality of our work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We are truly thankful for invaluable comments provided by reviewers. During the rebuttal, we have made our best effort to address all the comments raised by reviewers. In this global response, we summarize our main contribution and emphasize essentiality of our work. 

**<Main contribution>**

 - In this work, we have aimed to mitigate computational overhead of the conventional block-wise weight-rounding optimization method (BRECQ [1]), thereby achieving the balance between accuracy and efficiency for the quantization of LLMs. We believe that this contribution is essential and not negligible because without the proposed mitigation, the conventional BRECQ would not be suitable for the quantization of hyper-scale LLMs due to the following reasons:

   - BRECQ requires too much processing time. Even for relatively small-sized models such as OPT-1.3B, BRECQ requires more than 10 hours, which is about 8 times longer than that required by our method (see Table 11(a) in Appendix K). Obviously, it would be difficult to conduct multiple times of hyper-parameter searches (e.g., learning rate, the weight of the rounding loss ($\lambda$ in Eq. (27)), and the choice of calibration datasets), and without such a time-consuming hyper-parameter search, BRECQ shows inferior performance than ours (see Table 1).

   - Multi-GPUs are indispensable for BRECQ while our method can quantize hyper-scale LLMs using a single A100 GPU. This is because BRECQ needs to save forward and backward pass information of all layers inside the attention module to perform back-propagation during the quantization. In fact, we encountered the out-of-memory (OOM) issue when trying to quantize LLMs that have more than 7B parameters (see Table 11(b)). Due to the lack of scalability, BRECQ has not been considered as a feasible solution for the quantization of LLMs in many pieces of literature [2]-[5]. In contrast, the proposed method conducts quantization in a layer-wise manner (yet still targeting the reconstruction of the attention output) and thus back-propagation information of only one layer needs to be stored at a time, which enables us to quantize LLMs with 30B parameters using a single GPU.

 - In this work, to strike an optimal balance between efficiency and accuracy, we propose a novel layer-wise quantization approach that considers cross-layer dependency, and we also present an efficient loss computation method through a novel mathematical analysis (see Sections 3.3 and 3.4 in the main text). In doing so, we could achieve robust quantization performance even at the low bit-width (see Table 1 and Table 7) while conducting the quantization using a single GPU with a much shorter processing time than that required by BRECQ (see Table 11(a)).

[1] Y. Li et. al., "BRECQ: Pushing the limit of post-training quantization by block reconstruction,'' ICLR 2021.

[2] E. Frantar et. al., "OPTQ: Accurate quantization for generative pre-trained Transformers,'' ICLR 2023.

[3] J. Chee et. al., "QuIP: 2-bit quantization of large language models with guarantees,'' NeurIPS 2023.

[4] Y. Jeon et. al., "A frustratingly easy post-training quantization scheme for LLMs,'' EMNLP 2023.

[5] W. Shao et. al., "OmniQuant: Omnidirectionally calibrated quantization for large language models,'' ICLR 2024.

Meta-review: This paper proposes a layer-wise post-training quantization (PTQ) method for transformer models, called aespa, focusing on quantizing the attention modules to account for cross-layer dependencies. The algorithm is designed to achieve a balance between accuracy and efficiency, particularly in low-bit scenarios (e.g., 3-bit and 2-bit). Extensive experiments demonstrate that the proposed aespa, outperforms several baselines in low-bit quantization settings. Before the rebuttal, reviewers expressed concerns regarding several aspects, including comparisons with additional PTQ baselines, higher bit-width results, and the discussion of differences from prior PTQ methods. The authors effectively addressed these issues in the rebuttal, and the reviewers were satisfied with the responses. Therefore, I would like to recommend accepting this paper.
Decision: Accept (poster)