A Frustratingly Easy Post-Training Quantization Scheme for LLMs
Authors: Yongkweon Jeon
Chungman Lee
Kyungphil Park
Ho-young Kim
Venue: EMNLP 2023 Main (2023)
Keywords: Quantization
Efficient LLM
Model Compression
TL;DR: We propose a post-training quantization scheme for LLMs

Abstract: Efficient inference has become crucial for hyper-scale AI models, including large language models, as their parameter count continues to increase for enhanced performance.
This necessity holds true regardless of the computing environment, whether it be mobile devices or cloud servers.
Quantization emerges as a solution to alleviate the computational burden during inference.
By representing models with a reduced bit-width, quantization minimizes the frequency of DRAM access while fully exploiting the parallelism of operations through a dense matrix format.
Consequently, quantized models achieve low end-to-end latency and optimize resource utilization by addressing both memory and computing bottlenecks.
In this paper, we propose a straightforward post-training quantization scheme, called \textsc{Z-Fold}, that fully utilizes the feature of the Transformer structure widely employed in large language models.

Z-FOLD: A Frustratingly Easy Post-Training Quantization Scheme for LLMs
Yongkweon Jeon,*
Chungman Lee,"
Kyungphil Park,"
Ho-young Kim
Samsung Research
daraawoo.jeonn chungman. lee, k_phil. park, hoyoung4. Brgbougg.kkiJJessmuunnnn kimmessmmuugg.comm
Abstract
Output
softmax
Norm
W
Efficient inference has become crucial for hyper-scale AI models, including large lan- guage models, as their parameter count contin- to increase for enhanced performance. This necessity holds true regardless of the comput- ing environment, whether it be mobile devices cloud servers. Quantization emerges solution to alleviate the computational burden during inference. By representing models with a reduced bit- bit-width, quantization minimizes the frequency of DRAM access while fully ex- ploiting the parallelism of operations through a dense matrix format. Consequently, quan- tized models achieve low end-to-end latency and optimize resource utilization by addressing both memory and computing bottlenecks. In this paper, we propose straightforward post- training quantization scheme, called Z-FOLD, that fully utilizes the feature the Transformer structure widely employed in large language models. The code will be available at https: //gi //g.thub. com/ SamsungLabs//--Fr
Linear W
Urusing
Layer Norm
W₁
W2
Transformer Block
Vusing
Linear W,
Linear W
Linear W
Layer Norm
MatMull
Attention
Mathul
To
Linear Rix
Layer Norm
Rig
We
Embedding
Tinput
Figure: The Transformer Architecture (Pre- LayerNorm). We fully exploit the feature of the architec- ture to prevent accuracy drop from lowered bit-precision of weights.
1 Introduction
and He, 2020), pipeline parallelism (Huang et al., 2019), model parallelism (Shoeybi et al., 2019), me-normmaiizaiinn (Xiong et al., 2020), and AdamW (Loshchilov and Hutter, 2017) are em- ployed and resulting in models with up 100 tril- lion parameters.
The Transformer (Vaswani al., 2017) mod- els have revolutionized machine learning across various domains by leveraging attention mecha- nisms (Bahdanau et al., 2014). Ncural language processing (NLP) tasks, including large language models (LLMs) and neural machine translation (NMT), heavily rely on the Transformers, extend- ing their impact to vision tasks as well (Khan et al., 2022; Rombach et al., 2022). In essence, hyper- scale Al models heavily depend on the Transformer architecture.
Nonetheless, these hyperscale models encounter limitations that hinder their applicability. Among the key challenges are the memory and computa- tional costs associated with processing an exten- sive number of parameters; these lead to increased end-to-end latency and power consumption, regard- less of the available computing resources. Specif- ically, considerable number of parameters need be transferred between memory and processors, and computed with long input sequences. Thus, hyperscale models impose a burden on hardware during inference, despite the performance bene- fits of parameter scaling. Consequently, efficient
To enhance the performance of general lan- guage tasks, large language models progressively increase the number of parameters, which are achieved with various techniques such as pre- venting performance saturation or enabling hyper- scale model to be trained; layer dropping (Zhang
"Equal Contribution, *Corresponding Author

---

inference techniques have gained significant at- tention as much as for scaling out the models, whether in resource-constrained (e.g, edge devices) or rexource-abundant (e.g., cloud server) environ- ments.
The loss surface of pre-trained models can be approximated using the Taylor series. Assuming successful convergence and generalization of the pre-trained model, we can simplify the relationship between loss degradation (AC) and perturbation of flattened weights Aw (i.e. Aw å vec(A by quantization to linear as follows (LeCun et al.. 1989; Nagel et al., 2020):
Quantization emerges as promising and attrac- tive solution among various techniques aimed at improving model efficieney (e.g., pruning (Frantar and Alistarh, 2023), low-rank approximation (Chen et al., 2018) and knowledge distillation (Hsieh et al., 2023)). By representing models with a low bit-width, quantization reduces the frequency of DRAM access, while fully leveraging the paral- lelism of operations through dense matrix format. As result, quantized models alleviate the bottle- neck caused by both memory and computational overheads.
1 AC ~ Aww H(w), Aw, 2
(I)
where (wr) denotes the Hessian matrix of weights. When quantizing models in the absence of datasets, the Hessian can be approximated to c I, where and I denote constant and the Identity matrix, respectively. We thus set the out out_ channel-wise step size a E RRdoux1 for the weights W C Rd X to minimize weight perturbation as follows:
In this paper, we propose a post-training quan- tization method for LLMs, called Z-FOLD, which exploits the feature of the Transformer architecture (pre-LayerNorm) to enhance the quantized models without introducing additional quantization param- eters computational costs. In nutshell, quantize weights into low bit-width (down 2-bit) using more parameters (() than existing approaches. This contributes to improving quantized networks by further minimizing the loss perturbation as a result of quantization. However, we fold or fuse these additional parameters (O) into other existing parameters (a or see Figure 1) in advance of in- ference, to avoid imposing further hardware costs, As result, our approach achieves state-of- of-the-art performance among quantized LLMs without any additional parameters or hardware costs.
a arg min IW Wine Al F
(2)
W int = clip [ A A n.p),
where [-] represent the cound-to-nearee function and A denotes A diag(a).. And and p repre- sent the lower and upper bounds, respectively, for the integer weights Wine Once ax is determined, we employ the nearest-rounding scheme to map the weights onto uniformly distributed quantiza- tion grids, which we refer to as MMSE quantiza- tion. Note that, the step size a can be set simply as min-max quatization as follows:
Q max W) min W
(4)
2
2
Related Works
where min() and max() return the minimum and maximum values of each column respectively. Based on the step size a set by MMSE or min- max, can further optimize the quantized network by various approaches approximating the Hessian with the calibration set (few-shot), which leads quantized networks to be more robust than data- free quantization (Nagel et al., 2020; Li et al., 2021; Hubara et al., 2021; Frantar and Alistarh, 2022: Jeon et al., 2023). Among them, OPTQ (Fran- tar et al., 2023), also known as GPTQ, achieves reasonable performance for quantized LLMs with- out gradient-based optimization in a relatively short period by simplifying Optimal Brain Quan- tization (Frantar and Alistarh, 2022). Consider WT E doo xdin as the transposed weight matrix and X E Idd xn as the input batch of the layer,
2.1 Quantization
When the entire dataset is available, quantization- aware training (QAT) (Jin et al., 2021) could be use- ful, potentially outperforming
quan- tization (PTQ). However, given the vast number of parameters to be optimized in LLMs, retraining them with the entire dataset would not be a realistic option. QAT with few samples also suffers from overfitting due to its extensive search space. Con- sequently, most approaches for quantizing LLMs rely on PTQ with either calibration set (few-shot) without any dataset (data-free quantization). By limiting the search space to the region adjacent to the convergence point of traitrained models, over- fitting caused by few data can be prevented in PTQ.

---

Folding
quantize
in B...hhnnelll wise
ag a
Woo
Woi
Woz
Woo
W91
Wo2
So
So
00
Yo
W10
W11
W12
W10
W11
W12
step size (i
X
Wax
a1
Wij E [n. pl
Y/
d1
S1
out whinneel-wise step size
a2
W₂ SOW (3 =3 ow where {=
R1
In_ nn_channels
Y/
out_ cot_chhnnels
Rank-1 decompasition
Proposed
Figure 2: Decomposing the step size S into Ç and followed by folding Ç to the preceding layer (if available)
where din and represent the input dimension and input sequence length, respectively. By assuming there is no interaction between each row of WT, OPTQ approximates the Hessian H corresponding to each row of WT
mixed-precision scheme, in which outlier vec- tors are preserved as FP16. SmoothQuant (Xiao et al., 2022) and Quadapter (Park et al., 2022) have identified the challenges associated with handling the dynamic range of activations, and then suggest solutions to mitigate the variance. QuIP (Chee et al., 2023) has tried quantize LLMs into low bit-width (i.e., 2-bit) by introducing additional pa- rameters (random orthogonal matrices). However, QulP needs to store or generate such additional parameters and impose computational costs during the inference. ZeroQuant-V V2 (Yao et al., 2023) has summarized and analyzed the properties of each method and bit configuration (c.g., E8A8, W4A16). While all of these approaches apply post-training quantization LLMs, a QAT method for LLMs has been proposed in LLM-QAT (Liu et al., 2023).
H ~ 2XXT + AI R din xdm
(5)
We thus have H () as w)) = & H. where @ is the Kronecker product. OPTQ quantizes WT on column-by-colunn in ascending order (0 to -1). Whenever each column of weights w; E X1 is quantized, they measure the errоr e as
e (w, w H
(6)
where H-1 Cholesky(H IT, and then all the remaining (not yet quantized) column weights wj (j i+1 to din -1) are updated to reflect the pertur- bation of quantized weights as follows:
Z-FOLD
3.1 The Newly Introduced Parameter Ç
(w. w. H
1yde-1
To achieve even greater reductions in the loss per- turbation due to quantization (Eq. (I)), we intro- duce scaling factor Ç E Rm corresponding the in in_channel in addition to existing out channel- wise scaling factors E gax1 The scaling factor is then fused or folded into a of the previous layer in advance of inference to eliminate the over- head by additional parameters Ç. which we call Z-FOLD (see Figure 2). In essence, our objective is to obtain the bi-directional (in and out) step size matrix S, which can be formulated follows:
where H denotes the entry in the ith row and jth column of the Hessian inverse.
2.2 Quantization for LLMs
OPTQ enhances the robustness of quantized LLMs by approximating the Hessian matrix using small sample of sentences, where activations are maintained FP16 precision since they are not the bottleneck. ZeroQuant (Yao et al., 2022) adopts group-wise quantization for weights and wosen-wise quantization for activation to suggest a hardware-friendle quantization scheme. Despite having larger number of scaling parameters com- pared other methods, ZuroQuant adjusts the ber of scaling parameters to match the hardware architecture used during inference, resulting in min- imal overhead. LLM. int8() (Dettmers et al., 2022) propose a vector-wiss wise quantization approach with
S arg min A
H (w)) Aw,
(8)
Aw = vec W - S S Win)
(9)
W ant = clip ( W 0 Sl. m.
(10)
where and denote the Hadamard division and product, respectively. To determine S in Eq. (8),
'step size-Folding

---

Algorithm Alternating Least Square for S
Folding
Input: W R Kihul
Norm
affinity
Yo
Limear
Output: Rdoutx] S (Ç. (1=
Yi
1: procedure Z- Z-FOLD(W, H = +
2: Initialize and
3: repeat
Yo Bn
Karo
[xg
x
[x0
X1
Gin
4: W Wma=Quantize(WW W. Ç c.xe)
SaBo
6.
5: Ç Least__Suarree WT, (Wint (i)a"
Folding
6: W Ww..QuuntizeWW (.a))
Figure 3: Norm-to-Linear Folding- The in channel-- wise parameters can be fused into the affine transforma- tion parameters the previous normalization layer.
7: a = Least Squares W, diag(() Wot.. H)
8:
until converged
9:
return a, Ç
10: procedure QUANTIZE(W.S)
11: return clip (1 [W 2 pt, H)
factorized the step size from weights matrix W. obtain quantized weights matrix W as follows:
12: procedure HEEATTSSQARRSTI
for cach column vector (wt; thss) (W, W) do 14: 3/4 (m Hw:: w
13:
W S Wint
(14)
15:
return y
= diag(C) int diag(a)
(15)
int A
(16)
we decompose the matrix S into two vectors (the rank-1 approximation) by alternating least square (ALS) factorization (Zachariah et al., 2012). Specif- ically, determine those value of a R=x1 and Ç E Rmx1 follows:
Our schemn also supports per-out cou_ncattteellllaaaaaaaaaaaaaael asym- metric quantization, in which each column weight wi can be quantized follows:
w = a( Wiiint - -0) C
(17)
AW W
int A 12
(11)
Wiint clip
we 2 n.
(18)
when Z A diag(() and A A diag(a), (12)
where indicates the zero point vector (all-o tor), computed by
which can be achieved by vector-wise least squares. Considering vector w; E mm representing the ith column vector of W pmxx, can leverage the weighted least squares determine a using the Hessian as follows:
min(w, e() For asym- metric quantization, we can employ grid search to identify the optimal values of a and o, aiming to minimize the loss perturbation caused by quan- tization. This grid search replaces the use of least squares in Line 7 of Algorithm 1.
a; = Hw: L. w.
(13)
where w denotes wz diag(C) Windi The weighted least square is used only for a and al- ternately refined with Ç. Note that, because the weights sharing also share the entries of the Hes- sian, whether or not the Hessian is used to deter- mine does not affect the result for Ç. Algorithm provides a detailed overview of the factorization algorithm. We initialize Ç as all-1 vec- Norm-to-Linear Let consider latent token E R m the input of normalization layer followed by linear layer, the normalized vector, denoted by & is computed as follows:
3.2 Parameter Integration (ç-Folding)
Figures 2, 3 and illustrate the mechanism of Z- FOLD. The figures demonstrate how in channel- wise parameters can be integrated into either the affine transformation parameters and f (Norm- to-Linear) or the out ww__channe--wse parameters (Linear-to- testarttt-iingg tonerr) of the preceding layer.
tor and then employ Min-Max quantization for a in Eq. (4) (Line 2). Subsequently, and are alternately updated using least square solution until convergence (Line 3-8), such that minimiz- ing the loss as Eq. (8). Note that column-win least square (Line 12-14) offers loop-level paral- lelism allowing for parallel processing via batch matrix-matrix multiplication (i.e., bmm). With the
y r 3 à +
(19)
where 7 E R and G E gx denote affine transform parameters â (i.e. the scale and shift).

---

Concatenating
Folding
Wgo
W
Wgo
Woz
Wgo
W
Wgz
60
X0
Linear
Yn
Linear
W110
Wis
W10
W12
W10
W12
4
Q.11
2x22
Figure 4: Concatenating Wq. Wx. and Wv to extract a mutual Ç for fusion into the preceding normalization layer.
[x]
x1
[Cg
Suu 0
[x0
x1]
Boat
d1
G
Folding
Figure 5: Linear-to- LinLir Folding. The quantization parameters between two adjacent linear layers can fused if there is non-lincar function except ReLU.
Eq. (19) can be expressed as a matrix operation, as follows:
yd
o 1]
(20)
Algorithm 2 Fine Tuning Each Layer
0
Input: W R Xioaat
X E Ria X (Y = .T.W)
71
where r
diag(y) gr
(21)
Output: a, S. W
I: procedure FINE-TUNING WITH OPTQ(W,
7m-1
50
81
Am-1-
= 2XXT +
a, (=Z-FOLD(W.
Algorithm
The output y from the normalization layer is then multiplied by quantized weights W e Rmxn of the subsequent linear layer as follows:
H Cholesky(H
for each row vector wr W do the step size of wn
6:
(4 a:) R
7:
w = s clip [w s,,,,,))
T W y (Z mit A)
(22)
8:
(w: w/) H,
= ([ 1] r) (Z Win A)
(23)
9:
{w; wi H 1 13m441
((tT 1).(1 Z)) (Wint int A).
(24)
10: retum C. W
As in Eq. (24), we can update the affine transform parameters using the of the subsequent linear. As result, We obtain the revised parameters
excepting ReLU between adjacent two linear lay- ers. Considering W 2 and W R= as the weights two consecutive linear layers and æ E Raxi as the input the first layer, the output can be computed as follows:
CoCo
M
7'=
(25)
Con-1
(xT) (WW ( (2)
(26)
CoCo
BiCi
3m
1m
= = (W] A (W"
(27)
Essentially, we can update the affinity parameters without relying on the dataset to compensate for the errors caused by quantization. By folding Ç into y and (3 of the preceding normalization layer, we can reduce the loss by quantization weights without increasing the number of parameters and computational cost.
As shown in Figure can also fuse the pair of layers Wo and Wv in addition to W1 and W2. Suppose and X denote and the input of the attention layer, respectively. Then, the output of linear, Wo. remained consis- tent even after parameter incorporation follows:
softmax Q, KT)
Notably, by concatenating the weight matrices of Q, K, and V, the shared Ç can be extracted; consequently, the shared can be seamlessly incor- porated into the y and [3 of the preceding normal- ization layer (see Figure 4).
(Y (XT. W Av) (Zo W3 - (XT Wy Zo) (W3
(28)
(29)
3.3 Fine-Tuning with a Calibration Set
After determining the bi-directional step size ma- trix S in Algorithm tized LLMs with a small calibration set by per- forming optimization using the approximated Hes- sian (Nagel et al., 2020; et al., 2021: Hubara
Linear-to-Linear It can be easily demonstrated that the in_channel-wise step size Ç can be fused into the out out_channel-wise step size a the preced- ing layer, provided there no non-linear function
further optimize quan-

---

Table 1: Quantization performance of Z-FOLD on OPT models (Perplexity 4).
(a) WikiText-2
Method
Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Baseline
FP16
27.65
22.00
14.63
12.47
10.86
10.13
9.56
RTN
37.28
25.94
48.17
16.92
12.10
11.32
10.98
OPTQ'
INT4
31.12
24.24
15.47
12.87
11.39
10.31
9.63
Z-FOLD
30.18
23.64
14.92
12.41
10.95
10.15
9.52
RIN!
1.3e3
64.57
1.3c4
1.6e4
5.8e3
3.4e3
1.6e3
OPTQT
INT3
53.85
33.79
20.97
16.88
14.86
11.61
10.27
Z-FOLD
37.19
28.18
16.27
13.53
11.43
10.65
RTN
5.5e3
2.8e4
1.1e5
9.5e3
2.8e4
1.9e5
1.7e5
OPTQ
INT2
5.6e3
1.7c4
8.0c3
9.0e3
3,4c3
348.2
61.53
Z-FOLD
152.3
127.9
36.91
28.00
19.36
15.84
13.13
(b) PTB
Method Precision
125M
350M
1.38
2.7B
6.7B
13B
30B
Baseline
FP16
38.99
31.08
20.29
17.97
15.77
14.52
14.04
RIN
53.89
36.79
57.30
31.05
18.84
16.51
15.40
OPTQ
INT4
45.17
34.52
21.85
19.14
16.56
14.94
14.26
Z.FOLD
42.83
34,03
21.11
18.45
15.85
14.67
14.21
RTN
1.4e3
88,04
1.3e4
1.4e4
5.7e3
2.8e3
1.2c3
OPTQ'
INT3
73.19
47.08
32.10
24.81
21.88
16.68
15.36
Z-FOLD
53.74
40.23
23.15
20.30
15.19
14.58
RTN
4.3c3
2.8c4
1.1c4
6.8c3
1.8c4
1.2c5
1.7c5
OPTQ
INT2
3.8e3
1.5e4
7.1e3
8.6e3
2.4e4
275.7
87.54
Z.-FODD
210.1
190.2
60.42
48.53
27.89
23.57
19.15
(c) C4
Method Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Baseline
FP16
26.56
22.59
16.07
14.34
12.71
12.06
11.44
RIN!
33.91
26,21
24.51
18.43
14.36
13.36
13.46
OPTQ
INT4
29.22
24.63
16.97
15.00
13.18
12.26
11.57
Z-FOLD
27.96
24.12
16.50
14.65
12.86
12.15
11.50
RTN
834.0
55.49
5.2c3
1.1c4
5.3c3
3.1c3
1.4c3
OPTQ*
INT3
42.41
31.33
21.63
18.17
17.14
13.34
12.23
Z-FOLD
32.75
27.33
17.49
15.64
12.47
11.74
RTN
3.7c3
1.6e4
7.7c3
7.7e3
1.4e4
9.7c4
5.6e4
OPTQ
INT2
2.3c3
5.5c3
4.3e3
4.2c3
568.1
124.5
29.03
Z- FOLD
106.8
87.96
34.53
26.73
21.60
17.11
14.10
Results are taken from (Frantar et al.. 2023). In OPT-350M, post-LayerNorm architecture has been used that only apply Z-FOLD for Wo and W2 (see Figure 1).
et al., 2021; Wei et al., 2022; Frantar et al., 2023; Jeon et al., 2023). Among them, we utilize OPTQ, which is time- efficient solution for hyper-scale
quently apply a few-shot quantization method to
refine quantized models further. Algorithm 2 ex-
plains our approach when given calibration set.
models such LLMs. In essence, utilize Z- FOLD the initial quantization state and subsc-
We initialize quantized models by Z-FoLD,, where use the Hessian for minimize the degrada-

---

Table 2: Quantization performance of Z-FoLD on LLaMA models (Perplexity 4).
WikiText-2
PTB
C4
Method
Precision
7B
13B
30B
7B
13B
30B
7B
13B
30B
Baseline
FP16
5.68
5,09
4.10
8.80
8.07
7.30
7.08
6.61
5.98
RTN
5.53
4.54
9.70
8.63
7.70
7.73
6.99
6.33
OPTQ
INT4
6.11
5.35
4.46
9.31
8.42
7.53
7.43
6.84
6,20
Z-FOLD
6.07
5.28
4.34
9.35
8.31
7.52
7.47
6.77
6.13
RTN
25.62
11.78
14.87
61.71
22.38
30.68
28.24
13.24
28.57
OPTQ
INT3
8.11
6.71
5.68
12,06
9.99
8.90
9.56
8.22
7.36
Z-FOLD
6.79
5.76
4.92
10.43
8.96
7.97
8.22
7.21
6.59
RTN
1.1e5
5.7c4
2.7c4
1.2c5
7.8c4
2.604
1.1c5
5.9c4
2.8c4
OPTQ
INT2
1.0c4
7703.
2065.
7620.
1.4e4
5666.
1031.
1185.
349.0
Z-FOLD
14.58
13.25
9.65
24.36
20.98
17.01
15.19
13.10
10.82
tion of the loss (Eq. (1)) (Line 2-3). We then apply the OPTQ to minimize the loss further (Line 4-9). Our scheme can be applicd to not only uniform quantization but also multi-level binary quantiza- tion (Xu et al., 2018; Jeon et al., 2022). Both for- mats can be accelerated by dedicated kernels (Jcon et al., 2020; Frantar et al., 2023) when performing LLMs inference without quantizing activations. As well, Z-FOLD be utilized in QLoRA (Dettmers al., 2023) when quantizing and freezing the pre- trained weights of LLMs.
mark datasets (WikiText-2 (Merity et al., 2016),
C4 (Raffel al., 2019), and PTB (Marcus et al., 1993)) and several zero-shot tasks.
4.2 Comparison with Others
In Table 1, we summarize quantization results on OPT models of various sizes ( (125M to 30B). For comparison, we also summarize the results for con- ventional RTN and OPTQ schemes. Overall, Z- FOLD and OPTQ outperform RTN by large mar- gin since they minimize the loss degradation AC, rather than the weight quantization error Au, by exploiting the estimated Hessian. Furthermore, we can observe that the proposed Z-FOLD outperforms OPTQ, and the performance gap is getting larger when quantizing models into lower bit width.
4 Experiments
4.1 Experimental Setup
To evaluate the performance of the proposed method, we quantize publicly available LLMs (e.g.. OPT (Zhang et al., 2022), LLaMA (Touvron et al., 2023), and BLOOM (Scao al., 2022)) using Z- FOLD, As calibration dataset to estimate the Hessian, we use 128 random 2048 token segments from the C4 dataset (Raffel et al., 2019) as in OPTQ (Frantar et al., 2023). We conduct all exper- iments using a single NVIDIA A100 GPU (80 GB) and implement Z-FOLD using PyTorch.
In the case of 2-bit quantization, for example, OPTQ collapses completely (perplexity is larger than 2,000 regardless of the model size and the test dataset) while Z-FOLD still shows reasonable perplexity. The outstanding performance of Z- FOLD results from dividing quantization levels more precisely using an additional parameter Ç, yet imposing no extra cost via folding (see Section 3.2). Such an outperformance can be also observed for LLaMA (see Table 2) and BLOOM models (sec Table in Appendix 1.1).
In our experiments, we quantize only weights and retain activations as full-precision (FP16) since activations do not pose a significant bottleneck dur- ing the inference of LLMs (Frantar et al., 2023). We break the alternating updates of a and Ç in Z- FOLD (see lines 4-7 in Algorithm 1) when the loss perturbation AC (see equation 1) increases after the update or the total number of updates exceeds 30 iterations.
Z-FOLD performs better than OPTQ almost all cases, and the performance gap significant in a low bit width. We note that the quantized models obtained by OPTQ perform much worse than the full-precision model even for large-scale
Forr BLOOM, GeLU is used as an activation function hetween the adjacent linear layers (W, and W, in Figure l). In this case, Ç of Wa cannot he folded into Wi so we do not apply Z-FOLD for Wt.
After the quantization, we evaluate the perfor- mance of the quantized LLMs using three bench-

---

Table 3: Quantization (INT2) results for Z-FOLD com- bined with AdaRound/BRECQ (Perplexity 4)
Table 4: Ablation study for Z-FOLD on 2-bit quantiza- tion of OPT-125M (Perplexity 4)
Method
WikiText-2
PTB
C4
Quant. Grid
Method
WikiText-2
PTB
C4
Baseline (OPT-125M)
27.65
38,99
26.56
OPTQ Z-FOLD
5606.
3829.
2333.
AdaRound
570.5
877.6
298.6
Min-Max
3254.
2940.
1181.
AdaRound + Z-FOLD
158.8
227.6
100.6
OPTQ Z-FOLD
978.3
1192.
628.4
BRECQ
53.95
82.19
41.50
MMSE
493.3
680.9
314.9
BRECQ Z-FOLD
45.43
60.00
36.73
OPTQ
231.1
375.6
186.0
Hessian
Z-FOLD
152.3
210.1
106.8
models (e.g., LLaMA-30B) that might be overpa- rameterized and thus expected be quantized with marginal performance. For example, the perplexity the 2-bit LLaMA-30B model is 2065 while that of the full-precision model is only 4.10. Whereas, the perplexity obtained by the proposed Z-FOLD is 9.65, which means that the model quantized by Z-FOLD performs very close to the full-precision model.
We only consider OPT- 125M since AdaRound and BRECQ require more computing resources and time to complete the quantization of large-scale models than OPTQ. As in Table 3, we observe that Z-FOLD greatly enhances the quantization per- formance of AdaRound and BRECQ. In particu- lar, when combined with BRECQ, the quantized model obtained by Z-FOLD shows comparable per- formance with the full-precision model.
4.3 Generalizability of Z-FOLD
We investigate the generalizability of LLMs quan- tized by Z-FOLD using zero-shot tasks such as LAMBADA (Paperno et al., 2016), PIQA (Bisk et al., 2020), StoryCloze (Mostafazadeh et al.. 2017), and ARC (Easy and Challenge) (Clark et al., 2018). We recall that the calibration data we used for the quantization (i.e., random token segments from the C4 dataset) is excerpts from randomly crawled websites, which means that our calibration data is not task-specific (generic) and zero-shot setting is kept our experiments.
4.5 Ablation Study
We recall that two key features of the proposed Z-FOLD are 1) additional (foldable) parameters Ç and 2) Hessian-basee computation of quantization grids (i.e., Ç and ax). Note that Z.-FOI. utilizes the Hessian information when determining the step size while existing works such as AdaRound and OPTQ use MMSE or Min-Max for the step size.
To examine the impact of each feature, duct ablation studies (see Table 4). From the results, we observe that the quantization performance can be enhanced by exploiting Ç. which demonstrates the effectiveness of the proposed Ç in refining quan- tization grids. We also note that the performance of Z-FOLDD can be improved dramatically by uti- lizing the Hessian H in computing Ç and a.. In other words, networks by using the Hessian H in searching the quantization grid (i.e., a and (). Which contributes to reducing the loss perturbation AC rather than the quantization error A AW.
In Tables 6 to 8 (see Appendix A.2), we summa- rize the zero-shot performance of quantized LLMs. From the results, we observe that the models quan- tized by the proposed Z-FOLD are much more ro- bust than those obtained by conventional RTN and OPTQ approaches. In particular, for the LAM- BADA task, while RTN and OPTQ completely fail the 2-bit quantization even for 30B models, Z-FOLD shows reasonable performance owing to more sophisticated quantization capability induced by Ç.
further optimize the quantized
Conclusion
4.4 Portability of Z.-FOD
So far, have used Z-FOLD together with OPTQ quantize LLMs. To show the versatility of the proposed Z-FOLD, we perform the quantization by combining Z.FFLLD with other well-known quanti- zation approaches such as AdaRound (Nagel et al., 2020) and BRECQ (Li et al., 2021) (see Table 3).
In this paper, have proposed post-training quantization scheme called Z-FOLD. By capturing the unique features of the Transformer and fully leveraging its architecture (pre-LayerNorm and suc- cessivity of lincar layers), we developed a simple yet powerful approach that enhances the perfor-

---

mance of quantized models. Our scheme stands out by quantizing models using& greater number of parameters than existing approaches, resulting in enhanced sophistication. Importantly, these ad- ditional parameters are intelligently fused into ex- isting ones, eliminating the need for extra hardware costs. Our findings demonstrate that the Z-FOLD scheme achieves state-of-the-an thrt performance in quantized LLMs.
Elias Frantar and Dan Alistarh. 2022. Optimal brain compression: framework for accurate post- training quantization and pruning. arXiv preprint arXiv:2208. 11580.
Elias Frantar and Dan Alistarh. 2023. Sparsegpt: Mas- sive language models can be accurately pruned in one-shot.
Elias Frantar, Salch Ashkboos, Torsten Hoefler, and Dan Alistarh. 2023, Optq: Accurate quantization for generative pre-trained transformers. In The Eleventh International Conference on Learning Representa- tions.
6 Limitations
Our proposed scheme has a dependence on the tar- get architecture although the Pre-LN Transformer we target (e.g., OPT, BLOOM, and LLaMA) is broadly used in generative large language models.
Cheng Yu Hsieh, Chun-Liang Li, Chih-Kuu Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Ye Lec, and Tomas Pfister. 2023. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint 2rXiv:2.35..023
Acknowledgements
We would like to thank Junhan Kim, Ph.D for his helpful discussion.
Yanping Huang, Youlong Cheng, Ankur Bapna, Orhan Firat, Dehao Chen, Mia Chen, HyoukJoong Lee, Ji- quan Ngiam, Quoc Le, Yonghui Wu, et al. 2019. Gpipe: Efficient training of giant neural networks using pipeline parallelism. Advances in neural infor- mation processing systems, 32.
References
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Ben- gio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Itay Hubara, Yury Nahshan, Yair Hanani, Ron Banner, and Daniel Soudry. 2021. Accurate post training quantization with small calibration sets. In Inter- national Conference on Machine Learning, pages 4466-4475. PMLR.
Yonatan Bisk, Rowan 7ellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. Piqa: Reasoning about physical commonsense in natural language. In Thirty-
Yongkweon Jeon, Chungman Lee, Eulrang Cho, and Yeonju Ro. 2022. Mr.BiQ: Post-training non-uniform quantization based minimizing the reconstruction error. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 12329-12338.
Fourth AAAI Conference on Artificial Intelligence.
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. 2023, Quip: 2-bit quantization of large language models with guarantees. arXiv preprint arXiv:2307.13300
Yongkweon Jeon, Chungman Lee, and Ho-young Kim. 2023. GENIE: Show me the data for quantization. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR).
Patrick Chen, Si, Yang Li, Ciprian Chelba, and Cho- Jui Hsieh. 2018. Groupreduce: Block-wiss wise low-rank approximation for neural language model shrinking. Advances Neural Information Processing Sys- tems, volume 31. Curran Associates, Inc.
Yongkweon Jeon, Baeseong Park, Se Kwon, Byeong- wook Kim, Jeongin Yun, and Dongsoo Lee. 2020. BiQGEMM: Matrix multiplication with lookup ta- ble for binary-coding based quantized DNNs. In The International Conference for High Performance Computing, Networking, Storage, and Analysis (SC), pages 1343-1356 IEEE Computer Society.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? try arc, the ai2 reasoning challenge. 180..0557v...
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Gpt3. int8 (): 8-bit matrix mul- tiplication for transformers scale. Advances in Neural Information Processing Systems, 35:30318- 30332.
Jing Jin, Cai Liang, Tiancheng Wu, Liqin Zou, and Zhil- iang Gan. 2021. KDLSQ-BERT: quantized BERT combining knowledge distillation with learned step size quantization. arXiv preprint arXiv:2101.05938
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. 2022. Transformers in vision: A survey. ACM computing surveys (CSUR), 54(10s):1- 41.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. QLoRa: Efficient finetuning of quantized Ilms. arXiv preprint arXiv:2305.1e 14314.

---

Yann LeCun, John S Denker, Sara A Solla, Richard Howard, and Lawrence D Jackel. 1989. Optimal brain damage. In Advances in Neural Information Processing Systems (NIPS), volume 2. pages 598- 605.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. 2022. High- resolution image synthesis with latent diffusion mod- els. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10684-10695.
Yuhang Li, Ruihao Gong, Xu Tan, Yang Yang. Peng Hu, Qi Zhang, Fengwei Yu. Wei Wang, and Shi Gu. 2021. BRECQ: Pushing the limit of post-training quantiza- tion by block reconstruction. In International Con- ference Learning Representutions (ICLR).
Teven Le Scao, Angela Fan, Christopher Akiki, EI- lie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022. Bloom: A 176b- parameter open-access multilingual language model. arXiv preprint arXiv:2211.05/1 1.05100.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang. Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. 2023. Llt-qat: Data-free quantization aware training for large language models.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catan- zaro. 2019. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint arXiv: 1909.08053...
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
arXi:/171..05101
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and effi- cient foundation language models. arXiv preprint arXiv:2302.13971
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated cor- pus of English: The Penn Treebank. Computational Linguistics, 13-3333
13-330.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2016. Pointer sentinel mixture mod- cls. arXiv preprint 10Xix:660,,08443
Ashish Vaswani. Neam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Pro- cexsing Systems, volume 30, Curran Associates, Inc.
Nasrin Mostafazadeh, Michael Roth, Annie Louis. Nathanael Chambers, and James Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop Linking Models of Lexical, Sentential and Discourse-level Semantics, pages 46-51.
Xiuying Wei, Ruihao Gong, Yuhang Li, Xianglong Liu, and Fengwei Yu. 2022. Qdrop: randomly dropping quantization for extremely low-bit post-training quan- tization. arXiv preprint arXiv:2203.055: 05740..
Guangxuan Xiao, Ji Lin, Mickael Seznec, Julien De- mouth, and Song Han. 2022. Smoothquant: Accurate and efficient post-training quantization for large lan- guage models. arXiv preprint 1tt::2/111104333
Markus Nagel, Rana Ali Amjad, Mart Van Baalen, Christos Louizos, and Tijmen Blankevoort. 2020. Up or down? adaptive rounding for post-training quan- tization. In International Conference on Machine Learning (ICML.,, pages 7197-7206.
Ruibin Xiong, Yunchang Yang. Di He, Kai Zheng, Shuxin Zheng, Chen Xing. Huishuai Zhang, Yanyan Lan, Liwei Wang. and Tieyan Liu. 2020. On layer normalization in the transformer architecture. In In- ternttional Conference on Machine Learning, pages 10524- 10533. PMLR.
Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, Sandro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016 The LAMBADA dataset: Word prediction requiring a broad discourse context, In Proceedings of the 54th Anmual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 1525-1534, Berlin, Germany. Association for Computational Linguistics.
Chen Xu, Jianqiang Yao, Zhouchen Lin, Wenwu Ou, Yuanbin Cao. Zhirong Wang, and Hongbin Zha. 2018. Alternating multi-bit quantization for recurrent neural networks. In International Conference on Learning Representations (ICLR).
Minseop Park, Jaeseong You, Markus Nagel, and Simyung Chang. 2022. Quadapter: Adapter for GPT-
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. 2023. Zeroquant-v2: Exploring post- training quantization in Iims from comprehensive study to low rank compensation.
2 quantization. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 2510-2517, Abu Dhabi, United Arab Emirates. As- sociation for Computational Linguistics.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li. and Yuxiong He. 2022. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168- 27 1833
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text trans- former. arXiv e-prints.

---

Dave Zachariah, Martin Sundin, Magnus Jansson, and Saikat Chatterjee. 2012. Alternating least-squares for low-rank matrix reconstruction. IEEE Signal Processing Letters, 19442233-
Minjia Zhang and Yuxiong He. 2020. Accelerating training of transformer-based language models with progressive layer dropping. Advances in Neural In- formation Processing Systems, 33:14011-14025.
Susan Zhang. Stephen Roller, Naman Goyal. Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li. Xi Victoria Lin, et al. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXwv22005.00068

---

A
Additional Experimental Results
the Appendix, we provide experimental results excluded the main text due to the page limitation.
A.1
Results for BLOOM Models
Table Quantization performance Z.-OOLD on BLOOM models (Perplexity }
(a) WikiText-2
Method
Precision
560M
1.1B
1.7B
3B
7.1B
Baseline
FP16
22.42
17.69
15.39
13.48
11.37
RTN'
25.90
22.00
16.97
14.76
12.10
OPTQ'
INT4
24.03
19.05
16.48
14.20
11.73
Z-FOLD
23.58
18.61
15.99
13,90
11.59
RTN
57.08
50.19
63.59
39.36
17.38
OPTQ'
INT3
32.31
25.08
21.11
17.40
13.47
Z-FOLD
27.16
21.24
17.75
15.21
12.35
RTN
7.9c5
9.9c5
3.5e5
1.4e5
2.1c5
OPTQ
INT2
2646.
2220.
1791.
889.5
218.0
Z-FOLD
76.63
51.41
33,66
27.69
19.69
(b) PTB
Method Precision
560M
1.1B
1.7B
3B
7.1B
Baseline
FP16
43.69
57.96
30.00
25.34
20.83
RTN
51.10
66.85
33.58
27.68
22.42
OPTQ*
INT4
46.97
62.47
31.84
26.49
21.67
Z-FOLD
43.40
49.87
29.19
24.08
19.87
RTN
126.0
185.0
106.0
66.78
35.04
OPTQ'
INT3
70.35
87.04
46.11
34.02
26. 14
Z-FOLD
51.43
58.14
34.44
26.58
21.29
RTN
1.le6
2.0e6
3.2e5
1.2e5
1.3e5
OPTQ
INT2
4548.
5202.
4254.
2619.
579.2
Z-FOLD
184.6
154.5
68.05
55.57
38.23
(c) C4
Method Precision
560M
1.1B
1.7B
3B
7.1B
Baseline
FP16
26.60
22.05
19.49
17.49
15.20
RTN
29.89
24.44
21.26
18.76
16.06
OPTQ'
INT4
28.00
23.25
20.55
18.10
15.60
Z-FOLD
25.24
20.96
18.53
16.50
14.36
RTN
67.49
60.71
113.0
80.49
22.59
OPTQ'
INT3
35.78
28.83
25.34
21.25
17.67
Z-FOLD
27.96
22.77
19.93
17.51
15.01
RTN
1.le6
1.9c6
2.6e5
8.8c4
1.3e6
OPTQ
INT2
763.9
699.6
499.4
332.5
108.0
Z.FOLD
62.17
41.46
33.24
27.12
21.23
Results are taken from (Frantar et al., 2023).

---

A.2 Results on Zero-Shot Tasks
Table 6: Zero-shot performance of quantized OPT models (Accuracy 1).
(a) LAMBADA
Method
Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Baseline
FP16
39.16
46.67
58.80
64.82
68.72
70.23
72.39
RTN
18.34
40.62
36.31
59.27
64.66
67.38
70.48
OPTQ
INT4
34.74
48.38
56.45
62.97
66.37
69.12
72.40
Z-FOLD
33.70
48.03
57.83
66.83
69.45
71.14
72.87
RTN
0.10
27.36
0.00
0.06
1.46
OPTQ
INT3
13.93
32.31
37.26
52.26
54.98
64.18
69.69
Z-FOLD
24.37
51.10
55.33
63.59
69.96
70.48
72.29
RIN
0.00
0.00
0.00
0.00
0.00
0.00
0.00
OPTQ
INT2
0.00
0.00
0.00
0.00
0.83
4.70
26.94
Z-FOLD
6.42
16.67
28.10
43.10
51.58
54.69
65.75
(b) PIQA
Method
Precision
125M
1.3B
2.7B
6.7B
Baseline
FP16
62.02
64.74
72.36
74.81
76.39
76.88
78.18
RIN 1
61.43
63.44
67.63
73.72
76.44
76.01
77.26
OPTQ
INT4
61.26
63.71
70.73
73.99
76.28
76.61
79.00
Z-FOLD
62.35
64.47
71.33
75.35
76.39
76.55
78.07
RIN t
52.77
$1.90
50.49
56.37
OPTQ
INT3
59.25
61.32
68.34
71.38
73.29
75.24
77.58
Z-FOLD
60.55
63.17
70.78
73.50
76.77
76.01
78.29
RIN
50.44
50.76
49.62
49.56
49.29
50.38
47.82
OPTQ
INT2
52.61
55.71
55.01
56.58
59.58
62.57
67.41
Z-FOLD
57.62
56.80
64.15
64.42
70.78
69.15
74.92
(c) ARC-easy
Method
Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Bascline
FP16
39.69
50.93
54.34
60.14 14
61.83
65.40
RIN 1
36.32
38.55
49.20
$2,90
57.68
61.31
61.11
OPTQ
INT4
39.02
37.92
59.97
53.11
59.72
61.32
65.11
Z-FOLD
39,02
38.85
50.17
53.24
59.64
61.41
64.81
RTN t
30.43
36.07
27.97
26.05
25.04
30.60
34.22
OPTQ
INT3
36.15
36.91
46.17
48.19
53.41
56.82
59.72
Z-FOLD
36.83
38.05
47.90
51.56
58.80
59,93
62.75
RTN
25.84
25.72
25.42
25.34
25.59
26.05
25.29
OPTQ
INT2
28.07
25.93
26.05
26.18
30.30
31,06
40.24
Z-FOLD
32.41
31.78
40.74
40.36
47.31
54.80

---

(d) ARC-challenge
Method
Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Baseline
FP16
22.87
24.06
29.44
31.31
34.56
35.75
38.14
RTN
22,44
23.81
24.91
29.18
32.59
35.24
35.41
OPTQ
INT4
22.95
24.83
28.24
30.12
33.70
34.90
37.80
Z-FOLD
23.46
24.15
29.10
29.86
33.96
34.64
37.37
21.76
22.18
23.55
25.43
25.85
23.81
19.97
OPTQ
INT3
22.53
25.09
27.65
27.82
31.91
33,02
35.84
Z-FOLD
22.35
23.46
28.07
28.92
34.04
33.53
34.98
RTN
26.37
25.60
23.81
26.71
25.94
27.22
26.54
OPTQ
INT2
24.40
24.74
23.98
26.79
23.81
24.06
26.54
Z-FOLD
23.04
21.84
25.60
24.83
26.88
30.38
(e) Cloze
Method Precision
125M
350M
1.3B
2.7B
6.7B
13B
30B
Bascline
FP16
59.96
63.21
70.78
71.74
74.60
76.64
77.28
RTN
60.02
63.08
59.13
70.78
73.65
74.47
75.37
OPTQ
INT4
59.58
63.46
69.64
70.46
73.90
76.19
77.08
Z-FOLD
61.23
62.44
69.89
71.93
74.54
76.96
RIN
49.65
56.78
47.61
46.98
48.12
49.20
49.84
OPTQ
INT3
$7.03
60.15
65,25
68.43
70.97
73.07
75.68
Z-FOLD
59.00
60.92
68.68
70.66
74.92
74.79
76.64
RIN
48.12
48.76
47.74
48.19
OPTQ
INT2
51.94
54.23
53.47
52.51
54.04
61.17
65.63
Z-FOLD
53.15
54.55
62.19
64.42
67.79
68.87
72.12
Results are taken from (Frantar et al., 2023).

---

Table 7: Zero-shot performance of quantized BLOOM models (Accuracy t).
(a) LAMBADA and PIQA
LAMBADA
PIQA
Method
Precision
560M
1.IB
1.7B
3B
7.IB
560M
1.1B
1.7B
3B
7.1B
Baseline
FP16
34.06
42.85
46.71
$2.12
57.70
65.07
67.14
69.97
70.51
73.72
RTNt
26.00
39.06
41.92
45.84
50.48
63.11
65.29
67.74
69.86
72.69
OPTQ
INT4
31.75
39.80
46.28
51.41
54.65
64.31
66.05
68.77
69.42
72.96
Z-FOLD
35.34
43.94
47.53
51.43
58.14
63.98
66.97
68.34
69.91
72.69
RINI
9.10
15.95
15.02
24.55
29.90
58.60
60.80
60.88
66.28
69.70
OPTQ
INT3
21.31
33.65
43.12
47.41
61.62
62.62
65.18
68.34
70.95
Z-FOLD
31.73
34.66
45.12
50.13
58.72
62.57
65.45
67.14
69.10
70.89
RTN
0.00
0.00
0.00
0.00
0.00
51.09
48.53
49.23
49.73
50.22
OPTQ
INT2
0.06
0.39
0.12
1.16
4.44
51.03
50.76
50.76
$1.90
55.66
Z-FOLD
14.55
19.02
34.43
31.59
42.52
56.37
60.12
61.21
64.42
66.21
(b) ARC-casy and StoryCloze
ARC-easy
StoryCloze
Method Precision
560M
1.7B
3B
7.1B
560M
1.7B
Baseline
FP16
41.71
45.41
48.11
53.24
57.37
61.94
63.27
65.44
67.79
71.99
RTNI
39.40
42.51
44.70
51.35
56.14
60.15
60.66
62.95
67.09
70.72
OPTQ
INT4
40.24
44.49
44.49
52.82
56.14
61.17
62.32
64,48
67.22
71.36
Z-FOLD
40.82
44.07
45.62
52.48
55.47
61.43
63.21
64.67
67.60
70.97
RTN
45.44
46.87
37.58
45.08
48.61
54.87
56.08
55.79
59.83
66.20
OPTQ
INT3
39.14
41.79
42.85
46.63
$1.56
$7.80
59.77
61.81
63.97
69.26
Z-FOLD
39.94
42.56
44.23
48.82
53.20
59.96
62.44
63,21
67.03
RTN
26.81
27.10
27.27
26.56
27.95
47.29
46.53
45.83
46.15
46,40
OPTQ
INT2
31.61
33.84
34.34
35.65
38.05
49.71
49.97
48.89
50.80
53.34
Z-FOLD
34.05
35.69
39.06
40.91
43.94
53.47
56.46
59.26
61.11
63.33
Results taken from (Frantar al., 2023).

---

Table 8: Zero-shot performance of quantized LLAMA models (Accuracy 1).
ARC-challenge and HellaSwag
ARC-challenge
HcllaSwag
Method
Precision
13B
30B
7B
13B
30B
Baseline
FP16
44.48
48.83
52.17
76.19
79.06
82.64
RIN
43.13
44.48
52.83
74.17
77.06
81.54
OPTQ
INT4
43.81
46.15
48.83
75.09
77.87
81.32
Z-FOLD
43.81
47.83
52.84
75.40
78.25
82.40
RTN
20.40
37.79
28.43
45.42
62.06
31.49
OPTQ
INT3
34.78
34.78
45.82
68.74
73.15
78.05
Z-FOLD
39.46
45.82
49.83
72.35
75.62
79.60
RTN
26.42
28.43
29.10
26.47
26.10
25.89
OPTQ
INT2
29.43
26.42
33.11
25.32
26.35
26.03
Z-FOLD
28.43
31.44
38.80
53.87
57.09
64.53
(b) PIQA and OpenBookQA
PIQA
OpenBookQA
Method
Precision
7B
13B
30B
7B
13B
30B
Baseline
FP16
79.16
80.14
82.21
42.60
43.80
45.60
RTN
78.24
79.54
80.69
42.60
42.40
44.20
OPTQ
INT4
78.40
79.76
81.01
42.40
44.20
45,80
Z-FOLD
79.05
80.09
81.23
43.40
43,20
47.80
RTN
65.83
72.20
65.34
29.40
35,00
32.00
OPTQ
INT3
75.14
77.64
79.00
40.40
39.80
Z-FOLD
78.02
79.27
79.05
40.60
40.80
45.20
RTN
49.24
50.00
50.33
23.20
24.40
25.40
OPTQ
INT2
50.49
50.44
48.97
24.80
22.80
23.40
Z-FOLD
70.24
72.09
73.99
32.80
33.80
36.60


Reviews:
[{"review_id": "ddmxv8JS1M", "reviewer": "Reviewer_XxAx", "summary": "", "questions": "1) what is the precision of variables $S$, $Z$ and $A$ in equations 8 and 10? \nIf $W_int$ has the same precision, why not directly determine the correct values for $W_int$ based on SGD or Adam?\n\n2) The variable $\\zeta \\in R^{m \\times 1}$ does not consume a lot of memory, and there are few benefits to fold $\\zeta$ into previous modules, such as normalization and linear layers in Figures 4 and 5? Can you provide more specific experiments/statistics to demonstrate the claim?  \n\n3) I do not understand the feature in \"Z-Fold fully utilize the feature of the Transformer structure\". Does it refer to the pre-LayerNorm setting? If so, how about post-LayerNorm?\n\n4) Please describe the hardware evaluation platform, software used, benchmarks and evaluation objectives.\n\n5) Please provide quantified results on inference time,  GPU memory footprint etc. \n\n6) Please provide the GPU hours required to compute $\\zeta$ and $\\alpha$ for various model sizes.\n\n7) This paper's model sizes range from 125M to 30B. Most of these models can be loaded directly into an A100 GPU. Can you test your approach on a larger model, e.g., a 66B or 175B model? \n\n8) Please extend your evaluation to cover some of the latest benchmarks like TruthfulQA and MMLU.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for the valuable feedback that will be helpful in strengthening our paper.\n\nQ1. What is the precision of variables $S$, $Z$, $A$ in equations 8 and 10? If $W_{int}$  has the same precision, why not directly determine the correct values for $W_{int}$ based on SGD or Adam?\n\nA1) Generally, Post-training quantization can be divided into two stages: 1) setting the step size and 2) Optimization, where we can optimize $W_{int}$ by gradient-based optimization using SGD or ADAM. Eq (8) and (10) describe how to set the step size with newly introduced Z (or Zeta) along with A (or alpha) that is commonly used in per-channel quantization, where $A$, $Z$, and $S=(A\\circ Z)$ are FP16. After setting the step sizes, we can optimize W with a calibration set using existing algorithms such as OPTQ (or GPTQ) and AdaRound. In other words, we are contributing to the process of determining the step size, and then we can choose one of the optimization algorithms for quantization, of which we adopt OPTQ, which is a time-efficient solution. Unlike the existing algorithm, we use the Hessian to determine the step size as in Eq (29) and introduce Z (or zeta) parameters to quantize models elaborately, which summarizes our contributions.\n\n\nQ2. The variable zeta does not consume a lot of memory, and there are few benefits to fold zeta into previous modules, such as normalization and linear layers in Figures 4 and 5? Can you provide more specific experiments/statistics to demonstrate the claim?\n\nA2. While Z-Fold introduces the additional parameter $\\zeta$ in the quantization, the additional cost (e.g., DRAM R/W) is not incurred by $\\zeta$ at the inference time since $\\zeta$ can be fused into the parameters of other layers (e.g., LayerNorm or Linear) before the inference.\nFor example, when folded into the Linear layer (see Figure~5), all we need is to load fused parameters $\\zeta_i\\alpha_i$ instead of $\\alpha_i$, which means that additional DRAM R/W operation is not incurred by $\\zeta$ at the inference.\nTherefore, the total cost of Z-Fold is exactly the same as those required by other conventional per-channel quantization methods (e.g., OPTQ).\n\nQ3. I do not understand the feature in “Z-Fold fully utilizes the feature of the Transformer structure”. Does it refer to the pre-LayerNorm setting? If so, how about post-LayerNorm? \n\nA3. In our approach, we exploit the features of the Transformer structure to avoid the additional computational cost incurred by the newly introduced parameter $\\zeta$.\nOne such feature is the pre-LayerNorm that the reviewer pointed out.\nSpecifically, we incorporated $\\zeta$ of the layers right behind the normalization layer (e.g., $W_q$, $W_k$, $W_v$ and the linear layer $W_1$; see Figure 1) into the parameter $\\gamma$ of the normalization layer (see Figures 3 and 4).\nThe other feature that we utilized is the successive linear layers.\nSince there is no non-linear function between two linear layers (i.e., the value embedding $W_v$ and the output embedding $W_o$, and the linear layers $W_1$ and $W_2$; see Figure 1), $\\zeta$ of later layers $W_o$ and $W_2$ could be folded into the previous layers $W_v$ and $W_1$, respectively (see Eqs. (26) and (28)).\nIn summary, by utilizing the pre-LayerNorm architecture and the successivity of linear layers, we fused all $\\zeta$'s inside the Transformer block into the previous modules offline not at inference time.\n\nWhen the post-LayerNorm is used or a non-linear activation function exists between linear layers (e.g., GELU in BLOOM models), it is not easy to directly fuse $\\zeta$ into the previous modules.\nIn such cases, we quantize unfoldable weights using only out-channel step size $\\alpha$, but $\\zeta$ can still be used for foldable pairs (e.g., value and output embeddings ($W_v$, $W_o$) and linear layers with ReLU activations ($W_{1}$-$\\text{ReLU}$-$W_{2}$)).\n\nFinally, we would like to mention that since pre-LayerNorm Transformer structures, rather than post-LayerNorm architectures, have been prevalent in the latest AI models regardless of the domain (language or vision), the proposed Z-Fold can be also utilized for the quantization of various AI models.\n\n\nQ4. Please describe the hardware evaluation platform, software used, benchmarks and evaluation objectives.\n\nA4. All experiments have been conducted on NVIDIA A100 GPU (80~GB), and we implemented the proposed Z-Fold using PyTorch.\nWe quantized pre-trained LLMs (e.g., OPT and BLOOM) with Z-Fold and used perplexity as a metric to evaluate the performance of quantized models.\nWe compared our results with several public LLMs (such as OPTQ and BLOOM) and also used various zero-shot tasks (e.g., LAMBADA, PIQA, Story Cloze, and ARC) to verify the generalization performance of quantized models (see Tables 5-9). \nWe will add the details of our experimental setups to the main manuscript.\n\n\nQ5. Please provide quantified results on inference time, GPU memory footprint etc.\n\nA5. Since the proposed Z-Fold does not impose any additional memory or latency (see A3), the model quantized via Z-Fold can be implemented by exploiting well-known runtime implementation (e.g., ``llama.cpp'') or various dedicated kernels without any modification.\nFurthermore, the inference time and memory footprint of Z-Fold would be exactly the same as those of other per-channel quantization schemes on which Z-Fold is applied.\nThis means that if Z-Fold is used together with OPTQ, then as reported in [OPTQ paper], the obtained INT3 models would perform the inference 3.2 times faster than FP16 models.\nAlso, if Z-Fold is applied to Llama, the obtained INT4 models would achieve 2.3 times reduction in the inference time on MacBook M1 (see https://github.com/ggerganov/llama.cpp).\n\n\nQ6. Please provide the GPU hours required to compute and for various model sizes.\n\nA6. When compared to the existing method (e.g., OPTQ), the proposed Z-Fold requires additional processing time to determine step size parameters $\\zeta$ and $\\alpha$.\nIn our method, this is done by taking the alternative least squares (ALS) method (see Algorithm 1), and we set the number of iterations to 15 in our experiments.\nWe measure the actual processing time to perform ALS and summarize the result as follows:\n\nModel-OPT@A100|13B|30B|66B\n--|--|--|--\nOPTQ|20.0m|44.0m|1.6h\nZfold+OPTQ|48.7m|1.7h|3.6h\n\n\nModel-BLOOM@A100|1.7B|3B|7.1B\n--|--|--|--\nOPTQ|2.9m|5.2m|10.0m\nZfold+OPTQ|4.7m|8.3m|16.2m\n\nQ7-Q8 Additional experiments\n\nA7-A8. We conducted more experiments as below.  We measure the quality of models when bit-width equals 2 and add the counterpart and benchmark to verify our performance as follows:\n\nllama - WikiText2 (PPL ($\\downarrow$) |[W4]|7B|13B|30B|[W3]|7B|13B|30B|[W2]|7B|13B|30B\n--|--|--|--|--|--|--|--|--|--|--|--|--\nFP16||5.68|5.09|4.10||5.68|5.09|4.10||5.68|5.09|4.10\nRTN(Min-Max)||6.29|5.53|4.54||25.62|11.78|14.87||106441|57256|26805\nOPTQ||6.09|5.36|4.45||8.07|6.63|5.69||74.81|63.89|39.86\nZ-Fold+OPTQ||5.95|5.24|4.31||6.66|5.75|4.87||15.41|19.11|9.64\n\n\nllama - C4 (PPL ($\\downarrow$)|[W4]|7B|13B|30B|[W3]|7B|13B|30B|[W2]|7B|13B|30B\n--|--|--|--|--|--|--|--|--|--|--|--|--\nBase||7.34|6.80|6.13||7.34|6.80|6.13||7.34|6.80|6.13\nRTN(Min-Max)||8.12|7.23|6.54||30.85|14.46|30.10||107515.00|58832.00|27869.00\nOPTQ||8.35|8.75|7.56||14.42|11.24|8.85||80.36|78.96|31.62\nZ-Fold+OPTQ||7.65|6.97|6.28||8.57|7.51|6.84||17.78|16.88|16.52\n\n\nllama - PTB (PPL ($\\downarrow$)|[W4]|7B|13B|30B|[W3]|7B|13B|30B|[W2]|7B|13B|30B\n--|--|--|--|--|--|--|--|--|--|--|--|--\nBase||10.12|9.08|8.16||10.12|9.08|8.16||10.12|9.08|8.16\nRTN(Min-Max)||11.25|9.78|8.65||98.75|28.94|28.77||99310.00|80496.00|32852.00\nOPTQ||10.89|11.43|9.57||10.89|15.28|11.02||75.89|132.73|53.33\nZ-Fold+OPTQ||10.59|9.30|8.33||12.59|10.12|8.87||29.41|28.93|12.77\n\nTruthfulQA-mc1 | INT4  | 7B | 13B | 30B | INT3   | 7B | 13B | 30B\n-- | -- | -- | -- | -- | -- | -- | -- | --\nBase |  | 22.03 | 25.70 | 28.27 | | 22.03 | 25.70 | 28.27\nOPTQ | |21.79 | 24.85 | 28.52 || 20.69 | 23.75 | 23.50\nZ-Fold+OPTQ || 23.26 | 25.34 | 27.05 ||20.56 | 25.46 | 25.95\n\n\nTruthfulQA-mc2 | INT4  | 7B | 13B | 30B | INT3   | 7B | 13B | 30B\n-- | -- | -- | -- | -- | -- | -- | -- | --\nBase | |34.07 | 39.91 | 42.77 || 34.07 | 39.91 | 42.77\nOPTQ || 33.57 | 35.96 | 43.01 || 31.45 | 37.80 | 37.68\nZ-Fold+OPTQ || 35.41 | 39.90 | 42.38 || 31.18 | 39.22 | 39.19\n\n\n\ndue to limited time, we were not able to complete all the experiments you suggested. We will conduct additional experiments with the latest benchmark and the results will appear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "To minimize the objective function $||W-S \\odot W_{quant}||_{F}^{2}$, the proposed approach adopts a matrix decomposition approach, representing $S$ through two vectors: $\\zeta \\in R^{m \\times 1}$ and $\\alpha \\in R^{n \\times 1}$. It utilizes the Min-Max and MSE methods to compute $\\zeta$ and $\\alpha$, respectively. This technique involves a low-rank decomposition combined with quantization, enabling efficient optimization of the weights. By breaking down the complex problem into two simpler vector representations and leveraging appropriate methods, the work provides an effective and computationally feasible approach to enhance the quantization process.", "reasons_to_reject": "Limited novelty. \nPoor evaluation. The majority of baseline results were derived from the GPTQ paper. The paper does not provide quantified results for the most recent benchmarks and models. It does not evaluate the overhead of decomposing stage and the inference performance.", "questions_to_authors": "1) what is the precision of variables $S$, $Z$ and $A$ in equations 8 and 10? \nIf $W_int$ has the same precision, why not directly determine the correct values for $W_int$ based on SGD or Adam?\n\n2) The variable $\\zeta \\in R^{m \\times 1}$ does not consume a lot of memory, and there are few benefits to fold $\\zeta$ into previous modules, such as normalization and linear layers in Figures 4 and 5? Can you provide more specific experiments/statistics to demonstrate the claim?  \n\n3) I do not understand the feature in \"Z-Fold fully utilize the feature of the Transformer structure\". Does it refer to the pre-LayerNorm setting? If so, how about post-LayerNorm?\n\n4) Please describe the hardware evaluation platform, software used, benchmarks and evaluation objectives.\n\n5) Please provide quantified results on inference time,  GPU memory footprint etc. \n\n6) Please provide the GPU hours required to compute $\\zeta$ and $\\alpha$ for various model sizes.\n\n7) This paper's model sizes range from 125M to 30B. Most of these models can be loaded directly into an A100 GPU. Can you test your approach on a larger model, e.g., a 66B or 175B model? \n\n8) Please extend your evaluation to cover some of the latest benchmarks like TruthfulQA and MMLU.", "paper_topic_and_main_contributions": "This paper introduces a post-training quantization (PTQ) methodology, originally referred to as OPTQ (now known as GPTQ). Building upon GPTQ, the paper proposes a new approach for minimizing the mean squared error $||W-S \\odot W_{quant}||_{F}^{2}$, where $\\odot$ denotes the Hadamard product. This is achieved by decomposing the step size $S$ into two scaling factors: $\\zeta$, a vector of dimensions $m \\times 1$, and $\\alpha$, a vector of dimensions $n \\times 1$. Through iterative updates to $\\zeta$ and $\\alpha$, the weights $W$ can be quantized to more optimal values.\n\nThe proposed approach incorporates $\\zeta$ into the weights by employing a folding technique for norm-to-linear and linear-to-linear scenarios. Additionally, this method enables fine-tuning with calibration data by leveraging GPTQ to minimize loss, thereby enhancing the overall quantization performance.\n\nThe experiments in this paper were carried out on the OPT and Bloom models, which vary in size from around 125M to 13B parameters. Employing 4-bit and 3-bit quantization, the proposed Z-Fold(+GPTQ) approach demonstrates superior performance over the RTN and GPTQ baselines across nearly all downstream tasks. These results highlight the effectiveness of the proposed methodology in achieving improved quantization outcomes for various model architectures.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1) In equation2 6, the second $W_1^{q}$ should be $W_2^{q}$ \n\n2) Inconsistent symbols, such as $\\lambda||$ and  $c||$, confuse readers."}, "extra_text": {"Missing_References": "1) GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers\n2) QLoRA: Efficient Finetuning of Quantized LLMs\n3) ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensation\n4) LLM-QAT: Data-Free Quantization Aware Training for Large Language Models"}}, {"review_id": "PwjPUKvfmZ", "reviewer": "Reviewer_GYyE", "summary": "", "questions": "The results in Table 5 show that small OPT Models (125M ~ 1.3B) on LAMBADA has lower accuracy when using the Z-FOLD scheme with INT4 precision, but larger OPT Models (2.7B ~ 13B) have better results. Do you have an idea on the reasons for that?\n", "limitations": "", "rating": 3, "confidence": 2, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's valuable comments and positive evaluation on our manuscript.\n\nQ1. Z-Fold would cause DRAM R/W.\n\nA1. While Z-Fold introduces the additional parameter $\\zeta$ in the quantization, the additional cost (e.g., DRAM R/W) is not incurred by $\\zeta$ at the inference time since $\\zeta$ can be fused into the parameters of other layers (e.g., LayerNorm or Linear) before the inference.\nFor example, when folded into the Linear layer (see Figure~5), all we need is to load fused parameters $\\zeta_i\\alpha_i$ instead of $\\alpha_i$, which means that additional DRAM R/W operation is not incurred by $\\zeta$ at the inference.\nTherefore, the total cost of Z-Fold is exactly the same as those required by other conventional per-channel quantization methods (e.g., OPTQ).\n\nQ2. Some benchmark has lower accuracy when using Z-FOLD.\n\nA2.  In our experiments, we used generic text data (i.e., excerpts from randomly crawled websites), not task-specific data, as a calibration set for the quantization to keep the zero-shot setting.\nThis means that the Hessian matrix computed with the calibration set may differ from the real Hessian matrix, which leads to the inaccurate estimation of the step size. Unlike OPTQ, we use the Hessian to determine the step size as in Eq. (29). Namely, such a case may occur because the assumption does not always hold true.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The results look good, and also seem to be fairly reproducible.\n\n2. Arguments are solid and details are included.\n\n3. This paper also shows a certain level of novelty by proposing the Z-FOLD PTQ scheme.", "reasons_to_reject": "I assume the operation of Z-FOLD would cause DRAM R/W, hence there should be tests to show how this method will affect the memory efficiency.\n\n", "questions_to_authors": "The results in Table 5 show that small OPT Models (125M ~ 1.3B) on LAMBADA has lower accuracy when using the Z-FOLD scheme with INT4 precision, but larger OPT Models (2.7B ~ 13B) have better results. Do you have an idea on the reasons for that?\n", "paper_topic_and_main_contributions": "This paper introduces a straightforward post-training quantization scheme Z-FOLD that could enhance memory efficiency of Transformer structure, and hence enhance LLMs like OPT and BLOOM.\n\nThe main contribution of this paper is that it introduces a new approach for memory efficiency: Folding, which is used between layers to quantize weights into desirebale precision.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "n/a", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "DqIFYWbEuJ", "reviewer": "Reviewer_UHNK", "summary": "", "questions": "1. Why are there no comparisons with LLM.int8() or SmoothQuant in Table 1 and 2? \n\n2. What is the time it takes to learn the scalars and recover accuracy?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for constructive feedback.\n\nQ1. Why are there no comparisons with LLM.int8() or SmoothQuant in Table 1 and 2?\n\nA1. While LLM.int8() and SmoothQuant tried to quantize both weights and activations (i.e., W8A8), Z-Fold focuses on quantizing weights into a lower bit without the activation quantization (i.e., W4A16). \neach quantization format has its own advantages.\nSpecifically, noting that the LLM inference can be divided into the context and decoding stages, the W8A8 format could be a good option for the context stage, where compute-bound GEMM operations are dominant (i.e., multiple tokens are processed jointly). Thus, W8A8 would be more suitable in the stage.\nIn contrast, the W4A16 format can be better for the decoding stage.\nSince tokens are generated one-by-one in the decoding stage, memory-bound GEMV operations are dominant. Thus, reducing memory footprint by quantizing weights into a lower bit could be more beneficial than quantizing activations in this case.\nIn short, it is not an apple-to-apple comparisons between two format, which were also discussed in the appendix of SmoothQuant. \nInstead, we compare our method with other approaches targeting W4A16.\n\nLLAMA-PIQA (Acc (%) $\\uparrow$) |[W4A16]|7B|13B|30B\n--|--|--|--|--\nFP16||76.18|79.06|82.64\nRTN(Min-Max)||77.3*|79.1*|80.1*\nOPTQ||78.24|79.82|80.09\nLLM-QAT||78.3*|79.4*|81*\nZ-Fold+OPTQ||79.27|80.20|81.23\n\n\nLLAMA-HellaSwag (Acc (%) $\\uparrow$) |[W4A16]|7B|13B|30B\n--|--|--|--|--\nFP16||76.18|79.06|82.64\nRTN(Min-Max)||72.7*|76.8*|81.6*\nOPTQ||74.25|77.90|80.85\nLLM-QAT||74*|77.7*|81.8*\nZ-Fold+OPTQ||74.61|78.28|82.13\n\n\n\nLLAMA-C4 (PPL, $\\downarrow$)|[W4A16]|7B|13B|30B\n--|--|--|--|--\nFP16||7.34|6.80|6.13\nRTN(Min-Max)||8.12|7.23|6.54\nOPTQ||8.35|8.75|7.56\nLLM-QAT||7.4*|6.5*|6.5*\nZ-Fold+OPTQ||7.67|6.97|6.28\n\n\nOPT-WikiText2 (PPL, $\\downarrow$) |[W4A16]|125m|350m|1.3b|2.7b|6.7b|13b|30b\n--|--|--|--|--|--|--|--|--\nWFP16||27.65|22.00|14.63|12.47|10.86|10.13|9.56\nRTN(Min-Max)||37.28|25.94|48.17|16.92|12.10|11.32|10.98\nOPTQ||31.12|24.24|15.47|12.87|11.39|10.31|9.63\nZeroQuant-V2||36.71|25.51|19.38|17.92|11.91|10.67|10.10\nZfold+OPTQ||30.26|22.90|14.88|12.56|10.95|10.26|9.59\n\nBLOOM-WikiText2 (PPL, $\\downarrow$) |[W4A16]|560m|1.1b|1.7b|3b|7.1b\n--|--|--|--|--|--|--\nWFP16||22.42|17.69|15.39|13.48|11.37\nRTN(Min-Max)||25.90|22.00|16.97|14.76|12.10\nOPTQ||24.03|19.05|16.48|14.20|11.73\nZeroQuant-V2||25.31|23.90|16.93|14.65|12.06\nZfold+OPTQ||23.84|18.82|16.24|14.01|11.70 \n\n\n\n\n\n\n\n\nFinally, we would like to mention that we chose OPTQ (or GPTQ) as our main counterpart and focused on adding Z-Fold to OPTQ since it is a time- and memory-efficient solution to quantize hyper-scale AI models rapidly and also achieves the SOTA performance regarding the quantization of LLMs.\nNevertheless, we emphasize that the proposed Z-Fold can also be applied to any other quantizers such as AdaRound, BRECQ, and AdaQuant, in addition to OPTQ.\nWe believe that no matter which quantizer we choose, Z-Fold could be used to enhance the quantization performance of the underlying quantizer.\nThus, it might be more suitable to evaluate our paper by focusing on how much Z-Fold can enhance the performance of existing quantizers, not being evaluated by absolute figures.\n\nC4  (INT2), OPT models | 125M | 350M | 1.3B | 2.7B | 6.7B\n-- | -- | -- | -- | -- | --\nOPTQ | 2260.00 | 8418.00 | 4028.00 | 4388.00 | 500.70\nOPT+Zfold | 215.17 | 88.03 | 41.04 | 25.46 | 18.93\n\n\n\nWikiText2 (INT2) OPT models | 125M | 350M | 1.3B | 2.7B | 6.7B\n-- | -- | -- | -- | -- | --\nOPTQ | 4563.00 | 18687.00 | 7856.00 | 8949.00 | 2958.00\nOPT+Zfold | 255.06 | 134.18 | 46.53 | 25.30 | 16.32\n\n\n\nQ2. What is the time it takes to learn the scalars and recover accuracy?\n\nA2. When compared to the existing method (e.g., OPTQ), the proposed Z-Fold requires additional processing time to determine step size parameters $\\zeta$ and $\\alpha$.\nIn our method, this is done by taking the alternative least squares (ALS) method (see Algorithm 1), and we set the number of iterations to 15 in our experiments.\nWe measure the actual processing time to perform ALS and summarize the result as follows:\n\nModel-OPT@A100|13B|30B|66B\n--|--|--|--\nOPTQ|20.0m|44.0m|1.6h\nZfold+OPTQ|48.7m|1.7h|3.6h\n\n\nModel-BLOOM@A100|1.7B|3B|7.1B\n--|--|--|--\nOPTQ|2.9m|5.2m|10.0m\nZfold+OPTQ|4.7m|8.3m|16.2m", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A very easy to implement idea", "reasons_to_reject": "1. The authors dont evaluate against accepted baselines (see questions for Authors). As a result it is hard to understand the value add this work provides. The value add over OPTQ only show when we move from INT4 to INT3, which does not provide a significant improvement in terms of storage reduction.", "questions_to_authors": "1. Why are there no comparisons with LLM.int8() or SmoothQuant in Table 1 and 2? \n\n2. What is the time it takes to learn the scalars and recover accuracy?", "paper_topic_and_main_contributions": "This paper proposes a post training quantization scheme by adding additional scaling factors to the input channel of the matrix. These additional scaling factors can be folded into the scaling factors on the output channel of previous layer. To learn these scaling factors that minimize the distance from the original weights, the authors use alternating least square factorization. To recover accuracy using a calibration set, the authors use technique proposed in OPTQ. \n\nThe authors evaluate their technique on BLOOM models (upto 7.1B) and OPT models (upto 30B)", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a simple PTQ pipeline for the LLM quantization. Reviewers initially had concerns about missing experiments, references, and limited novelty. The authors provided a rebuttal to address them. Based on the unanimous agreement among reviewers, the AC deemed this paper should be accepted.
Decision: Accept-Main