QERA: an Analytical Framework for Quantization Error Reconstruction
Authors: Cheng Zhang
Jeffrey T. H. Wong
Can Xiao
George Anthony Constantinides
Yiren Zhao
Venue: ICLR 2025 Poster (2025)
Keywords: parameter-efficient fine-tuning
PEFT
LoRA
QLoRA
quantization
post-training quantization
PTQ
low-rank approximation
Primary area: foundation or frontier models, including LLMs
TL;DR: Analytical solution to low-rank quantization error reconstruction and its applications

Abstract: The growing number of parameters and computational demands of large language models (LLMs) present significant challenges for their efficient deployment.
Recently, there is an increasing interest in quantizing weights to extremely low precision while offsetting the resulting error with low-rank, high-precision error reconstruction terms.
The combination of quantization and low-rank approximation is now popular in both adapter-based, parameter-efficient fine-tuning methods such as LoftQ and low-precision inference techniques including ZeroQuant-V2.
Usually, the low-rank terms are calculated via the singular value decomposition (SVD) of the weight quantization error,
minimizing the Frobenius and spectral norms of the weight approximation error.
Recent methods like LQ-LoRA and LQER introduced hand-crafted heuristics to minimize errors in layer outputs (activations) rather than weights, resulting improved quantization results.
However, these heuristic methods lack an analytical solution to guide the design of quantization error reconstruction terms.
In this paper, we revisit this problem and formulate an analytical framework, named Quantization Error Reconstruction Analysis (QERA),
and offer a closed-form solution to the problem.
We show QERA benefits both existing low-precision fine-tuning and inference methods --
QERA achieves a fine-tuned accuracy gain of $\Delta_{\text{acc}}$ = 6.05\% of 2-bit RoBERTa-base on GLUE compared to LoftQ;
and obtains $\Delta_{\text{acc}}$ = 2.97\% higher post-training quantization accuracy of 4-bit Llama-3.1-70B on average than ZeroQuant-V2 and $\Delta_{\text{ppl}}$ = $-$ 0.28 lower perplexity on WikiText2 than LQER.

Published as a conference paper at ICLR 2025
QERA: AN ANALYTICAL FRAMEWORK FOR QUANTI- ZATION ERROR RECONSTRUCTION
Cheng Zhang, Jeffrey T. H. Wong, Can Xiao, George A. Constantinides & Yiren Zhao Department of Electrical and Electronic Engineering
Imperial College London
London, UK (cheng. zhang 12,, ssz 1izzworr wong20, can. xiao22, g. constantinides,, a. zhao) ac. uk
ABSTRACT
The growing number of parameters and computational demands of large lan- guage models (LLMs) present significant challenges for their efficient deploy- ment. Recently, there is an increasing interest in quantizing weights to extremely low precision while offsetting the resulting error with low-rank, boghh-precision error reconstruction terms. The combination of quantization and low-rank ap- proximation now popular both adapter- adapter-based, parameter-efficien fine-tuning methods such as LoftQ (Li et al. (023) and low-precision inference techniques including ZeroQuant-V V2 (Yao et al. 2023). Usually, the low-rank terms are cal- culated via the singular value decomposition (SVD) of the weight quantization error, minimizing the Frobenius and spectral norms of the weight approximation error. Recent methods like LQ-LoRA (Guo et al. 2023) and LQER (Zhang et al.) 2024a) introduced hand-crafted heuristics to minimize errors in layer outputs (ac- livations) rather than weights, resulting improved quantization results. However, these heuristic-based methods lack an analytical solution to guide the design of quantization
reconstruction terms. In this paper, we revisit this problem and formulate an analytical framework, named Quantization Error Reconstruc- tion Analysis (QERA), and offer closed-form solution to the problem. We show QERA benefits both existing low-precision fine-tuning and inference methods QERA achieves fine-tuned accuracy gain for A = 6.05% of 2-bit RoBERTa- base on GLUE compared to LoftQ; and obtains Aace 2.97% higher post-training quantization accuracy of 4-bit Llama-3. 1--0BB compared to ZeroQuant-V V2 and Appl 0.28 lower perplexity WikiText2 compared to LQER. We open-source our code and models at ggithub.com/ChennZhanggg 98//ERAA
INTRODUCTION
The demand for efficient deployment of large language models (LLMs) has been increasing (Faiz jet al.) 2023). LLMS now typically contain billions parameters (Kaplan et al. 2020 Dubey et al. 2024), making their fine-tuning and inference computationally expensive and resource- intensive (Ding et al., 2023). To address these challenges, there has been a surge of interest in building efficient fine-tuning and inference methods. One popular formulation is apply low- rank term to reconstruct the error after quantization. Given a linear layer y = æW, the weight matrix W E m is quantized to W, and we rewrite W R W + B₁ such that both A and B₁ are low-rank yet high-precision terms with rank is < min(m,n).
We call the problem of finding the optimal A₁ and B quantization error reconstruction. Inter- estingly, this problem has, coincidentally, seen widespread application in two actively researched areas: quantized parameter-efficient fine-tuning (QPEFT) and post-training quantization (PTQ) for model inference. QPEFT refers to fine-tuning techniques that adapt LLMs to specific tasks by quan- tizing pretrained weights and updating only a small number of extra parameters, hence significantly reducing memory requirements and training time, such as QLoRA (Guo et al. (003). On the other side, PTQ is training-fre free method that reduces the model size and may accelerate the forward pass the underlying hardware supports it. Recently, researchers combined PTQ with quantization error
reconstruction Yao al. 2023 Liu et al. 2023al Zhang et al. 2024a)
further reduce weight

---

Published as a conference paper at ICLR 2025
precision. Works such ZeroQuant- V2 Yao et al. 2023) and LQER (Zhang al. 2024a) have shown that adding a high-precision low rank component, as low as 8 or 32, can recover considerable model performance for 3- or 4-bit weight quantization.
Although both the QPEFT and PTQ methods have demonstrated substantial performance improve- ments in lowering the computational overhead of LLMs, a theoretical analysis of quantization error reconstruction is lacking. Usually, A and B₁ are calculated by applying truncated singular value decomposition (SVD) to the weight quantization error (W W). mínimizing the Frobenius and spectral norms of the weight approximation error. However, recent work on activation-aware quan- tization and knowledge distillation implies that minimizing layer output error may lead to a greater
performance gain than minimizing weight approximation error (Lin et al. 2024 Liu et al. 2023a
Shao et al. 2023).
Besides the unsettled minimization objective, it has remained unclear whether there exists a theo- retically optimal solution for the values of A& and Bg, and if so, how can solve for it. A better initialization or theoretically grounded initialization of A₁ and Bg brings direct benefits for both OPEFT and PTQ. In QPEFT, the initialization of LoRA (Hu et al., [021], which uses element-wise Gaussian random values for As and zeros for BR, struggles under aggressive quantization since the quantization error can derail fine-tuning. In PTQ, the quantized model performance is based on the computation of the low-rank terms, given a specific quantization function q(-) and rank k.
In this paper, aim to provide an analytical framework for the quantization error reconstruction problem. To demonstrate the effectiveness of our theoretical framework, we further apply our ana- lytical solutions to state-of-the-as ofrtt--art QPEFT and PTQ methods and show the significant performance improvements under the same computational budget. Specifically, our contributions are as follows:
We show that the commonly used objective for solving the quantization error recon- struction problem in prior work i.e., minimizing the weight approximation error (e.g.,
I - WlIp), does not guarantee a reduced model output error. Instead, we show that minimizing the layer output error (e.g., ly pl is closely related to minimizing the model output error.
We derive the analytical solution to the low-rank terms A₁ and B₁ by minimizing the layer output error. We demonstrate that under statistical assumption, this solution can be found in a particularly computationally efficient manner, also explaining the success of LQER.
We empirically demonstrate the effectiveness of our solutions by applying them to state- of-the-art art QPEFT and PTQ methods. Our analytical framework, QERA, significantly im- proves the performance of these methods. For example, QERA achieves Aace = 6.05% higher accuracy of 2-t RoBERTa-bass on GLUE compared to LoftQ. improving the fine- tuning accuracy and efficiency. Moreover, QERA obtains A 2.97% higher accuracy than ZeroQuant- V2, when quantizing LLaMA-3-70B to bits, averaged across six tasks. This narrows the model performance gap between emoorreconssruucioon-beee post-training quantization and full-precision models.
2 RELATED WORK
In this section, we review the existing methods that combine weight quantization and low-rank enror reconstruction. These methods can be roughly categorized into two groups based on their applications: QPEFT for training and PTQ for inference.
1021 is a representative PEFT method that introduces train- LoRA and QPEFT LoRA Hu et al. able low-rank terms to adapt the model to specific task. Take linear layer as an example.
y = ( As
where W m is the pretrained weight matrix, row vector x I and y E R" are the input and output, and Ar E mm and B, E 4 are trainable low-rank matrices ("adapter") with rank & & min(m, n). During fine-tuning, the pretrained W is frozen and only the adapter A and Bş are updated. To make the adapted layer's 's output match the original one at the start of fine- tuning, LoRA initializes Aş with Gaussian random values and Bg with zeros. Once the fine-tuning completed, the adapter merged into the pre-trained weights.
2

---

Published as a conference paper at ICLR 2025
QLoRA (Guo et al., 2023) extends LoRA by quantizing the pretrained weights stored in GPU mem- ory to reduce memory footprint.
W₂ (W)
(2)
One difference between QLoRA and LoRA is that during fine-tuning, Wa needs to be dequantized before involved into matrix multiplications:
W = dd ,), x: ( + BB)
(3)
where dq(-) is the dequantization function. QLoRA introduces weight quantization error W
W), shifting the starting point of fine-tuning. To address this problem, LoftQ (Li al. 2023) initializes the adapter using the SVD-based low-rank approximation of (W W) to reduce the weight approximation error:
arg min W - Ax B,
Specifically, LoftQ uses a heuristic-basee algorithm to iteratively update the quantized weights and the adapter (Algorithmm]]in the Appendix). Their experiments show that larger number of iterations leads to a smaller weight error.
LQ-LoRA (Guo et al. 2023) also adopts LoftQ's iterative method but keeps track of scaled variant of the objective, arg min B, || row W W AA, BB Deall F. where Drow and cot are heuristic homogenous row/column matrices from activation statistics. LQ-LoRA exits the iteration when the scaled objective function stops decreasing due to the lack of a theoretical justification for LoftQ.
Quantization Error Reconstruction for PTQ Similar to the forward pass of fine-tuning in QLoRA, there are also PTQ methods that quantize the pretrained weights to low-precision for- mats and recover the model performance with additional low-rank terms. With a small enough rank k, the additional computation introduced is negligible. Note that unlike QPEFT which can utilize fine-tuning to correct the quantization error, PTQ methods aim to recover the model performance as much as possible without any training.
V2 ZeroQuant-V V2 ( Yao et al.) 2023) is the earliest weight-only quantization method introducing low- rank quantization error reconstruction the PTQ problem. They apply SVD the weight quanti- zation error (W- W) to calculate A₁ and B, (equivalent to LoftQ with one iteration). Combin- ing low-rank terms and fine-grained quantization, ZeroQuant-V2 recovers the performance of 4-bit LLMs to a level comparable to 8-bit.
Recent quantization works have shown that activation statistics play crucial role in weight-only LLM quantization (Liu et al., 2023b Lin et al., 2024). QLLM (Liu et al.) 2023a) trains the low- rank terms using gradient descent with loss function that minimizes the output error of the at- tention layer. LQER (Zhang et al.] 2024a) applies an activation-induced heuristic scale matrix S to the quantization error before calculating SVD, UEVT = SVD(S(W W)), and as- signs A := S- B₁ := Ex.. V.k., (Refer to Algorithm 2 in the Appendix). LQER achieves significant improvement over ZeroQuant- V2 and observes that in some layers singular values are shaped toward a more desirable distribution where singular values decay faster. Note that ZeroQuant- V2 can also be considered as special case where in LQER is an identity matrix. To our knowledge, CALDERA (Saha et al. (024) is the concurrent work close to ours. CALDERA focuses on a different problem setup find optimal W, As. B, all in low-precision formats that minimizes output error, with a lemma agreeing with our exact solution. We elaborate the connection and difference between CALDERA and QERA Appendix A.3
In summary, to solve the quantization error reconstruction (QER) problem, most of existing methods target the minimization of the weight approximation error. Several recent works such as LQ-LoRA, QLLM, and LQER introduce activation-induced heuristics to the calculation of adapters/low-rank terms, but justification for the optimization objective and the corresponding analytical frame- work are still missing.
OUR ANALYTICAL FRAMEWORK
In this section, we formulate the optimization objective of quantization error reconstruction and derive the analytical solution to the low-rank term Ck Br
3

---

Published as a conference paper at ICLR 2025
3.1 PROBLEM STATEMENT
Given a pretrained linear layer - a: W with input vector æ E R", output vector y E IK" and weight matrix W E R our aim is to approximate it with high-rank low-precision W and low-rank high-precision term Cj E m with rank & < min(m, n).
ÿ = x: W Ck)
(5)
This raises the question of actual optimization target: Should we minimize the weight recon- struction error | - F or the output reconstruction error y Ul2 We separate these two problems and introduce them formally below.
Problem 1 (Minimization of weight emor). For pretrained linear layer y æW and its ap- proximated form y z(W W + C*), reconstructing the quantization error by minimizing weight approximation error has the following objective:
arg min I| C2
-- Cx
where || || denotes the Frobenius norm.
Solution to Problem I From the Eckart- Eckart-Young-- Young-Mirsky theorem (Eckart & Young 1936). the optimal solution to Problem with respect to rank k the truncated SVD of the weight error matrix:
C₁ U k. V.J.:
where U, Σ, and V form the SVD of the weight quantization error, UEVT SVD(W W).
As noted in Section 2 most existing works (Li et al. 2023 Yao et al. 2023 Guo et al. 2023 in QPEET and PTQ adopt this solution. However, we know that minimizing the weight approximation enror is not equivalent to minimizing the layer output error. Furthermore, does minimizing the weight approximation error for each layer in a network effectively reduce the final model output error? We will show that the answer is negative in Section 4.2
Problem 2 (Minimization of layer output error). For a pretrained linear layer æW and its approximated form ÿ x W + Ck), approximating the layer by minimizing the error between y and ý is to minimize the following expectation.
arg E y112) C2
(8)
where || 112 denotes la norm, and Y C RR is output space of the layer. We expand Equation (8) by substituting and y:
arg min Ex-x [ W + C²) x
C₁
where C R" is the input space. practice, the expectation can be approximated as sample mean on a calibration dataset like a subset of the pretraining data set.
Problem 2 motivates some recent works (Liu et al. 2023a Guo et al. 2023 Zhang et al. 2024 to
involve activation-inducel heuristics in the optimization of C2 but without theoretical foundation.
In the following two sections, we will derive the analytical solution to Problem [ More precisely, present solutions: exact solution in Section 3.2 and approximated solution based on
a suitable statistical assumption in Section 3.3
3.2 QERA-EXACT: ANALYTICAL SOLUTION
QERA-exact is our exact solution Problem 2 QERA-exact is computationally expensive as calculates the autocorrelation matrix of the input space X. However, we will show in Section QERA-exact recovers significant model performance in extremely low-precision quantization.
Theorem (QERA-exact solution).
The solution Problem is
C₁
R
U.ik E.cck
(10)
4

---

Published as a conference paper at ICLR 2025
where Ryg is the autocorrelation matrix respect to the input space X.
Rxx Ex-x { (x" (x"x)
(11)
Rix represents the unique symmetric positive semi-definite matrix square root of Rxx, and Un Dak,ko and form the truncated SVD of the following scaled weight error matrix,
UEV SVD( (W--))
(12)
Remark 1. Rigx positive semi-definite. In the event that it has zero eigenvalue, would be normal to add a small diagonal perturbation to recover invertibility. In practice, we ran extensive experiments and find that Rixx is invertible for all the pretrained models and datasets we present in Section 4
Proof of Theorem
Proof. Define P = W C - W. and p: == P.. is the i-th row of P. Then we substitute
(W + Ck W) in the expanded objective Equation (9 of Problem with P:
B y1l2) Ex (
Rexx (1)
i=1
(13)
Exrx
2
Pip
j=1
We rewrite the last line of Equation 13 as:
Exsoy y
Em-x {e: ((x"x)) (PP")) .e²
(14)
where e = [1 1 1] is row vector of m ones, and @ denotes the element- product.
Using the property of the element-wiss product (Styan 1973), the RHS of the above can be simpli- fied.
Ey-r (1 y112 Exnx { Tr ((x (PPT))))
Tr (Bo~x x (x=x)
x) PPT)
(15)
Tr (Rxx
where Tr(.)) denotes trace and Rxx E { x at) is the autocorrelation matrix with respect the input space X.
Since Rxx is symmetric positive semi-definite matrix, always has precisely matrix square root, denoted Rix. that is also symmetric and positive semi-definite (Horn & Johnson, 2012). We reorganize Equation 15 as the following since both Ryx and (PP?') are symmetric and positive semi-definite:
Ey-zill
=11 = Tr R
PP
R
Tr
R
PP
R
(16)
Now the objective of Problem 2
Equation 8
is equivalent to:
min E
v1
arg min C.
R 2
C
(17)
arg min C₁
R W + C. W
5

---

Published as a conference paper at ICLR 2025
we assign Q == R W W) and Q₁ == R2x Ck. the objective becomes:
arg min ||Q
Q1
(18)
Qx
Note that multiplication by the invertible matrix R (Remark 1 does not change the rank of the matrix Cs. According to the Eckart- Eckart-Young- Young- Boung-Mirsky theorem (Eckart & Young 1936), the optimal rank & approximation to Qx is the truncated SVD of Q:
Q: = U.. Sekcck V.k.:
(19)
where UEV = SVD(Q) = SVD
R Rxx (W- - W))
Thus the optimal rank-k solution to C3 is:
C = R
Qs = (R U.
(20)
In practice, we assign A
R
U and B₁ d V se.." Note that QERA adds
constraints the quantization (and dequantization) function q(-) (and dq(-)), ie., the low-precision W can be obtained by any quantization method.
3.3 QERA-APPROX: AN ANALYTICAL SOLUTION WITH THE UNCORRELATED ASSUMPTION
DERA-approx is analytical solution to Problem 2 based on the assumption that different em- bedding dimensions are uncorrelated. This solution is more computationally efficient than the exact solution, and the assumption is testable real-world datasets. The complete proof of QERA-approx in Appendix A.2
Assumption 1. For a pretrained linear layer y as W, the expectation of the product of different embedding dimensions is zero:
E F = 0, Vi +
(21)
where 2; and xy the i-th and j-th elements of the input vector
We test this assumption on L.MSs in Section 5
Theorem (QERA-approx solution). The solution Problem 2 based on Assumption 7 is:
C = S- Ex
(22)
where S is a diagonal matrix built from activation statistics,
S diag( V /Baxxaa), V /Ea-x(=?),... E xx
(23)
and U, E, V form the SVD of the following scaled weight error matrix,
UEVT = SV -
(24)
Remark 2. For the diagonal matrix S in Theorem 2 to be invertible, we need Ex-x(a?) # 0 for all dimension i. practice, this is almost always true for pretrained layers because no dimension in the input embeddings is always zero.
For implementation, we assign A := S and B; = Ex V.R.: to form the low-rank terms to save the memory and computation cost. Interestingly. QERA-approx solution is similar to the activation-induced heuristics LQER (Zhang al. 2024a), which calibrates the average absolute value on the embedding dimension (Refer to Algorithm 2 in the Appendix). In Section 4.3) we will show that our solution is more effective in practice and resolves the discrepancy between the recovered model performance and the number of calibration samples in LQER.
EXPERIMENTS
In this section, we first introduce the experiment setup Section 4.1 Then we present the results of our experiments QPFFT and PTQ in Section 4.2 and Section 4.3 respectively.

---

Published as a conference paper at ICLR 2025
22
4-bit Weight
3-bit Weight
3.5
-K- OLORA
--- QUORA
Model output error
18
Lotto (mm4)
2.0
Luft [3-iar]
Model outgut
Loft (5-ibr)
Lotto (x-16)
2.5
QERA
14
OERA (km4)
QERA (k-8i
2.0
QENA (k-161
1.5
0
1.0
Rank
LoftQ num iterations
(a) Model output error rank
(b) Model output error LoftQ iterations
The model output error of RoBERTa- base before fine-tuning. We feed 128 samples from RoBERTa's 's pretraining dataset and profile the output logits error between the adapted and the FP32 model. We sweep the rank k and the iteration number of LoftQ on 4-bit and 3-bit models. In LoftQ. neither more iterations approximation eтrоr of every layer decreases. In contrast, QERA-appros consistently has the lowest model output error across all settings, and the error monotonically decreases as the rank increases.
Figure 1:
higher rank guarantees lower model output error, though the weight
4.1 EXPERIMENT SETUP
We perform QPEFT and PTQ experiments separately, and compare with their respective SoTA meth- ods. The experiments take around 6400 GPU hours in total. The hardware platform, separate GPU hours, software dependencies, and random seed settings can be found in Appendix A.4
For QPEFT experiments, we use Theorem noted as QERA-approx, to initialize low-rank terms, and compare with full-finetuning, LoRA (Hu et al. 1021), QLoRA (Dettmers et al., 2024), and LoftQ (Li et al. 2023). Specifically, we adopt 5-iteration LoftQ. which is the officially recom- mended setup. We include both encoder-only model experiments (fine-tuning RoBERTa- base (Liu 2019) on GLUE Ye et al. 2019) and decoder-only LLM experiments (fine-tuning LLaMA-2 (Tou-
vron et al. 2023) and LI 1 (Dubey et al.) 2024 on continuous pretraining task SlimPa- jama (Soooleva et al. 2023) and supervised fine-tuning task GSM8K (Cobbe et al. (202]) For each method/baseline, we sweep the learning rate and record the best result. The final results are averaged over three random seeds. The learning rate ranges and batch sizes are listed in Appendix A.4.I
For PTQ experiments, we use both Theorem [ noted as QERA-exact, and Theorem (QERA- approx) to calculate the low-rank error reconstruction terms and report results separately. We compare with BF16, quantized model without error reconstruction terms (w-only), ZeroQuant- V2 (Yao et al., (2023), and LQER (Zhang et al.) (024a) at different precision setups. We also include HQQ (Badri & Shajil, (003) a leading 4-bit method that does not use quantization error reconstruction. We quantize I.L.Ms of various sizes and model family, including Timy (Zhang ct al. 2024b). Gemma-2 (Team et al. 2024). Phi-3.5 (Abdin et al. 2024) and LLaMA-2/-3.1 (Tou- vron et al.] 2023 Dubey et al.
2024). We use Lm-ovaluat con-harness to report results on ARC (challenge) (Clark et al. 2018), BoolQ (Clark et al. 2019), CommonSenseQAA TTlmor et al. 2019). Winogrande (Sakaguchi et al. 2019). MMLU (Hendrycks)
Wikitext2 MMertyy et al. (016).
(et al. 2021), and BigBench- Hard (Suzgun et al.] 2022). We also evaluate instruction-tun tuned model,
Vicuna-v1.5 (Zheng al., 2023). with AlpacaEval 2.0 (Dubois et al.) 2024), which is automatic
evaluation tool for instruction-follovii tasks. Detailed setup is in Appendix A.4.2
4.2 IMPROVED QPEFT
We first identify a pitfall in the commonly-used iterative Algorithm [] that is, minimizing the weight approximation error for each layer does not necessarily minimize the model output error. Then we show that our QERA initialization enables a clear reduction the model output error at the start of fine-tuning, leading to better fine-tuned accuracy/perplexity and faster convergence.
Reduced layer weight error & reduced model output error We apply 4-bit and 3-bit QLORA, LoftQ, and QERA-approx RoBERTa-base and inspect the model output error on RoBERTa's pretraining dataset before fine-tuning at rank k = 4, 8, 16, 32. For LoftQ, we also sweep the number we observe that
of iterations from
In Figure [
For LoftQ, given a specific rank, increasing the optimization iterations does not guarantee a reduced model output error. Though all the layers' weight approximation errors monoton-
7

---

Published as a conference paper at ICLR 2025
Table I:
Fine-tuning results of RoBERTa-baas on GLUE. QERA-approx outperforms LoftQ across all bit widths, and the improvement more obvious with aggressive quantization. QERA achieves Ance = 4.12% higher than LoftQ at 3-bit and 6.05% at 2-bit.
MNLI Ace
QNLI
RTE
SST
MRPC Acc
CoLA Matt
QQP
STSB
Rank
W-bits
Method
Avg.
Acc
Acc
Acc
Acc
P/S Corr
16
Full FT
87.61
92.95
73.16
94.88
92.15
60.41
91.61
90.44/90.25
85,38
LoRA
87.85
92.84
69.55
94.46
89.99
$7.52
89.83
89.92/89.83
84.00
QLoRA
87.21
92.32
63,90
94.08
88. 88.24
56.08
90.55
$9.59/89.56 .66
82.75
4.25
LoftQ (5-iter)
87.27
92.48
67.13
94.38
88.24
54.59
90.51
88.75/88.79 79
82.92
QFRA-approx
87.28
92.45
70.40
94.38
88.97
55.99
90.39
89.83/89.72
83.71
QLoRA
84.87
89.58
$3.67
91.02
73.94
3.12
89.31
84,80 14.80/84.38
71.29
3.25
LoftQ (5-iter)
85,24
89.65
58.24
92.05
75.82
11.00
88,93
85.55/85.27
73.31
QFRA-approx
85.58
90.74
58.48
92.59
82.19
32.98
89.41
87,43/87.08
77.43
QLoRA
77.87
85.26
54.15
90.02
71.00
87,93
74.72/75.31
67 62.62
64
2.50
LoftQ (5-iter)
80.15
87.65
52.95
90.94
74.35
3.43
89.17
82.76/82.90
70.18
QERA-exact
84.64
90.05
58.48
92.32
84.72
26.43
89.69
86.48/86.40
76.23
Table 2: Fine-tuning results of LLaMA-2-777 2-BB and LLaMA-3.1 1-8B on SlimPajama and GSM8K. A trend similar to RoBERTa experiments are observed, i.e., QERA outperforms QLoRA and LoftQ and the improvement is more obvious on aggressive quantization.
LLaMA-2-7B
LLaMA-3.1-8B
W-bits
Method
SlimPajama (Agu)
GSM8K (Axc)
SlimPajama pApi)
GSM8K (Aucc
16
LoRA
6.17
39.40
8.07
55.72
QLoRA
6,44 (+0.27)
30.71 (-8.69)
8.70 (+0.63)
54.81 (-0,91)
4.25
LoftQ (5-iter)
6.39 (+0.22)
28.58 (-10.82)
8.73 (+0.66)
54.23 (-1.49)
QERA-approx
6.33 (+0.16)
32,26 (-7.14)
8.68 (+0.61)
55.24 (-0.48)
QLoRA
53.95 (+47.78)
12.79 (-18.31)
71.90 (+63.83
5,08 (-50.64)
2.25
LoftQ (5-iter)
12.30 (+6.13)
18.37 (-12.73)
27.16 (+19.09)
13.72 (-42.00)
QERA-approx
10.56 (+4.39)
18.78 (-12.32)
20.07 (+12.00)
19.41 (-36.31)
ically decrease with the number of iterations (as illustrated in Figure in Appendix), the model output error does not monotonically decrease. For example, in Figure la the model output error of LoftQ (5-iter) is larger than LoftQ (3-iter) at rank k 8.
For LoftQ, given a specific number of iterations, increasing the rank does not guarantee reduced model output error. For example, in Figure [b the output error of LoftQ (rank k = 16) is larger than LoftQ (rank & = 4) and & = 8 at 2, 3, 4, and iterations.
The model output error of our QERA-approx always smaller than LoftQ and QLoRA, across all precision and rank settings. Moreover, the output error of QERA-approx mono- tonically decreases as the rank increases.
This empirical evidence suggests a strong correlation between the reduction of layer output error and the decrease model output error QER problem. Conversely, minimizing weight approximation enror using LoftQ does not have a comparable impact overall model performance.
Better optimization quality Table T and Table 2 sum- marize the fine-tuning experiments of RoBERTa- GLUE, and LL 1-88 fine-tuned on SlimPajama and GSM8K, respectively. QERA outperforms both LoftQ and QLoRA. In GLUE experiments, at 4-bit, QERA enables an average accuracy gain of 0.96% and 0.79% higher than
Spearman Correlation
0.8
0.6
0.4
0.2
QLoRA and LoftQ respectively, close to BF16 LoRA; At 3- bit and b-t QERA achieves 4.12% and 6.05% higher aver- age accuracy than LoftQ respectively. Similar trends are ob- served on LLM fine-tuning experiments, i.e., QERA outper- forms OLoA and LoftQ, and the advantage of QERA over LoftQ is more obvious with more aggressive quantization.
Loft (S-ite)
0.0
Q Qppr-x
Epoch
Figure 2: Faster convergence of
QERA-approx on STSB.
Faster Convergence QERA initialization also speeds up the training convergence. For LLM tnn-ttunig, this is expected as QERA initialization closer to the full-precision model. Interestingly, in encoder-only experiments GLUE where the model
8

---

Published as a conference paper at ICLR 2025
Table 3: Perplexity (L) of LLMs on WikiText2. w-only denotes the quantized model without low- rank error reconstruction. QERA-approx outperforms LQER almost all setups and QERA-exact achieves the lowest perplexity. The advantage of QERA is pronounced at 3-bit.
TinyLlama
Gemma-2 2
Phi-3.5
LLaMA-2
LLaMA-3.1
W-bits
Method
Rank
1.1B
2B
3.8B
7B
13B
8B
80B
BF16
13.98
13.08
11.50
8.71
7.68
7.55
3.06
4.25
HQQ
15.02
14.29
14.63
9.59
8.27
8.72
3.97
w-only
19.40
16.23
14.16
9.45
8.06
8.78
4.55
ZeroQuant- V2
18.03
15.71
14.09
9.42
8.07
8.83
4.48
4.25
LQER
16.23
14.55
12.88
9.22
7.96
8.45
4.10
32
QERA-approx
15.66
14.60
12.81
9.17
7.95
8.45
4.10
QERA-cxact
16.16
14.12
12.30
9.12
7.93
8.33
3.82
w-only
32.82
41.13
47.78
13.32
10.24
18.96
16.46
ZeroQuant- V2
27.80
33.56
42.64
13.00
10.03
19.29
10.12
3,25
LQER
20.60
21.99
18.27
14.00
9.09
11.86
7.05
64
QERA-approx QERA-exac
20.43
21.93
17.99
10.99
11.73
6.99
19.51
19.97
20.37
10.67
8.97
11.39
6.68
classifier head is randomly initialized, we also observe that QERA converges faster, especially on small subsets such as STSB and MRPC where only a few thousand samples are available (in compar- ison MNLI has 393k samples and QQP has 364k samples). For example, in Figure the Spearman correlation coefficient of QERA on STSB increases and converges faster than LoftQ and QLoRA, as the green line plateaus first.
4.3 IMPROVED PTQ
In this part, first demonstrate that LQER, which depends on heuristics derived from activa- tion values, does not guarantee improved performance with a larger calibration dataset. However, show that QERA consis-
QERA exhibits the opposite trend. Through extensive experiments, tently outperforms ZeroQuant- V2 and LQER, and QERA-exace exhibits better model performance than QERA-approx the cost of more computation in the guantization process. These results veri- fied the effectiveness of our analytical solution.
Model performance vs. calibration set size As men-
10.78
LQER
tioned at the end of Section 3.3 the scale matrix in
10.76
QERA-approx
LOER (Zhang et al.) 2024a) is similar to the one in QERA- approx, but is based on hand-crafted heuristics. As a result,
Perplexity
10.74
10.72
observe that the model performance of LQER varies randomly as the number of calibration samples increases (the purple curve in Figure 3). On the contrary, more cal- ibration samples consistently lead to better model perfor- mance for QERA until convergence.
10.70
10.68
2a
2
2
Num calibration samples
Figure 3: QERA resolves the discrep- ancy between the recovered model performance and the number of cali- bration samples in LQER.
Improved perplexity and downsteam task accuracy We apply QERA-approx and QERA-exact to a range of models and evaluate both pretraining task and down- stream tasks in Table 3 and Table 4 respectively. We also compare to HQQ, a SoTA method that does use quantization error reconstruction. On most models, QERA-approx outperforms ZeroQuant-Vas and LQER, while QERA-exact achieves the best performance. 4-bit, QERA-exact is nearly lossless. 3-bit, QERA-exact's improvement over QERA-approx (Table 3 is clear, indicating the superiority of QERA-exact for aggressive quantiza- tion.
Higher win rate on AlpacaEval 2.0 To better understand the impact on instruction-tune models, we present the results of Vicuna-7b-vl.5 on AlpacaEval 2.0. In Figure[4] we evaluate the QER QER-based methods against the w-only quantization counterpart. QERA outperforms ZeroQuant-V and LQER by higher win rate, indicating better response quality.
'The average accuracy of Jama-. 1.1B excludes BoolQ, CommonsenseQA, and MMLU since TimyL.Jama1.. .BB has random guess accuracy on these tasks.

---

Published as conference paper at ICLR 2025
Table 4: Average accuracy (1) of LLMs on six downstream tasks. QERA-exact outperforms other quantization-error reconstruction-base based methods
almost all models. We also compare to HQQ (Badri & Shaji 2023). SoTA PTQ method that does not adopt quantization-error recon-
struction or activation heuristics. QERA-exact achieves an average accuracy on par with HQQ.
TinyLlama
Gemma-2
Phi-3.5
LLaMA-2
LLaMA-3.1
W-bits
Method
Rank
1.IB
3.8B
7B
13B
80B
BF16
40.59
53.96
66.91
49.61
55.74
63.88
72.05
4.25
HQQ
40.35
52.54
59.17
48.26
54.53
62.59
71.31
w-only
36.56
48.33
64.52
47.62
55.12
61.53
68.46
ZeroQuant-V
37.26
48.24
64.44
47.43
55.15
61.70
68.45
4.25
LQER
40.45
49.77
64.46
48.47
55,40
61.75
70.94
32
QERA-approx
40.02
49.29
64.53
48.52
55.20
61.68
70.80
QERA-exact
40.36
51.73
65.08
48.91
55.42
62.05
71.42
DISCUSSION
In this section, we revisit the arguments, design choices, and observations made the previous sections, including a test of Assumption T and the choice of the calibration set for PEFT. We offer an extended discussion of the numeric stability and scalability in Appendix [A.7] and LoRA rank and model choices PEFT experiments in Appendix A.9
ZeroQ
52.5%
44.4%
LOER
54,3%
43.3%
QERA
56.2%
40.8%
Test of Assumption 1 To test Assumption D we profile the autocorrelation matrix Rxx of the linear layer inputs in LLaMA-2-7B and ILLMA-3-8B. Note that Rxxi, E [x,, ). which assumes to be zero for i * sumption Figure [] shows the normalized Ryx mag- nitude, ||xx||> of four representative layers in LLaMA-
Figure 4: AlpacaEval 2.0 evalua- tion results. We compare quantized models to the counterpart without quantization-error reconstruction. A higher win rate (t) indicates better instruction-folloviii performance.
As-
Rbx()
3-8B where darker elements denote values closer to zero.
There are several layers with some input dimensions strongly correlated with others, such as the inputs to the third attention layer Figure 5a but for most layers, our assumption holds, especially the MLP layers, such as Figures tb 5d More Ryx plots are in Appendix A.I1
3. o.o_proj
(b) kb)/k///vvprro
7.o.po)
(d) ..gtt..proj
Figure 5: Normalized abs( abs(R)x) of the layer inputs in LLaMA-3-8B. Dark elements denotes value close to zero. There are few layers with input dimensions strongly correlated with others, such as the third attention layer in (a), but for most layers, our assumption zero-expectation holds.
Choice of calibration set for QPEFT One problem is to determine the calibration set for QERA before tuniitunng. bit RoBERTa-base fine-tuni fine-tuning experiment on SST2 (Appendix(A.6) we find that calibrating the pretraining dataset, WikiText2, helps the loss to decrease. However, the loss of the model calibrated on the fine-tuning dataset does follow the same trend. We hypothesize that the massive padding tokens in preprocessed SST2 samples cause this discrepancy, especially considering that the sequence length the raw SST2 dataset changes fiercely.
CONCLUSION
In this paper, we formulate the problem of quantization error reconstruction and propose QERA as analytical solution. Applying QERA to related works for efficient fine-tuning inference, show that QERA resolves the discrepancy in existing methods, and outperforms SoTA methods in both fine-tuning and quantization tasks by clear margin.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
This work was sponsored by Advanced Research + Invention Agency (ARIA), UK ARIA for their research network.
We also thank
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint 1rXiv2240..14219, 2024.
Hicham Badri and Appu Shaji. Half-quadratic quantization of large machine learning models, November 2023. htt.hhtpps//mmbbuuss....g/ htt.g//iihtttii.ooggooooooo
Jerry Chee, Yaohui Cai, Volodymyr Kuleshov, and Christopher De Sa. Quip: 2-bit quantization of large language models with guarantees. Advances in Neural Information Processing Systems, 36, 2024.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint 1rXiv:1905.10044 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. 17315:1803. 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.1410 2021.
Bita Darvish Rouhani, Ritchie Zhao, Venmugil Elango, Rasoul Shafipour, Mathew Hall, Maral Mes- makhosroshahi, Ankit More, Levi Mclnick, Maximilian Golub, Girish Varatkar, et al. With shared microcxponents. little shifting goes long way. In Proceedings of the 50th Annual International Symposium on Computer Architecture, pp. 1-13, 2023.
Edvin Deadman, Nicholas J Higham, and Rui Ralha. Blocked schur algorithms for computing the matrix square root. In International Workshop on Applied Parallel Computing. pp. 171-182. Springer, 2012.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Advances in Neural Information Processing Systems, 36, 2024.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235. 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 1783, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled al- pacaeval: A simple way debias automatic evaluators. arXiv preprint arXiv:2404.0447.. 04475, 2024.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- Irika, 1(3):211-218, 1936.
Ahmad Faiz, Sotaro Kaneda, Ruhan Wang, Rita Osi, Parteek Sharma, Fan Chen, and Lei Jiang. LImcarbon: Modeling the end-to-end carbon footprint of large language models. arXiv preprint arXiv:2309.14393 2023.
Han Guo, Philip Greengard, Eric P Xing, and Yoon Kim. Lq-lora: Low-rank plus quantized matrix decomposition for efficient language model finetuning. arXiv preprint arXiv:2311. / 2023.
11

---

Published as a conference paper at ICLR 2025
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with donddden-disentanlled embedding sharing. arXiv preprint arXiv:2111.09543, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. Proceedings of the Interna- lional Conference on Learning Representations (ICLR), 2021.
Roger A Horn and Charles R Johnson. Matrix analysis. Cambridge university press, 2012.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. Xrviv preprint arXiv:2001.0836 2020.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tuning arXiv:2310.08659 2023.
quantization for large language models. arXiv preprint
Ji Lin, Jiaming Tang, Haotian Tang, Shang Yang, Wei-Ming Chen, Wei-Chen Wang, Guangxuan Xiao, Xingyu Dang, Chuang Gan, and Song Han. Awq: Activation-aware weight quantization for on-device ilm compression and acceleration. Proceedings of Machine Learning and Systems, 6: 87-100, 2024.
Jing Liu, Ruihao Gong, Xiuying Wei, Zhiwei Dong, Jianfei Cai, and Bohan Zhuang. Qllm: Accurate and efficient low-bitwidth quantization for large language models. arXiv preprint arXiv:2310.0804/ 2023a.
Yinhan Liu. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 197. 2019.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Emnie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. LIm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305.17888 2023b.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. arXiv preprint arXiv:2404.02945 2024.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Rajarshi Saha, Naomi Sagan, Varun Srivastava, Andrea J Goldsmith, and Mert Pilanci. Compress- ing large language models using low rank and low precision decomposition. arXiv preprint arXiv:2405. 18886, 2024.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. An adversarial winograd schema challenge at scale. arXiv preprint arXiv: 1907. 10641, 2019.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint arXiv:2308. 13137, 2023.
Daria Soboleva, Faisal Al-Khateeb, Robert tness, and Nolan Dey. SlimPajama: plicated version of RedPajama. a http::/www.....aaa...........aaa 2023. /
Myers, Jacob R Steeves, Joel Hes- A 627B token cleaned and dedu-
https://ww.cerebraas
www. www.wrrreeea..eeee net /bb/blog// oeet
http:///gggggaoaeoooo////aaaarrr imma jama- ama 1aaa--a 627B
George PH Styan. Hadamard products and multivariate statistical analysis. Linear algebra and its applications, 6:217-240, 1973.
12

---

Published as a conference paper at ICLR 2025
Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc Le. Ed H Chi, Denny Zhou, and Jason Wei. Challenging big- bench tasks and whether chain-of-thought can solve them. arXiv preprint arXtv:2210.02661, 2022.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. QommonsenseeQ:: A ques- tion answering challenge largeting commonsense knowledge. In Proceedings of the 2019 Con- ference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4149-4158, Minneapolis, Min- nesota, June 2019. Association for Computational Linguistics. doi: 10. 10.18633/11NN19-14222 URL ht
1421
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhu- patiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.001 18, 2024.
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykow. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288. 2023.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization Ilms from comprehensive study to low rank compensation. arXiv preprint arXiv:2303.08302 2023.
Zhi-Xiu Ye. Qian Chen, Wen Wang, and Zhen-Hua Ling. Align, mask and select: A simple method for incorporating commonsense knowledge into language representation models. arXiv preprini arXiv:1908.065 067255000006.....0 2019.
Cheng Zhang, Jianyi Cheng, George Constantinides, and Yiren Zhao. Lqer: Low-rank quantiza- tion error reconstruction for Ilms. arXiv preprint arXiv:2402.0244 2024a.
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024b.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning, arXiv preprint arXiv:2303. 2023.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Erie Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-46623, 2023.
13

---

Published as a conference paper at ICLR 2025
A
APPENDIX
A.1 ALGORITHMS IN RELATED WORK
Here we summarize the algorithm of LoftQ (Li 2023
Algorithm I and LQER (Zhang al.
2024a) in Algorithm 2 respectively. LQ-LoRA (Guo et al. (023) adopts variant of Algorithm ZeroQuant- V2 Yao al. 2023) can be considered Algorithm
] with one iteration, or special
case of Algorithm 2 where the scale matrix S is identity matrix.
Algorithm LoftQ (Li
al. 2023)
Require: Pretrained weight W, target rank & quantization function q(-). dequantization function dq(-). num-
ber of iterations 7'
1: A. <- B,
for to
3:
W. f- q(W Bs)
b Update quantized weight matrix
4:
W t- d)(W))
5:
U, V +- SVD(W W)
SVD-based rank-k approximation
A& +- U. E Be E.t. Vile
7: end for
Algorithm LQQR (Zhang al. 2024a)
Require:
Pretrained weight W. target rank quantization function 4(-). dequantization function dq(-). cali-
bration dataset X (æ; R
1, I...,N)
I: Initialize vector 8 <-
2: for sample in X do
> Calibration Accumulate activation magnitudc on cach dimension
3:
abs(x)
end for
< diag((
Construct a diagonal matrix
6: W + q(W)
7: W < dg(WW))
8: U, E, V + SVW(S(W- - W))
> SVD on the scaled weight error
A 5 U,,K, Be E. Vik.:
Rank-k approximation with un-scaling
A.2 PROOF OF THEOREM 2
Here we present the full proof of QERA-approx. QERA-approx is an approximated solution to Problem based Assumption [ which is suitable to initialize the low-rank terms in fine-tuning for lower computation complexity.
Proof of Theorem 2
Proof. We continue at Equation 13 Since Ee~x is the expectation with respect to the input space, move the expectation inside the summation of RHS Equation (3p.
E By-rfl v1
E Bm-x 10 }P.p"
(25)
Under Assumption Ex-x (,r = 0 for i 전 j. the RHS of Equation 25 simplifies to:
m
Ey y 2)
Exax (2] ppp
(26)
We can define diagonal matrix S diag( / Ex (1f)
Ex v Ex-x(121
and
rewrite the RHS Equation 26) as:
Eysy ( vll T(SPPP S") S
(27)
where Tr(:) denotes the trace of matrix.
14

---

Published as a conference paper at ICLR 2025
Therefore, the objective of Problem 2 (Equation
8
is equivalent to:
arg min E v
yl y1 = arg min |PP||
C₁
C2
(28)
arg min SS( + C2 - W)
C
If we assign Q = S(W -W) and Qx SC2, the objective becomes:
arg min Q
||Qu Q1
(29)
Note that the invertible matrix S in Qs does not change the rank of the matrix Cj. According to the Eckart- Youog- theorem, the optimal rank approximation to Q the truncated SVD of Q:
Q U. B.cck V.k.:: where UEVT = QVD(Q) = SVD (s(W- - W))
(30)
Finally, we get the optimal solution to the low-rank term C₁:
C === = S E
(31)
A.3 CONNECTION AND DIFFERENCE BETWEEN CALDERA AND QERA
CALDERA (Saha et al. 2024) is the concurrent work close QERA. Here elaborate the nection and difference between CALDERA and QERA, and highlight the contributions of QERA.
CALDERA focuses different problem setup. Specifically, CALDERA focuses on the following problem:
min W. A XW - X(W + A b,,k
(32)
where X E b denotes batch calibration samples, and W, At.g. and B are all quantized variables to optimize. Note that this problem setup is different from QERA (Equation (
arg min E a| W + C2) x C
(33)
where only the low-rank high- nigh-precision C₁ A₁ By is the variable to optimize, and the quantized weight W is predefined given a quantization method.
Table 5: Notation Table for the Equivalence Derivation
Notation
Description
Comments
Number of calibration samples (vectors)
Layer input feature size
Layer output feature size Calibration set
X
Shape: m
E
A sample the calibration set Original full-precision layer weights
Shape: x m
W
Shape: m x
Y
Layer output matrix corresponding to X
Shape: b x
Y
Layer output vector corresponding Rank of low-rank approximation Approximated rank-k weight SVD decomposition of X Truncated rank & SVD
Shape: X
&
C₁
Shape: m x n
U,E.V V SVD&(-)
We find that CALDERA's Lemma 4.2 is equivalent to Theorem I in QERA. Note that the proof of QERA-exxcct is different from Caldera's Lemma 4.2, though the final closed-form solution is
15

---

Published as a conference paper at ICLR 2025
equivalent. Here additionally show the derivation of the equivalence between QERA-exace and
Caldera's s Lemma 4.2 using the notation table Table 5 For convenience, we remove the quantized
which does not change the proof. Now the problem becomes finding the optimal low-rank approximation of the weight matrix, C that minimizes the layer output error.
weight term W from QERA (Problem 2 in Equation p
First we note that the objective of QERA, Equation by constant
9 equivalent to CALDERA's Eq(5) scaled
QERA ming, X W)
(34)
CALDERA minc, X (Cx W
Then we show that Theorem ]
(QERA-exact) is equal to Caldera's Lemma 4.2.
QERA-exact: Cj (R xX
) SVD R W)
(35)
CALDERA: C₁ VE (U
(36)
We firstly show that (R (-
-1 in Equation 33 equals to VΣ in Equation (36) scaled by a constant
vb:
1
Rxx = X x) = VEU = VΣ
x
b
R 1
EVT
(37)
vb
(Rxx) -1 -iVE-
Then show that R W in Equation 35 equals to UT in Equation (66 scaled by the constant
vB
UTY = -''X=U''EEV"N WV- VOR W
(38)
W
U
vb
Therefore Cs equals to C²,, and the two solutions are equivalent. Despite of the equivalence, we
shortlist the differences between CALDERA and work:
Different problem setup (Equation
34
We simplify QERA-exact and derive QERA-approx, which is computationalll-effiicen approximated solution. Specifically, QERA-approx is more suitable for parameter-effice fine-tuning than DERAA=xacC/CALLERR Moreover, QERA-approx overcomes the pitfalls in existing methods and explains why previous heuristic methods like LQER work.
The optimization objective is similar (vector form vs matrix form), and the final closed- form solution equivalent, but the proof of QERA-exact different from CALDERA.
A.4 DETAILED EXPERIMENT SETUP
We mainly use PyTorch fransformers PEPT and |Accelleaatt to implement QERA. We use
SoiPy implementation of blocked Schur algorithm (Deadman al. 2012) to caleulate the matrix square root, which runs on CPUs. The evaluation is performed with mn-eva.luation-haee Evaluate| and AlpacaEval 2.0 (Dubois al. 2024).
A.4.1 QPEFT HYPERPARAMETERS
We perform fine-tuning experiments on four NVIDIA A100 80GB GPUs with AMD EPYC 64-Core Processor with 1024GB RAM. The total fine-tuning time is around 2100 GPU hours.
16

---

Published as a conference paper at ICLR 2025
RoBERTa-base GLUE We sweep learning rates for each (nethod, task), and collect the best accuracy. Thus each (method, task) pair has its own tailored learning rate, ensuring the best performance of baselines and QERA under the same trainable parameter budget. The reported accuracy is the average value across random seeds 42, 1, and 2. The total batch size is 64 for all GLUE experiments and train the models for 5 epochs. For 4-bit experiments, we use 4-bit floating point from the QLoRA implementation in PEFT. For 3-bit experiments, we use emulated MXINT (Darvish Rouhani et al. 2023) with block size 32 and for 2-bit experiments we use MXINT with block size = 16. Table 6 lists the learning rates for each experiment.
LLaMA-2-7BB-331--BB- on SlimPajama and GSM8K We adopt the learning rates in Meng et al. 2024). The reported perplexity/accuran is the average value across random seeds 42, 1, and 2. For SlimPajama we fine-tune the model on a subset for 1000 steps with rank = 8, total batch size 64, sequence length 1024, learning rate 3e-5. For GSM8K, fine-tune the model for 10 epochs with rank = 64, total batch size = 128, sequence length 384, and learning rate = 3e-5.
Table 6: Learning rates of RoBERTa-base experiments on GLUE
Rank
W-bits
Method
Learning rates
16
Full FT
7e-5, 5e-5, 2e-5
8
16
LoRA
le-4, 2e-4, 3e-4
8
4.25
OLoRA/LofIQ/QERA apprrxx
le-4. 2e-4, 3e-4
8 3.25
QLoRA//.oRQQQQERA A-P-:QQRRA--PPPPP Aapprox
le-4, 2e-4, 3e-4
64
2.50
QUoRAII.ffI/QOERA-ccco
2c-5, 3c-5, 4e-5, 5c-5, 6e-5, 9c-5, 1e-4
A.4.2 PTQ HYPERPARAMETERS
We perform PTQ experiments on eight NVIDIA A6000 48GB GPUs with AMD EPYC 256-Core Processor with 1024GB RAM. The total quantization and evaluation time is around 4500 GPU hours. We report 0-shot accuracy or normalized accuracy (if available) for all tasks except Wiki- Text2, in which we report word perplexity. The sequence length for reporting word perplexity is the model's context length by default, except for Phi-3.5 and LLaMA-3.1. For these two models, set the sequence length 2048. We the HuggingFace former implementation of HQQ, and reimplement ZeroQuant-V V2 and LQER as baselines. We use MXINT with block size 32 as the quantization format for all quantization methods except HQQ. which uses its built-in INT format with group size = 64. Thus, both formats have an average W-bits of 4.25. We evaluate quantized Vicuna-v1.5-- use GPT4-Turbo as the evaluator. The reported win rate is the cength-controlled win rate, which is debiased version of the win rate that controls for the length the generated outputs.
which is an instruction tuned LLaMA-2-7B, with AlpacaEval 2.0. and
A.5 DECREASING WEIGHT ERROR 7 DECREASING OUTPUT ERROR FOR LOFTQ
We provide the weight approximation error, W W - -xIlF in Figure 6 of all linaer layers in RoBERRa-base by sweeping the number of iterations. We observe that the weight approximation emor monotonically decreases with the number of iterations, but as shown in Figure the model out- put error may increase. This observation indicates that the commonly used objective of minimizing the weight approximation eтror and the corresponding algorithm not ideal for the quantization error reconstruction problem.
A.6 CHOICE OF CALIBRATION SET
We compare the QaRA-addpted models calibrated on the pretraining dataset and the downstream dataset. Specifically, we fine-tune two QERA-adapted 2-bit RoBERTa-base models. One is cali- brated on its pretraining dataset, WikiText2, and the other on SST2. Figure (ssurs the loss curves of the two models across three learning rates. None loss curves of the models calibrated on SST2 de- creases, but the ones calibrated on WikiText2 successfully decrease and converge. We hypothesize that this is due to the massive padding tokens in preprocessed SST2 considering that the raw sample lengths change fiercely. However, WikiText2 samples were preprocessed in the masked language modeling style, which means that only few special tokens added to the grouped texts.
17

---

Published as a conference paper at ICLR 2025
Layer Layer
Layer Layer
Layer Layer 11
Layer 1
Layer
Layer
out_proj
key_proj
query_pro)
Weight reconstruction error
9.0
22
8.5
20
8.0
18
16
14
14
6.5
LoftQ iterations value_proj
LoftQ iterations
LoftQ iterations
fc1
fc2
Weight reconstruction error
12
11
22
18
20
16
18
14
2
5
2
5
LoftQ num iterations
LaftQ num iterations
LoftQ num iterations
Figure 6: Weight approximation error of 3-bit rank-16 16 LoftQ with different numbers of iterations on RoBERTa- base.. We observe that the weight reconstruction error of all the layers decreases as the number of iterations increases, but as shown in Figure [b] the model output error (k=16) increases from the 4-th 5-th iteration.
Downstream, lr-le-4 Pretraining Ir-1e444--11----
Downstream, Jr-2e-4
Downstream Jr-3
Pretraining Ir-2e-4
Pretr vetrainingg_Ir33 Ir-e
0.6
0.5
Loss
0.4
0.3
0.2
1000
2000
3000
4000
5000
Step
The fine-tuning loss curves of QERA-adapted 2-bit RoBERTa-base on SST2. The loss fails to decrease if the calibration is performed on the downstream task SST2 due to the massive padding tokens in preprocessed SST2 samples. In pretraining dataset, there are only a few special tokens like padding tokens and mask tokens.
Figure 7:
18

---

Published as a conference paper at ICLR 2025
Calibration Matrix
layer
layer
1.8.00
Rue error rañlo log Frodty
layer 12
10-29
16
Quantization time
6n
4.7h
10:22
29.0m
28.2m
X
A
3.30
130
7B
13B
Layer hidden size
Model size
(a) Estimated error ratio of the square root of Rxxx
(b) QERA quantization time
Figure 8: Scalability of QERA. (a) plots the estimated enror ratio of the matrix square root calcu- lation of Rxx of some layers where the eтror increases as the model goes larger. (b) compares the quantization time of QERA-approx and QERA-exact if all layers are quantized sequentially. The matrix square root is time-consuming since executed on CPUs. One key optimization for accel- erating the quantization process QERA-exact will be the GPU-accelerated matrix square root.
A.7 SCALABILITY AND NUMERICAL STABILITY OF QERA
One may notice the diminishing model performance improvement of QERA-exact over QERA- approx as the model size increases. The main reason is that larger LLMs are more resistant to quantization (Chee al.) 2024). Another reason can be the error ratio of the matrix square root calculation of the autocorrelation matrix increases with model hidden size (Figure Sa).
We find that the data type used the calibration important for the numeric stability of QERA- exact due to the calculation of the matrix square root and SVD. To improve the stability of the calculation QERA-exact good practice we find to perform the outer product Rxx FP32, accumulated outer product in FP64, and calculate the matrix square root in FP64 using the blocked Schur algorithm [Deadman et al. 2012). Figure Sblilustttss the quantization time QERA-approx and QERA-exact on the platform deseribed in Appendix A.4 where the linear layers are quantized sequentially. QERA-exact is slow due to the calculation of matrix square roots on CPUs. GPU- accelerated matrix square root will be the key optimization to reduce the quantization time. Note that in QERA, the quantization of individual layers is independent, allowing more parallelization and acceleration of the quantization process.
A.8 CHOICE SOLUTIONS FOR QPEFT AND PTQ
QPEFT PTQ are two different application scenarios of QERA. recommend QERA-approx for QPEFT and QERA-exact for PTQ. PTQ aims to recover the model performance much as possible without re-training. For PTQ, it is desirable to recover more model performance even if takes longer compute low-rank terms. Note that the low-rank terms pre-computed once offline. At inference time, QeRA-exact introduces no overhead to the hardware since LQER, QERA-approx, and QERA-exact all takes the same form of a: ( + A Bg).
However, for QPEFT experiments, unreasonable to pay long time for initializing the low-rank terms for the limited improvement in output approximation error (i.e., WAR.A-exet/CALLDRR.... because 1) fine-tuning can recover the error, and 2) instead of spending much time on initialization, increasing training steps or increasing the rank number brings more gain in the fine-tuned accuracy. We run controlled experiments to support this claim. In Table 7 and Table g we run QPEFT ex- periments of RoBERTA-baas to QERA-exact (Caldera's Lemma 4.2), QERA-approx achieves better accuracy/perplexity while taking? 3/ of the time.
MRPC and LLaMA-2-7B on ShimPajmm respectively. Compared
19

---

Published as a conference paper at ICLR 2025
Table 7: Runtime comparison of QERA-exact and QERA-approx on MRPC. It is recommended using QERA-approx for QPEFT instead of QERA-exact.
Method
Rank
Epochs
Init. time
Training time
Total time (l)
Acc. (1)
QERA-exact
8
4
1.6min
2.2min
3.8min
88.97
QERA-approx
12
4
21s
2.2min
2.6min
89.95
QERA-approx
8
5
21s
2.7min
3.1min
89.97
Table 8: Runtime comparison of QERA-exact and QERA-approx on SlimPajama. mended using QERA-approx for QPEET instead of QERA-exact.
It is recom-
Method
Rank
Epochs
Init. time
Training time
Total time (4)
PPL. (4)
QERA-exact
16
2
4.9h
1.9h
6.8h
6.31
-aprrox
64
2
29.6min
2.1h
2.6h
6.18
QFRA-approx
16
4
28.2min
4.0h
4.5h
6.21
A.9 CHOICES OF LORA RANKS, MODELS, AND PRECISIONS FOR QPEFT
Rank = 8 for GLUE experiments We notice LoftQ paper uses a large rank of 16 and 32 for fine-tuning on GLUE, which is larger than the commonly-used rank value of LoRA (4 or 8 in LoRA paper (Hu et al. 202T)). If we consider LoRA as the upper limit of QLoRA-like QPEFT methods (including LoftQ and QERA), effectively compare these QPEFT methods, one easy way is to set the rank as the minimum value required by LoRA and check which QPEFT method achieves accuracy closest to LoRA. This is why we choose rank for GLUE experiments (For 2-bit GLUE experiments we use a large rank 64 since the quantization very aggressive). If we use rank 32, LoRA and all the QPEFT methods may be over-parameterrzed and it will be hard to make a fair comparison in terms of fine-tuned accuracy. To support this claim, we sweep the rank of LoRA-adapted RoBERTA base on SST2 and MRPC and show large rank k like in LoftQ has over-parallelization problem in Table 9 and Table I0
RoBERTa vs. DeBERTa
When investigating the related work, we find that both RoBERTa and DeBERTaV3 V3 (He et al. 2021) are used in QPEFT experiments (Guo et al.) 2023 Li et al. 2023
Meng et al. 2024; Guo et al. 2023 Zhang et al. 2023). The reason why we chose RoBERTa is that the RoBERTa checkpoint
HuggingFace is complete and compatible with both Hugging- Face's s official examples of sequence classification" and masked language modeling Specifically, the RoBERTa checkpoint contains both the base model and the masked language modeling head but the DeBERTaV3's checkpoint only contains the base model. As we know, the base model is enough for fine-tuning on downstream tasks. However, to calibrate on the pretraining dataset, need the language modeling head to verify if our implementation of data preprocessing and calibra- tion matches how the model was originally pretrained. Note that the quality of the statistic values in QERA like Rxx depends on the quality of the calibration set. Thus, without the language mod- eling head in the checkpoint, we cannot perform the QERA's 's calibration ensure the correctness of statistics in QERA, and explore the effect of the choice of calibration sets.
DeBERTaVs properly,
A. DETAILED PTQ RESULTS
Here we offer the detailed evaluation results for each downstream task in Tables [I
to
A.11 TEST OF ASSUMPTION 1
We provide more plots of normalized Rx magnitude,
aba(Rxx)) across LLaMA-3.1-88 3.1-BB..BBB...... LLaMA-2- |R
7B, Mistral-7B-v0.3, and TinyLlama- I. in Figures to where dark pixels are elements close
"RoBERTa-base checkpoint: link
'HuggingFace example of sequence classification: link
*HuggingFace example of masked language modeling: link
'DBBRTTaV3ss checkpoint: link
20

---

Published as a conference paper at ICLR 2025
Table 9: Over-parameleciizatoo problem. We sweep the rank i of LoRA on SST2 and reported fine-tuned accuracy. The highest accuracy at rank ho = 12 indicates over-parameterization happens for k 2 12.
Method
Rank k
Learning rates
Best Acc.
4
1c-4422-44/4c444444446----66
94.38
8
1c-4/22-44/3--14444444-4-----
94,46
LoRA
12
94.73
94.50
20
10-4422-443e-44444445----
94.50
Table 10: Over-parameterriation problem. We sweep the rank of LoRA on MRPC and reported
fine-tuned accuracy. The highest accuracy rank k 12 indicates wver-parameterization happens for k 2 12.
Method
Rank
Learning rates
Best Acc.
4
1c-4/2e-443c44444445----4
87.99
8
88.97
LoRA
12
40-4222-443e-44444445-------
89,95
16
89,46
20
(c-4722-443-44444446444---
89.71
to zeros. There are strongly correlated embedding channels in some proj and proj layers. The assumption fits better in MI. layers (gate proj, up proj, and down proj), and holds for over 60% of the layers in LLMs.
1 N
yor Caper (1)
K Day
5.0% Cap 20
Figure 9: Normalized abs(Rxx) of inputs of k-proj layers in -LaMA-3-8B. Note that the q-proj and v. v.proj share the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
21

---

Published as a conference paper at ICLR 2025
Table 11: Post-training quantization evaluation of TinyLLama-l 1.....
ARC (challenge)
BoolQ
CommonSenseQA
BBH
MMLU
WikiTex12
Winogrande
rank
Method
w-bits
Acc.norm
Acc
Acc
Accm
Acc
Word
Acc
BF16
16
32.51
55.93
20.07
29.68
25.35
13.98
59.59
HQQ
32.00
58.13
20.15
29.70
25.75
15.02
59.35
u--nlyy
28.67
58.23
19.49
28.99
23.81
19,40
52.01
ZoroQuant-V2
29.69
$7.86
19.41
29.53
24.85
18.03
52.57
4.25
LOER
32.00
52.42
59.75
32
QERA approx
31.83
$2.08
17.20
29.51
25.22
15.66
58.72
QERA-exact
32.00
Table 12: Post-training quantization evaluation of Gemma-2-2B.
Method
ARC (challenge)
BoolQ
CommonSemseQA
BBH
MMLU
WikiTex12
Winogrande
Acc norm
Acc
Acc
Acc norm
Acc
Word ppl
Ace
BF16
16
49.91
72.60
50.29
32.67
49.44
13.08
68.82
HQQ
48.81
71.77
48.40
32.32
46.52
14.29
67.40
w-only
44.62
69.91
34,07
31.96
42,90
16.23
66.54
ZeroQuant-V
44.45
69.94
34,07
31.50
43.27
15.71
66.22
4.25
LQER
46.08
68.84
37.59
32,60
45.78
14.55
67.72
32
QERA-appmn approx
45.31
68.99
36.20
32.04
45.80
14.60
67.40
QERA-exact
46.84
72.32
42.75
33.36
47.29
14.12
67.80
Table 13: Post-training quantization evaluation of Phi3-3.5-min
rank
Method
W-bits
ARC (challenge)
BoolQ
CommonSenseQA
BBH
MMLU
WikiText2
Winogrande
Acc_norm
Acc
Acc
Acc.norm
Acc
Word ppl
Acc
BF16
59.39
84.65
71.91
48.19
64.58
11.50
72.77
74.34
60.20
69.61
W only
59.73
82.72
68.22
44.45
61.54
14.16
70.48
ZeroQuant-Vy
4.25
LOER
59.39
84.01
70.76
45.67
62.21
12.88
70.74
32
QERA-approx
59.45
84.82
70.84
45.67
62.26
12.81
70.17
QERA-exact
58.70
83.73
69.45
45.37
62.01
13.00
71.19
Table 14: Post-training quantization evaluation of LLaMA-2-7B.
ARC (challenge)
BoolQ
CommonSenseCAA
BBH
MMLU
Wikilext2
Winogrande
rank
Method
W-bits
Acc. norm
Acc
Acc
Acc norm
Acc
Word ppl
Acc
77.83
HQQ
44.03
75.87
29.40
30.50
40.14
9.59
69.61
w-only
45.22
75.87
25.47
30.71
40.03
9.45
68.43
ZeroQuant-V2
45.82
75.90
24.82
29.99
39.84
9.42
68.19
LOER
44.28
76.15
29.81
30.72
40.66
9.22
69.22
32
QERA-approx
75.96
68.59
QERA exact
44,80
76.39
31.61
30.57
40,86
9.12
69.22
Table 15: Post-training quantization evaluation of LLaMA-2-13BB
rank
Method
W-bits
ARC (challenge)
BoolQ CommenSense@A
BBH
MMLU
Wikilex12
Winogrande
norm
Word ppl
BF16
16
49.49
80.58
47.34
32.65
52.18
7.68
72.22
HQQ
49.06
78.69
45.05
32.41
50.85
8.27
71.11
w-only
50.43
80.58
44.06
33.45
50.21
8.06
71.98
ZeroQuant-Ve V2
50.00
81.04
44.47
33,50
50.31
8.07
71.59
32
LQER
$1.02
81.25
44.47
32.41
51.24
7.96
71.98
approxp
$1.11
80.83
44.06
32.48
$1.07
7.95
71.67
QERA exact
50.77
81.10
44.55
32.91
51.23
7.93
71.98
Table 16: Post-training quantization evaluation of LLaMA-3.1--0
ARC (challenge)
BoolQ
CommonSenseQA
WikiText2
Winogrande
rank
Method
W-bits
Acc_norm
Acc
Acc_norm
Word
Acc
BF16
16
$3.50
82.05
71.42
39.07
63.27
7.35
73.95
IQQQ
$2.73
81.19
69.86
35.60
62.14
8.72
74.03
w-only
81.31
67.24
37.34
ZeroQuant V2
$1.11
81.25
66.99
38.43
58,94
8.83
73.48
4.25
LQQR
32
QERA-approx
50.77
81.04
66.75
37.94
60.09
8.45
73.48
QERA-cxact
51.28
80.18
68.83
37.48
60.60
8.33
73.95
22

---

Published as a conference paper at ICLR 2025
Table 17: Post-training quantization evaluation of LLaMA-3.1-70B.
rank
Method
W-bits
ARC (challenge)
BoolQ
CommonSenseQA
BBH
MMLU
WikiText2
Winogrando
Acc_norm
Acc
Acc
Acc_norm
Acc
Word ppl
Acc
BF16
65.10
85.38
78.46
48.53
75.28
3.06
79.56
IQQ
w-only
60.58
83.82
73.63
41.28
73.06
4.55
78.37
ZeroQuant
42.75
77.74
LQER
62.97
83.88
76.25
48.67
74.26
4.10
79.64
QERA-approx
62.12
83.79
76.74
48.53
73.98
4.10
79.64
1. Staeert
4.3% 24
Hugar 21
Gupeer
2,59/U5e 25:
2.5% 221
c.po| layer 221
0,9% GU0 271
Figure 10: Normalized abs(Ixxx) of inputs of c_proj layers in LLaMA-3-8B, Layers are sampled and only the first 96 dimensions are plotted for clarity.
23

---

Published as a conference paper at ICLR 2025
Figure 11: Normalized abs(Rxx) of inputs of gate_proj layers in LLLMA-3-8B. Note that the up-proj shares the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Ober pww.ttaeee
Haper
Swn and Bayer 221
CO Laper
SoweLor Jrd Haye
200 Kayez2)
DAI Lave 31
Figure 12: Normalized abs (xxx) of inputs of down-proj layers in LLaMA-3-8B. Layers are sam- pled and only the first 96 dimensions are plotted for clarity.
24

---

Published as a conference paper at ICLR 2025
ky T9
Figure 13: Normalized (bs(()x)) of inputs of k k-proj layers in LLaMA-2-77. Note that the q-proj and v_proj share the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
1,Sr 25:
Jaye 231
c,pol Layer 221
0,0% Gaper
2,5r//laser 30>
Figure 14: Normalized abs (IRxx) of inputs of c.pro layers in LLaMA-2-7B. Layers are sampled and only the first 96 dimensions are plotted for clarity.
25

---

Published as a conference paper at ICLR 2025
Figure 15: Normalized abs(Rxx) of inputs of gate_proj layers in LLaMA-2-7B. Note that the up-proj shares the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Oker Lww.ttgeer
Haper
Bwnr Bayer
COWELET Layer
SoweLon Hoye
200r Kayezz)
DH Lave 3I
Figure 16: Normalized abs(Rxx) of inputs of down-proj layers in LLaMA-2-7B. Layers are sam- pled and only the first 96 dimensions are plotted for clarity.
26

---

Published as a conference paper at ICLR 2025
Caper
k T9
Figure 17: Normalized (bs(( of inputs of k-proj layers Mistral-7B-vO.3. Note that the q-proj and v-proj share the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
2,5r//45ee 25:
2.5% 231
sppol layer 221
2,5T//USe 22
Figure 18: Normalized abx(()xx) of inputs of o.proj layers in Mistral-7B-v0.3, Layers are sampled and only the first 96 dimensions are plotted for clarity.
27

---

Published as a conference paper at ICLR 2025
Uayer
Figure 19: Normalized als((Rgx) of inputs of gate_proj layers Mistral-7B-vO.3. Note that the up-proj shares the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Oker pww.ttgeeeeee
Hager
awwrann Bayer 121
COWELETE Layer
SoweLon Hoye
S0PP Saye
200r Kayez2)
E7II FM Laaee 312
Figure 20: Normalized (xxx) of inputs of downspro layers in Mistral-7B-v0.3. Layers are sampled and only the first 96 dimensions are plotted for clarity.
28

---

Published as a conference paper at ICLR 2025
KEPR of
Kupa GAper
Cartilage 21
Koin Gayer
Figure 21: Normalized abs(Exx) of inputs of k-proj layers in TinyLlama-I.I Note that the q q-proj and v proj share the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
25
Figure 22: Normalized abs(Rxxx of inputs of o.proj layers in TinyLlama-1, 1.B.. Layers are sampled and only the first 96 dimensions plotted for clarity.
GEALON (Lye LO
pETE Los
Figure 23: Normalized (Rxx)) of inputs of gate.proj layers in TinyLlama-1.1B. Note that the up-proj shares the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
29

---

1.10aa11.BBB. IB. Layers are Published as a conference paper at ICLR 2025
Lawi Jayar
2id Aaper
coel sot Hapa
Bost poi Haper
Figure 24: Normalized abs(Rxx) of inputs of down.proj layers in TinyLlama- sampled and only the first 96 dimensions are plotted for clarity.
30


Reviews:
[{"review_id": "jFYgY7zxuT", "reviewer": "Reviewer_8w8V", "summary": "This work proposes a framework for analyzing the quantization error after it is compensated using low-rank, high-precision terms. In such a decomposition, a weight matrix $\\mathbf{W}$ is approximately split as $\\mathbf{\\tilde{W}} + \\mathbf{A}_k\\mathbf{B}_k$, where $\\mathbf{\\tilde{W}}$ is the quantized weight, and $\\mathbf{A}_k\\mathbf{B}_k$ is the low-rank compensation. This work aims to minimize the Frobenius norm error of the outputs of each layer (in contrast to just the weight quantization error), and propose closed-form solutions for the low-rank terms. The improved benefits are validated with numerical experiments of both encoder-only (RoBERTa) and decoder-only (LLaMa family) models. The experiments involve both post-training quantization (PTQ), as well as parameter-efficient fine-tuning (PEFT).", "questions": "Please see the Weaknesses section. I would be happy to readjust my score if they are satisfactorily addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper analytically considers the problem of compensating the quantization error using low-rank high-precision components. The paper is generally well-written, although the work will benefit if it takes into account and compares with more recent works which takes into account the same problem (see weaknesses below).  \n\nThe numerical experiments are comprehensive, and the results on a wide variety of models are presented. They are also compared with some other prior works, and show improved benefits.", "weaknesses": "My major concern with this paper is that it fails to take into account more recent works in this area, and justify how it compares with those works. The contribution of not really clear in light of a more recently proposed algorithm, Caldera (https://arxiv.org/abs/2405.18886) solves the optimization problem (9) optimally, i.e., the output error is minimized and closed form solutions for the low-rank factors are obtained (ref. Lemma 4.2 in the paper). Could the authors highlight the difference in their result of QERA-exact solution (Thm. 1) with Lemma 4.2 (Caldera)? \n\nFurthermore, the autocorrelation matrix, $\\mathbf{R}_{\\mathbb{XX}}$ (which is also referred to as Hessians, because it is the Hessian of the quadratic loss in (9)), does need to be computed using a calibration dataset -- but this is a one-time cost. Additionally, the approximation in Assumption 1 for QERA-Approx approximates the autocorrelation matrix as a diagonal matrix -- this is not necessarily true as shown in Figs. 5, 7 and 8. \n\nSecondly, minimizing error in layer outputs for PTQ is not really a recent idea as mentioned in the paper. It has been around for a few years now. See for example, https://arxiv.org/abs/2004.10568. \n\nDespite the weaknesses mentioned here, the numerical evaluations on the models and the regimes of 3/4-bit quantization are most likely new.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OBSCpQn2Ma", "reviewer": "Reviewer_8H2Q", "summary": "This paper proposes QERA that analytically finds the proper low rank terms by minimizing the layer output error. They show that minimizing the layer output error is more closely related to minimizing the model output error than minimizing the weight approximation error. They empirically demonstrate their method is better than other previous methods in both QPEFT and PTQ perspectives.", "questions": "N/A", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and easy to follow.\n2. The idea of deriving the analytical solution to the low rank terms by minimizing the layer output error is new.", "weaknesses": "The weaknesses of this paper mostly come from the experiment part.\n\n1. The numbers in Table 1 and Table 2 don't match with the loftq original paper. Is that because you change the experimental setup? Could you please show your method outperforms loftq in their setup?\n2. In the original loftq paper, they includes some experimental results about 2bit fine-tuning. Could you also show some results about 2bit fine-tuning?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ro1pYOLePN", "reviewer": "Reviewer_tFvD", "summary": "In this paper, authors give the  analytical solution to solve the minimizing the errors in layer outputs via low rank terms. And authors prove that  minimizing the layer output error is better than minimizing the weight approximation error from the aspect of model performance.", "questions": "1.How to prove that the reason of worse performance of weight approximation methods is rooted in the model's nature instead of your weight approximation methods choice in experiments?\n\n2.In first paragraph, k<<min(m,n). I think it should be k << $\\frac{mn}{m+n}$ because the computation cost of W is mn, the computation cost of $A_KB_K$is mk+nk, we want (m+n)k<mn, k should satisfy k <$\\frac{mn}{m+n}$", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This article is excellent in aspect of motivation, problem solving, and paper writing, and is also highly recommended for  its algorithm engineering work. \n1. In terms of motivation, this article chooses to use theoretical methods to solve problems that can only be solved using heuristic algorithms at this stage, and determines the theoretical extreme value of the problem and the method to reach it. \n\n2. This article provides a very solid analytical method and gives an algorithm for solving the extreme value according to this method, which is solid and reliable. \n\n3. Paper writing aspect, this article has been detailed and concise, simplifying the repetitive proof of Theorem 2 and placing the proof of Theorem 1 as an important part of the text, allowing readers to fully understand the contribution of this article. In addition, the structure of this article is clear, and the questions and answers are clearly stated. \n\n4.The biggest advantage of this article is that it reasonably engineers the algorithm and provides a reasonable simplified algorithm that is easy to implement in engineering.", "weaknesses": "The authors insight that minimizing the output error is better than weight approximation error is is consistent with our practical experience in the aspect of model performance. However, this point is hard to prove via experiments, because we cannot enumerate all weight approximation methods on every models. The conclusion is so strong. \nSo, two suggestions are that 1. give a mathematical proof of this point. 2. avoiding discuss this conclusion in paper, and only show your work is better than SOTA low rank methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EcnEPn1aoz", "reviewer": "Reviewer_KXUq", "summary": "This paper directly gives the analytic solutions of the linear compensation matrix for quantization error correction.  The analytic solutions involve both exact and approximate solutions to tradeoff compensation quality and efficiency. The analytic solutions are solidly validated for two scenarios, post-training and fine-tuning-based quantization.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The work formally demonstrates that optimizing to reduce the model layer output error is more effective than minimizing the weight quantization error. Given the fundamental importance of the optimization objective in this field, the work has a significant impact.\n\n2. The assumption used to approximate the exact solution appears reasonable and can be validated through practical testing.\n\n3. The experiments conducted in the work are extensive and comprehensive.", "weaknesses": "1. The efficiency of the exact and approximate solutions, like computational complexity and the concrete execution time compared with related works, should be further clarified.\n2.  Eq. (18), $\\mathop{\\arg\\min}_{Q_k} $. \n3. Can the analytic solution be used in LLM pruning? Given that pruning and quantization usually have similar mathematical modeling to some extent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aTPwP7HmxB", "reviewer": "Reviewer_1yFJ", "summary": "This paper proposes Quantization Error Reconstruction Analysis (QERA), an analytical framework for reconstructing quantization error with low-rank terms. QERA offers a closed-form solution to the problem of quantization error reconstruction and demonstrates that LLMs quantized with QERA achieve better linguistic capabilities compared to previous approaches.", "questions": "Please check the Weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tIt introduces an analytical framework for reconstructing quantization error by considering the layer output, not just the weight values.\n2.\tIt provides detailed mathematical proofs.", "weaknesses": "1.\tAlthough the paper claims that demonstrating the relationship between minimizing layer output error and minimizing model output error is a key contribution, this claim seems overstated. Many existing quantization approaches [1, 2] for LLMs already consider layer output error rather than focusing solely on weight approximation error to achieve more accurate quantization.\n2.\tThe overhead of considering layer output is not sufficiently discussed and is only briefly mentioned in Figure 12(b) of the appendix. This paper lacks a thorough analysis of the overhead (e.g., memory requirements and runtime) associated with the error reconstruction procedures. While weight approximation error has limitations in reconstructing errors, it offers more efficient procedures since it does not require a calibration dataset. However, approaches that consider layer input/output for error reconstruction do require a calibration dataset and more computation. Despite this discrepancy in calibration overhead, the paper does not provide a comprehensive analysis of the error reconstruction overhead in the proposed method compared to previous works.\n3.\tAccording to Figure 12(b) of the appendix, as QERA-exact appears to be very slow, it seems fair to compare LQER and QERA-approx. However, QERA-approx offers minimal advantage over LQER in terms of quantization.\n4.\tThe paper does not compare the proposed method with LQ-LoRA, a more advanced method that also uses a calibration dataset for error reconstruction. A comparison between the proposed method and LQ-LoRA is important for evaluating QERA.\n5.\tThe paper uses confusing terminology when categorizing experiments. It should use standard terms such as \"quantization-aware fine-tuning\" and \"post-training quantization\" to classify experiments, rather than \"fine-tuning experiments\" and \"quantization experiments,\" as fine-tuning experiments also include quantization.\n\n[1] Frantar, Elias, et al. \"Gptq: Accurate post-training quantization for generative pre-trained transformers.\" ICLR 2023.\n\n[2] Lin, Ji, et al. \"AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration.\" MLSys 2024", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)