QERA: an Analytical Framework for Quantization Error Reconstruction

## Abstract

Abstract The growing number of parameters and computational demands of large language models (LLMs) present significant challenges for their efficient deployment.
Recently, there is an increasing interest in quantizing weights to extremely low precision while offsetting the resulting error with low-rank, high-precision error reconstruction terms.
The combination of quantization and low-rank approximation is now popular in both adapter-based, parameter-efficient fine-tuning methods such as LoftQ [ 1 ] and low-precision inference techniques including ZeroQuant-V2 [ 2 ] .
Usually, the low-rank terms are calculated via the singular value decomposition (SVD) of the weight quantization error,
minimizing the Frobenius and spectral norms of the weight approximation error.
Recent methods like LQ-LoRA [ 3 ] and LQER [ 4 ] introduced hand-crafted heuristics to minimize errors in layer outputs (activations) rather than weights, resulting improved quantization results. However, these heuristic-based methods lack an analytical solution to guide the design of quantization error reconstruction terms.
In this paper, we revisit this problem and formulate an analytical framework, named Quantization Error Reconstruction Analysis (QERA),
and offer a closed-form solution to the problem.
We show QERA benefits both existing low-precision fine-tuning and inference methods –
QERA achieves a fine-tuned accuracy gain for Δ acc subscript Δ acc \Delta_{\text{acc}} roman_Δ start_POSTSUBSCRIPT acc end_POSTSUBSCRIPT = 6.05% of 2-bit RoBERTa-base on GLUE compared to LoftQ;
and obtains Δ acc subscript Δ acc \Delta_{\text{acc}} roman_Δ start_POSTSUBSCRIPT acc end_POSTSUBSCRIPT = 2.97% higher post-training quantization accuracy of 4-bit Llama-3.1-70B comapred to ZeroQuant-V2 and Δ ppl subscript Δ ppl \Delta_{\text{ppl}} roman_Δ start_POSTSUBSCRIPT ppl end_POSTSUBSCRIPT = − - - 0.28 lower perplexity on WikiText2 compared to LQER.

###### Index Terms:

## I Introduction

The demand for efficient deployment of large language models (LLMs) has been increasing [].
LLMs now typically
contain billions of parameters [, ], making their fine-tuning and inference computationally expensive and
resource-intensive [].
To address these challenges, there has been a surge of interest in building efficient fine-tuning and inference methods.
One popular formulation is to apply a low-rank term to reconstruct the error after quantization.
Given a linear layer ${\bm{y}}={\bm{x}}{\bm{W}}$,
the weight matrix ${\bm{W}}\in{\mathbb{R}}^{m\times n}$ is quantized to $\widetilde{{\bm{W}}}$, and we rewrite ${\bm{W}}\approx\widetilde{{\bm{W}}}+{\bm{A}}_{k}{\bm{B}}_{k}$ such that both ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ are low-rank yet high-precision terms with rank $k\ll\min(m,n)$.

We call the problem of finding the optimal ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ quantization error reconstruction. Interestingly, this problem has, coincidentally, seen widespread application in two actively researched areas: quantized parameter-efficient fine-tuning (QPEFT) and post-training quantization (PTQ) for model inference.
QPEFT refers to fine-tuning techniques that adapt LLMs to specific tasks by quantizing pretrained weights and updating only a small number of extra parameters, hence significantly reducing memory requirements and training time, such as QLoRA [].
On the other side, PTQ is a training-free method that reduces the model size and may accelerate the forward pass if the underlying hardware supports it.
Recently, researchers combined PTQ with quantization error reconstruction [, , ] to further reduce weight precision.
Works such as ZeroQuant-V2 [] and LQER [] have shown that adding a high-precision low-rank component, as low as 8 or 32, can recover considerable model performance for 3- or 4-bit weight quantization.

Although both the QPEFT and PTQ methods have demonstrated substantial performance improvements in lowering the computational overhead of LLMs, a theoretical analysis of quantization error reconstruction is lacking.
Usually, ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ are calculated by applying truncated singular value decomposition (SVD) to the weight quantization error $({\bm{W}}-\widetilde{{\bm{W}}})$, minimizing the Frobenius and spectral norms of the weight approximation error. However, recent work on activation-aware quantization and knowledge distillation implies that minimizing layer output error generally leads to a greater performance gain than minimizing weight approximation error [, , ].

Besides the unsettled minimization objective, it has remained unclear whether there exists a theoretically optimal solution for the values of ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$, and if so, how one can solve for it.
A better initialization or theoretically grounded initialization of ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ brings direct benefits for both QPEFT and PTQ.
In QPEFT, the initialization of LoRA [], which uses element-wise Gaussian random values for ${\bm{A}}_{k}$ and zeros for ${\bm{B}}_{k}$, struggles under aggressive quantization since the quantization error can derail fine-tuning.
In PTQ, the quantized model performance is based on the computation of the low-rank terms, given a specific quantization function $\mathrm{q}(\cdot)$ and rank $k$.

In this paper, we aim to provide an analytical framework for the quantization error reconstruction problem.
To demonstrate the effectiveness of our theoretical framework, we further apply our analytical solutions to state-of-the-art QPEFT and PTQ methods and
show the significant performance improvements under the same computational budget. Specifically, our contributions are as follows:

- •
We show that the commonly used objective for solving the quantization error reconstruction problem in prior work , i.e., minimizing the weight approximation error (e.g., $||{\bm{W}}-\widetilde{{\bm{W}}}||_{p}$), does not guarantee a reduced model output error. Instead, we show that minimizing the layer output error (e.g., $||{\bm{y}}-\widetilde{{\bm{y}}}||_{p}$) is closely related to minimizing the model output error.
- •
We derive the analytical solution to the low-rank terms ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ by minimizing the layer output error. We demonstrate that under a statistical assumption, this solution can be found in a particularly computationally efficient manner, also explaining the success of LQER.
- •
We empirically demonstrate the effectiveness of our solutions by applying them to state-of-the-art QPEFT and PTQ methods. Our analytical framework, QERA, significantly improves the performance of these methods. For example, QERA achieves $\Delta_{\text{acc}}$ = 6.05% higher accuracy of 2-bit RoBERTa-base on GLUE compared to LoftQ, improving the fine-tuning accuracy and efficiency. Moreover, QERA obtains $\Delta_{\text{acc}}$ = 2.97% higher accuracy than ZeroQuant-V2, when quantizing LLaMA-3-70B to 4 bits, averaged across six tasks. This narrows the model performance gap between error-reconstruction-based post-training quantization and full-precision models.

## II Related Work

In this section, we review the existing methods that combine weight quantization and low-rank error reconstruction.
These methods can be roughly categorized into two groups based on their applications:
QPEFT for training and PTQ for inference.

##### LoRA and QPEFT

LoRA [] is a representative PEFT method that
introduces trainable low-rank terms to adapt the model to a specific task.
Take a linear layer as an example,

$$ ${\bm{y}}={\bm{x}}({\bm{W}}+{\bm{A}}_{k}{\bm{B}}_{k})$ (1) $$

where ${\bm{W}}\in{\mathbb{R}}^{m\times n}$ is the pretrained weight matrix,
row vector ${\bm{x}}\in{\mathbb{R}}^{m}$ and ${\bm{y}}\in{\mathbb{R}}^{n}$ are the input and output,
and ${\bm{A}}_{k}\in{\mathbb{R}}^{m\times k}$ and ${\bm{B}}_{k}\in{\mathbb{R}}^{k\times n}$ are trainable low-rank matrices (“adapter”) with rank $k\ll\min(m,n)$.
During fine-tuning, the pretrained ${\bm{W}}$ is frozen and only the adapter ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$ are updated.
To make the adapted layer’s output match the original one at the start of fine-tuning,
LoRA initializes ${\bm{A}}_{k}$ with Gaussian random values and ${\bm{B}}_{k}$ with zeros. Once the fine-tuning is completed, the adapter
is merged into the pre-trained weights.

QLoRA [] extends LoRA by quantizing the pretrained weights stored in GPU memory to reduce memory footprint.

$$ ${\bm{W}}_{q}=\mathrm{q}({\bm{W}})$ (2) $$

One difference between QLoRA and LoRA is that during fine-tuning, ${\bm{W}}_{q}$ needs to be dequantized before involved into matrix multiplications:

$$ $\begin{split}\widetilde{{\bm{W}}}=\mathrm{dq}({\bm{W}}_{q}),\;{\bm{y}}={\bm{x} }(\widetilde{{\bm{W}}}+{\bm{A}}_{k}{\bm{B}}_{k})\end{split}$ (3) $$

where $\mathrm{dq}(\cdot)$ is the dequantization function.
QLoRA introduces weight quantization error $({\bm{W}}-\widetilde{{\bm{W}}})$,
shifting the starting point of fine-tuning.
To address this problem, LoftQ [] initializes the adapter
using the SVD-based low-rank approximation of $({\bm{W}}-\widetilde{{\bm{W}}})$ to reduce the weight approximation error:

$$ $\operatorname*{arg\,min}_{{\bm{A}}_{k},{\bm{B}}_{k}}||{\bm{W}}-\widetilde{{\bm {W}}}-{\bm{A}}_{k}{\bm{B}}_{k}||_{F}$ (4) $$

Specifically, LoftQ uses a heuristic-based algorithm to iteratively update the quantized weights and the adapter (Algorithm 1 in the Appendix).
Their experiments show that a larger number of iterations leads to a smaller weight error.

LQ-LoRA [] also adopts LoftQ’s iterative method but keeps track of a scaled variant of the objective,
$\operatorname*{arg\,min}_{{\bm{A}}_{k},{\bm{B}}_{k}}||{\bm{D}}_{\text{row}}({
\bm{W}}-\widetilde{{\bm{W}}}-{\bm{A}}_{k}{\bm{B}}_{k}){\bm{D}}_{\text{col}}||_
{F}$,
where ${\bm{D}}_{\text{row}}$ and ${\bm{D}}_{\text{col}}$ are heuristic homogenous row/column matrices from activation statistics.
LQ-LoRA exits the iteration when the scaled objective function stops decreasing
due to the lack of a theoretical justification for LoftQ.

##### Quantization Error Reconstruction for PTQ

Similar to the forward pass of fine-tuning in QLoRA, there are also PTQ methods that quantize the pretrained weights
to low-precision formats and recover the model performance with additional low-rank terms.
With a small enough rank $k$, the additional computation introduced is negligible.
Note that unlike QPEFT which can utilize fine-tuning to correct the quantization error,
PTQ methods aim to recover the model performance as much as possible without any training.

ZeroQuant-V2 [] is the earliest weight-only quantization method introducing low-rank quantization error reconstruction to the PTQ problem.
They apply SVD to the weight quantization error $({\bm{W}}-\widetilde{{\bm{W}}})$ to calculate ${\bm{A}}_{k}$ and ${\bm{B}}_{k}$
(equivalent to LoftQ with one iteration). Combining low-rank terms and fine-grained quantization,
ZeroQuant-V2 recovers the performance of 4-bit LLMs to a level comparable to 8-bit.

Recent quantization works have shown that activation statistics play a crucial role in weight-only LLM quantization [, ].
QLLM [] trains the low-rank terms using gradient descent with a loss function that minimizes the output error of the attention layer.
LQER [] applies an activation-induced heuristic scale matrix ${\bm{S}}$ to the quantization error before calculating SVD,
${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{S}}({\bm{W}}-\widetilde{{
\bm{W}}}))$, and assigns ${\bm{A}}_{k}:={\bm{S}}^{-1}{\bm{U}}_{:,:k},\;{\bm{B}}_{k}:={\bm{\Sigma}}_{:k,:
k}{\bm{V}}^{T}_{:k,:}$ (Refer to Algorithm 2 in the Appendix).
LQER achieves significant improvement over ZeroQuant-V2 and
observes that in some layers singular values are shaped toward a more desirable distribution where singular values decay faster.
Note that ZeroQuant-V2 can also be considered as a special case where ${\bm{S}}$ in LQER is an identity matrix.

In summary, most of existing methods on QPEFT and PTQ
target the minimization of the weight approximation error.
Several recent works such as LQ-LoRA, QLLM, and LQER introduce activation-induced heuristics
to the calculation of adapters/low-rank terms,
but a justification for the optimization objective and the corresponding analytical framework are still missing.

## III Our Analytical Framework

In this section, we formulate the optimization objective of quantization error reconstruction
and derive the analytical solution to the low-rank term ${\bm{C}}_{k}:={\bm{A}}_{k}{\bm{B}}_{k}$.

### III-A Problem Statement

Given a pretrained linear layer ${\bm{y}}={\bm{x}}{\bm{W}}$ with input vector ${\bm{x}}\in{\mathbb{R}}^{m}$,
output vector ${\bm{y}}\in{\mathbb{R}}^{n}$, and weight matrix ${\bm{W}}\in{\mathbb{R}}^{m\times n}$, our aim is to
approximate it with a high-rank low-precision $\widetilde{{\bm{W}}}$
and a low-rank high-precision term ${\bm{C}}_{k}\in{\mathbb{R}}^{m\times n}$ with rank $k\ll\min(m,n)$.

$$ $\widetilde{{\bm{y}}}={\bm{x}}(\widetilde{{\bm{W}}}+{\bm{C}}_{k})$ (5) $$

This raises the question of the actual optimization target:
Should we minimize the weight reconstruction error $||{\bm{W}}-\widetilde{{\bm{W}}}||_{F}$ or the output reconstruction error $||{\bm{y}}-\widetilde{{\bm{y}}}||_{2}$?
We separate these two problems and introduce them formally below.

###### Problem 1 (Minimization of weight error) .

For a pretrained linear layer ${\bm{y}}={\bm{x}}{\bm{W}}$ and its approximated form $\widetilde{{\bm{y}}}={\bm{x}}(\widetilde{{\bm{W}}}+{\bm{C}}_{k})$,
reconstructing the quantization error by minimizing weight approximation error has the following objective:

$$ $\operatorname*{arg\,min}_{{\bm{C}}_{k}}||{\bm{W}}-\widetilde{{\bm{W}}}-{\bm{C} }_{k}||_{F}$ (6) $$

where $\|\cdot\|_{F}$ denotes the Frobenius norm.

Solution to Problem 1. From the Eckart-Young-Mirsky theorem [],
the optimal solution to Problem 1 with respect to rank $k$ is the truncated SVD of the weight error matrix:

$$ ${\bm{C}}_{k}={\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}$ (7) $$

where ${\bm{U}}$, ${\bm{\Sigma}}$, and ${\bm{V}}^{T}$ form the SVD of the weight quantization error,
${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{W}}-\widetilde{{\bm{W}}})$.

As noted in Section II,
most existing works [, , ] in QPEFT and PTQ adopt this solution.
However, we know that minimizing the weight approximation error is not equivalent to minimizing the layer output error.
Furthermore, does minimizing the weight approximation error for each layer in a network effectively reduce the final model output error?
We will show that the answer is negative in Section IV-B.

###### Problem 2 (Minimization of layer output error) .

For a pretrained linear layer ${\bm{y}}={\bm{x}}{\bm{W}}$ and its approximated form $\widetilde{{\bm{y}}}={\bm{x}}(\widetilde{{\bm{W}}}+{\bm{C}}_{K})$,
approximating the layer by minimizing the error between ${\bm{y}}$ and $\widetilde{{\bm{y}}}$ is to minimize the following expectation.

$$ $\operatorname*{arg\,min}_{{\bm{C}}_{k}}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{ ||\widetilde{{\bm{y}}}-{\bm{y}}||_{2}^{2}\}$ (8) $$

where $||\cdot||_{2}$ denotes $l_{2}$ norm, and ${\mathbb{Y}}\subseteq{\mathbb{R}}^{n}$ is output space of the layer.
We expand Equation 8 by substituting $\widetilde{{\bm{y}}}$ and ${\bm{y}}$:

$$ $\operatorname*{arg\,min}_{{\bm{C}}_{k}}\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{ ||{\bm{x}}(\widetilde{{\bm{W}}}+{\bm{C}}_{k})-{\bm{x}}{\bm{W}}||_{2}^{2}\}$ (9) $$

where ${\mathbb{X}}\subseteq{\mathbb{R}}^{m}$ is the input space.
In practice, the expectation can be approximated as a sample mean on a calibration dataset like a subset of the pretraining data set.

Problem 2 motivates some recent works [, , ] to involve activation-induced heuristics
in the optimization of ${\bm{C}}_{k}$. However, to our best knowledge,
there is no work providing an analytical solution to this problem.
In the following two sections, we will derive the analytical solution to Problem 2.
More precisely, we present two solutions: one exact solution in Section III-B and an approximated solution based on a suitable statistical assumption in Section III-C.

###### Problem 1 (Minimization of weight error) .

###### Problem 2 (Minimization of layer output error) .

### III-B QERA-exact: Analytical Solution

QERA-exact is our exact solution to Problem 2.
QERA-exact is computationally expensive as it calculates the autocorrelation matrix of the input space ${\mathbb{X}}$.
However, as we will show in Section IV, QERA-exact recovers significant model performance in extremely low-precision quantization.

###### Theorem 1 (QERA-exact solution) .

The solution to Problem 2 is

$$ $\begin{split}{\bm{C}}_{k}&=\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1} {2}}\right)^{-1}{\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}\end{split}$ (10) $$

where ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$ is the autocorrelation matrix respect to the input space ${\mathbb{X}}$,

$$ ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left \{{\bm{x}}^{T}{\bm{x}}\right\}$ (11) $$

${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}$ represents the unique symmetric positive semi-definite matrix square root of ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$, and
${\bm{U}}_{:,:k}$, ${\bm{\Sigma}}_{:k,:k}$, and ${\bm{V}}_{:k,:}$ form the truncated SVD of the following scaled weight error matrix,

$$ ${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{R}}_{{\mathbb{X}}{\mathbb{ X}}}^{\frac{1}{2}}({\bm{W}}-\widetilde{{\bm{W}}}))$ (12) $$

###### Remark 1 .

${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}$ is positive semi-definite. In the event that it has a zero eigenvalue, it would be normal to add a small diagonal perturbation to recover invertibility.
In practice, we ran extensive experiments and find that ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}$ is invertible for all the pretrained models and datasets we present in Section IV.

Proof of Theorem 1

###### Proof.

Define ${\bm{P}}:=\widetilde{{\bm{W}}}+{\bm{C}}_{k}-{\bm{W}}$,
and ${\bm{p}}_{i}:={\bm{P}}_{i,:}$ is the $i$-th row of ${\bm{P}}$.
Then we substitute $(\widetilde{{\bm{W}}}+{\bm{C}}_{k}-{\bm{W}})$ in the expanded objective Equation 9 of Problem 2 with ${\bm{P}}$:

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{||{\bm{x}}{\bm{P}} ||_{2}^{2}\}\\ &=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{||\sum_{i=1}^{m}x_{i}{\bm{p}}_{i}||_{ 2}^{2}\}\\ &=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left\{\sum_{i=1}^{m}\sum_{j=1}^{m}x_{i }x_{j}{\bm{p}}_{i}{\bm{p}}_{j}^{T}\right\}\end{split}$ (13) $$

We rewrite the last line of Equation 13 as:

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left\{{\bm{e}}\cdot \left(({\bm{x}}^{T}{\bm{x}})\odot({\bm{P}}{\bm{P}}^{T})\right)\cdot{\bm{e}}^{T }\right\}\end{split}$ (14) $$

where ${\bm{e}}=\begin{bmatrix}1&1&\ldots&1\end{bmatrix}$ is a row vector of $m$ ones, and $\odot$ denotes the element-wise product.

Using the property of the element-wise product [], the RHS of the above can be simplified.

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left\{\operatorname {Tr}\left(({\bm{x}}^{T}{\bm{x}})({\bm{P}}{\bm{P}}^{T})^{T}\right)\right\}\\ &=\operatorname{Tr}\left(\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left\{{\bm{x}}^ {T}{\bm{x}}\right\}{\bm{P}}{\bm{P}}^{T}\right)\\ &=\operatorname{Tr}\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}{\bm{P}}{\bm{P}}^{ T}\right)\end{split}$ (15) $$

where $\operatorname{Tr}(\cdot)$ denotes trace and ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}=\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\left
\{{\bm{x}}^{T}{\bm{x}}\right\}$ is the autocorrelation matrix with respect to the input space ${\mathbb{X}}$.

Since ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$ is a symmetric positive semi-definite matrix, it always has precisely one matrix square root,
denoted as ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}$,
that is also symmetric and positive semi-definite []. We reorganize Equation 15 as the following
since both ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$ and $({\bm{P}}{\bm{P}}^{T})$ are symmetric and positive semi-definite:

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\operatorname{Tr}\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}} }^{\frac{1}{2}}{\bm{P}}{\bm{P}}^{T}{\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{ 1}{2}}\right)\\ &=\operatorname{Tr}\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}{\bm {P}}{\bm{P}}^{T}({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}})^{T}\right) \\ &=||{\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}{\bm{P}}||_{F}^{2}\end{split}$ (16) $$

Now the objective of Problem 2 ( Equation 8) is equivalent to:

$$ $\begin{split}\operatorname*{arg\,min}_{{\bm{C}}_{k}}\mathbb{E}_{{\bm{y}}\sim{ \mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{\bm{y}}||_{2}^{2}\}&=\operatorname*{arg \,min}_{{\bm{C}}_{k}}||{\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}{\bm{P }}||_{F}^{2}\\ &=\operatorname*{arg\,min}_{{\bm{C}}_{k}}||{\bm{R}}_{{\mathbb{X}}{\mathbb{X}}} ^{\frac{1}{2}}(\widetilde{{\bm{W}}}+{\bm{C}}_{k}-{\bm{W}})||_{F}^{2}\end{split}$ (17) $$

If we assign ${\bm{Q}}:={\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}({\bm{W}}-
\widetilde{{\bm{W}}})$ and ${\bm{Q}}_{k}:={\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}{\bm{C}}_{k}$, the objective becomes:

$$ $\operatorname*{arg\,min}_{{\bm{Q}}}||{\bm{Q}}_{k}-{\bm{Q}}||_{F}^{2}$ (18) $$

Note that multiplication by the invertible matrix ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}$ (Remark 1) does not change the rank of the matrix ${\bm{C}}_{k}$.
According to the Eckart-Young-Mirsky theorem [],
the optimal rank $k$ approximation to ${\bm{Q}}_{k}$ is the truncated SVD of ${\bm{Q}}$:

$$ ${\bm{Q}}_{k}={\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}$ (19) $$

where ${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{Q}})=\mathrm{SVD}\left({
\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}({\bm{W}}-\widetilde{{\bm{W}}}
)\right)$.
Thus the optimal rank-$k$ solution to ${\bm{C}}_{k}$ is:

$$ $\begin{split}{\bm{C}}_{k}=\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{ 2}}\right)^{-1}{\bm{Q}}_{k}=\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1 }{2}}\right)^{-1}{\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}\end{split}$ (20) $$

∎

In practice, we assign ${\bm{A}}_{k}:=\left({\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}^{\frac{1}{2}}\right)^{
-1}{\bm{U}}_{:,:k}$ and ${\bm{B}}_{k}:={\bm{\Sigma}}_{:k,:k}{\bm{V}}_{:k,:}^{T}$.
Note that QERA adds no constraints to the quantization (and dequantization) function $\mathrm{q}(\cdot)$ (and $\mathrm{dq}(\cdot)$),
i.e., the low-precision $\widetilde{{\bm{W}}}$ can be obtained by any quantization method.

###### Theorem 1 (QERA-exact solution) .

###### Remark 1 .

###### Proof.

### III-C QERA-approx: An Analytical Solution with the Uncorrelated Assumption

QERA-approx is our analytical solution to Problem 2
based on the assumption that different embedding dimensions are uncorrelated.
This solution is more computationally efficient than the exact solution,
and the assumption is testable on real-world datasets.
The complete proof of QERA-approx is in Section A-B.

###### Assumption 1 .

For a pretrained linear layer ${\bm{y}}={\bm{x}}{\bm{W}}$, the expectation of the product of different embedding dimensions is zero:

$$ $\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{i}x_{j}\}=0,\quad\forall i\neq j$ (21) $$

where $x_{i}$ and $x_{j}$ are the $i$-th and $j$-th elements of the input vector ${\bm{x}}$.

We test this assumption on LLMs in Section V.

###### Theorem 2 (QERA-approx solution) .

The solution to Problem 2 based on Assumption 1 is:

$$ $\begin{split}{\bm{C}}_{k}&={\bm{S}}^{-1}{\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{ \bm{V}}^{T}_{:k,:}\end{split}$ (22) $$

where ${\bm{S}}$ is a diagonal matrix built from activation statistics,

$$ ${\bm{S}}=\mathrm{diag}(\sqrt{\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{1}^{2} \}},\sqrt{\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{2}^{2}\}},\ldots,\sqrt{ \mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{m}^{2}\}})$ (23) $$

and ${\bm{U}}$, ${\bm{\Sigma}}$, ${\bm{V}}^{T}$ form the SVD of the following scaled weight error matrix,

$$ ${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{S}}({\bm{W}}-\widetilde{{ \bm{W}}}))$ (24) $$

###### Remark 2 .

For the diagonal matrix ${\bm{S}}$ in Theorem 2 to be invertible,
we need $\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{i}^{2}\}\neq 0$ for all dimension $i$.
In practice, this is almost always true for pretrained layers because
no dimension in the input embeddings is always zero.

For implementation, we assign ${\bm{A}}_{k}:={\bm{S}}^{-1}{\bm{U}}_{:,:k}$ and ${\bm{B}}_{k}:={\bm{\Sigma}}_{:k,:k}{\bm{V}}_{:k,:}$ to
form the low-rank terms to save the memory and computation cost.
Interestingly, QERA-approx solution is similar to the activation-induced heuristics
in LQER [], which calibrates the average absolute value on the embedding dimension (Refer to Algorithm 2 in the Appendix).
In Section IV-C, we will show that our solution is more effective in practice and
resolves the discrepancy between the recovered model performance and the number of calibration samples in LQER.

###### Assumption 1 .

###### Theorem 2 (QERA-approx solution) .

###### Remark 2 .

## IV Experiments

In this section, we first introduce the experiment setup in Section IV-A.
Then we present the results of our experiments on QPEFT and PTQ in Section IV-B and Section IV-C respectively.

### IV-A Experiment Setup

We perform fine-tuning and quantization experiments separately, and compare with their respective SoTA methods. The experiments take around 6400 GPU hours in total. The hardware platform, separate GPU hours, software dependencies, and random seed settings can be found in Section A-C.

For fine-tuning experiments, we use Theorem 2, noted as QERA-approx, to initialize low-rank terms, and compare with full-finetuning, LoRA [], QLoRA [], and LoftQ []. Specifically, we adopt 5-iteration LoftQ,
which is the officially recommended setup.
We include both encoder-only model experiments (fine-tuning RoBERTa-base [] on GLUE [])
and decoder-only LLM experiments (fine-tuning LLaMA-2 [] and LLaMA-3.1 [] on continuous pretraining task SlimPajama [] and supervised fine-tuning task GSM8K []).
For each method/baseline, we sweep the learning rate and record the best result. The final results are averaged over three random seeds.
The learning rate ranges and batch sizes are listed in Section A-C1.

For quantization experiments, we use both Theorem 1, noted as QERA-exact, and Theorem 2 (QERA-approx)
to calculate the low-rank error reconstruction terms and report results separately. We compare with BF16, quantized model without error reconstruction terms ($w$-only),
ZeroQuant-V2 [], and LQER [] at different precision setups. We also include HQQ [], a leading 4-bit method that does not use quantization error reconstruction.
We quantize LLMs of various sizes and model family, including TinyLlama [], Gemma-2 [], Phi-3.5 [] and LLaMA-2/-3.1 [, ].
We use lm-evaluation-harness to report results on Wikitext2 [], ARC (challenge) [], BoolQ [], CommonSenseQA [],
Winogrande [], MMLU [], and BigBench-Hard [].
We also evaluate instruction-tuned model, Vicuna-v1.5 [], with AlpacaEval 2.0 [], which is an automatic evaluation tool for instruction-following tasks.
Detailed setup is in Section A-C2.

### IV-B Improved QPEFT

Figure: (a) Model output error vs. rank
Refer to caption: x1.png

We first identify a pitfall in the commonly-used iterative Algorithm 1, that is,
minimizing the weight approximation error for each layer does not necessarily minimize the model output error.
Then we show that our QERA initialization enables a clear reduction in the model output error at the start of fine-tuning,
leading to better fine-tuned accuracy/perplexity and faster convergence.

**TABLE I: Fine-tuning results of RoBERTa-base on GLUE. QERA-approx outperforms LoftQ across all bit widths, and the improvement is more obvious with aggressive quantization. QERA achieves $\Delta_{\text{acc}}$ = 4.12% higher than LoftQ at 3-bit and 6.05% at 2-bit.**
| Rank | W-bits | Method | MNLI | QNLI | RTE | SST | MRPC | CoLA | QQP | STSB | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc | Acc | Acc | Acc | Acc | Matt | Acc | P/S Corr |  |  |  |  |
| - | 16 | Full FT | 87.61 | 92.95 | 73.16 | 94.88 | 92.15 | 60.41 | 91.61 | 90.44/90.25 | 85.38 |
| 8 | 16 | LoRA | 87.85 | 92.84 | 69.55 | 94.46 | 89.99 | 57.52 | 89.83 | 89.92/89.83 | 84.00 |
| 8 | 4.25 | QLoRA | 87.21 | 92.32 | 63.90 | 94.08 | 88.24 | 56.08 | 90.55 | 89.59/89.56 | 82.75 |
| LoftQ (5-iter) | 87.27 | 92.48 | 67.13 | 94.38 | 88.24 | 54.59 | 90.51 | 88.75/88.79 | 82.92 |  |  |
| QERA-approx | 87.28 | 92.45 | 70.40 | 94.38 | 88.97 | 55.99 | 90.39 | 89.83/89.72 | 83.71 |  |  |
| 8 | 3.25 | QLoRA | 84.87 | 89.58 | 53.67 | 91.02 | 73.94 | 3.12 | 89.31 | 84.80/84.38 | 71.29 |
| LoftQ (5-iter) | 85.24 | 89.65 | 58.24 | 92.05 | 75.82 | 11.00 | 88.93 | 85.55/85.27 | 73.31 |  |  |
| QERA-approx | 85.58 | 90.74 | 58.48 | 92.59 | 82.19 | 32.98 | 89.41 | 87.43/87.08 | 77.43 |  |  |
| 64 | 2.50 | QLoRA | 77.87 | 85.26 | 54.15 | 90.02 | 71.00 | 0 | 87.93 | 74.72/75.31 | 67.62 |
| LoftQ (5-iter) | 80.15 | 87.65 | 52.95 | 90.94 | 74.35 | 3.43 | 89.17 | 82.76/82.90 | 70.18 |  |  |
| QERA-exact | 84.64 | 90.05 | 58.48 | 92.32 | 84.72 | 26.43 | 89.69 | 86.48/86.40 | 76.23 |  |  |

**TABLE II: Fine-tuning results of LLaMA-2-7B and LLaMA-3.1-8B on SlimPajama and GSM8K. A trend similar to RoBERTa experiments are observed, i.e., QERA outperforms QLoRA and LoftQ and the improvement is more obvious on aggressive quantization.**
| W-bits | Method | LLaMA-2-7B | LLaMA-3.1-8B |  |  |
| --- | --- | --- | --- | --- | --- |
| SlimPajama ($\Delta_{\text{ppl}}$) | GSM8K ($\Delta_{\text{acc}}$) | SlimPajama ($\Delta_{\text{ppl}}$) | GSM8K ($\Delta_{\text{acc}}$) |  |  |
| 16 | LoRA | 6.17 | 39.40 | 8.07 | 55.72 |
| 4.25 | QLoRA | 6.44 (+0.27) | 30.71 (-8.69) | 8.70 (+0.63) | 54.81 (-0.91) |
| LoftQ (5-iter) | 6.39 (+0.22) | 28.58 (-10.82) | 8.73 (+0.66) | 54.23 (-1.49) |  |
| QERA-approx | 6.33 (+0.16) | 32.26 (-7.14) | 8.68 (+0.61) | 55.24 (-0.48) |  |
| 2.25 | QLoRA | 53.95 (+47.78) | 12.79 (-18.31) | 71.90 (+63.83) | 5.08 (-50.64) |
| LoftQ (5-iter) | 12.30 (+6.13) | 18.37 (-12.73) | 27.16 (+19.09) | 13.72 (-42.00) |  |
| QERA-approx | 10.56 (+4.39) | 18.78 (-12.32) | 20.07 (+12.00) | 19.41 (-36.31) |  |

##### Reduced layer weight error ≠ \neq ≠ reduced model output error

Figure: Figure 2: Faster convergence of QERA-approx on STSB.
Refer to caption: x3.png

We apply 4-bit and 3-bit QLoRA, LoftQ, and QERA-approx to RoBERTa-base and
inspect the model output error on RoBERTa’s pretraining dataset before fine-tuning at rank $k=4,8,16,32$.
For LoftQ, we also sweep the number of iterations from 1 to 5.
In Figure 1, we observe that

- •
For LoftQ, given a specific rank, increasing the optimization iterations does not guarantee a reduced model output error. Though all the layers’ weight approximation errors monotonically decrease with the number of iterations (as illustrated in Figure 6 in Appendix), the model output error does not monotonically decrease. For example, in Figure 1(a), the model output error of LoftQ (5-iter) is larger than LoftQ (3-iter) at rank $k=8$.
- •
For LoftQ, given a specific number of iterations, increasing the rank does not guarantee a reduced model output error.
For example, in Figure 1(b), the output error of LoftQ (rank $k=16$) is larger than LoftQ (rank $k=4$) and $k=8$ at 2, 3, 4, and 5 iterations.
- •
The model output error of our QERA-approx is always smaller than LoftQ and QLoRA, across all precision and rank settings.
Moreover, the output error of QERA-approx monotonically decreases as the rank increases.

This empirical evidence suggests a strong correlation between the reduction of layer output error and the decrease in model output error. Conversely, minimizing weight approximation error does not demonstrate a comparable impact on overall model performance.

##### Better optimization quality

Table I and Table II summarize the fine-tuning experiments of RoBERTa-base on GLUE,
and LLaMA-2-7B/-3.1-8B fine-tuned on SlimPajama and GSM8K, respectively.
QERA outperforms both LoftQ and QLoRA. In GLUE experiments, at 4-bit, QERA enables an average accuracy gain of 0.96% and 0.79% higher than QLoRA and LoftQ respectively, close to BF16 LoRA;
At 3-bit and 2-bit, QERA achieves a 4.12% and 6.05% higher average accuracy than LoftQ respectively.
Similar trends are observed on LLM fine-tuning experiments, i.e.,
QERA outperforms QLoRA and LoftQ, and
the advantage of QERA over LoftQ is more obvious with more aggressive quantization.

##### Faster Convergence

QERA initialization also speeds up the training convergence.
For LLM fine-tuning, this is expected as QERA initialization is closer to the full-precision model.
Interestingly, in encoder-only experiments on GLUE where the model classifier head is randomly initialized,
we also observe that QERA converges faster,
especially on small subsets such as STSB and MRPC where only a few thousand samples are available
(in comparison MNLI has 393k samples and QQP has 364k samples).
For example, in Figure 2,
the Spearman correlation coefficient of QERA on STSB increases and converges faster than LoftQ and QLoRA, as the green line plateaus first.

### IV-C Improved PTQ

In this part, we first demonstrate that LQER, which depends on heuristics derived from activation values, does not guarantee improved performance with a larger calibration dataset. However, QERA exhibits the opposite trend.
Through extensive experiments, we show that
QERA consistently outperforms ZeroQuant-V2 and LQER,
and QERA-exact exhibits better model performance than QERA-approx at the cost of more computation in the quantization process.
These results verified the effectiveness of our analytical solution.

Figure: Figure 3: QERA resolves the discrepancy between the recovered model performance and the number of calibration samples in LQER.
Refer to caption: x4.png

##### Model performance vs. calibration set size

As mentioned at the end of Section III-C,
the scale matrix in LQER [] is similar to
the one in QERA-approx, but is based on hand-crafted heuristics. As a result, we observe that
the model performance of LQER varies randomly as the number of calibration samples increases
(the purple curve in Figure 3).
On the contrary, more calibration samples consistently lead to better model performance for QERA until convergence.

##### Improved perplexity and downsteam task accuracy

We apply QERA-approx and QERA-exact to a range of models and
evaluate on both pretraining task and downstream tasks in Table III
and Table IV respectively.
We also compare to HQQ, a SoTA method that does not use quantization error reconstruction.

**TABLE III: Perplexity ($\downarrow$) of LLMs on WikiText2. $w$-only denotes the quantized model without low-rank error reconstruction. QERA-approx outperforms LQER on almost all setups and QERA-exact achieves the lowest perplexity. The advantage of QERA is pronounced at 3-bit.**
| W-bits | Method | Rank | TinyLlama | Gemma-2 | Phi-3.5 | LLaMA-2 | LLaMA-3.1 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.1B | 2B | 3.8B | 7B | 13B | 8B | 80B |  |  |  |
| - | BF16 | - | 13.98 | 13.08 | 11.50 | 8.71 | 7.68 | 7.55 | 3.06 |
| 4.25 | HQQ | - | 15.02 | 14.29 | 14.63 | 9.59 | 8.27 | 8.72 | 3.97 |
| 4.25 | $w$-only | - | 19.40 | 16.23 | 14.16 | 9.45 | 8.06 | 8.78 | 4.55 |
| ZeroQuant-V2 | 32 | 18.03 | 15.71 | 14.09 | 9.42 | 8.07 | 8.83 | 4.48 |  |
| LQER | 16.23 | 14.55 | 12.88 | 9.22 | 7.96 | 8.45 | 4.10 |  |  |
| QERA-approx | 15.66 | 14.60 | 12.81 | 9.17 | 7.95 | 8.45 | 4.10 |  |  |
| QERA-exact | 16.16 | 14.12 | 12.30 | 9.12 | 7.93 | 8.33 | 3.82 |  |  |
| 3.25 | $w$-only | - | 32.82 | 41.13 | 47.78 | 13.32 | 10.24 | 18.96 | 16.46 |
| ZeroQuant-V2 | 64 | 27.80 | 33.56 | 42.64 | 13.00 | 10.03 | 19.29 | 10.12 |  |
| LQER | 20.60 | 21.99 | 18.27 | 14.00 | 9.09 | 11.86 | 7.05 |  |  |
| QERA-approx | 20.43 | 21.93 | 17.99 | 10.99 | 9.04 | 11.73 | 6.99 |  |  |
| QERA-exact | 19.51 | 19.97 | 20.37 | 10.67 | 8.97 | 11.39 | 6.68 |  |  |

**TABLE IV: Average accuracy ($\uparrow$) of LLMs on six downstream tasks. QERA-exact outperforms other quantization-error reconstruction-based methods across almost all models. We also compare to HQQ [], a SoTA PTQ method that does not adopt quantization-error reconstruction or activation heuristics. QERA-exact achieves an average accuracy on par with HQQ.**
| W-bits | Method | Rank | TinyLlama(^1^11The average accuracy of TinyLlama-1.1B excludes BoolQ, CommonsenseQA, and MMLU since TinyLlama-1.1B has random guess accuracy on these tasks.) | Gemma-2 | Phi-3.5 | LLaMA-2 | LLaMA-3.1 |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 1.1B | 2B | 3.8B | 7B | 13B | 8B | 80B |  |  |  |
| - | BF16 | - | 40.59 | 53.96 | 66.91 | 49.61 | 55.74 | 63.88 | 72.05 |
| 4.25 | HQQ | - | 40.35 | 52.54 | 59.17 | 48.26 | 54.53 | 62.59 | 71.31 |
| 4.25 | $w$-only | - | 36.56 | 48.33 | 64.52 | 47.62 | 55.12 | 61.53 | 68.46 |
| ZeroQuant-V2 | 32 | 37.26 | 48.24 | 64.44 | 47.43 | 55.15 | 61.70 | 68.45 |  |
| LQER | 40.45 | 49.77 | 64.46 | 48.47 | 55.40 | 61.75 | 70.94 |  |  |
| QERA-approx | 40.02 | 49.29 | 64.53 | 48.52 | 55.20 | 61.68 | 70.80 |  |  |
| QERA-exact | 40.36 | 51.73 | 65.08 | 48.91 | 55.42 | 62.05 | 71.42 |  |  |

On most models, QERA-approx outperforms ZeroQuant-V2 and LQER,
while QERA-exact achieves the best performance.
At 4-bit, QERA-exact is nearly lossless.
At 3-bit, QERA-exact’s improvement over QERA-approx (Table III) is clear,
indicating the superiority of QERA-exact for aggressive quantization.

##### Higher win rate on AlpacaEval 2.0

To better understand the impact on instruction-tuned models,
we present the results of Vicuna-7b-v1.5 on AlpacaEval 2.0.
In Figure 4, we evaluate the quantization-error-reconstruction-based methods against the $w$-only quantization counterpart.
QERA outperforms ZeroQuant-V2 and LQER by a higher win rate, indicating a better response quality.

## V Discussion

Figure: Figure 4: AlpacaEval 2.0 evaluation results. We compare quantized models to the counterpart without quantization-error reconstruction. A higher win rate ($\uparrow$) indicates better instruction-following performance.
Refer to caption: x5.png

In this section, we revisit the arguments, design choices, and observations made
in the previous sections, including a test of Assumption 1,
and the choice of the calibration set for PEFT.
An extended discussion of numeric stability and scalability of QERA is in Section A-G, and LoRA rank and model choices of PEFT experiments is in Section A-H.

##### Test of Assumption 1

To test Assumption 1,
we profile the autocorrelation matrix ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$ of the linear layer inputs in LLaMA-2-7B and LLaMA-3-8B.
Note that ${\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}}_{i,j}=\mathbb{E}_{{\bm{x}}\sim{\mathbb
{X}}}\{x_{i}x_{j}\}$,
which assumes to be zero for $i\neq j$ in Assumption 1.
Figure 5
shows the normalized ${\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}}$ magnitude, $\frac{\mathrm{abs}(\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}})}{||\mathbb{R}_{{
\mathbb{X}}{\mathbb{X}}}||_{F}}$, of four representative layers in LLaMA-3-8B
where darker elements denote values closer to zero. There are several layers with some input dimensions strongly correlated with others,
such as the inputs to the third attention layer in Figure 5(a),
but for most layers, our assumption holds, especially the MLP layers, such as Figures 5(b), 5(c) and 5(d).
More ${\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}}$ plots are in Section A-E.

##### Choice of calibration set for QPEFT

One problem is to determine the calibration set for QERA before fine-tuning.
In 2-bit RoBERTa-base fine-tuning experiment on SST2 (Section A-F),
we find that calibrating on the pretraining dataset, WikiText2, helps the loss to decrease.
However, the loss of the model calibrated on the fine-tuning dataset does not follow the same trend.
We hypothesize that the massive padding tokens in preprocessed SST2 samples cause this discrepancy, especially considering
that the sequence length of the raw SST2 dataset changes fiercely.

Figure: (a) 3.o_proj
Refer to caption: x6.png

## VI Conclusion

In this paper, we formulate the problem of quantization error reconstruction and propose QERA as an analytical solution.
Applying QERA to related works for efficient fine-tuning or inference,
we show that QERA resolves the discrepancy in existing methods,
and outperforms SoTA methods in both fine-tuning and quantization tasks by a clear margin.

## Appendix A Appendix

### A-A Algorithms in Related Work

Here we summarize the algorithm of LoftQ [] in Algorithm 1 and LQER [] in Algorithm 2 respectively.
LQ-LoRA [] adopts a variant of Algorithm 1.
ZeroQuant-V2 [] can be considered as Algorithm 1 with one iteration, or a special case of Algorithm 2 where the scale matrix ${\bm{S}}$ is an identity matrix.

Figure: Algorithm 1 LoftQ []

Figure: Algorithm 2 LQER []

### A-B Proof of Theorem 2

Here we present the full proof of QERA-approx. QERA-approx is an approximated solution to Problem 2 based on Assumption 1, which is suitable to initialize the low-rank terms in fine-tuning for lower computation complexity.

Proof of Theorem 2

###### Proof.

We continue at Equation 13.
Since $\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}$ is the expectation with respect to the input space,
we move the expectation inside the summation of RHS of Equation 13.

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}=\sum_{i=1}^{m}\sum_{j=1}^{m}\mathbb{E}_{{\bm{x}}\sim{ \mathbb{X}}}\{x_{i}x_{j}\}{\bm{p}}_{i}{\bm{p}}_{j}^{T}\end{split}$ (25) $$

Under Assumption 1,
$\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{i}x_{j}\}=0$ for $i\neq j$,
the RHS of Equation 25 simplifies to:

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\sum_{i=1}^{m}\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{i }^{2}\}{\bm{p}}_{i}{\bm{p}}_{i}^{T}\end{split}$ (26) $$

We can define diagonal matrix ${\bm{S}}=\mathrm{diag}(\sqrt{\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{1}^{2}
\}},\sqrt{\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{2}^{2}\}},\ldots,\sqrt{
\mathbb{E}_{{\bm{x}}\sim{\mathbb{X}}}\{x_{m}^{2}\}})$
and rewrite the RHS of Equation 26 as:

$$ $\begin{split}\mathbb{E}_{{\bm{y}}\sim{\mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{ \bm{y}}||_{2}^{2}\}&=\operatorname{Tr}({\bm{S}}{\bm{P}}{\bm{P}}^{T}{\bm{S}}^{T })=||{\bm{S}}{\bm{P}}||_{F}^{2}\end{split}$ (27) $$

where $\operatorname{Tr}(\cdot)$ denotes the trace of a matrix.

Therefore, the objective of Problem 2 (Equation 8) is equivalent to:

$$ $\begin{split}\operatorname*{arg\,min}_{{\bm{C}}_{k}}\mathbb{E}_{{\bm{y}}\sim{ \mathbb{Y}}}\{||\widetilde{{\bm{y}}}-{\bm{y}}||_{2}^{2}\}&=\operatorname*{arg \,min}_{{\bm{C}}_{k}}||{\bm{S}}{\bm{P}}||_{F}^{2}\\ &=\operatorname*{arg\,min}_{{\bm{C}}_{k}}||{\bm{S}}(\widetilde{{\bm{W}}}+{\bm{ C}}_{k}-{\bm{W}})||_{F}^{2}\end{split}$ (28) $$

If we assign ${\bm{Q}}={\bm{S}}({\bm{W}}-\widetilde{{\bm{W}}})$ and ${\bm{Q}}_{k}={\bm{S}}{\bm{C}}_{k}$, the objective becomes:

$$ $\operatorname*{arg\,min}_{{\bm{Q}}}||{\bm{Q}}_{k}-{\bm{Q}}||_{F}^{2}$ (29) $$

Note that the invertible matrix ${\bm{S}}$ in ${\bm{Q}}_{k}$ does not change the rank of the matrix ${\bm{C}}_{k}$.
According to the Eckart-Young-Mirsky theorem,
the optimal rank $k$ approximation to ${\bm{Q}}$ is the truncated SVD of ${\bm{Q}}$:

$$ ${\bm{Q}}_{k}={\bm{U}}_{:,:k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}$ (30) $$

where ${\bm{U}}{\bm{\Sigma}}{\bm{V}}^{T}=\mathrm{SVD}({\bm{Q}})=\mathrm{SVD}\left({
\bm{S}}({\bm{W}}-\widetilde{{\bm{W}}})\right)$.

Finally, we get the optimal solution to the low-rank term ${\bm{C}}_{k}$:

$$ $\begin{split}{\bm{C}}_{k}={\bm{S}}^{-1}{\bm{Q}}_{k}={\bm{S}}^{-1}{\bm{U}}_{:,: k}{\bm{\Sigma}}_{:k,:k}{\bm{V}}^{T}_{:k,:}\end{split}$ (31) $$

∎

###### Proof.

### A-C Detailed Experiment Setup

We mainly use [PyTorch](https://pytorch.org/), [Transformers](https://github.com/huggingface/transformers), [PEFT](https://github.com/huggingface/peft), and [Accelerate](https://github.com/huggingface/accelerate/tree/main) to implement QERA. We use [SciPy](https://docs.scipy.org/doc/scipy/index.html)’s implementation of blocked Schur algorithm [] to calculate the matrix square root, which runs on CPUs. The evaluation is performed with [lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness), [Evaluate](https://github.com/huggingface/evaluate/tree/main), and [AlpacaEval 2.0](https://github.com/tatsu-lab/alpaca_eval) [].

#### A-C 1 QPEFT Hyperparameters

We perform fine-tuning experiments on four NVIDIA A100 80GB GPUs with AMD EPYC 64-Core Processor with 1024GB RAM.
The total fine-tuning time is around 2100 GPU hours.

##### RoBERTa-base on GLUE

We sweep learning rates for each baseline/method, and collect the best accuracy. The reported accuracy is the average value across random seeds 42, 1, and 2. The total batch size is 64 for all GLUE experiments and we train the models for 5 epochs. For 4-bit experiments, we use 4-bit floating point from the QLoRA implementation in PEFT.
For 3-bit experiments, we use emulated MXINT [] with block size = 32 and for 2-bit experiments we use MXINT with block size = 16.
Table V lists the learning rates for each experiment.

##### LLaMA-2-7B/-3.1-8B on SlimPajama and GSM8K

We adopt the learning rates in meng2024pissa []. The reported perplexity/accuracy is the average value across random seeds 42, 1, and 2. For SlimPajama, we fine-tune the model on a subset for 1000 steps with rank = 8, total batch size = 64, sequence length = 1024, learning rate = 3e-5. For GSM8K, we fine-tune the model for 10 epochs with rank = 64, total batch size = 128, sequence length = 384, and learning rate = 3e-5.

**TABLE V: Learning rates of RoBERTa-base experiments on GLUE.**
| Rank | W-bits | Method | Learning rates |
| --- | --- | --- | --- |
| - | 16 | Full FT | 7e-5, 5e-5, 2e-5 |
| 8 | 16 | LoRA | 1e-4, 2e-4, 3e-4 |
| 8 | 4.25 | QLoRA/LoftQ/QERA-approx | 1e-4, 2e-4, 3e-4 |
| 8 | 3.25 | QLoRA/LoftQ/QERA-approx | 1e-4, 2e-4, 3e-4 |
| 64 | 2.50 | QLoRA/LoftQ/QERA-exact | 2e-5, 3e-5, 4e-5, 5e-5, 6e-5, 9e-5, 1e-4 |

#### A-C 2 PTQ Hyperparameters

We perform PTQ experiments on eight NVIDIA A6000 48GB GPUs with AMD EPYC 256-Core Processor with 1024GB RAM.
The total quantization and evaluation time is around 4500 GPU hours. We report 0-shot accuracy or normalized accuracy (if available) for all tasks except WikiText2, in which we report word perplexity.
The sequence length for reporting word perplexity is the model’s context length by default, except for Phi-3.5 and LLaMA-3.1. For these two models, we set the sequence length = 2048.
We use the HuggingFace Transformers’s implementation of HQQ, and reimplement ZeroQuant-V2 and LQER as baselines.
We use MXINT with block size = 32 as the quantization format for all quantization methods except HQQ, which uses its built-in INT format with group size = 64.
Thus, both formats have an average W-bits of 4.25.
We evaluate quantized Vicuna-v1.5-7B, which is an instruction-tuned LLaMA-2-7B, with AlpacaEval 2.0. and use GPT4-Turbo as the evaluator. The reported win rate is the length-controlled win rate, which is a debiased version of the win rate that controls for the length of the generated outputs.

### A-D Decreasing Weight Error ≠ \neq ≠ Decreasing Output Error for LoftQ

We provide the weight approximation error, $||{\bm{W}}-\widetilde{{\bm{W}}}-{\bm{C}}_{k}||_{F}$, in Figure 6, of all linaer layers in RoBERTa-base by sweeping the number of iterations.
We observe that the weight approximation error monotonically decreases with the number of iterations, but as shown in Figure 1, the model output error may increase.
This observation indicates that the commonly used objective of minimizing the weight approximation error and the corresponding algorithm are not ideal for the quantization error reconstruction problem.

Figure: Figure 6: Weight approximation error of 3-bit rank-16 LoftQ with different numbers of iterations on RoBERTa-base. We observe that the weight reconstruction error of all the layers decreases as the number of iterations increases, but as shown in Figure 1(b), the model output error ($k$=16) increases from the 4-th to 5-th iteration.
Refer to caption: x10.png

### A-E Test of Assumption 1

We provide more plots of normalized ${\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}}$ magnitude, $\frac{\mathrm{abs}(\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}})}{||\mathbb{R}_{{
\mathbb{X}}{\mathbb{X}}}||_{F}}$, in Figures 7, 8, 9 and 10,
where dark pixels are elements close to zeros. There are strongly correlated embedding channels in some k_proj and o_proj layers.
The assumption fits better in MLP layers (gate_proj, up_proj, and down_proj).

Figure: Figure 7: Normalized $\mathrm{abs}({\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}})$ of inputs of k_proj layers in LLaMA-3-8B. Note that the q_proj and v_proj share the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Refer to caption: x11.png

Figure: Figure 8: Normalized $\mathrm{abs}({\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}})$ of inputs of o_proj layers in LLaMA-3-8B. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Refer to caption: x12.png

Figure: Figure 9: Normalized $\mathrm{abs}({\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}})$ of inputs of gate_proj layers in LLaMA-3-8B. Note that the up_proj shares the same inputs. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Refer to caption: x13.png

Figure: Figure 10: Normalized $\mathrm{abs}({\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}})$ of inputs of down_proj layers in LLaMA-3-8B. Layers are sampled and only the first 96 dimensions are plotted for clarity.
Refer to caption: x14.png

### A-F Choice of Calibration Set

We compare the QERA-adapted models calibrated on the pretraining dataset and the downstream dataset.
Specifically, we fine-tune two QERA-adapted 2-bit RoBERTa-base models. One is calibrated on its pretraining dataset, WikiText2, and the other on SST2.
Figure 11 shows the loss curves of the two models across three learning rates.
None loss curves of the models calibrated on SST2 decreases,
but the ones calibrated on WikiText2 successfully decrease and converge.
We hypothesize that this is due to the massive padding tokens in preprocessed SST2 considering that the raw sample lengths change fiercely.
However, WikiText2 samples were preprocessed in the masked language modeling style, which means that only a few special tokens are added to the grouped texts.

Figure: Figure 11: The fine-tuning loss curves of QERA-adapted 2-bit RoBERTa-base on SST2. The loss fails to decrease if the calibration is performed on the downstream task SST2 due to the massive padding tokens in preprocessed SST2 samples. In pretraining dataset, there are only a few special tokens like padding tokens and mask tokens.
Refer to caption: x15.png

Figure: (a) Estimated error ratio of the square root of $\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}$
Refer to caption: x16.png

### A-G Scalability and Numerical Stability of QERA

One may notice the diminishing model performance improvement of QERA-exact over QERA-approx as the model size increases.
The main reason is that larger LLMs are more resistant to quantization [].
Another reason can be the error ratio of the matrix square root calculation of the autocorrelation matrix increases with model hidden size (Figure 12(a)).

We find that the data type used in the calibration is important for the numeric stability of QERA-exact due to the calculation of the matrix square root and SVD.
To improve the stability of the calculation in QERA-exact, a good practice we find is to perform the outer product of ${\bm{R}}_{{\mathbb{X}}{\mathbb{X}}}$ in FP32,
accumulated outer product in FP64, and calculate the matrix square root in FP64 using the blocked Schur algorithm []. Figure 12(b) illustrates the quantization time of QERA-approx and QERA-exact on the platform described in Section A-C where the linear layers are quantized sequentially. QERA-exact is slow due to the calculation of matrix square roots on CPUs. GPU-accelerated matrix square root will be the key optimization to reduce the quantization time. Note that in QERA, the quantization of individual layers is independent, allowing more parallelization and acceleration of the quantization process.

### A-H Choices of LoRA Ranks and Models

##### Rank = 8 for GLUE experiments

We notice LoftQ paper uses a large rank of 16 and 32 for fine-tuning on GLUE, which is larger than the commonly-used rank value of LoRA (4 or 8 in LoRA paper []). If we consider LoRA as the upper limit of QLoRA-like QPEFT methods (including LoftQ and QERA), to effectively compare these QPEFT methods, one easy way is to set the rank as the minimum value required by LoRA and check which QPEFT method achieves an accuracy closest to LoRA. This is why we choose rank = 8 for GLUE experiments (For 2-bit GLUE experiments we use a large rank 64 since the quantization is very aggressive). If we use rank = 32, LoRA and all the QPEFT methods may be over-parameterized and it will be hard to make a fair comparison in terms of fine-tuned accuracy.

##### RoBERTa vs. DeBERTa

When investigating the related work, we find that both RoBERTa and DeBERTaV3 [] are used in QPEFT experiments [, , , , ]. The reason why we chose RoBERTa is that the RoBERTa checkpoint on HuggingFace(^2^22RoBERTa-base checkpoint: [link](https://huggingface.co/FacebookAI/roberta-base)) is complete and compatible with both HuggingFace’s official examples of sequence classification(^3^33HuggingFace example of sequence classification: [link](https://github.com/huggingface/transformers/tree/main/examples/pytorch/text-classification)) and masked language modeling(^4^44HuggingFace example of masked language modeling: [link](https://github.com/huggingface/transformers/tree/main/examples/pytorch/language-modeling)).
Specifically, the RoBERTa checkpoint contains both the base model and the masked language modeling head but the DeBERTaV3’s checkpoint(^5^55DeBERTaV3’s checkpoint: [link](https://huggingface.co/microsoft/deberta-v3-base)) only contains the base model. As we know, the base model is enough for fine-tuning on downstream tasks.
However, to calibrate on the pretraining dataset, we need the language modeling head to verify if our implementation of data preprocessing and calibration matches how the model was originally pretrained. Note that the quality of the statistic values in QERA like ${\mathbb{R}_{{\mathbb{X}}{\mathbb{X}}}}$ depends on the quality of the calibration set.
Thus, without the language modeling head in the checkpoint, we cannot perform the QERA’s calibration for DeBERTaV3 properly, ensure the correctness of statistics in QERA, and explore the effect of the choice of calibration sets.

### A-I Detailed PTQ Results

Here we offer the detailed evaluation results for each downstream task in Tables VI, VII, VIII, IX, X, XI and XII.

**TABLE VI: Post-training quantization evaluation of TinyLlama-1.1B.**
| rank | Method | w-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 32.51 | 55.93 | 20.07 | 29.68 | 25.35 | 13.98 | 59.59 |
| - | HQQ | 4.25 | 32.00 | 58.13 | 20.15 | 29.70 | 25.75 | 15.02 | 59.35 |
| - | $w$-only | 28.67 | 58.23 | 19.49 | 28.99 | 23.81 | 19.40 | 52.01 |  |
| 32 | ZeroQuant-V2 | 29.69 | 57.86 | 19.41 | 29.53 | 24.85 | 18.03 | 52.57 |  |
| LQER | 32.00 | 52.42 | 18.59 | 29.60 | 25.31 | 16.23 | 59.75 |  |  |
| QERA-approx | 31.83 | 52.08 | 17.20 | 29.51 | 25.22 | 15.66 | 58.72 |  |  |
| QERA-exact | 32.00 | 51.31 | 19.33 | 29.42 | 25.19 | 16.16 | 59.67 |  |  |

**TABLE VII: Post-training quantization evaluation of Gemma-2-2B.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 49.91 | 72.60 | 50.29 | 32.67 | 49.44 | 13.08 | 68.82 |
| - | HQQ | 4.25 | 48.81 | 71.77 | 48.40 | 32.32 | 46.52 | 14.29 | 67.40 |
| - | w-only | 44.62 | 69.91 | 34.07 | 31.96 | 42.90 | 16.23 | 66.54 |  |
| 32 | ZeroQuant-V2 | 44.45 | 69.94 | 34.07 | 31.50 | 43.27 | 15.71 | 66.22 |  |
| LQER | 46.08 | 68.84 | 37.59 | 32.60 | 45.78 | 14.55 | 67.72 |  |  |
| QERA-approx | 45.31 | 68.99 | 36.20 | 32.04 | 45.80 | 14.60 | 67.40 |  |  |
| QERA-exact | 46.84 | 72.32 | 42.75 | 33.36 | 47.29 | 14.12 | 67.80 |  |  |

**TABLE VIII: Post-training quantization evaluation of Phi3-3.5-mini.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 59.39 | 84.65 | 71.91 | 48.19 | 64.58 | 11.50 | 72.77 |
| - | HQQ | 4.25 | 57.00 | 74.34 | 60.20 | 38.22 | 56.00 | 14.63 | 69.61 |
| - | w-only | 59.73 | 82.72 | 68.22 | 44.45 | 61.54 | 14.16 | 70.48 |  |
| 32 | ZeroQuant-V2 | 59.64 | 82.94 | 68.06 | 44.58 | 62.00 | 14.09 | 69.77 |  |
| LQER | 59.39 | 84.01 | 70.76 | 45.67 | 62.21 | 12.88 | 70.74 |  |  |
| QERA-approx | 59.45 | 84.82 | 70.84 | 45.67 | 62.26 | 12.81 | 70.17 |  |  |
| QERA-exact | 58.70 | 83.73 | 69.45 | 45.37 | 62.01 | 13.00 | 71.19 |  |  |

**TABLE IX: Post-training quantization evaluation of LLaMA-2-7B.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 46.25 | 77.83 | 33.09 | 30.74 | 40.64 | 8.71 | 69.14 |
| - | HQQ | 4.25 | 44.03 | 75.87 | 29.40 | 30.50 | 40.14 | 9.59 | 69.61 |
| - | w-only | 45.22 | 75.87 | 25.47 | 30.71 | 40.03 | 9.45 | 68.43 |  |
| 32 | ZeroQuant-V2 | 45.82 | 75.90 | 24.82 | 29.99 | 39.84 | 9.42 | 68.19 |  |
| LQER | 44.28 | 76.15 | 29.81 | 30.72 | 40.66 | 9.22 | 69.22 |  |  |
| QERA-approx | 44.28 | 75.96 | 30.96 | 30.72 | 40.59 | 9.17 | 68.59 |  |  |
| QERA-exact | 44.80 | 76.39 | 31.61 | 30.57 | 40.86 | 9.12 | 69.22 |  |  |

**TABLE X: Post-training quantization evaluation of LLaMA-2-13B.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 49.49 | 80.58 | 47.34 | 32.65 | 52.18 | 7.68 | 72.22 |
| - | HQQ | 4.25 | 49.06 | 78.69 | 45.05 | 32.41 | 50.85 | 8.27 | 71.11 |
| - | w-only | 50.43 | 80.58 | 44.06 | 33.45 | 50.21 | 8.06 | 71.98 |  |
| 32 | ZeroQuant-V2 | 50.00 | 81.04 | 44.47 | 33.50 | 50.31 | 8.07 | 71.59 |  |
| LQER | 51.02 | 81.25 | 44.47 | 32.41 | 51.24 | 7.96 | 71.98 |  |  |
| QERA-approx | 51.11 | 80.83 | 44.06 | 32.48 | 51.07 | 7.95 | 71.67 |  |  |
| QERA-exact | 50.77 | 81.10 | 44.55 | 32.91 | 51.23 | 7.93 | 71.98 |  |  |

**TABLE XI: Post-training quantization evaluation of LLaMA-3.1-8B.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 53.50 | 82.05 | 71.42 | 39.07 | 63.27 | 7.55 | 73.95 |
| - | HQQ | 4.25 | 52.73 | 81.19 | 69.86 | 35.60 | 62.14 | 8.72 | 74.03 |
| - | w-only | 50.68 | 81.31 | 67.24 | 37.34 | 59.03 | 8.78 | 73.56 |  |
| 32 | ZeroQuant-V2 | 51.11 | 81.25 | 66.99 | 38.43 | 58.94 | 8.83 | 73.48 |  |
| LQER | 50.34 | 80.98 | 67.49 | 38.05 | 60.23 | 8.45 | 73.40 |  |  |
| QERA-approx | 50.77 | 81.04 | 66.75 | 37.94 | 60.09 | 8.45 | 73.48 |  |  |
| QERA-exact | 51.28 | 80.18 | 68.83 | 37.48 | 60.60 | 8.33 | 73.95 |  |  |

**TABLE XII: Post-training quantization evaluation of LLaMA-3.1-70B.**
| rank | Method | W-bits | ARC (challenge) | BoolQ | CommonSenseQA | BBH | MMLU | WikiText2 | Winogrande |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Acc_norm | Acc | Acc | Acc_norm | Acc | Word ppl | Acc |  |  |  |
| - | BF16 | 16 | 65.10 | 85.38 | 78.46 | 48.53 | 75.28 | 3.06 | 79.56 |
| - | HQQ | 4.25 | 63.99 | 85.02 | 77.48 | 48.19 | 75.20 | 3.97 | 77.98 |
| - | w-only | 60.58 | 83.82 | 73.63 | 41.28 | 73.06 | 4.55 | 78.37 |  |
| 32 | ZeroQuant-V2 | 59.90 | 83.61 | 73.55 | 42.75 | 73.15 | 4.48 | 77.74 |  |
|  | LQER | 62.97 | 83.88 | 76.25 | 48.67 | 74.26 | 4.10 | 79.64 |  |
|  | QERA-approx | 62.12 | 83.79 | 76.74 | 48.53 | 73.98 | 4.10 | 79.64 |  |

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the Weaknesses section. I would be happy to readjust my score if they are satisfactorily addressed.
soundness: 3
strengths: The paper analytically considers the problem of compensating the quantization error using low-rank high-precision components. The paper is generally well-written, although the work will benefit if it takes into account and compares with more recent works which takes into account the same problem (see weaknesses below).  

The numerical experiments are comprehensive, and the results on a wide variety of models are presented. They are also compared with some other prior works, and show improved benefits.
summary: This work proposes a framework for analyzing the quantization error after it is compensated using low-rank, high-precision terms. In such a decomposition, a weight matrix $\mathbf{W}$ is approximately split as $\mathbf{\tilde{W}} + \mathbf{A}_k\mathbf{B}_k$, where $\mathbf{\tilde{W}}$ is the quantized weight, and $\mathbf{A}_k\mathbf{B}_k$ is the low-rank compensation. This work aims to minimize the Frobenius norm error of the outputs of each layer (in contrast to just the weight quantization error), and propose closed-form solutions for the low-rank terms. The improved benefits are validated with numerical experiments of both encoder-only (RoBERTa) and decoder-only (LLaMa family) models. The experiments involve both post-training quantization (PTQ), as well as parameter-efficient fine-tuning (PEFT).
weaknesses: My major concern with this paper is that it fails to take into account more recent works in this area, and justify how it compares with those works. The contribution of not really clear in light of a more recently proposed algorithm, Caldera (https://arxiv.org/abs/2405.18886) solves the optimization problem (9) optimally, i.e., the output error is minimized and closed form solutions for the low-rank factors are obtained (ref. Lemma 4.2 in the paper). Could the authors highlight the difference in their result of QERA-exact solution (Thm. 1) with Lemma 4.2 (Caldera)? 

Furthermore, the autocorrelation matrix, $\mathbf{R}_{\mathbb{XX}}$ (which is also referred to as Hessians, because it is the Hessian of the quadratic loss in (9)), does need to be computed using a calibration dataset -- but this is a one-time cost. Additionally, the approximation in Assumption 1 for QERA-Approx approximates the autocorrelation matrix as a diagonal matrix -- this is not necessarily true as shown in Figs. 5, 7 and 8. 

Secondly, minimizing error in layer outputs for PTQ is not really a recent idea as mentioned in the paper. It has been around for a few years now. See for example, https://arxiv.org/abs/2004.10568. 

Despite the weaknesses mentioned here, the numerical evaluations on the models and the regimes of 3/4-bit quantization are most likely new.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: N/A
soundness: 2
strengths: 1. The paper is generally well-written and easy to follow.
2. The idea of deriving the analytical solution to the low rank terms by minimizing the layer output error is new.
summary: This paper proposes QERA that analytically finds the proper low rank terms by minimizing the layer output error. They show that minimizing the layer output error is more closely related to minimizing the model output error than minimizing the weight approximation error. They empirically demonstrate their method is better than other previous methods in both QPEFT and PTQ perspectives.
weaknesses: The weaknesses of this paper mostly come from the experiment part.

1. The numbers in Table 1 and Table 2 don't match with the loftq original paper. Is that because you change the experimental setup? Could you please show your method outperforms loftq in their setup?
2. In the original loftq paper, they includes some experimental results about 2bit fine-tuning. Could you also show some results about 2bit fine-tuning?

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1.How to prove that the reason of worse performance of weight approximation methods is rooted in the model's nature instead of your weight approximation methods choice in experiments?

2.In first paragraph, k<<min(m,n). I think it should be k << $\frac{mn}{m+n}$ because the computation cost of W is mn, the computation cost of $A_KB_K$is mk+nk, we want (m+n)k<mn, k should satisfy k <$\frac{mn}{m+n}$
soundness: 3
strengths: This article is excellent in aspect of motivation, problem solving, and paper writing, and is also highly recommended for  its algorithm engineering work. 
1. In terms of motivation, this article chooses to use theoretical methods to solve problems that can only be solved using heuristic algorithms at this stage, and determines the theoretical extreme value of the problem and the method to reach it. 

2. This article provides a very solid analytical method and gives an algorithm for solving the extreme value according to this method, which is solid and reliable. 

3. Paper writing aspect, this article has been detailed and concise, simplifying the repetitive proof of Theorem 2 and placing the proof of Theorem 1 as an important part of the text, allowing readers to fully understand the contribution of this article. In addition, the structure of this article is clear, and the questions and answers are clearly stated. 

4.The biggest advantage of this article is that it reasonably engineers the algorithm and provides a reasonable simplified algorithm that is easy to implement in engineering.
summary: In this paper, authors give the  analytical solution to solve the minimizing the errors in layer outputs via low rank terms. And authors prove that  minimizing the layer output error is better than minimizing the weight approximation error from the aspect of model performance.
weaknesses: The authors insight that minimizing the output error is better than weight approximation error is is consistent with our practical experience in the aspect of model performance. However, this point is hard to prove via experiments, because we cannot enumerate all weight approximation methods on every models. The conclusion is so strong. 
So, two suggestions are that 1. give a mathematical proof of this point. 2. avoiding discuss this conclusion in paper, and only show your work is better than SOTA low rank methods.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Please see the weaknesses.
soundness: 4
strengths: 1. The work formally demonstrates that optimizing to reduce the model layer output error is more effective than minimizing the weight quantization error. Given the fundamental importance of the optimization objective in this field, the work has a significant impact.

2. The assumption used to approximate the exact solution appears reasonable and can be validated through practical testing.

3. The experiments conducted in the work are extensive and comprehensive.
summary: This paper directly gives the analytic solutions of the linear compensation matrix for quantization error correction.  The analytic solutions involve both exact and approximate solutions to tradeoff compensation quality and efficiency. The analytic solutions are solidly validated for two scenarios, post-training and fine-tuning-based quantization.
weaknesses: 1. The efficiency of the exact and approximate solutions, like computational complexity and the concrete execution time compared with related works, should be further clarified.
2.  Eq. (18), $\mathop{\arg\min}_{Q_k} $. 
3. Can the analytic solution be used in LLM pruning? Given that pruning and quantization usually have similar mathematical modeling to some extent.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Please check the Weakness.
soundness: 3
strengths: 1.	It introduces an analytical framework for reconstructing quantization error by considering the layer output, not just the weight values.
2.	It provides detailed mathematical proofs.
summary: This paper proposes Quantization Error Reconstruction Analysis (QERA), an analytical framework for reconstructing quantization error with low-rank terms. QERA offers a closed-form solution to the problem of quantization error reconstruction and demonstrates that LLMs quantized with QERA achieve better linguistic capabilities compared to previous approaches.
weaknesses: 1.	Although the paper claims that demonstrating the relationship between minimizing layer output error and minimizing model output error is a key contribution, this claim seems overstated. Many existing quantization approaches [1, 2] for LLMs already consider layer output error rather than focusing solely on weight approximation error to achieve more accurate quantization.
2.	The overhead of considering layer output is not sufficiently discussed and is only briefly mentioned in Figure 12(b) of the appendix. This paper lacks a thorough analysis of the overhead (e.g., memory requirements and runtime) associated with the error reconstruction procedures. While weight approximation error has limitations in reconstructing errors, it offers more efficient procedures since it does not require a calibration dataset. However, approaches that consider layer input/output for error reconstruction do require a calibration dataset and more computation. Despite this discrepancy in calibration overhead, the paper does not provide a comprehensive analysis of the error reconstruction overhead in the proposed method compared to previous works.
3.	According to Figure 12(b) of the appendix, as QERA-exact appears to be very slow, it seems fair to compare LQER and QERA-approx. However, QERA-approx offers minimal advantage over LQER in terms of quantization.
4.	The paper does not compare the proposed method with LQ-LoRA, a more advanced method that also uses a calibration dataset for error reconstruction. A comparison between the proposed method and LQ-LoRA is important for evaluating QERA.
5.	The paper uses confusing terminology when categorizing experiments. It should use standard terms such as "quantization-aware fine-tuning" and "post-training quantization" to classify experiments, rather than "fine-tuning experiments" and "quantization experiments," as fine-tuning experiments also include quantization.

[1] Frantar, Elias, et al. "Gptq: Accurate post-training quantization for generative pre-trained transformers." ICLR 2023.

[2] Lin, Ji, et al. "AWQ: Activation-aware Weight Quantization for On-Device LLM Compression and Acceleration." MLSys 2024

Meta-review: Dear Authors,

Thank you for your valuable contribution to the ICLR and the ML community. Your submitted paper has undergone a rigorous review process, and I have carefully read and considered the feedback provided by the reviewers.

This paper proposes a framework for analyzing the quantization error after it is compensated using low-rank, high-precision terms. The paper received overall positive scores (6,6,6,8,8). Given this positive assessment, I am willing to recommend the acceptance of your paper for publication. However, I urge you authors to carefully review Reviewer 8w8V's feedback again. On page 4 of the current paper, it is stated that "... to our best knowledge, there is no work providing an analytical solution to this problem.". However, the reference Saha et al. 2024 pointed out by the reviewer (which can be considered concurrent work) needs to be cited here since the theoretical result is near-identical (except a tiny difference in the objectives considered).

I would also like to remind you to carefully review all the other reviewer feedback and the resulting discussion. While most reviews were positive, the reviewers have offered valuable suggestions that can further strengthen the quality of the paper. Please take another careful look a the 'weaknesses' section of each reviewer comment. I encourage you to use this feedback to make any necessary improvements and refinements before submitting the final version of your paper.

Once again, thank you for submitting your work to ICLR.

Best,
Area Chair

Decision: Accept (Poster)