Low-Rank Quantization-Aware Training for LLMs

## Abstract

Abstract Large language models (LLMs) are omnipresent, however their practical deployment is challenging due to their ever increasing computational and memory demands.
Quantization is one of the most effective ways to make them more compute and memory efficient.
Quantization-aware training (QAT) methods, generally produce the best quantized performance, however it comes at the cost of potentially long training time and excessive memory usage, making it impractical when applying for LLMs.
Inspired by parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) literature, we propose LR-QAT – a lightweight and memory-efficient QAT algorithm for LLMs.
LR-QAT employs several components to save memory without sacrificing predictive performance: (a) low-rank auxiliary weights that are aware of the quantization grid; (b) a downcasting operator using fixed-point or double-packed integers and (c) checkpointing.
Unlike most related work, our method (i) is inference-efficient, leading to no additional overhead compared to traditional PTQ; (ii) can be seen as a general extended pretraining framework, meaning that the resulting model can still be utilized for any downstream task afterwards; (iii) can be applied across a wide range of quantization settings, such as different choices quantization granularity, activation quantization, and seamlessly combined with many PTQ techniques.
We apply LR-QAT to LLaMA-1/2/3 and Mistral model families and validate its effectiveness on several downstream tasks.
Our method outperforms common post-training quantization (PTQ) approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage.
Specifically, we can train a 7B LLM on a single consumer grade GPU with 24GB of memory.
Our source code is available at https://github.com/qualcomm-ai-research/LR-QAT .

## 1 Introduction

In recent years, large language models (LLMs) have emerged as a powerful tool for a plethora of natural language processing tasks.
As these models continue to grow in size and capability, addressing their ever increasing computational and memory demands becomes crucial for practical deployment, especially when considering resource-constrained edge devices.

One of the most effective methods to tackle this problem is neural network quantization, which uses low-bit precision for weight and activation tensors.
While recent post-training quantization (PTQ) methods can help with decreasing the model size and improving the computational efficiency of LLMs, they typically lead to subpar performance, especially in the case of low-bit ($\leq 4$) quantization.
Quantization-aware training (QAT), conversely, yields significantly better model performance compared to PTQ.
However, due to extreme model sizes of modern LLMs, using traditional QAT is very computationally expensive and requires a prohibitively high GPU memory usage, making it impractical.

Inspired by parameter-efficient fine-tuning (PEFT) and low-rank adaptation (LoRA) literature, we propose Low-Rank Quantization-Aware Training (LR-QAT) – a lightweight memory-efficient and inference-efficient QAT algorithm for LLMs.
LR-QAT reduces the memory requirements of training a 7B LLM from >98GB of GPU memory to <21GB without degrading the predictive performance compared to traditional full-model QAT, making it possible to train such models on a single consumer grade GPU.
Unlike most related work that combines low-rank adaptation with quantization, our method is also inference-efficient.
After the training is complete, the auxiliary matrices are naturally absorbed into the quantized weight tensor
without loss of accuracy and no extra overhead at inference time. LR-QAT is positioned as a general *extended pretraining* method, as opposed to being strictly a fine-tuning method – the resulting model is a low-bit general pretrained LLM, that can still be utilized for any task afterwards.
If needed, our resulting low-bit pretrained LLM can be fine-tuned on specific downstream tasks or used with multiple LoRA adapters for rapid switching between various tasks.

LR-QAT introduces and combines several innovations designed to reduce memory use without sacrificing model performance:
(1) a form of QAT with low-rank reparameterization, in which we place the low-rank weights in the integer domain to ensure they align with the quantization grid of the pretrained weights. This allows for seamless fusion during inference into a single low-bit integer matrix.
(2) A downcasting operator that represents the frozen pretrained weights as low-bit INT-$b$ ($b\leq 4$) double-packed into INT8 or as fixed-point values stored in INT8.
(3) Finally, we combine the proposed quantization formulation with gradient checkpointing
to avoid aggressive memory spikes from storing some of the intermediate results in memory for the backward pass.

We apply LR-QAT to LLaMA-1/2/3 and Mistral model families and demonstrate its effectiveness on several general language modeling datasets and zero-shot evaluation on some of the common reasoning downstream tasks.
Our method outperforms recent PTQ approaches and reaches the same predictive performance as full-model QAT at the fraction of its memory usage.
Finally, our method can be applied across a wide range of quantization settings, including per-channel or per-block weight quantization, activation quantization, and can be combined with most of other PTQ techniques.

## 2 Background and related work

Figure: Figure 1: *Left:* A schematic illustration of our proposed LR-QAT. $\mathbf{x}$ and $\mathbf{y}$ denote the input and the output of the linear layer. *Right:* Memory requirements for training with various QAT techniques on LLaMA-2 7B, assuming batch size 1, sequence length 1024, $r=32$, and BF16 compute data type. ‘Intermediate results’ refer to the results of some intermediate computations, *e.g*. after rounding/clipping in (3), which are saved in memory for the backward pass.
Refer to caption: extracted/5828679/img/01_lora_qat_yell.png

Neural network quantization is one of the most powerful ways to reduce model footprint, data transfer and compute requirements.
By quantizing a model, high bit-width floating point weights and activations can be represented using low-bit numbers.
On top of that, by using low-bit fixed-point representations, such as INT8, one can further reduce energy consumption since the fixed-point operations are more efficient than their floating-point counterparts.
Quantizing to 8 bits or lower, however, typically introduces quantization noise in the model, resulting in a potential drop in accuracy/perplexity.

In this section, we provide a brief overview of uniform affine quantization and a summary of recent methods for LLM quantization.
We will discuss some of the trade-offs of those techniques.
Finally, we touch upon the challenges of LLM quantization and some of the limitations of current approaches.

#### Uniform affine quantization

We use the following definition of the quantization function:

$$ $\widehat{\mathbf{x}}:=q\left(\mathbf{x};\,s,z,b\right)=s\cdot\vphantom{\Bigg{( }}\Big{(}\,\smash{\underbrace{\operatorname{clip}\!\left(\left\lfloor{\frac{ \mathbf{x}}{s}}\right\rceil+z;-2^{b-1},2^{b-1}-1\right)}_{\text{\normalsize$=: \mathbf{x}_{\mathbb{Z}}$}}}-z\Big{)},$ (1) $$

where $\mathbf{x}$ denotes the quantizer input (i.e., network weights or activations),
$s$ the higher precision quantization scale,
$z$ the integer zero offset, and $b$ the bitwidth.
$\left\lfloor{\cdot}\right\rceil$ denotes the round-to-nearest-integer operator.
Quantization parameters $s$, $z$ can be shared across the components of $\mathbf{x}$.
One can see that such a quantizer approximates an original floating point vector as $\mathbf{x}\approx s\cdot\left(\mathbf{x}_{\mathbb{Z}}-z\right)$, where each element in $\mathbf{x}_{\mathbb{Z}}$ is a $b$-bit integer value.
This quantization scheme is called *uniform affine* or *asymmetric* quantization [, , ] and is one of the most commonly used quantization schemes because it allows for efficient implementation of fixed-point arithmetic.
In the case of *symmetric* quantization, we restrict the quantization grid to be symmetric around $z=0$.

#### Post-training quantization methods

Post-training quantization (PTQ) algorithms take a pretrained high precision (FP32 / FP16 / BF16) network and convert it directly into a fixed-point network without the need for the original training pipeline [, , , , , , , , , ].
These methods are either data-free or only require a small calibration dataset and are generally quite easy to use.
Having almost no hyperparameter tuning makes them usable via a single API call as a black-box method to quantize a pretrained neural network in a computationally efficient manner.

Post-training quantization of LLMs is a challenging task due to presence of numerical outliers in weights and activations [, , , , ].
Existing LLM PTQ methods can be broadly categorized into *weights-only* quantization and *weight-activation* quantization.

Weights-only quantization focuses on converting weights to low-bit values.
For instance, GPTQ [] employs second-order information to iteratively round grouped weights and correct the quantization error in the remaining groups.
SpQR [], AWQ [] and OWQ [] emphasize the importance of so-called “salient” weights that correspond to high-magnitude activations.
Other recent W-only methods include [, , , ].

Weight-activation quantization compresses both weights and activations.
SmoothQuant [], LLM.int8() [] and Outlier Suppression [] achieve W8A8 quantization by managing activation outliers. LLM.int8() uses mixed-precision decomposition, while the other two employ channel-wise scaling.
OmniQuant [] modulates the extreme values of weights by optimizing the clipping threshold and shifts the challenge of quantization from activations to weights by employing the learnable equivalent transformation.
Some of the other recent W&A PTQ methods are [, , , , , , ].

#### Quantization-aware training methods

Quantization-aware training (QAT) methods [, , , , ] simulate quantization during training, allowing the model to find more optimal solutions compared to PTQ approaches.
However, better accuracy/perplexity comes at the cost of neural network training, i.e., longer training times, increased memory usage, need for labeled data and hyperparameter search.

The excessive training cost and memory usage of traditional QAT methods make them unsuitable for quantizing modern LLMs.
A few works that apply QAT to LLMs include LLM-QAT [] that combine QAT with data-free knowledge distillation, and EdgeQAT [] that only considers tiny (sub 100M parameter) language models.

#### Low-rank adapters for fine-tuning

Low-rank adaptation (LoRA) [] is a parameter efficient fine-tuning (PEFT) method that reduces memory requirements compared to standard training.
LoRA freezes the pretrained weights $\bm{W}=\bm{W_{0}}$, and only trains a small set of low-rank trainable parameters, often termed *adapters*.
Given a linear projection $\mathbf{y}=\bm{W_{0}}\mathbf{x}$ with $\bm{W_{0}}\in\mathbb{R}^{m\times k}$, LoRA computes

$$ $\displaystyle\mathbf{y}=\bm{W_{0}}\mathbf{x}+\frac{\alpha}{r}\bm{A}\bm{B} \mathbf{x},$ (2) $$

where $\bm{A}\in\mathbb{R}^{m\times r}$, $\bm{B}\in\mathbb{R}^{r\times k}$, $r<\min\left\{m,k\right\}$ – rank, and $\alpha$ is a scalar that is constant in $r$.
The benefits of LoRA are that it is much cheaper and often performs on par with or better than full fine-tuning. Additionally, the fine-tuned (floating-point) model can be deployed without extra cost, as the low-rank matrices can be fused into the pretrained weights after fine-tuning ($\bm{W}:=\bm{W_{0}}+\frac{\alpha}{r}\bm{A}\bm{B}$).

Naturally, there have been several works that explored the combination of LoRA and quantization.
QLoRA [] quantizes the pretrained weights to 4 bit using (a non-uniform) NF4 format and dequantizes them in the forward pass to further reduce fine-tuning memory footprint.
QA-LoRA [] uses INT4 quantization and introduces
group-wise operators to enable quantization during inference
stage.
LoftQ [] proposed an iterative SVD-based procedure for initializing $\bm{A}$, $\bm{B}$ that yields faster fine-tuning convergence when used together with low-bit quantization.
LQ-LoRA [] further extends initialization technique from LoftQ to mixed precision and data aware cases.
Other recent works include [, ].

Finally, the closest work to ours is PEQA [], that attempts to combine the benefits of inference-efficiency of QAT together with memory-efficiency of PEFT methods.
However, their approach is different since they focus on a task-specific fine-tuning as opposed to being a general extended pretraining method.
In addition to that, PEQA has significantly less degrees of freedom compared to our method, leading to a subpar performance.

#### Motivation

While generally fast and simple, PTQ suffers from limited performance in low-bit scenarios.
Although QAT methods still perform well in low-bit regimes, their high training costs and memory usage make them impractical for LLMs.

LoRA-based methods address memory issues for efficient fine-tuning.
However, in most cases they don’t consider efficient inference.
The adapters $\bm{A}$ and $\bm{B}$ are typically stored in higher precision formats, such as BF16. During inference, the frozen low-bit pretrained weights $\bm{W_{0}}$ are dequantized to match the same data format, resulting in runtime overhead.

Simply quantizing adapters after training will lead to a different quantization grid compared to $\bm{W_{0}}$, and quantizing them using the same quantization grid as $\bm{W_{0}}$ will lead to a high error.
QA-LoRA is the only work we are aware of that attempts to fuse auxiliary LoRA weights back into the frozen $\bm{W}_{\mathbb{Z}}$.
Yet, their method is designed to only work with group-wise quantization with high number of groups (a small group size of $32$).
In addition to that, QA-LoRA and most of LoRA-based methods combine their proposed techniques with the task-specific fine-tuning, whereas we propose LR-QAT as an *extended pretraining* method.

We are inspired by LoRA-based methods to make QAT more memory- and runtime-efficient.
In addition to that, our goal is to design a method that is *inference-efficient*, *i.e*. where the low-rank adapters can be fused back into a low-bit integer matrix $\bm{W}_{\mathbb{Z}}$ without any loss of accuracy/perplexity, yielding PTQ level of inference efficiency.
Contrary to QA-LoRA [], we are not relaxing the quantization constraints – our method is applicable at any weight quantization granularity.
Finally, we see our method as a general extended pretraining framework.
The resulting model can afterwards still be used on any task.
We summarize different trade-offs for the discussed techniques in Table 1.

**Table 1: A comparison between existing approaches and the proposed method.**
| Method | Accuracy | Memory efficiency | Inference efficiency |
| --- | --- | --- | --- |
| PTQ | ✕ | ✓ | ✓ |
| (Full-model) QAT | ✓ | ✕ | ✓ |
| LoRA / PEFT | ✓ | ✓ | ✕ |
| LR-QAT (ours) | \textpdfrender<br>TextRenderingMode=FillStroke,<br>LineWidth=.5pt, ✓ | \textpdfrender<br>TextRenderingMode=FillStroke,<br>LineWidth=.5pt, ✓ | \textpdfrender<br>TextRenderingMode=FillStroke,<br>LineWidth=.5pt, ✓ |

## 3 Method

We now discuss the components of LR-QAT followed by a formal definition of LR-QAT.

#### QAT with low-rank adapters

Let’s recall how traditional QAT [] works.
Given a linear layer with a weight matrix $\bm{W}\in\mathbb{R}^{m\times k}$ and assuming $b$-bit symmetric uniform affine quantization, the quantization is simulated as follows:

$$ $\displaystyle\widehat{\bm{W}}:=\mathbf{s}\cdot\operatorname{clip}\left(\left \lfloor{\frac{\bm{W}}{\mathbf{s}}}\right\rceil,-2^{b-1},2^{b-1}-1\right),$ (3) $$

where weights $\bm{W}$ are trainable parameters and the quantization scale $\mathbf{s}$ can be either fixed or also learned.
To be able to backpropagate through round-to-nearest operation in (3), it is common to use *straight-through estimator* (STE, Bengio et al. ), where it is assumed that $\frac{\partial\!\left\lfloor{t}\right\rceil}{\partial t}=1$.
When applied to LLMs, it is straightforward to see that this procedure is very expensive: we have to learn a comparable number of parameters that was used for pretraining, leading to excessive memory usage.

To make this approach more practical we *freeze* the pretrained weights $\bm{W}$ (denote $\bm{W_{0}}$) and introduce low-rank adapters $\bm{A}\in\mathbb{R}^{m\times r}$, $\bm{B}\in\mathbb{R}^{r\times k}$, $r\ll\min\left\{m,k\right\}$.
We have to be careful where exactly those adapters are placed.
As discussed in Section 2, after the training is complete, we want $\bm{A}$ and $\bm{B}$ to be seamlessly integrated into a single $b$-bit integer matrix $\bm{W}_{\mathbb{Z}}$ without loss of accuracy to facilitate efficient inference.
To accommodate that, we put the auxiliary matrices inside the rounding operator as follows

$$ $\displaystyle\widehat{\bm{W}}:=\mathbf{s}\cdot\operatorname{clip}\left(\left \lfloor{\frac{\bm{W_{0}}}{\mathbf{s}}+\frac{\alpha}{r}\bm{A}\bm{B}}\right \rceil,-2^{b-1},2^{b-1}-1\right),$ (4) $$

where we are using STE assumption for the rounding operation to compute the gradients of the loss w.r.t. $\bm{A}$, $\bm{B}$ and $\mathbf{s}$. We further employ a scaling factor $\alpha/r$ used in LoRA [] to reduce the need to retune hyperparameters as we vary the rank $r$.
After training is complete, (4) can be represented as regular fixed point tensor, $\widehat{\bm{W}}=\mathbf{s}\cdot\bm{W}_{\mathbb{Z}}$, without any extra effort or loss of accuracy and therefore enabling efficient inference without any extra overhead.
Note that this is different to most of the literature, such as QLoRA [], where adapters are placed outside of the quantization function (such as $\mathbf{y}=\widehat{\bm{W}}\mathbf{x}+\bm{A}\bm{B}\mathbf{x}$) and are typically stored in higher precision formats such as BF16.

#### Downcasting operator

The formulation (4) is already significantly more memory efficient compared to standard full-model QAT (3).
We don’t need to compute neither gradients w.r.t. weights $\bm{W}$ nor the respective first or second-order momentum terms for Adam-based optimizers, and only need to do so for the auxiliary matrices $\bm{A}$ and $\bm{B}$, which is noticeably more affordable provided $r\ll\min\!\left\{m,k\right\}$.

Given that the weight matrix $\bm{W_{0}}$ is frozen, the next natural step to further reduce the memory utilization is to store it in a lower-precision format.
One could directly apply downcasting to $\bm{W_{0}}$ in (4).
However, it’s important to note that these weights are divided by the scale $\mathbf{s}$ during every forward pass.
To ensure stable training, the scale generally needs to be stored in a high-precision format.
Therefore, to simplify further, we propose the following variant of low-rank QAT:

$$ $\displaystyle\widehat{\bm{W}}:=\mathbf{s}\cdot\operatorname{clip}\left(\left \lfloor{\frac{\bm{W_{0}}}{\mathbf{s_{0}}}+\frac{\alpha}{r}\bm{A}\bm{B}}\right \rceil,-2^{b-1},2^{b-1}-1\right),$ (5) $$

where we use the initial scale(^1^11A frozen scale obtained after initial range estimation before the training begins.)
$\mathbf{s_{0}}$ instead of learned scale $\mathbf{s}$ inside the rounding operator, and the rest is the same as in (4).
Now the entire fraction $\bm{W_{0}}/\mathbf{s_{0}}$ is fixed and we can store it in a lower-precision format.
Note that the scale $\mathbf{s}$ outside of the clipping operator can still be learned.
Empirically, we found that (5) performs consistently on par with or even slightly better compared to (4).

During training the pretrained weights are represented and stored as follows

$$ $\displaystyle\bm{\Phi_{0}}:=\varphi\!\left(\frac{\bm{W_{0}}}{\mathbf{s_{0}}} \right),$ (6) $$

where $\varphi\!\left(\cdot\right)$ is a *downcasting operator* that encapsulates a choice of different numeric formats or other preprocessing computations.
In the simplest form, $\varphi\!\left(\cdot\right)$ would cast the input to one of pre-existing floating-point formats, such as FP16, BF16, FP8 etc.

Inspired by traditional fixed point quantization, we also explore integer representations for $\varphi\!\left(\cdot\right)$.
Specifically, $\varphi\!\left(x\right)=\operatorname{clip}\left(\left\lfloor{x}\right\rceil,-
2^{b-1},2^{b-1}-1\right)$ corresponds to a standard $b$-bit integer quantization and can be stored as INT-$b$ number.
We denote this approach $\varphi={\text{{\small{INT-}}$b$}}$ for brevity.
In addition to that, in case of low-bit quantization ($b\leq 4$), which is a primary focus of this work,
two INT-$b$ numbers can be *double-packed*
into a single INT8 number, leading to further memory savings.
This is helpful because most of the common deep learning frameworks like PyTorch, at the time of writing this paper, don’t natively support low-bit formats such as INT4 yet.

Using $\varphi={\text{{\small{INT-}}$b$}}$ naturally leads to aggressive memory reduction by only keeping the integer part of (clipped) $\bm{W_{0}}/\mathbf{s_{0}}$.
In our preliminary experiments, we found that this setting, combined with the standard initialization for $\bm{A}$ and $\bm{B}$ used in [], did not work as well compared to $\varphi={\text{{\small{BF16}}}}$.
This indicates the importance of keeping some information of the fractional part of $\bm{W_{0}}/\mathbf{s_{0}}$ and potentially the need for better initialization of auxiliary matrices.

We address this problem in two distinct ways:
We adapt and experiment with a variant of SVD-based initialization for low-rank matrices $\bm{A}$, $\bm{B}$ proposed in [] before we apply a downcasting operator to $\bm{W_{0}}/\mathbf{s_{0}}$, to capture some of the information of it’s fractional part.
With this approach we can still employ a double-packing since we are still using $\varphi={\text{{\small{INT-}}$b$}}$.

Another way is to use INT8 storage type, allocate $b$ bits to represent the integer part as before, but utilize the remaining $8-b$ bits for storing the approximate fractional part ($2\leq b\leq 7$).
In other words, we represent $\bm{\Phi_{0}}$ using fixed-point numbers.
Assuming the rest of the computation is performed in BF16, we define the downcasting and the corresponding upcasting operators as follows:

$$ $\displaystyle\varphi\!\left(x\right)$ $\displaystyle=\text{{\small{INT8}}}\!\left(\left\lfloor{2^{8-b}\cdot \operatorname{clip}\left(x,-2^{b-1},2^{b-1}-1\right)}\right\rceil\right),$ (7) $\displaystyle\varphi^{-1}\!\left(x\right)$ $\displaystyle=\text{{\small{BF16}}}\!\left(x\right)\!/{2^{8-b}}.$ $$

A fixed-point number where $n$ bits are used for the integer part of the value and $m$ bits are used for the fractional part are commonly denoted [] as Qn.m.
For brevity, we will refer to (7) as $\varphi={\text{{\small{Q$b$.$(8-b)$}}}}$.
In this work we will be mainly focusing on $b\in\left\{3,4\right\}$, which corresponds to Q3.5 and Q4.4, respectively.

#### Gradient checkpointing

Note that both in the original LoRA paper [] and in the related work like QLoRA [], there is no need to compute the product $\bm{A}\bm{B}$ explicitly.
Instead, those matrices are multiplied with the activations $\mathbf{x}$ as $\bm{A}\left(\bm{B}\mathbf{x}\right)$.
However, we have to compute a product $\bm{A}\bm{B}$ in (5), and in the naïve implementation of our method, this product together with the results of some intermediate computations (e.g., after rounding and clipping) will be automatically kept in memory for the backward pass, leading to increased memory usage.
To prevent this, we employ gradient checkpointing [] on (5).
In other words, we recompute the quantizer function in the backward pass, leading to a slight runtime overhead but avoiding significantly increased memory usage.

#### LR-QAT

Using the components described above, we define LR-QAT for a single layer with a (pretrained) weight matrix $\bm{W_{0}}$ as follows

$$ $\displaystyle\widehat{\bm{W}}:=\mathbf{s}\cdot\operatorname{clip}\left(\left \lfloor{\bm{\Phi_{0}}+\frac{\alpha}{r}\bm{A}\bm{B}}\right\rceil,-2^{b-1},2^{b- 1}-1\right),$ (8) $$

where $\mathbf{s}$ – trainable or frozen quantization scale with the initial value of $\mathbf{s_{0}}$, $\bm{A}$, $\bm{B}$ – trainable rank $r$ auxiliary matrices, $\bm{\Phi_{0}}:=\varphi\!\left(\bm{W_{0}}/\mathbf{s_{0}}\right)$ – frozen representation of the original pretrained weights, and $\varphi$ is the downcasting operator.
To avoid excessive memory allocation for the results of intermediate computations in (8) involving the product $\bm{A}\bm{B}$, we apply checkpointing on $\widehat{\bm{W}}$.
After the training is complete, low-rank adapters are naturally integrated into a single integer matrix $\bm{W}_{\mathbb{Z}}=\operatorname{clip}\left(\cdots\right)$ without loss of accuracy.
Note, while we presented our method for symmetric quantization which is commonly used for weights [], it can equally be applied for asymmetric quantization by adding a zero offset $z$ outside the rounding operation as shown in (1).

## 4 Experiments

We assess the effectiveness of LR-QAT by conducting experiments on LLaMA 7B [], LLaMA-2 7B/13B [], LLaMA-3 8B [], and Mistral-0.1 7B [].
We first explore the impact of the choice of rank $r$, a downcasting operator $\varphi\!\left(\cdot\right)$, and the initialization of auxiliary matrices $\bm{A}$, $\bm{B}$.
We then compare our method in terms of accuracy to standard full-model QAT, other baselines, and the related work.
All detailed hyperparameters of our experiments are in Appendix B.

#### Quantization

We experiment with both weight-only and weight-activation quantization.
The default settings are INT4 / INT3 / INT2 per-channel (denoted ‘pc’) and group-wise weight quantization with a group size of 128 (denoted ‘g128’).
We use symmetric quantization, except the INT2 case, where we use asymmetric quantization (1), for a fair comparison with related work.
We quantize all linear layers, except the classification head.
In weight-activation quantization, defaults are INT4 per-channel weight and
per-token activation quantization [].
Following OmniQuant [], we quantize all inputs to matmuls with exception of the softmax output and additionally quantize the KV-cache as in LLM-QAT [].

#### Datasets and training

We apply our method to all linear layers in the attention blocks, both in self-attention and in the feed-forward network.
We only train low-rank auxiliary matrices $\bm{A}$, $\bm{B}$ and the quantization parameters $\mathbf{s}$ and keep embedding layers, final classification head and RMSNorm parameters frozen.
In the case of asymmetric weight quantization, a zero offset $\mathbf{z}$ is set during range estimation phase and kept frozen throughout training ($\mathbf{z}=\mathbf{z_{0}}$).

We train on a small subset of SlimPajama [],
which is an open-source dataset similar to the original one used for pretraining LLaMA models.
In all experiments we train using batch size $32$ and a maximum sequence length of $1024$.
For all weight-only and weight-activation quantization results, we train for $10^{4}$ steps.
For ablation studies in Sections 4.1 and 4.2 we use shorter training of $10^{3}$ steps.
We select hyperparameters based on the perplexity of a small subset of Wikipedia validation set (512 sequences).

#### Evaluation

Following the previous work [, , , ], we evaluate quantized models by reporting the perplexity of language generation on WikiText-2 [], using a sequence length of $2048$.
We also report zero-shot accuracy on a set of common sense reasoning tasks including BoolQ [], PIQA [], Winogrande [], ARC [], and HellaSwag [].
For zero-shot evaluation, we use the LM Evaluation Harness framework [].
Specifically, we use lm_eval v0.4.2 and report acc_norm for tasks where it’s available (PIQA, ARC-e, ARC-c, HellaSwag) and otherwise acc (BoolQ and Winogrande).

#### Baselines

We compare with round-to-nearest quantization (RTN), where we set the ranges based on minimizing the $L^{p}$-norms between quantized and unquantized weights and report the best performing configuration.
We also use that as initialization for LR-QAT.
For weight-only quantization, we compare with GPTQ [], AWQ [], and OmniQuant [].
We also compare with our implementation of PEQA [] and full-model QAT (LSQ) [], where we follow the same experimental setup as for our method, together with the best RTN initialization, for a fair comparison.

For weight-activation quantization, we compare our method with RTN, SmoothQuant [], LLM-QAT [], Outlier Suppression$+$ [], OmniQuant [], and our implementation of PEQA [].
Following [], we compare to them in several different settings, where the weights, activations and KV cache values are quantized to different bitwidths (denoted as W-A-KV).

### 4.1 The impact of rank r 𝑟 r italic_r

**Table 2: *Left*: The performance of LR-QAT ($\varphi{}={\text{{\small{Q4.4}}}}$) depending on the rank $r$ of auxiliary matrices $\bm{A}$ and $\bm{B}$ on LLaMA-2 7B with W4 per-channel quantization. We report mean and standard deviation over 5 runs with different random seeds. *Right*: The performance of LR-QAT applied to LLaMA-2 7B depending on the choice of downcasting operator $\varphi\!\left(\cdot\right)$, compute data type, and initialization method for low-rank auxiliary matrices. We report WikiText-2 test set perplexity, lower is better, and average zero-shot accuracy of 6 tasks, higher is better. Numbers marked in bold are the best results.**
| $\varphi\!\left(\cdot\right)$ | dtype | $\bm{A}$, $\bm{B}$ init. | WikiText-2 $\downarrow$ | Zero-shot acc. $\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| W4 pc | W3 pc | W4 pc | W3 pc |  |  |  |
| FP32 | FP32 | LoRA | 5.69 | 6.21 | 69.28 | 66.62 |
| FP16 | FP32 | LoRA | $+$0.00 | $+$0.01 | $-$0.13 | $-$0.01 |
| BF16 | FP32 | LoRA | $-$0.01 | $+$0.01 | $+$0.11 | $+$0.45 |
| Q4.4 / Q3.5 | FP32 | LoRA | $-$0.01 | $+$0.01 | $+$0.16 | $+$0.31 |
| Q4.4 / Q3.5 | BF16 | LoRA | $-$0.01 | $+$0.01 | $+$0.15 | $+$0.31 |
| INT-4 / INT-3 | FP32 | LoRA | $+$0.02 | $+$20.5 | $-$0.04 | $-$22.8 |
| INT-4 / INT-3 | FP32 | LoftQ ($T=1$) | $+$0.28 | $+$0.18 | $-$0.67 | $+$0.26 |
| INT-4 / INT-3 | FP32 | LoftQ ($T=64$) | $+$0.40 | $+$1.37 | $-$1.40 | $-$2.01 |

We investigate the effect of different values of rank $r$ of the auxiliary matrices $\bm{A}$ and $\bm{B}$ and present results in Figure 2.
Increasing the rank from $1$ to $32$ leads to progressively slightly better performance, excluding one outlier.
The fact that using $r>32$ doesn’t lead to further improvement in perplexity is likely because of the limited number of training steps we used for this experiment ($10^{3}$), and that more steps might be needed for the procedure to fully converge.
Interestingly, a rank $r$ as small as 1 already performs really well.
We hypothesize that this is the case because of the following.
Even though $\operatorname{rank}\!\left(\bm{A}\bm{B}\right)=1$, by applying a low-rank approximation inside the rounding and clipping operators in (8), this can overall lead to a high-rank perturbation to the original weights $\bm{\Phi_{0}}$ (in the integer domain).
Finally, for all ranks we observe only a small standard deviation between $0.005$ and $0.008$ ppl., indicating the robustness of LR-QAT to a random initialization of $\bm{B}$.
Going forward, we use $r=32$ in all our experiments(^2^22This amounts to only 1.2% of the total number of parameters for 7B LLaMA model.).

### 4.2 The choice of the downcasting operator φ ⁢ ( ⋅ ) 𝜑 ⋅ \varphi\!\left(\cdot\right) italic_φ ( ⋅ ) and 𝑨 𝑨 \bm{A} bold_italic_A , 𝑩 𝑩 \bm{B} bold_italic_B initialization

We study the effect of several choices of the downcasting operators discussed in Section 3 and summarize results in Table 2.
We can see that by going from FP32 to BF16, and finally to an 8-bit fixed-point representation of $\bm{\Phi_{0}}$, aside from memory savings we also maintain the same WikiText-2 perplexity and even slightly improve zero-shot accuracy.
The latter is likely due to a slight regularization effect caused by the fact that we discard some of the information in the fractional part in $\bm{W_{0}}/\mathbf{s_{0}}$, some of which might be noise.
One step further, however, while $\varphi{}={\text{{\small{INT-$b$}}}}$ still leads to a good model performance in the case of 4-bit weight quantization, it completely breaks for W3.

So far, we initialized matrices $\bm{A}$ and $\bm{B}$ following the procedure proposed in LoRA [] where $\bm{B}$ is initialized to zero, and $\bm{A}$ is initialized randomly as in [].
We refer to this initialization scheme as ‘LoRA’.
We hypothesize that a poor performance of $\varphi{}={\text{{\small{INT3}}}}$ can be explained by the fact that we lose all the information in the fractional part of $\bm{W_{0}}/\mathbf{s_{0}}$ and that without that information it is difficult for low-rank approximation to learn.
To address this, we adapt and experiment with a variant of SVD-based initialization proposed in LoftQ [].
We see that using LoftQ initialization with $T=1$ step recovers almost all the predictive performance compared to a fixed-point representation.
Increasing number of LoftQ steps, or applying it to a 4-bit case, however, did not help.

Finally, when using the fixed point representation for $\bm{\Phi_{0}}$, we still maintain the same model performance by switching the compute data type(^3^33A data type used for activations, gradients, and frozen parameters.) from FP32 to BF16, where the latter is what is commonly used for LLMs.
Going forward, we use $\varphi={\text{{\small{Q$b$.$(8-b)$}}}}$ with ‘LoRA’ initialization and BF16 compute data type.

### 4.3 Comparison with full-model QAT

**Table 3: A comparison of the proposed method ($\varphi{}={\text{{\small{Q4.4}}}}$) with the full-model QAT on LLaMA-2 7B with W4 and W3 per-channel quantization. We report mean and standard deviation over 5 runs with different random seeds. We also report the maximum GPU memory with (without) gradient checkpointing and the runtime on a Nvidia A100 80GB GPU.**
| Method | GPU mem., | Time/100 steps, | WikiText-2 ppl. $\downarrow$ | Zero-shot acc. $\uparrow$ |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| GB | sec | W4 pc | W3 pc | W4 pc | W3 pc |  |
| Full-model QAT (LSQ) | 62.2 (98.5) | 3248${}^{\scriptsize\pm{}\text{7}}$ | 5.77${}^{\scriptsize\pm{}\text{0.02}}$ | 6.14${}^{\scriptsize\pm{}\text{0.01}}$ | 68.96${}^{\scriptsize\pm{}\text{0.29}}$ | 67.14${}^{\scriptsize\pm{}\text{0.13}}$ |
| LR-QAT (ours) | 20.5 | 1522${}^{\scriptsize\pm{}\text{5}}$ | 5.66${}^{\scriptsize\pm{}\text{0.00}}$ | 6.13${}^{\scriptsize\pm{}\text{0.02}}$ | 69.72${}^{\scriptsize\pm{}\text{0.32}}$ | 67.70${}^{\scriptsize\pm{}\text{0.25}}$ |

Finally, before presenting our main set of results, we compare our method with a standard full-model QAT (LSQ) [].
For full-model QAT, we follow the same training setup as for our method.
We also tune the maximum value of the learning rate for $\bm{W}$ using the following search space $\left\{\text{1e-5, {5e-5}, 1e-4, 5e-4, 1e-3}\right\}$ and select the best configuration based on Wikipedia validation perplexity.
Note that we use the same learning rate for $\mathbf{s}$ for both full-model QAT and our method.

As we can see in Table 3, training with our method leads to on par or better predictive performance at a significantly lower memory usage and runtime compared to full-model QAT.
We include results for other models and bitwidths in Table 4.
A more detailed runtime comparison can also be found in Table A8.

### 4.4 Main results

**Table 4: Weight-only quantization results for LLaMA and Mistral models. We report WikiText-2 test set perplexity (lower is better) and average zero-shot accuracy (higher is better). Models marked ‘L1’/‘L2’/‘L3’, and ‘M’ denote LLaMA-1/2/3 and Mistral, respectively. Numbers marked in bold are SOTA or on par (within $0.05$). ^§Uses asymmetric weight quantization. ✕ denotes out of memory.**
| # Bits | Method | WikiText-2 perplexity $\downarrow$ | Avg. zero-shot accuracy $\uparrow$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| L1-7B | L2-7B | L2-13B | L3-8B | M-7B | L1-7B | L2-7B | L2-13B | L3-8B | M-7B |  |  |
| FP16 |  | 5.68 | 5.47 | 4.88 | 6.14 | 5.25 | 69.68 | 70.47 | 73.18 | 74.22 | 75.69 |
| W4 pc | RTN | 6.33 | 6.14 | 5.21 | 7.53 | 5.91 | 68.51 | 68.88 | 71.73 | 72.19 | 73.44 |
| GPTQ^§ | 6.13 | 5.83 | 5.13 | - | - | 64.95 | - | - | - | - |  |
| AWQ | 6.08 | 6.15 | 5.12 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 5.86 | 5.74 | 5.02 | - | - | - | - | - | - | - |  |
| LSQ (our impl.) | 5.94 | 5.77 | ✕ | 6.87 | 5.73 | 68.37 | 68.96 | ✕ | 73.28 | 72.88 |  |
| PEQA (our impl.) | 5.86 | 5.71 | 5.03 | 7.51 | 5.56 | 68.49 | 69.23 | 72.51 | 72.79 | 73.73 |  |
|  | LR-QAT (ours) | 5.84 | 5.66 | 5.03 | 6.78 | 5.46 | 68.54 | 69.72 | 73.19 | 73.84 | 74.44 |
| W4 g128 | RTN | 6.05 | 5.78 | 5.04 | 6.96 | 5.49 | 68.93 | 69.75 | 72.94 | 72.30 | 75.07 |
| GPTQ^§ | 5.85 | 5.61 | 4.98 | - | - | - | - | - | - | - |  |
| AWQ | 5.81 | 5.62 | 4.97 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 5.77 | 5.58 | 4.95 | - | - | - | - | - | - | - |  |
| LSQ (our impl.) | 5.76 | 5.61 | ✕ | 6.58 | 5.67 | 69.17 | 69.68 | ✕ | 73.31 | 72.90 |  |
| PEQA (our impl.) | 5.75 | 5.67 | 5.02 | 6.89 | 5.48 | 69.19 | 69.64 | 72.80 | 72.99 | 73.34 |  |
|  | LR-QAT (ours) | 5.75 | 5.59 | 4.97 | 6.57 | 5.37 | 69.15 | 69.88 | 72.91 | 73.66 | 75.28 |
| W3 pc | RTN | 12.88 | 26.73 | 8.71 | 34.10 | 9.49 | 54.66 | 43.87 | 55.01 | 47.46 | 64.58 |
| GPTQ^§ | 8.06 | 8.37 | 6.44 | - | - | - | - | - | - | - |  |
| AWQ | 11.88 | 24.00 | 10.45 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 6.49 | 6.58 | 5.58 | - | - | - | - | - | - | - |  |
| LSQ (our impl.) | 6.29 | 6.14 | ✕ | 8.14 | 6.06 | 66.29 | 67.14 | ✕ | 69.58 | 71.61 |  |
| PEQA (our impl.) | 6.56 | 6.45 | 5.73 | 26.20 | 6.51 | 65.75 | 65.44 | 69.81 | 51.05 | 71.02 |  |
|  | LR-QAT (ours) | 6.27 | 6.13 | 5.54 | 8.12 | 6.03 | 66.60 | 67.70 | 71.22 | 70.46 | 71.87 |
| W3 g128 | RTN | 7.96 | 7.61 | 6.20 | 15.11 | 6.77 | 63.50 | 63.20 | 67.60 | 57.74 | 69.35 |
| GPTQ^§ | 6.55 | 6.29 | 5.42 | - | - | - | - | - | - | - |  |
| AWQ | 6.46 | 6.24 | 5.32 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 6.15 | 6.03 | 5.28 | - | - | - | - | - | - | - |  |
| LSQ (our impl.) | 6.20 | 6.02 | ✕ | 8.08 | 5.90 | 66.53 | 68.36 | ✕ | 70.11 | 71.96 |  |
| PEQA (our impl.) | 6.22 | 6.05 | 5.58 | 9.64 | 5.85 | 66.66 | 68.10 | 70.29 | 67.19 | 72.21 |  |
|  | LR-QAT (ours) | 6.17 | 5.98 | 5.32 | 7.74 | 5.80 | 66.81 | 68.62 | 71.51 | 70.48 | 72.41 |
| W2 pc | RTN^§ | 4.9e3 | 5.2e3 | 5.2e3 | 6.4e4 | 6.8e3 | 37.92 | 36.52 | 36.27 | 36.80 | 36.59 |
| GPTQ^§ | 2.1e3 | 7.7e3 | 2.1e3 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 15.47 | 37.37 | 17.21 | - | - | - | - | - | - | - |  |
| PEQA (our impl.)^§ | 8.24 | 9.34 | 7.51 | 14.47 | 8.08 | 59.83 | 57.40 | 62.29 | 57.72 | 64.48 |  |
| LR-QAT (ours)^§ | 7.99 | 8.51 | 7.16 | 12.52 | 7.98 | 61.77 | 60.03 | 65.28 | 58.49 | 65.11 |  |
| W2 g128 | RTN^§ | 708 | 2.5e3 | 115.6 | 1.4e4 | 369 | 39.74 | 37.94 | 41.12 | 36.97 | 41.30 |
| GPTQ^§ | 44.01 | 36.77 | 28.14 | - | - | - | - | - | - | - |  |
| AWQ | 2.6e5 | 2.2e5 | 1.2e5 | - | - | - | - | - | - | - |  |
| OmniQuant^§ | 9.72 | 11.06 | 8.26 | - | - | - | - | - | - | - |  |
| PEQA (our impl.)^§ | 8.12 | 7.87 | 6.83 | 12.39 | 8.05 | 60.69 | 60.74 | 64.74 | 57.81 | 64.65 |  |
| LR-QAT (ours)^§ | 7.86 | 7.62 | 6.57 | 11.09 | 7.92 | 61.60 | 61.70 | 66.75 | 60.46 | 65.27 |  |

#### Weight-only quantization

We summarize our results in Table 4.
As we can see, in almost most cases LR-QAT outperforms or is on par with prior weight-only quantization methods across various LLM families and quantization settings, including both per-channel and group-wise quantization.
In the case of extremely low-bitwidth regime, our method consistently and significantly outperforms related work, across all settings.

In a few cases, especially in case of group-wise quantization our method did not outpeform OmniQuant.
However, OmniQuant uses asymmetric quantization which provides extra degrees of freedom compared to symmetric quantization, which are very helpful in the case of low-bit quantization.
In practice, however, symmetric weight quantization yields more efficient inference [].
Additionally, techniques like OmniQuant and related techniques are orthogonal to our method and can be used as as initialization of LR-QAT.

#### Weight-activation quantization

**Table 5: Weight and activation quantization results for LLaMA-1/2 (denoted ‘L1’/‘L2’, respectively). We report WikiText-2 test set perplexity and zero-shot accuracy of 6 tasks. Numbers marked in bold are SOTA. ^§Uses asymmetric weight quantization. ^*Uses a maximum sequence length of 1024 for evaluation.**
| # Bits<br>(W-A-KV) | Method | WikiText-2 perplexity $\downarrow$ | Avg. zero-shot accuracy $\uparrow$ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| L1-7B | L2-7B | L2-13B | L1-7B | L2-7B | L2-13B |  |  |
| FP16 |  | 5.68 | 5.47 | 4.88 | 69.68 | 70.47 | 73.18 |
| 4-8-8 | RTN | 6.88 | 6.17 | 5.23 | 65.83 | 68.55 | 71.54 |
| SmoothQuant | 13.7^* | - | - | 65.17 | - | - |  |
| LLM-QAT | 11.2^* | - | - | 68.18 | - | - |  |
| PEQA (our impl.) | 5.89 | 5.72 | 5.08 | 68.53 | 69.11 | 72.49 |  |
| LR-QAT (ours) | 5.85 | 5.67 | 5.04 | 68.58 | 69.32 | 73.18 |  |
| 4-8-4 | RTN | 7.66 | 6.85 | 5.78 | 62.78 | 65.16 | 67.06 |
| SmoothQuant | 163.6^* | - | - | 45.35 | - | - |  |
| LLM-QAT | 11.6^* | - | - | 64.75 | - | - |  |
| PEQA (our impl.) | 6.15 | 6.03 | 5.27 | 66.54 | 67.56 | 71.35 |  |
| LR-QAT (ours) | 6.07 | 5.90 | 5.24 | 67.16 | 69.84 | 71.65 |  |
| 4-4-4 | RTN | 17.75 | 18.98 | 11.37 | 49.60 | 51.75 | 55.07 |
| SmoothQuant | 25.25 | 83.12 | 35.88 | 38.42 | - | - |  |
| LLM-QAT | - | - | - | 41.27 | - | - |  |
| LLM-QAT + SQ | - | - | - | 46.43 | - | - |  |
| Outlier Suppression+ | - | - | - | 48.43 | - | - |  |
| OmniQuant^§ | 11.26 | 14.26 | 12.30 | 52.65 | - | - |  |
|  | PEQA (our impl.) | 8.60 | 8.72 | 7.23 | 58.39 | 57.93 | 62.26 |
|  | LR-QAT (ours) | 8.47 | 8.46 | 7.15 | 59.00 | 58.98 | 62.65 |

We present our results for weight-activation quantization applied to LLaMA-1/2 models in Table 5.
LR-QAT demonstrates superior performance compared to all the PTQ and QAT baselines, consistently outperforming them across all model families and the bitwidth settings.
In addition to that, as we decrease the activation bitwidths, the improvement in model performance compared to prior work becomes more pronounced.

This indicates LR-QAT’s versatility, being readily applicable not only to weight-only quantization but also weight-activation quantization, a setting that allows for a very efficient inference using fixed-point arithmetic.
Further, our method can still be combined with most of the related PTQ methods including OmniQuant that shift the difficulty of activation quantization to weight quantization, and will likely lead to even better results.

## 5 Discussion

#### Limitations

A core assumption of LR-QAT is that a low-rank approximation can compensate the introduced quantization noise. While quantization noise follows a random uniform distribution and is theoretically not low-rank, our results and several prior works [, , ] suggest that in an end-to-end training setup, low-rank approaches can effectively compensate quantization noise.
In our work, we demonstrated the effectiveness of LR-QAT for LLMs up to 13B parameters. It is unclear how it scales to significantly larger LLMs, though we do not see any reason why our findings should not hold beyond 13B models.
We evaluate LR-QAT as extended pretraining technique with several thousands of iterations. It is unclear how it would perform in case it is used during pretraining for millions of iterations.

#### Impact

As LR-QAT improves memory and inference efficiency of LLMs, we expect mainly positive outcomes from our work.
Efficiently deploying LLMs will help with reducing their high power consumption at inference time. It further helps to move inference from the cloud to edge devices which can overcome potential privacy concerns.
In some cases, quantization might lead to biased predictions, see [] for further discussion.

## 6 Conclusions

In this paper we propose LR-QAT, a lightweight and memory-efficient QAT algorithm for LLMs which enables training a 7B LLM on a single consumer grade GPU with 24GB of memory.
Inspired by PEFT methods, we introduce a low-rank reparameterization that is aware of the quantization grid. We further reduce the memory requirements by introducing a downcasting operator involving fixed-point or double-packed integers, and applying checkpointing.
In almost all cases, our method outperforms common PTQ approaches and reaches the same model performance as full-model QAT at the fraction of its memory usage.

## Acknowledgements

We would like to thank Tijmen Blankevoort, Paul Whatmough, Jorn Peters, and Ties van Rozendaal for valuable discussions and support.

## Appendix A Extended results

In this section, we provide additional detailed results.

**Table A1: LM-eval weight-only quantization results for LLaMA-1 7B. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization.**
| # Bits | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 |  | 75.05 | 79.16 | 70.01 | 72.85 | 44.80 | 76.21 | 69.68 |
| W4 pc | RTN | 73.18 | 78.78 | 69.14 | 71.38 | 44.37 | 74.22 | 68.51 |
| GPTQ^§ | 67.70 | 76.00 | 66.70 | 66.90 | 43.00 | 69.40 | 64.95 |  |
| LSQ (our impl.) | 73.91 | 78.24 | 69.22 | 70.84 | 43.26 | 74.74 | 68.37 |  |
| PEQA (our impl.) | 74.71 | 78.29 | 70.09 | 70.33 | 42.24 | 75.27 | 68.49 |  |
| LR-QAT (ours) | 74.13 | 78.29 | 70.01 | 71.21 | 42.41 | 75.16 | 68.54 |  |
| W4 g128 | RTN | 74.77 | 78.51 | 70.64 | 71.30 | 43.60 | 74.74 | 68.93 |
| LSQ (our impl.) | 75.90 | 79.22 | 70.01 | 71.42 | 43.34 | 75.15 | 69.17 |  |
| PEQA (our impl.) | 75.75 | 79.17 | 70.17 | 70.75 | 43.60 | 75.71 | 69.19 |  |
| LR-QAT (ours) | 75.29 | 78.62 | 69.61 | 71.59 | 44.11 | 75.67 | 69.15 |  |
| W3 pc | RTN | 58.93 | 70.40 | 55.72 | 55.01 | 32.17 | 55.75 | 54.66 |
| LSQ (our impl.) | 71.35 | 77.97 | 68.82 | 66.33 | 40.10 | 73.14 | 66.29 |  |
| PEQA (our impl.) | 72.69 | 77.15 | 65.90 | 68.27 | 38.91 | 71.60 | 65.75 |  |
| LR-QAT (ours) | 73.24 | 78.18 | 67.40 | 67.47 | 40.53 | 72.77 | 66.60 |  |
| W3 g128 | RTN | 69.48 | 76.33 | 64.40 | 64.44 | 38.65 | 67.67 | 63.50 |
| LSQ (our impl.) | 71.04 | 77.97 | 68.11 | 68.27 | 40.44 | 73.37 | 66.53 |  |
| PEQA (our impl.) | 71.65 | 78.24 | 68.51 | 68.18 | 40.10 | 73.30 | 66.66 |  |
| LR-QAT (ours) | 72.84 | 78.02 | 67.40 | 68.52 | 41.04 | 73.04 | 66.81 |  |
| W2 pc | RTN^§ | 43.24 | 52.61 | 51.62 | 27.48 | 26.28 | 26.26 | 37.92 |
| PEQA (our impl.)^§ | 67.19 | 74.32 | 61.80 | 59.97 | 32.42 | 63.28 | 59.83 |  |
| LR-QAT (ours)^§ | 68.07 | 74.27 | 64.80 | 61.03 | 36.95 | 65.48 | 61.77 |  |
| W2 g128 | RTN^§ | 41.16 | 58.05 | 50.20 | 34.97 | 23.12 | 30.93 | 39.74 |
| PEQA (our impl.)^§ | 67.40 | 74.21 | 62.04 | 60.98 | 35.32 | 64.16 | 60.69 |  |
| LR-QAT (ours)^§ | 67.65 | 74.86 | 61.80 | 62.37 | 37.03 | 65.90 | 61.60 |  |

**Table A2: LM-eval weight-only quantization results for LLaMA-2 7B. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization.**
| # Bits | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 |  | 77.74 | 79.11 | 69.14 | 74.58 | 46.25 | 75.98 | 70.47 |
| W4 pc | RTN | 76.36 | 78.07 | 68.19 | 71.21 | 44.80 | 74.65 | 68.88 |
| LSQ (our impl.) | 76.69 | 78.15 | 68.12 | 71.84 | 44.18 | 74.76 | 68.96 |  |
| PEQA (our impl.) | 77.49 | 78.24 | 69.61 | 70.96 | 43.52 | 75.54 | 69.23 |  |
| LR-QAT (ours) | 77.41 | 78.56 | 69.42 | 72.80 | 44.66 | 75.45 | 69.72 |  |
| W4 g128 | RTN | 76.76 | 78.18 | 69.77 | 72.60 | 45.73 | 75.43 | 69.75 |
| LSQ (our impl.) | 77.28 | 78.45 | 69.61 | 72.18 | 44.97 | 75.61 | 69.68 |  |
| PEQA (our impl.) | 76.88 | 78.89 | 69.85 | 72.18 | 44.11 | 75.95 | 69.64 |  |
| LR-QAT (ours) | 76.73 | 78.62 | 70.48 | 72.85 | 44.97 | 75.62 | 69.88 |  |
| W3 pc | RTN | 46.27 | 60.28 | 54.85 | 38.05 | 23.29 | 40.47 | 43.87 |
| LSQ (our impl.) | 74.39 | 77.91 | 66.85 | 69.15 | 41.53 | 73.04 | 67.14 |  |
| PEQA (our impl.) | 71.62 | 76.82 | 66.14 | 65.66 | 39.76 | 72.63 | 65.44 |  |
| LR-QAT (ours) | 75.08 | 77.73 | 67.50 | 69.97 | 42.70 | 73.24 | 67.70 |  |
| W3 g128 | RTN | 66.42 | 75.57 | 65.19 | 64.90 | 38.14 | 68.96 | 63.20 |
| LSQ (our impl.) | 74.68 | 77.80 | 68.35 | 71.76 | 44.11 | 73.46 | 68.36 |  |
| PEQA (our impl.) | 75.38 | 77.97 | 68.59 | 70.62 | 42.32 | 73.74 | 68.10 |  |
| LR-QAT (ours) | 76.61 | 77.31 | 68.98 | 72.05 | 42.58 | 74.20 | 68.62 |  |
| W2 pc | RTN^§ | 39.57 | 50.71 | 50.20 | 26.73 | 26.28 | 25.63 | 36.52 |
| PEQA (our impl.)^§ | 64.98 | 71.33 | 58.09 | 55.22 | 33.11 | 61.66 | 57.40 |  |
| LR-QAT (ours)^§ | 69.88 | 73.07 | 62.90 | 56.57 | 34.04 | 63.71 | 60.03 |  |
| W2 g128 | RTN^§ | 39.08 | 55.93 | 50.99 | 29.84 | 24.06 | 27.71 | 37.94 |
| PEQA (our impl.)^§ | 69.51 | 73.83 | 63.46 | 58.21 | 34.90 | 64.53 | 60.74 |  |
| LR-QAT (ours)^§ | 70.18 | 73.83 | 63.69 | 60.27 | 34.98 | 67.24 | 61.70 |  |

**Table A3: LM-eval weight-only quantization results for LLaMA-2 13B. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization. ✕ denotes out of memory.**
| # Bits | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 |  | 80.55 | 80.52 | 72.22 | 77.44 | 48.98 | 79.38 | 73.18 |
| W4 pc | RTN | 79.30 | 79.71 | 70.01 | 75.51 | 48.89 | 76.96 | 71.73 |
| LSQ (our impl.) | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ |  |
| PEQA (our impl.) | 78.99 | 80.14 | 71.27 | 76.43 | 48.98 | 79.24 | 72.51 |  |
| LR-QAT (ours) | 80.15 | 80.09 | 72.06 | 77.65 | 49.91 | 79.28 | 73.19 |  |
| W4 g128 | RTN | 81.10 | 79.82 | 72.38 | 76.73 | 49.06 | 78.52 | 72.94 |
| LSQ (our impl.) | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ |  |
| PEQA (our impl.) | 80.28 | 80.63 | 71.74 | 76.14 | 48.38 | 79.62 | 72.80 |  |
| LR-QAT (ours) | 80.73 | 80.30 | 71.74 | 76.14 | 49.06 | 79.51 | 72.91 |  |
| W3 pc | RTN | 55.05 | 71.06 | 54.22 | 56.19 | 32.25 | 61.27 | 55.01 |
| LSQ (our impl.) | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ |  |
| PEQA (our impl.) | 74.28 | 78.67 | 69.06 | 74.87 | 45.99 | 76.00 | 69.81 |  |
| LR-QAT (ours) | 78.62 | 79.49 | 72.61 | 73.99 | 45.56 | 77.05 | 71.22 |  |
| W3 g128 | RTN | 74.65 | 76.93 | 69.14 | 70.16 | 42.66 | 72.06 | 67.60 |
| LSQ (our impl.) | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ | ✕ |  |
| PEQA (our impl.) | 78.56 | 78.73 | 69.85 | 73.61 | 44.28 | 76.69 | 70.29 |  |
| LR-QAT (ours) | 79.79 | 79.60 | 70.64 | 74.24 | 46.76 | 78.00 | 71.51 |  |
| W2 pc | RTN^§ | 38.35 | 48.97 | 48.54 | 27.78 | 27.99 | 25.97 | 36.27 |
| PEQA (our impl.)^§ | 71.19 | 74.48 | 59.67 | 62.96 | 37.29 | 68.14 | 62.29 |  |
| LR-QAT (ours)^§ | 72.60 | 76.61 | 66.22 | 66.67 | 39.76 | 69.79 | 65.28 |  |
| W2 g128 | RTN^§ | 50.12 | 57.29 | 50.36 | 34.43 | 22.18 | 32.35 | 41.12 |
| PEQA (our impl.)^§ | 70.34 | 76.88 | 66.85 | 64.94 | 38.74 | 70.66 | 64.74 |  |
| LR-QAT (ours)^§ | 75.47 | 77.86 | 65.98 | 67.63 | 41.04 | 72.50 | 66.75 |  |

**Table A4: LM-eval weight-only quantization results for LLaMA-3 8B. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization.**
| # Bits | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 |  | 81.44 | 80.79 | 72.85 | 77.74 | 53.33 | 79.16 | 74.22 |
| W4 pc | RTN | 79.02 | 78.56 | 72.85 | 75.97 | 49.32 | 77.44 | 72.19 |
| LSQ (our impl.) | 80.58 | 80.03 | 72.69 | 78.24 | 50.17 | 77.94 | 73.28 |  |
| PEQA (our impl.) | 79.57 | 78.67 | 72.93 | 77.19 | 51.11 | 77.25 | 72.79 |  |
| LR-QAT (ours) | 81.62 | 79.98 | 72.85 | 78.32 | 52.05 | 78.19 | 73.84 |  |
| W4 g128 | RTN | 79.48 | 79.27 | 73.56 | 75.08 | 48.81 | 77.61 | 72.30 |
| LSQ (our impl.) | 80.24 | 80.36 | 73.01 | 77.23 | 50.43 | 78.60 | 73.31 |  |
| PEQA (our impl.) | 80.98 | 80.14 | 72.61 | 76.18 | 49.57 | 78.45 | 72.99 |  |
| LR-QAT (ours) | 80.40 | 80.90 | 73.48 | 77.44 | 51.11 | 78.60 | 73.66 |  |
| W3 pc | RTN | 58.65 | 61.75 | 56.04 | 39.60 | 23.81 | 44.91 | 47.46 |
| LSQ (our impl.) | 76.42 | 78.24 | 70.01 | 72.31 | 45.22 | 75.26 | 69.58 |  |
| PEQA (our impl.) | 63.18 | 64.74 | 57.62 | 43.39 | 26.88 | 50.48 | 51.05 |  |
| LR-QAT (ours) | 77.46 | 78.51 | 69.85 | 74.83 | 47.35 | 74.73 | 70.46 |  |
| W3 g128 | RTN | 65.47 | 68.39 | 65.19 | 54.00 | 33.45 | 59.96 | 57.74 |
| LSQ (our impl.) | 75.50 | 78.78 | 69.69 | 73.57 | 48.55 | 74.55 | 70.11 |  |
| PEQA (our impl.) | 72.26 | 76.06 | 67.80 | 69.02 | 46.08 | 71.89 | 67.19 |  |
| LR-QAT (ours) | 72.97 | 79.38 | 71.67 | 74.37 | 49.06 | 75.44 | 70.48 |  |
| W2 pc | RTN^§ | 44.89 | 48.97 | 47.51 | 25.29 | 27.73 | 26.41 | 36.80 |
| PEQA (our impl.)^§ | 62.54 | 72.96 | 59.51 | 56.82 | 33.70 | 60.77 | 57.72 |  |
| LR-QAT (ours)^§ | 65.81 | 72.31 | 64.72 | 53.54 | 33.45 | 61.11 | 58.49 |  |
| W2 g128 | RTN^§ | 38.47 | 53.32 | 51.78 | 28.75 | 22.70 | 26.81 | 36.97 |
| PEQA (our impl.)^§ | 58.53 | 72.52 | 61.96 | 57.41 | 35.32 | 61.14 | 57.81 |  |
| LR-QAT (ours)^§ | 67.98 | 73.83 | 62.83 | 56.94 | 36.43 | 64.77 | 60.46 |  |

**Table A5: LM-eval weight-only quantization results for Mistral 7B. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization.**
| # Bits | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| FP16 |  | 83.58 | 82.10 | 73.88 | 79.59 | 53.92 | 81.07 | 75.69 |
| W4 pc | RTN | 81.22 | 80.63 | 72.53 | 76.77 | 50.09 | 79.41 | 73.44 |
| LSQ (our impl.) | 81.96 | 80.41 | 73.09 | 75.08 | 48.55 | 78.19 | 72.88 |  |
| PEQA (our impl.) | 81.80 | 81.12 | 72.61 | 77.23 | 50.17 | 79.43 | 73.73 |  |
| LR-QAT (ours) | 81.99 | 81.28 | 73.56 | 78.20 | 51.02 | 80.57 | 74.44 |  |
| W4 g128 | RTN | 84.16 | 81.77 | 74.43 | 77.95 | 51.71 | 80.42 | 75.07 |
| LSQ (our impl.) | 80.40 | 80.63 | 73.16 | 76.26 | 48.72 | 78.24 | 72.90 |  |
| PEQA (our impl.) | 80.89 | 81.72 | 73.80 | 75.42 | 48.46 | 79.76 | 73.34 |  |
| LR-QAT (ours) | 83.55 | 81.61 | 74.51 | 78.28 | 52.90 | 80.84 | 75.28 |  |
| W3 pc | RTN | 68.13 | 77.64 | 63.93 | 63.93 | 41.13 | 72.73 | 64.58 |
| LSQ (our impl.) | 79.97 | 80.74 | 70.24 | 74.37 | 47.18 | 77.14 | 71.61 |  |
| PEQA (our impl.) | 80.03 | 80.09 | 69.93 | 72.90 | 45.82 | 77.32 | 71.02 |  |
| LR-QAT (ours) | 81.62 | 80.09 | 70.96 | 74.75 | 46.08 | 77.71 | 71.87 |  |
| W3 g128 | RTN | 78.44 | 79.60 | 69.14 | 71.17 | 43.00 | 74.75 | 69.35 |
| LSQ (our impl.) | 81.01 | 80.36 | 71.03 | 74.41 | 47.27 | 77.68 | 71.96 |  |
| PEQA (our impl.) | 81.99 | 81.18 | 69.61 | 74.92 | 47.18 | 78.37 | 72.21 |  |
| LR-QAT (ours) | 81.71 | 80.90 | 70.48 | 75.08 | 47.78 | 78.50 | 72.41 |  |
| W2 pc | RTN^§ | 39.42 | 51.03 | 49.49 | 26.22 | 26.71 | 26.67 | 36.59 |
| PEQA (our impl.)^§ | 74.34 | 75.95 | 66.22 | 62.21 | 38.48 | 69.68 | 64.48 |  |
| LR-QAT (ours)^§ | 75.08 | 76.39 | 65.98 | 64.48 | 39.59 | 69.14 | 65.11 |  |
| W2 g128 | RTN^§ | 54.98 | 56.91 | 51.70 | 31.27 | 23.21 | 29.70 | 41.30 |
| PEQA (our impl.)^§ | 75.41 | 75.84 | 64.96 | 65.45 | 38.23 | 68.00 | 64.65 |  |
| LR-QAT (ours)^§ | 75.81 | 77.15 | 64.64 | 63.59 | 39.51 | 70.89 | 65.27 |  |

**Table A6: LM-eval weight and activation quantization results for LLaMA models. We report zero-shot accuracy of 6 tasks (higher is better). ^§Uses asymmetric weight quantization.**
| Model | # Bits<br>(W-A-KV) | Method | BoolQ | PIQA | Winogrande | ARC-e | ARC-c | HellaSwag | Avg. |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaMA-1 7B | FP16 |  | 75.05 | 79.16 | 70.01 | 72.85 | 44.80 | 76.21 | 69.68 |
| 4-8-8 | RTN | 71.35 | 76.66 | 66.46 | 66.84 | 41.55 | 72.10 | 65.83 |  |
| SmoothQuant | 71.00 | 76.00 | 66.00 | 67.40 | 42.80 | 67.80 | 65.17 |  |  |
| LLM-QAT | 74.60 | 77.50 | 67.70 | 70.20 | 45.60 | 73.50 | 68.18 |  |  |
| PEQA (our impl.) | 74.86 | 78.24 | 70.01 | 70.12 | 42.83 | 75.14 | 68.53 |  |  |
| LR-QAT (ours) | 73.76 | 78.51 | 71.19 | 71.09 | 41.81 | 75.10 | 68.58 |  |  |
| 4-8-4 | RTN | 68.81 | 75.46 | 62.12 | 62.46 | 39.51 | 68.33 | 62.78 |  |
| SmoothQuant | 54.70 | 55.40 | 51.50 | 43.90 | 27.70 | 38.90 | 45.35 |  |  |
| LLM-QAT | 69.50 | 75.40 | 64.60 | 66.00 | 43.80 | 69.20 | 64.75 |  |  |
| PEQA (our impl.) | 72.97 | 77.80 | 67.72 | 67.13 | 40.27 | 73.35 | 66.54 |  |  |
| LR-QAT (ours) | 73.64 | 77.91 | 67.56 | 69.28 | 41.30 | 73.25 | 67.16 |  |  |
| 4-4-4 | RTN | 50.49 | 64.25 | 52.41 | 48.27 | 30.12 | 52.04 | 49.60 |  |
| SmoothQuant | 49.10 | 49.80 | 48.00 | 30.40 | 25.80 | 27.40 | 38.42 |  |  |
| LLM-QAT | 61.30 | 51.50 | 51.90 | 27.90 | 23.90 | 31.10 | 41.27 |  |  |
| LLM-QAT + SQ | 62.40 | 55.90 | 50.60 | 35.50 | 26.40 | 47.80 | 46.43 |  |  |
| Outlier Suppression+ | 60.21 | 62.73 | 52.96 | 39.98 | 30.29 | 44.39 | 48.43 |  |  |
| OmniQuant^§ | 63.51 | 66.15 | 53.43 | 45.20 | 31.14 | 56.44 | 52.65 |  |  |
| PEQA (our impl.) | 65.69 | 72.31 | 59.83 | 56.52 | 34.22 | 61.79 | 58.39 |  |  |
|  | LR-QAT (ours) | 67.16 | 71.76 | 59.59 | 58.42 | 34.73 | 62.34 | 59.00 |  |
| LLaMA-2 7B | FP16 |  | 77.74 | 79.11 | 69.14 | 74.58 | 46.25 | 75.98 | 70.47 |
| 4-8-8 | RTN | 75.87 | 77.91 | 67.88 | 71.09 | 44.03 | 74.51 | 68.55 |  |
| PEQA (our impl.) | 77.37 | 77.97 | 69.77 | 70.54 | 43.52 | 75.50 | 69.11 |  |  |
| LR-QAT (ours) | 77.00 | 78.13 | 69.14 | 72.10 | 44.11 | 75.44 | 69.32 |  |  |
| 4-8-4 | RTN | 70.37 | 76.01 | 63.38 | 68.94 | 41.47 | 70.76 | 65.16 |  |
| PEQA (our impl.) | 74.71 | 77.48 | 67.40 | 69.28 | 42.75 | 73.75 | 67.56 |  |  |
| LR-QAT (ours) | 74.46 | 77.69 | 68.51 | 69.78 | 42.75 | 73.82 | 67.84 |  |  |
| 4-4-4 | RTN | 57.86 | 64.91 | 54.46 | 49.62 | 31.83 | 51.83 | 51.75 |  |
| PEQA (our impl.) | 67.09 | 70.67 | 60.06 | 54.80 | 32.17 | 62.76 | 57.93 |  |  |
| LR-QAT (ours) | 66.94 | 71.98 | 60.77 | 57.20 | 33.87 | 63.10 | 58.98 |  |  |
| LLaMA-2 13B | FP16 |  | 80.55 | 80.52 | 72.22 | 77.44 | 48.98 | 79.38 | 73.18 |
| 4-8-8 | RTN | 79.24 | 79.27 | 70.01 | 75.51 | 48.29 | 76.92 | 71.54 |  |
| PEQA (our impl.) | 79.02 | 80.20 | 71.19 | 76.60 | 48.72 | 79.22 | 72.49 |  |  |
| LR-QAT (ours) | 81.10 | 79.76 | 71.35 | 77.36 | 50.60 | 78.91 | 73.18 |  |  |
| 4-8-4 | RTN | 77.25 | 76.61 | 66.69 | 67.72 | 41.13 | 72.98 | 67.06 |  |
| PEQA (our impl.) | 78.01 | 79.22 | 69.30 | 75.59 | 48.21 | 77.78 | 71.35 |  |  |
| LR-QAT (ours) | 78.59 | 79.54 | 70.80 | 75.29 | 48.04 | 77.64 | 71.65 |  |  |
| 4-4-4 | RTN | 62.60 | 67.90 | 53.20 | 57.32 | 34.13 | 55.25 | 55.07 |  |
| PEQA (our impl.) | 68.72 | 74.21 | 62.35 | 63.85 | 36.01 | 68.44 | 62.26 |  |  |
| LR-QAT (ours) | 70.64 | 73.88 | 63.14 | 61.78 | 38.14 | 68.31 | 62.65 |  |  |

**Table A7: A comparison between min-max and the best range setting used for round-to-nearest (RTN) initialization for LLaMA and Mistral models. We report WikiText-2 test set perplexity (lower is better) and average zero-shot accuracy (higher is better). ^§Uses asymmetric weight quantization.**
| # Bits | Range | WikiText-2 perplexity $\downarrow$ | Avg. zero-shot accuracy $\uparrow$ |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| estimator | L1-7B | L2-7B | L2-13B | L3-8B | M-7B | L1-7B | L2-7B | L2-13B | L3-8B | M-7B |  |
| FP16 |  | 5.68 | 5.47 | 4.88 | 6.14 | 5.25 | 69.68 | 70.47 | 73.18 | 74.22 | 75.69 |
| W4 pc | best est. | $L^{4}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{4}$ | $L^{4}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{4}$ |
| best | 6.33 | 6.14 | 5.21 | 7.53 | 5.91 | 68.51 | 68.88 | 71.73 | 72.19 | 73.44 |  |
| min-max | 6.85 | 7.14 | 5.40 | 10.53 | 6.33 | 66.23 | 66.41 | 72.19 | 67.44 | 71.84 |  |
| W4 g128 | best est. | $L^{5}$ | min-max | min-max | $L^{4}$ | $L^{5}$ | $L^{5}$ | min-max | min-max | $L^{4}$ | $L^{5}$ |
| best | 6.05 | 5.78 | 5.04 | 6.96 | 5.49 | 68.93 | 69.75 | 72.94 | 72.30 | 75.07 |  |
| min-max | 6.08 | 5.78 | 5.04 | 6.99 | 5.51 | 68.96 | 69.75 | 72.94 | 72.95 | 74.98 |  |
| W3 pc | best est. | $L^{3.5}$ | $L^{3.5}$ | $L^{5}$ | $L^{3.5}$ | $L^{4}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{5}$ | $L^{3.5}$ | $L^{4}$ |
| best | 12.88 | 26.73 | 8.71 | 34.10 | 9.49 | 54.66 | 43.87 | 55.01 | 47.46 | 64.58 |  |
| min-max | 2.4e4 | 1.9e4 | 2.3e3 | 1.6e5 | 3.2e3 | 36.02 | 35.71 | 37.85 | 35.78 | 36.78 |  |
| W3 g128 | best est. | $L^{5}$ | $L^{4}$ | $L^{5}$ | $L^{5}$ | $L^{5}$ | $L^{5}$ | $L^{4}$ | $L^{5}$ | $L^{5}$ | $L^{5}$ |
| best | 7.95 | 7.61 | 6.20 | 15.11 | 6.77 | 63.50 | 63.20 | 67.60 | 57.74 | 69.35 |  |
| min-max | 8.10 | 8.22 | 6.14 | 29.38 | 7.22 | 62.69 | 64.07 | 66.81 | 54.54 | 68.35 |  |
| W2 pc^§ | best est. | $L^{2.4}$ | $L^{3}$ | $L^{3}$ | $L^{3.5}$ | $L^{3.5}$ | $L^{2.4}$ | $L^{3}$ | $L^{3}$ | $L^{3.5}$ | $L^{3.5}$ |
| best | 4.9e3 | 5.2e3 | 5.2e3 | 6.4e4 | 6.8e3 | 37.92 | 36.52 | 36.27 | 36.80 | 36.59 |  |
| min-max | 1.1e5 | 2.5e4 | 4.9e4 | 1.4e6 | 7.5e4 | 39.07 | 36.53 | 39.01 | 37.69 | 38.13 |  |
| W2 g128^§ | best est. | $L^{4}$ | $L^{3.5}$ | $L^{5}$ | $L^{4}$ | $L^{5}$ | $L^{4}$ | $L^{3.5}$ | $L^{5}$ | $L^{4}$ | $L^{5}$ |
| best | 708 | 2.5e3 | 115.6 | 1.4e4 | 369 | 39.74 | 37.94 | 41.12 | 36.97 | 41.30 |  |
| min-max | 3.6e3 | 5.9e3 | 341 | 2.8e5 | 3.4e3 | 37.06 | 36.78 | 40.30 | 37.18 | 37.16 |  |

**Table A8: Runtime comparison between LR-QAT, full-model QAT (LSQ), full-precision training and full-precision LoRA for LLaMA 7B on Nvidia A100 80GB GPU, assuming effective batch size 32 and sequence length 1024. We repeat each experiment 5 times and report mean $\pm{}$ standard deviation. ✕ denotes out of memory.**
| # Bits | Method | Per-device batch size $\times{}$ | Time/100 steps, sec |
| --- | --- | --- | --- |
| grad. accumulation steps |  |  |  |
| FP16 | Full-model training | 1$\times{}$32 | 974${}^{\scriptsize\pm{}\text{3}}$ |
| FP16 | LoRA, $r=32$ | 1$\times{}$32 | 950${}^{\scriptsize\pm{}\text{3}}$ |
| W4 pc | Full-model QAT | 1$\times{}$32 | ✕ |
| W4 pc | Full-model QAT + CPU opt. offloading | 1$\times{}$32 | 7426${}^{\scriptsize\pm{}\text{216}}$ |
| W4 pc | Full-model QAT + CPU opt. offloading | 2$\times{}$16 | ✕ |
| W4 pc | Full-model QAT + checkpointing | 1$\times{}$32 | 3248${}^{\scriptsize\pm{}\text{7}}$ |
| W4 pc | Full-model QAT + checkpointing | 2$\times{}$16 | ✕ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=32$ | 1$\times{}$32 | 2938${}^{\scriptsize\pm{}\text{6}}$ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=32$ | 4$\times{}$8 | 1522${}^{\scriptsize\pm{}\text{5}}$ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=1$ | 4$\times{}$8 | 1519${}^{\scriptsize\pm{}\text{6}}$ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=4$ | 4$\times{}$8 | 1528${}^{\scriptsize\pm{}\text{3}}$ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=256$ | 4$\times{}$8 | 1546${}^{\scriptsize\pm{}\text{4}}$ |
| W4 pc | LR-QAT, $\varphi{}={\text{{\small{INT4}}}}$, $r=32$ | 4$\times{}$8 | 1518${}^{\scriptsize\pm{}\text{8}}$ |
| W4 g128 | LR-QAT, $\varphi{}={\text{{\small{Q4.4}}}}$, $r=32$ | 4$\times{}$8 | 1528${}^{\scriptsize\pm{}\text{5}}$ |

## Appendix B Experimental details

In this section, we list the details related to hyperparameters and other settings used in our experiments.
If not stated otherwise, the standard hyperparameters that we use are the one shown in Table B1.

**Table B1: Common hyperparameters used for experiments. ^*Is equivalent to freezing the quantization scale / zero offset obtained after initial range estimation ($\mathbf{s}=\mathbf{s_{0}}$, $\mathbf{z}=\mathbf{z_{0}}$).**
| Hyperparameter | Value / Search space |
| --- | --- |
| Optimizer | AdamW |
| Learning rate for $\bm{A}$, $\bm{B}$ (INT4 / INT3 / INT2) | $\{10^{-5},10^{-4},10^{-3},10^{-2}\}$ |
| Learning rate for $\mathbf{s}$ (INT4 / INT3) | $\{0{{\textsuperscript{*}}},10^{-5}\}$ |
| Learning rate for $\mathbf{s}$ (INT2) | $\{10^{-5},10^{-4}\}$ |
| Learning rate for $\mathbf{z}$ (INT2 only) | $0{{\textsuperscript{*}}}$ |
| Learning rate for $\bm{W}$ (full-model QAT only) | $\{1,5,10,50,100\}\cdot 10^{-5}$ |
| Learning rate schedule for $\bm{A}$, $\bm{B}$ | linear (with warmup) |
| Learning rate schedule for $\mathbf{s}$ | linear (with warmup) |
| Learning rate schedule for $\bm{W}$ (full-model QAT only) | linear (with warmup) |
| Weight decay for $\bm{A}$, $\bm{B}$ | 0 0 |
| Weight decay for $\mathbf{s}$ | 0 0 |
| Weight decay for $\bm{W}$ (full-model QAT only) | $0.1$ |
| Adam $\left(\beta_{1},\beta_{2}\right)$ | $\left(0.9,0.95\right)$ |
| Training steps | $10^{4}$ |
| Warmup steps | $10\%$ of Training steps |
| Batch size | $32$ |
| Maximum sequence length (during training) | $1024$ |
| $L^{2}$-norm gradient clipping (maximum norm) | $1.0$ |
| $\alpha$ in (8) | $1.0$ |

#### Quantization

We experiment with both weight-only and weight-activation quantization.
The default settings are INT4 / INT3 / INT2 per-channel (denoted ‘pc’) and group-wise weight quantization with a group size of 128 (denoted ‘g128’).
We use symmetric quantization, except the INT2 case, where we use asymmetric quantization (1), for a fair comparison with related work.
We quantize all linear layers, except the classification head.
In weight-activation quantization, defaults are INT4 per-channel weight and
per-token activation quantization [].
Following OmniQuant [], we quantize all inputs to matmuls with exception of the softmax output and additionally quantize the KV-cache as in LLM-QAT [].

#### Libraries

We implement our method in PyTorch [] and use training and evaluation pipelines from HuggingFace libraries [, , ].
For zero-shot evaluation, we use the LM Evaluation Harness framework [].
Specifically, we use lm_eval v0.4.2 and report acc_norm for tasks where it’s available (PIQA, ARC-e, ARC-c, HellaSwag) and otherwise acc (BoolQ and Winogrande).

#### Datasets and training

To optimize the learnable parameters, we use AdamW optimizer [] with weight decay set to zero, $(\beta_{1},\beta_{2})=(0.9,0.95)$ and linear learning rate warm up over the first $10\%$ steps, following by a linear decay to zero by the end of training.
We use a separate maximum learning rate for quantization scales and for low-rank adapters, which are tuned depending on the experiment.

We apply our methods to all linear layers in the attention blocks (both in self-attention and in the feed-forward network).
We only train low-rank auxiliary matrices $\bm{A}$, $\bm{B}$ and the quantization parameters $\mathbf{s}$.
Specifically, we freeze embedding layers, the final classification heads and RMSNorm parameters.
In the case of asymmetric weight quantization, a zero offset $\mathbf{z}$ is set during range estimation phase and kept frozen throughout training.

We train on a small subset of SlimPajama [], which is a close open-source replica of the dataset used for pre-training LLaMA models.
We select hyperparameters based on the perplexity of a small subset of Wikipedia validation set(^4^44Specifically, we use the English subset of Wiki40b, [https://huggingface.co/datasets/wiki40b](https://huggingface.co/datasets/wiki40b), that contains cleaned-up text of English Wikipedia and training/validation splits.) (512 sequences).
For all weight-only and weight-activation quantization results, including the comparison with full-model QAT in Section 4.3, we train for $10^{4}$ steps.
For ablation studies in Sections 4.1 and 4.2 we use shorter training of $10^{3}$ steps.
Since the full-model QAT experiment requires more than 80GB of GPU memory, we apply checkpointing on the quantization function $\widehat{\bm{W}}$ to be able to run the experiment on an Nvidia A100 80GB GPU.
We have also experimented with CPU optimizer state offloading, but that turned out to be significantly slower, see details in Table A8.
The rest of the hyperparameters and their search spaces are listed in Table B1.

#### PTQ initialization

We compare with vanilla round-to-nearest quantization (RTN), where we explore several choices of range setting and report the best one based on Wikipedia validation set perplexity, and also use that as initialization for our method.
Specifically, we experimented with min-max range estimator and with $L^{p}$-norm range estimator with the following values for $p$: $\{2.0,2.4,3.0,3.5,4.0,5.0\}$.

#### Computational Resources

All the experiments were executed on a single Nvidia A100 GPU equipped with 80GB of VRAM.
Models of size 7B, 8B and 13B needed respectively around 2.25, 2.75 and 4.5 days for $10^{4}$ training steps experiments.
For obtaining all the results in the paper, including the ablations, we needed 202 GPU days (A100).
Including preliminary experiments that did not make it in the final paper and hyperparameter turning we estimate the total compute costs of this research to approximately 750 GPU days.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses
soundness: 3
strengths: The paper is well-written and explained clearly. The approach is well-supported by a thorough experimental section, showcasing promising results that validate its efficiency and robustness across multiple large language models.
summary: This paper introduces LR-QAT, a quantization-aware training framework designed for large language models that optimizes memory efficiency without compromising model performance. It incorporates low-rank quantization-aware reparameterization, downcasting, and checkpointing techniques. The proposed technique reduces memory usage significantly often outperforming PQT techniques while allowing integration with such techniques. Additionally, it also does not incur any additional overhead during inference.
weaknesses: The novelty of the approach is somewhat limited as it incorporates several known techniques rather than introducing entirely new concepts. While the paper provides detailed memory and runtime comparisons with full-model QAT (e.g., LSQ), it does not compare these metrics against other implemented baselines like LSQ, or OmniQuant, which limits the assessment of its relative efficiency.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The questions have been shared in earlier section"Weakness"
soundness: 3
strengths: 1. The paper is well-written with adequate background and related work.

2. The paper combines LoRA + QAT techniques and applies down casting and gradient checkpointing for memory-efficient and inference-efficient LLMs

3. The key innovation lies in seamless fusion of the quantized low-rank adapters with the quantization field of frozen retrained weight unlike other LoRA inspired quantization works.

4. The empirical analysis is extensive with sufficient comparisons and ablation studies with impact of rank, and down casting operation.
summary: The authors propose a Low-Rank Quantization Aware Training (LR-QAT) algorithm to train LLM model on single consumer grade GPU. In particular, LR-QAT is based on quantizing low-rank adapters (LoRA) such that it could be fused into quantized frozen pre-trained weight. In addition, it uses a downcasting operator for frozen pretrained weight and gradient checkpointing for tackling increased memory usage owing to product computation of low rank adapters. In contrast to other works on Post training quantization, Quantization-aware training of full models, and PEFT-based (LoRa) quantization schemes, LR-QAT is geared towards better accuracy, memory-efficiency and inference-efficiency (via fusion of low-rank adapter to low-bit integer frozen weight)
weaknesses: 1. Figure 1 probably is not referenced in the main text. Probably a deeper discussion on Fig. 1 (right) is needed with respect to goal of training LLMS on single device with 24GB.

2. It might be helpful to have a comparison table or figure that depicts the what parts of the model are being quantized and quantization scheme used across various PTQ, QAT and LoRa-inspired works referenced and compared in experiments - this would help put related works in perspective.

3. LoRA is predominantly used for fine-tuning as part of PEFT methods, but in many places, the authors use training, pertaining and fine-tuning with LoRA interchangeably. It would be helpful to be consistent with keywords related to LoRA and proposed LR-QAT. eg. Line 187, should it be fine-tuning compared to training, title of the work, quantization-aware training or pretraining of LLMs when authors claim LR-QAT to be general purpose extended pretraining method versus fine-tuning method

4. PTQ and LR- abbreviations used in abstract without full form.

5. Line 48: Please specify which is most related work the authors are contrasting with, since they referenced both PEQA (Line 176) and QA-LoRA (Line 195) to be closest to theirs.

6. Where does PEQA fit in Table 1?

7. Lines 049 and 177: Define inference efficiency of QAT here earlier than Lines 201-202

8. Line 65: Define r as "rank" 

9. Line 193: Typo "is" ..This issue.....

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See the weakness.
soundness: 2
strengths: LR-QAT introduces and combines several innovations designed to reduce memory use without sacrificing model performance: (1) a form of QAT with low-rank reparameterization, in which it places the low-rank weights in the integer domain to ensure they align with the quantization grid of the pretrained weights. This allows for seamless fusion during inference into a single low-bit integer matrix. (2) A downcasting operator that represents the frozen pretrained weights as low-bit INT-b (b ≤ 4) double-packed into INT8 or as fixed-point values stored in INT8. 

It applies LR-QAT to LLaMA-1/2/3 and Mistral model families and demonstrate its effectiveness on several general language modeling datasets and zero-shot evaluation on some of the common reasoning downstream tasks. The method outperforms recent LLM quantization approaches and reaches the same predictive performance as full-model QAT at the fraction of its memory usage.
summary: Inspired by parameter efficient fine-tuning literature, this paper proposes LR-QAT – a lightweight and memory-efficient QAT algorithm for LLMs. LR-QAT employs several components to save memory without sacrificing predictive performance: (a) low-rank quantization-aware reparameterization; (b) downcasting operation using fixed-point or double-packing and (c) checkpointing.
weaknesses: The novelty may be limited. The proposed method combines traditional quantization and LoRA methods. The downcasting operator cast the input to one of pre-existing floating-point formats which follows previous works such as  (Oberstar, 2007) and  (Li et al., 2023).  The gradient checkpointing mainly follow the previous work (Chen et al., 2016). The technical contribution may be limited. 

It only has results which finetune on a single dataset. It is better to demonstrate the performance by finetuning on different datasets. For example, QLoRA and QALoRA finetunes on two different datasets. Finetuning on multiple different datasets can demonstrate the performance in general cases. If finetuning on only one single dataset, we are not sure about the general performance of the proposed method on other datasets. 

There are some typos such as  'leading to a subpar performance. paragraph' in Line 180. 

The claim of 'an extended pretraining method' for this work does not seem to be solid. It seems that 'task-specific fine-tuning' or 'extended pretraining' just depends on which dataset is used for further training. Using downstream datasets is  task-specific fine-tuning    and using pretraining dataset is the so called extended pretraining.  This difference is minor and the proposed method or the baselines can switch the finetuning datasets.  Thus these methods can either be task-specific fine-tuning or extended pretraining. It does not seem to be very difficult to switch the datasets.  It is better to provide more discussion about the claim of extended pretraining. 

The method is very similar to QA-LoRA. In QA-LoRA, weights merged with LoRA are still in a quantized form so that LLMs can be deployed with computational efficiency. The core idea of has been investigated in QA-LoRA. It is better to have a more detailed comparison with QA-LoRA to discuss the difference. It is better to compare with QA-LoRA in experiments.

---

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1- **Choice of $s_0$ in Equation 5**:  Could the authors clarify how  $s_0$ is chosen? Is it based on a range estimation technique or empirical tuning?

2- **Scaling Parameters for Integer Representation**:  I believe that the fixed point representation of the weights can be modeled using the integer representation with tuning of scaling parameters. Have the authors experimented with tuning the scaling parameters?

3- **Ignoring Quantization in Backward Pass**: Would omitting quantization for weights and low-rank adapters during the backward pass yield notable memory or computation savings?

4- **Trade-offs with Gradient Checkpointing**: I'm wondering what the effects of ignoring quantization of the weights (and low-rank adapters) are in the backward pass. This could lead to both memory and compute savings in the backward pass, since it allows efficient computation of low-rank multiplications.

5- **Adapter Rank and Dataset Size Scaling**: Based on Figure 2, the perplexity of the model w.r.t. the adapter rank is almost flat, and has perturbations in higher ranks. As the authors have pointed out, that might be an issue with the size of the fine-tuning set. I suggest the authors use scaling laws for finding the dataset sizes for different low-rank adapters and check if that solves the inconsistency with these results.

6- **Alternative Initializations for Quantized Weights**: Has the paper considered initialization methods beyond RTN? OmniQuant could offer improved results by leveraging adaptive rounding and error minimization techniques, respectively.
soundness: 3
strengths: 1- **Efficiency in Memory and Computation**: The approach effectively reduces the memory requirements for QAT, enabling large models to be trained on a single 24GB GPU without compromising predictive performance.

2- **Inference Efficiency**: Unlike many related low-rank and quantization methods, LR-QAT allows the fused low-bit pretrained model to be used directly in inference without additional overhead, which is a practical advantage.

3- **Comprehensive Experimental Validation**: The authors evaluate LR-QAT on multiple LLM families and bitwidth settings, demonstrating versatility across different quantization and low-rank configurations.

4- **Analytical Justifications**: The paper offers theoretical reasoning for observed outcomes, such as the choice of downcasting operators and low-rank approximation within the quantization operation, which contributes to understanding the method's impact on LLM quantization.
summary: The paper proposes Low-Rank Quantization-Aware Training (LR-QAT) to address the memory and computational inefficiencies of traditional quantization-aware training (QAT) methods when applied to large language models (LLMs). LR-QAT integrates low-rank adapters into the quantization process and utilizes fixed-point representations and gradient checkpointing to minimize memory usage. Applied to several LLM architectures, LR-QAT achieves model performance on par with full-model QAT, while being resource-efficient and inference-friendly, making it feasible for deployment on consumer-grade GPUs.
weaknesses: 1- **Limited Comparison in Initialization**: The paper relies on round-to-nearest (RTN) for initializing quantized weights without extensive exploration of alternatives. AdaRound [1] has shown that RTN is not necessarily and optimal quantization scheme, and even searching over random quantization schemes can lead to better results. The authors even pointed our using other method such as OmniQuant as a possible initialization scheme.

2- **Lack of Complete Results in Tables**: In Tables 4 and 5, several cells contain dashes (“-”), implying missing data. This restricts fair comparison across methods and limits insight into LR-QAT’s performance in certain configurations.

3- **Potential for Limited Practical Impact**: In cases where LR-QAT's fine-tuning time is significant, the marginal performance gain over zero-shot methods may not justify the overhead, questioning the scalability and applicability of LR-QAT in high-stakes deployments. A comparison between the quantization time required for different methods can help shed light to the trade off between different methods (in addition to ther fine-tuning timing results reported in the appendix).

4- **Fixed Point Representation Modeling**: I believe that the fixed point representation of the weights can be modeled using the integer representation with tuning of scaling parameters. This could potentially yield better accuracy and flexibility without increasing the complexity of implementation.

**Typos and Minor Issues**
- Please ensure consistent notation for mathematical symbols throughout equations.
- Avoid using “paragraph” as a placeholder, as seen in Section 2.

[1] Nagel et al., "Up or Down? Adaptive Rounding for Post-Training Quantization," ICML 2020

Meta-review: This paper proposes a combination of low-rank and quantization-aware training for LLMs, which  incorporates low-rank quantization-aware reparameterization, downcasting, and checkpointing techniques. The strength of the paper is the good empirical results, showing the proposed framework is better than previous quantization/low-rank training in several settings, while there are two major weaknesses pointed out by the reviewers: (1) Since the proposed method is based on a combination of several existing techniques, the novelty of the paper is limited, and (2) Several reviewers pointed out some missing numbers in the results and missing baselines/datasets, which makes it harder to conclude whether the proposed method is generally better than previous ones in general. 

Since the novelty is limited and the current empirical results are not that complete, we recommend to reject the paper and encourage the authors to improve the experimental results and resubmit their paper to another top ML conference.

Decision: Reject