QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models
Authors: Yuhui Xu
Lingxi Xie
Xiaotao Gu
Xin Chen
Heng Chang
Hengheng Zhang
Zhengsu Chen
XIAOPENG ZHANG
Qi Tian
Venue: ICLR 2024 poster (2024)
Keywords: Large Language Models
Quantization
LoRA
Instruct Fine-tuning
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: An efficient quantization-aware LoRA algorithm for fine-tuning LLMs; no post-training quantization is required and accuracy is improved

Abstract: Recently years have witnessed a rapid development of large language models (LLMs). Despite the strong ability in many language-understanding tasks, the heavy computational burden largely restricts the application of LLMs especially when one needs to deploy them onto edge devices. In this paper, we propose a quantization-aware low-rank adaptation (QA-LoRA) algorithm. The motivation lies in the imbalanced degrees of freedom of quantization and adaptation, and the solution is to use group-wise operators which increase the degree of freedom of quantization meanwhile decreasing that of adaptation. QA-LoRA is easily implemented with a few lines of code, and it equips the original LoRA with two-fold abilities: (i) during fine-tuning, the LLM's weights are quantized (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the LLM and auxiliary weights are naturally integrated into a quantized model without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model families and validate its effectiveness in different fine-tuning datasets and downstream scenarios. The code is made available at https://github.com/yuhuixu1993/qa-lora.

Published as a conference paper at ICLR 2024
QA-LORA: QUANTIZATION-AWARY LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Yuhui Xu Lingxi Xie X Xiaotao Gu
Xin Chen
Heng Chang
Hengheng Zhang Zhensu Chen Xiaopeng Zhang Huawei Inc.
Qi Tian
( corresponding author)
(xyh6666, 98808xc, guxt 994, chenxin061, {1mhmhm, chenzhengsul, 2xphi con.
.com
tian. wett.qii@huwwwr
ABSTRACT
Recently years have witnessed a rapid development of large language models (LLMs). Despite the strong ability in many umguuag--unerrtanddng tasks, the heavy computational burden largely restricts the application LLMs especially when needs deploy them onto edge devices. In this paper, propose quantization-awe - low-rank adaptation (QA-LoRA) algorithm. The motivation lies in the imbalanced numbers of parameters for quantization and adaptation, and the solution is to use group-wise operators to increase the number of parameters for quantization meanwhile decreasing that of adaptation. QA-LoRA is easily im- plemented with few lines of code, and it equips the original LoRA with two-fold abilities: (i) during senittnnng, the LLM's weights are quantized (e.g., into INT4) to reduce time and memory usage; (ii) after fine-tuning, the LLM and auxiliary weights are naturally integrated into quantized model without loss of accuracy. We apply QA-LoRA to the LLaMA and LLaMA2 model families and validate its effectiveness different fine-tuning datasets and downstream scenarios. The code is made available at https: /eggt//gthtttt com/ http/yuhuuxxxl990 993 //qaa -lora.
INTRODUCTION
Recently, large language models (LLMs) (Brown et al., 2020; Scao et al., 2022; Zhang et al., 2022; Touvron al., 2023a; Chowdhery al., 2022; OpenAI, 2023; Zeng et al., 2023) have shown un- precedented performance across a wide range of language understanding tasks (Wei et al., 2022a) and served as the foundation state-of-the-art chat systems (Bubeck et al., 2023). The diversity of real-world applications calls for pipeline in which LLMs can be fine-tuned to fit different scenarios and quantized to be deployed onto edge devices (e.g., mobile phones), and the key issue is to get rid of the heavy computational burden brought by the large number of parameters of LLMs.
There are two lines of research for this purpose. The first one is parameter-efficien fine-tuning (PEFT) (Houlsby et al., 2019; Li & Liang, 2021; Liu al., 2021; He al., 2022; Hu et al., 2021) which introduced a small number of learnable parameters while keeping most pre-trained param- eters unchanged. Among them, low-rank adaptation (LoRA) (Hu et al., 2021), a popular PEFT algorithm, proposed to fine-tune low-rank matrices to complement the pre-trained weights. Despite the comparable performance full-parameter fine-tuning, the memory usage of LoRA is still large, especially when the base LLM is large (e.g.. LLaMA-65B). The second one studies parameter quan- tization (Yao al., 2022: Dettmers et al., 2022: Wei al., 2022b; Frantar et al., 2023; Lin al., 2023; Xiao et al., 2023; Dettmers et al., 2023b) where the trained weights are quantized into low-bit integers or floating point numbers. Although these methods can alleviate the computational burden, they often report unsatisfying accuracy especially when the quantization bit width is low.
Hence, it is an important topic to integrate PEFT with quantization. naive solution to perform post-training quantization (PTQ) after PEFT, but it reports unsatisfying accuracy especially when the quantization bit width is low. Advanced methods exist, but they are either computationally ex- pensive in the fine-tuning stage (Liu et al., 2023) unable maintain the quantized property after fine-tuning (Dettmers et al., 2023a). In this paper, we propose a simple yet effective method for quantization-awan low-rank adaptation (QA-LoRA). Our idea is based on the imbalanced numbers

---

Published as a conference paper at ICLR 2024
QLARA
QUoRA (4-30)
QA-EARA (4-Ng)
QUURA (3-6i)
(3
QLORA (240)
QATTRRA (2-kg)
01.4 62
60
60
60
25:30
51A
50
41 (+22
40
35.9
40
11.4
30
30
30
(4) QI Q4--1.RAAA 4-bit
(b) Ql ARA QA-L-oRAin 3-bir
(e) QI ORA QA-LORA 2-bit
Figure I: The comparison of 5-shot MMLU accuracy (%) with different quantization bit widths based on the LLaMA model family. QLoRA (NF4 FP16) refers to the original QLoRA models with pre-trained weights in INT4 and adapter weights in FP16, and QLoRA (INT4) refers to per- forming post-training quantization (into INT4) upon the merged QLoRA models. All models are fine-tuned on the Alpaca dataset. Full results are provided in Table 1.
of parameters for quantization and adaptation. Specifically, each column of the pre-trained weight matrix is accompanied by only one pair of scaling and zero parameters but many more LoRA pa- rameters. This imbalance not only results in large quantization errors (which harm the LLM's s accu- racy), but also makes it difficult to integrate the auxiliary weights into the main model. QA-LoRA addresses the issue by introducing group-wise operators to increase the number of parameters for low-bit quantization (each group quantized individually) and decrease that of LoRA (each group shares the adaptation parameters). QA-LoRA enjoys two-fold benefits: (i) an efficient fine-tuning stage thanks to the LLM's weights being quantized into low-bit integers; (ii) a lightweight, fine- tuned model without the need for PTQ which often incurs loss of accuracy.
We evaluate QA-LoRA on the LLaMA and LLAMA2 model families (Touvron et al., 2023a:b) and validate it on various language understanding benchmarks. Figure shows the comparison of 5-shot accuracy on the MMLU benchmark between QA-LoRA and the direct baseline, QLoRA (Dettmers et al., 2023a). QA-LoRA consistently outperforms QLoRA with PTQ on top of LLMs of different scales (the advantage becomes more significant when the quantization bit width is lower) and is on par with QLoRA without PTQ. Note that during inference, QA-LoRA has exactly the same complexity as QLoRA with PTQ and is much more efficient than QLoRA without PTQ. Hence, QA-LoRÁ is effective and off-the-shelf method for joint quantization and adaptation of LLMs.
2 RELATED WORK
Large language models (LLMs) (Devlin et al., 2019; Brown et al., 2020; Zhao et al., 2023a; Hadi et al., 2023; Yu et al., 2023) have emerged as a dominant paradigm in natural language processing. It achieved the state-of-the-arr on various tasks (Zhao et al., 2023b; Zhou et al., 2023; Wang al., 2023) and served as the fundamental of chat systems (OpenAI, 2023). However, their deployment in real-world scenarios is hindered by the high computational and memory requirements during inference (Chang et al., 2023). To tackle this issue, various methods have been proposed, including distillation (Liu al., 2023). quantization (Yao et al., 2022: Dettmers et al., 2022: Wei et al., 2022b; Frantar et al., 2023; Lin et al., 2023; Xiao et al., 2023), pruning (Frantar & Alistarh, 2023; Ma et al., 2023; Sun al., 2023), etc. (Weng, 2023). This paper mainly focuses on the quantization of LLMs.
Fine-tuning LLMs with adapters. Parameter efficient fine-tuning (PEFT) is an important topic for LLMs. One of the most popular approaches low-rank adaptation (LoRA) (Hu et al., 2021; Valipour et 2022),, where the key insight is to decompose the adapter weights into the multiplication of two low-rank (and thus parameter-efficient matrices. LoRA has claimed comparable performance to full fine-tuning while using much fewer learnable parameters. Meanwhile, there are also other branches of adapters for LLMs such as the series adapter (Houlsby et al., 2019) and parallel adapter (He et al., 2022). Please refer to (Mangrulkar et al., 2022; Hu et al., 2023) for review of these adapters.
Quantization of LLMs. Quantization is a compression technique that reduces the bit width of the parameters and/or activations of LLMs improve their efficiency and scalability (Xiao et al., 2023: Dettmers et al., 2022; 2023a). Existing methods mostly focused on preserving or restoring the accuracy of quantized LLMS during the inference stage (Zhu et al., 2023), where the key is to
2

---

Published as a conference paper at ICLR 2024
reduce the memory footprint and computational costs without re-training the LLMs. One of the main challenges is to handle the outliers in the parameter distribution (Xiao et al., 2023), which can cause significant errors when quantized. To address this issue, some methods proposed to use either adaptive or dynamic quantization schemes that adjust the quantization range or precision according to the parameters (Xiao et al., 2023; Dettmers al., 2022). Other methods used sophisticated group- ing or clustering techniques to partition the parameters into different groups and applied different quantization strategies for each group (Park al., 2022; Yao al., 2022; Wu al., 2023).
Joint adaptation and quantization. This paper aims to achieve the objectives of both parameter- efficient adaptation and computation-efficient tuning and deployment, which can further improve the efficiency and scalability of LLMs as well as mitigate the negative impact of quantization er- rors. However, this also poses additional challenges, such as propagating gradients through discrete values and optimizing the quantization parameters. To overcome these challenges, lossy quantiza- tion methods proposed to use stochastic rounding (Shen et al., 2020) or learned rounding (Esser et al., 2019) to approximate the gradients and update the parameters, but applying these methods to LLMs often difficult. Other methods proposed to use switchback layers (Wortsman et al., 2023) or mixed-prectt precision inference (Dettmers et al., 2023a) to alternate between quantized and full/half- precision values, which often result in low inference speed.
To the best of our knowledge, the most related work is QLoRA (Dettmers et al., 2023a) which squeezed the pre-trained weights into NF4 and added LoRA. However, QLoRA added the adaption weights back pre-trained weights and turned them into FP16 again, and thus the deployed model is still slow. We solve this problem with the proposed QA-LoRA approach.
3 THE PROPOSED APPROACH
3.1 BASELINE: LOW-RANK ADAPTATION AND LOW-BIT QUANTIZATION
We follow the notation system used in LoRA (Hu et al., 2021) which assumed pre-trained weights to form a matrix W and the features form a vector x. The definition is easily applied to a wide range of scenarios and extended into is set of vectors (e.g., feature matrix). Let the size of W be Din x Dout and x has the length of Din. and thus the computation is easily written as = W x where y is the output vector with length of Dout-
The key idea of LoRA is to introduce a pair of matrices, A and B, to supplement W. A and B have sizes of Dm Diue and Dint x Dout, respectively, so that their multiplication, AB, has the same size as W. The intermediate dimensionality is often small (i.e., Dine < min Din Dout H), making AB low-rank matrix compared to W. During fine-tuning, we compute y W x s (ABB) x, where s is the coefficient for weight tuning, and W is fixed while A and B can be adjusted, arriving at the goal of parameter-efficient fine-tuning. After fine-tuning, the computation is reformulated into y (W + 8 AB) x, where W is replaced by W' = W + 8 AB for fast inference.
Another effective way to reduce computational costs lies in low-bit quantization. We only consider the quantization of weights throughout this paper. In particular, we apply a simple method named min-max quantization. Mathematically, given the bit width N and a pre-trained weight matrix W, we compute the minimum and maximum values across all elements of W, denoted as min (W) and max(W), respectively. Then, Wir is quantized into W by computing
W
W W = W + +
+ +
(I)
α
where (max (W)-
1) and min(W) are called the scaling and zero factors, respectively; 1-7 denotes the integer rounding operation. All elements in W are in the set of {0, 1. 2 1) and thus stored as B-bit integers. The computation, y W x, is approximated
as = W x = w x 3x. The quantization brings two-fold benefits, namely, the storage
of W is reduced (e.g., from FP16 to INT4) and the computation of W x becomes faster. The cost is that W is an approximation of W, which may harm the accuracy of language understanding.
To reduce the quantization loss between W and W, an effective strategy is to perform an individual quantization for each column of W. Let W [wig]] D.n Dews where (1,..., D and j
3

---

Published as a conference paper at ICLR 2024
LoRA
QLoRA
QA-LoRA
Fine-funie fee--uunng
Dout
Dout
Dout
B
Pre-trained Weights (FP16)
Quantized Weights W (NF4)
Quantized Weights W (INT4)
Dint
Dint
Pint
A
Dmn
Dun
Guli
X
X
D
Capy.
Inference
FP16
EP16
INT4
Figure 2: An illustration the goal QA-LoRA. Compared to prior adaptation methods, LoRA and QLoRA, our approach is computationally efficient in both the fine-tuning and inference stages. More importantly, it does not suffer an accuracy loss because post-training quantization not required. We display INT4 quantization in the figure, but QA-LoRA is generalized INT3 and INT2.
{I...., } are iterative variables. Let ay and Bg be the scaling and zero factors computed on the j-th column, wy. Hence, Equation 1 is updated as W [wj] Dous Oj
w,-2,
+ /3
Don
and the computation is rewritten y W x aj w
Byx
Compared to
Daus
the original (holistic) quantization, the computational cost is unchanged while the storage cost of the scaling and zero factors increases from 2 to 2D floating point numbers. This is negligible compared to the reduced cost of storing the full cell-precision W.
3.2 OBJECTIVE: EFFICIENT ADAPTATION AND DEPLOYMENT
As shown in Figure 2, we aim to achieve two goals. First, during the fine-tuning stage, the pre- trained weights W are quantized into low-bit representation so that LLMSs be fine-tuned as few GPUs as possible. Second, after the fine-tuning stage, the fine-tuned and merged weights W are still in a quantized form so that LLMs can be deployed with computational efficiency.
We note that QLoRA (Dettmers et al., 2023a), a recently proposed variant of LoRA, achieved the first goal. The idea is quantize W from FP16 NF4 (a highly squeezed type of floating point numbers) during the fine-tuning stage. We learn from QLoRA that joint optimization of quantization and adaptation is tractable because the accuracy loss between W and W is compensated by the low- rank weights, 8 AB. After fine-tuning, the side weights s AB must be added back to W, making the final weights W in FP16 again. Indeed, one can perform post-training quantization (PTQ) upon W', but this strategy can cause a significant loss in accuracy especially when the bit width is low. Please refer to the experiments for details. Additionally, there no operator-level optimization for NF4 yet, making it difficult to accelerate the fine-tuning and inference stages. In brief, the only benefü brought by QLoRA is the reduced memory cost for fine-tuning.
3.3 SOLUTION: GROUP-WTE QUANTIZATION WITH LOW-RANK ADAPTATION
From the above analysis, the key to achieving the second goal lies in that W (i.e., the quantized W) and s AB can be merged without using high-precision numbers (e.g., FP16). We first note that this impossible in the original setting, i.e. W is quantized into W in a column- manner while both A and B are unconstrained.
We write down the condition using the mathematical language. Since W = W + AB, we have w/y, W + Σg di,bby,j j for all (i.j). Here, for any j. all Way represented using the same set of scaling and zero factors, i.e., there exist aj and 3, so that Wi,j = aj x Wi + By, Wi E {0, 1....... N -1}. After each W; is added by s- s.Er ar,bb.. (abbreviated as ci,;), if we want to keep the property for quantization, must guarantee that for any j. all possible values of o.j
4

---

Published as a conference paper at ICLR 2024
Algorithm QA-LoRA Pseudocode in the PyTorch-like style
4 D_in, D_out, : the input, output, and low-rank adaptacion dimensions the quantization group numbers of weights (D_in is the group size)
4 the coefficient for adaptazion; N: the bit width of quantlzation
0A nn.AvgFool Id(fii///
1ora_A mn.Parameter (torch. empty: (D_int, 1)) (R_out, 1ora_R- nn,Paramster storch.empryy htp_emyyppytuttu,, (_int)))
def sallrr__erwardd
и, 1ora_A, 1ora_B);
K_tildo pre__puuee prre_quentizatinn
alpha, bets)
result = x 8 R_tilde
result (QA (D.1n//1) lora_A.trenaposee(0 crra__.cran
return result
def pre_maanization alpha, beta):
K_hat torch.round(1 alphal beta
return elpha (W_hat beta)
def merge_withe with_quantization.eeem (beta, Aora_A, lora_B):
beta_new - beta - a (Iora_B 8 Aora_A) (at.llttaanee .transpose (0, alpha return beta_new
form an arithmetic set with the common difference being ay This is intractable in continuous and gradient-based optimization unless we ask that C. a constant, i.e., C1.j = Cij = ............. C for any j. This is equivalent to set all row vectors of A to be same, i.e., ai = where = denotes element-wise equivalence between two vectors.
a....
The above strategy, while tractable, leads to a significant accuracy drop in practice. In particular,
with all rows of A being the same vector, we have rank (A) = 1 and thus rank(AB) ) = 1. whereas the rank of AB is correlated to the ability of fine-tuning W new data (Hu et al., 2021; Valipour al., 2022; Dettmers al., 2023a). To address this issue, straightforward idea is relax the constraints for both quantization and adaptation.
We partition each column of W into I. groups where, for of implementation, set L. to be divisor of Din- Instead of quantizing each column of W entirely, we use an individual pair of scaling and zero factors for quantization, i.e., the (-th group of factors, o1 and Bt.j. are computed for D L elements in the j-th column. Correspondingly. we only require the row vectors of A within the same group to have the same value. In our implementation, this is achieved by doing summation within each group of the input vector, x. This parameter-fre free operation reduces the dimension of x from Din to L, hence we can set to be L X Dint matrix without further constraints.
The proposed approach is named quantization-awan low-rank adaptation (QA-LoRA). It is im- plemented by inserting/modifyiing few lines of code beyond QLoRA, as shown in Algorithm 1 Compared to LoRA, QA-LoRA enjoys advantages in time and memory consumption. Compared to QLoRA, QA-L requires extra storage for L Dout pairs of scaling and zero factors but reduces the number of parameters of A from D., x Dine to L Diat since often set L < Din- the above change is negligible. Additionally, QA-LoRA is much faster in inference because merges s AB into W while keeping the merged matrix W' quantized in low bits.
The insight of QA-LoRA: balance, QA-LoRA is very similar to a variant of QLoRA in which NF4 quantization is replaced by INT4?). In this version, the number of parameters for quantization (Dout pairs of scaling and zero factors) is much smaller than that for adaptation (Di, Dim: + D X Dout parameters), i.e., there is a significant imbalance here. We introduce group-wise operations, increasing the number of parameters for quantization from Dout L Dout. meanwhile decreasing that for adaptation from Di x Dim + Dim x Dout to L x Duut + Dint x Doott As we shall see in experiments, moderate can achieve satisfying accuracy of language understanding meanwhile preserving computational efficiency.
'The exact conditions are two-fold. For any j. there exists new zero factor 8y and set of integers ory that ci. = aj G By. Additionally, the difference between the minimum and maximum of uhs + A.j still be quantized into B-bit integers.
not greater than 2 - 1 so that the summed weights
"The merge_wi t ch_quantiii i zat on function called after the training procedure for merging weights We implemented this version of QLoAA, and it reports very similar (+0.5%) accuracy compared to the original QLoRA in the few-shot experiments for MML.U.
5

---

Published as a conference paper at ICLR 2024
0-shot and 5-shot accuracy (%) the Massive Multitask Language Understanding (MMLU) dataset (Hendrycks et al., 2021). Each block is based on the same foundation model specified at the first row. We organize all results using the fine-tuning dataset (Alpaca or Flan-v2) and the bit width of quantization. The bit width of 4 16' refers to the original QLoRA where the final version for inference is in FP16. Some important numbers are plotted in Figures 4-7 (please see Appendix B).
Table 1:
Method
Dataset
#Bits
MMLU (0-shot)
MMLU (5-shot)
Hums.
STEM Social Other
Avg.
Hums.
STEM
Social Other
Avg.
LLaMA-7B
16
32,4
26.6
31.4
37.2
32.1
33.3
29.8
37.8
38.0
34,6
QLoRA
Alpaca
4+16,
38.1
31.1
41.6
46.9
39.4
36.7
31.9
42.0
44.3
38.4
OLoRA w/ GPTQ
Alpaca
35.7
30.9
38.0
44.0
37.1
33.8
31.3
37.4
42.2
36.0
PEQA
Alpaca
34.9
28.9
37.5
40.1
34.8
QA-LoRA
A Alpaca
36.9
31.4
40.3
44.9
38.3
36.6
32.4
44.8
44.9
39.4
OLoRA GPTQ
A Alpaca
31.5
28.9
31.8
36.8
32.2
31.6
30.1
35.6
39.8
34.0
QA-LORA
A Alpaca
36.0
34.1
42.0
42.3
38.3
35.6
30.5
41.5
42.7
37.4
OLORA w/ GPTQ
A ppaa
24.1
22.1
22.5
23.7
23.2
23.4
26.2
26.4
28.4
25.8
QA-LoRA
Alpaca
26.4
25.5
25.6
28.7
26.5
27.3
26.1
26.1
30.3
27.5
QL
PLAN v2 44161
40.9
32.5
47.8
49.5
42.6
"4..4
35.0
49.8
"22.0
"4.33
OLoRA w/ GPTQ
FLAN v2
39.7
32.5
46.4
48.1
41.6
36.5
33.7
46.9
50.3
41.4
QA-LORA
FLAN v2
44.0
35.3
$2.3
52.6
45.9
43.9
38.0
54.3
53.0
47.0
OLORA w/ GPTQ
FLAN V2
36.7
30.2
38.4
40.1
36.5
32.2
31.7
42.7
42.8
36.9
OA-LORA
FLAN V2
41.4
35.1
52.0
50.2
44.4
41.3
36.0
52.8
50.2
44.7
OLORA w/ GPTQ
FLAN V2
2
24.1
22.5
22.3
23.8
23.31
23.9
25.3
26.2
25.3
25.0
QA-LoRA
FLAN v2
34.1
30.0
37.2
39.8
35.2
31.8
38.1
34.5
38.5
33.2
LLaMA-13B
40.6
36.7
48.9
48.0
43.3
44.0
35.9
53.2
52.9
OLORA
Alpaca
4+
45.2
38.3
55.0
54.6
48.1
46.0
37.3
55.8
55.1
48.4
OLORA w/ GPTQ
Alpaca
4"'""""441 44.7
38.0
54.4
34.0
47.6
45.4
37.4
35.9
54.3
48.0
PEQA
A Alpaca
43.0
37.7
53.6
49.0
45.0
QA-LORA
Alpaca
44.3
38.0
55.1
55.5
47.9
48.4
38.3
54.9
55.2
49.2
OLORA GPTQ
ppca
43.5
36.2
52.3
52.6
45.9
43.6
36.1
53.0
52.7
46.1
QA-LORA
paca
43.9
37.3
53.1
54.3
46.9
44.3
38.8
53.4
53.8
47.3
OLORA w/ GPTQ
Alpaca
27.7
27.6
31.8
29.7
29.0
29.0
27.1
33.4
34.8
30.9
QA-LoRA
Alpaca
35.7
33.3
40.9
42.0
37.8
35.6
30.6
39.9
41.7
36,9
QLORA
FLAN v2 4+16 16'
48.0
39.2
58.2
56.7
50.3
49.9
40.1
60.2
57.9
51.9
OLORA W/ GPTO
FLAN v2
47..6
39.6
$9.6
56.0
50.0
49'4
40.9
59.7
51.6
QA-LoRA
FLAN v2
47.7
41.4
59.6
57.2
$1.1
50.0
41.5
60.5
58.4
52.4
OLORA w/ GPTQ
FLAN v2
46.6
37.9
55.9
55.7
48.9
46.5
38.2
57.2
56.1
49.3
QA-LoRA
FLAN V2
47.4
39.4
57.7
56.0
49.9
49.3
40.0
60.0
57.5
$1.5
OLORA w/ GPTQ
FLAN V2
36.2
30.3
40.8
44.1
37.8
36.6
32.0
43.8
44.2
38,9
QA-LoRA
FLAN V2
40.8
36.4
39.3
50.1
43.9
40.9
36.1
50.7
46.7
44.1
LLLMA-33B
16
51.0
42.7
63.3
60.4
54.1
56.2
45.9
67.1
63.9
58.2
OLoRA
Alpaca
4+16
52.2
44.9
64.3
61.8
55.3
35.4
46.0
66.4
63.6
37.7
OLoRA w/ GPTQ
Alpaca
51.7
44.7
63.4
61.0
54.9
53.9
46.6
66.3
62.9
57.1
OA-LORA
Alpaca
51.6
44
65.0
55.4
55.8
46.4
64.0
58.1
OLORA w/ GPTQ
Alpaca
1
44.5
43.3
63.1
61.0
53.8
53.3
45.0
64.1
61.4
55.8
LARLORA
Alpaca
50.6
44.6
64.0
54.7
45.8
62.6
OLoRA w/ GPTQ
Alpaca
32.0
31.6
35.8
32.8
32.9
37.5
34.9
45.3
44.9
40.4
QA-LORA
Alpaca
38.4
38.2
50.7
49.7
43.6
44.2
38.8
52.3
OLORA
FLAN v2
4+16
56.3
46.5
68.6
64.6
58.8
57.2
48.6
69.8
65.2
60.0
OLORA W/ GPTT
FLAN V2
54.9
46.4
68.2
63.6
58.0
37.4
48.6
69.2
64.9
59.8
QA-L
FLAN v2
$4.2
47.0
69.7
65.5
58.7
57.9
48.8
71.0
65.5
60.6
OLORA w/ GPTQ
FLAN v2
54.0
44.3
65.8
62.7
56.5
55.7
47.4
67.9
64.0
58.5
OA-LORA
FLAN v2
S3.1
45.0
66.9
63.0
56.7
56.8
46.9
68.9
63.7
58.9
OLORA W/ GPTQ
FLAN V2
37.9
35.0
47.6
42.9
40.6
42.8
37.0
54.3
51.
46.1
QA-LORA
FLAN V2
49.4
40.4
59.8
56.5
51.4
49.6
42.7
60.7
57.8
52.4
I.LMA-65B
16
56.4
45.2
68.0
64.1
58.3
61.4
51.9
73.6
67.6
63.4
QLoRA
Alpaca
4+16
55.5
49.3
70.4
66.9
60.1
10.11'60.3
60.3
52.7
72.9
63.7
63.7
OLORA w/ GPTQ
Alpaca
54.8
48.9
69.8
66.1
59.4
60.4
52.5
73.0
67.2
63.0
QA-LORA
Alpaca
57.1
48.2
70.7
64.9
60.0
60.8
50.5
72.5
66.7
62.5
OLORA w/ GPTQ
Alpaca
S7.4
47.9
67.2
65.1
59.3
59.6
50.0
70.6
66.1
61.4
OA-LORA
Alpaca
57.6
48.4
69.3
65.4
60.0
59.3
49.6
71.9
66.0
61.5
OLORA w/ GPTQ
Alpaca
43.9
38.0
42.6
51.1
46.2
47.3
40.8
58.9
57.0
$0.7
QA-LORA
Alpaca
48.6
42.5
60.7
58.6
52.2
51.3
43.4
63.4
60.7
54.4
QLORA
FLAN v2
4616
58.8
52.5
74.0
67.4
62.8
59.8
52.9
75.0
69.6
63.9
QLORA GPTQ
FLAN v2
57.8
51.9
73.5
667.8
62.3
59.
52..5
75.0
"
63.5
OA-LORA
FLAN v2
64.1
52.6
74.8
69.1
65.1
57.6
51.1
73.9
67.4
62.1
OLORA GPTQ
FLAN v2
58.5
50.2
71.5
66.9
61.5
59.9
51.7
73.4
67.9
63.0
QA-LORA
FLAN v2
57.5
49.5
72.4
66.9
61.2
61.7
51.1
73.8
68.4
63.6
OLORA w/ GPTQ
FLAN v2
47.9
43.1
56.0
51.4
$2.6
43,8
62.8
58.5
54.3
QA-LORA
FLAN v2
55,99
44.6
65.6
63.4
57.1
55.5
46.8
67.3
63.2
58.0

---

Published as a conference paper at ICLR 2024
4 EXPERIMENTS
4.1 SETTINGS
Foundation models. We establish QA-LoRA upon the LLaMA (Touvron et al., 2023a) and LLaMa2 (Touvron et al., 2023b) families. In particular, we fine-tune the 7B, 13B, 33B, and 65B models of LLaMA and the 7B and 13B models of LLaMA2.
Evaluation metries. Following QLoRA (Dettmers et al., 2023a), we evaluate both the zero- shot and few-shot performance of the LLMs on Massively Multitask Language Understanding (MMLU) benchmark (Hendrycks et al., 2021). It consists of 57 language tasks including human- ities, STEM, social science, etc. We use the official MMLU evaluation script and prompts". We further assess the zero-shot common sense reasoning ability on tasks covering HellaSwag (Zellers et al., 2019), PIQA (Bisk et al., 2020), WinoGrande (Sakaguchi et al., 2019), ARC (Clark et al., 2018), BoolQ (Clark et al.. 2019), and OpenBookQA (Mihaylov et al., 2018). We adopt Im-eval- harness (Gao et al., 2021) to produce the Common Sense QA results.
Quantization. We adopt GPTQ (Frantar et al., 2023) in the quantization step, and our approach is open to other PTQ methods such as (Lin ct al., 2023; Dettmers et al., 2023b). We use the same settings to quantize the QLoRA fine-tuned models and pre-trained LLaMA models. In the main experiments, we conduct group-wis asymmetric quantization (with a group size of 32). We set the act-order variable to be false and the sruu-sequentiaa variable to be true.
Datasets and training details. We choose Alpaca (Taori et al., 2023) and FLAN v2 (Longpre et al., 2023) as our fine-ture -teninnnng datasets. Alpaca contains 52K instruction-oollwwvii
data generated from text-davinci-003 (GPT 3.5) (Wang et al., 2022). FLAN v2 is collection of 1 1,836 tasks combining the mixture with CoT, Muffin, TO-SF, and NIV2. To save the tuning cost, we randomly sample a 320K subset from the FLAN v2 collection. Following QLoRA (Dettmers et al., 2023a), we use paged A optimizer, a maximum gradient norm of 0.3, and a batch size of 16 in the tuning period. We choose the constant learning rate schedule and set the learning rate to be 2 x 10-5 for the 7B and 13B models and x 10-5 for the 33B and 65B models. The number of fine- fine-tuning steps is 10K for Alpaca and 20K for FLAN v2. All experiments are conducted on Tesla V100 GPUs. We use one GPU for the 7B, 13B, and 33B models and two GPUs for the 65B models.
4.2 MAIN RESULTS AND EFFICIENCY
Comparison against recent competitors on LLaMA for MMLU. We first apply QA-LoRA to fine-tune the LLaMA models for MMLU. Table 1 summarizes the results with respect to different model sizes, fine-tuning datasets, and bit widths. Besides the base LLaMA models, we also compare QA-LoRA against QLoRA (Dettmers al., 2023a), the most related work, and PEQA (Kim al., 2023), a recent quantization method that does not use LoRA. We report both the original QLORA (the inference stage involves FP16 computation) and the variant after GPTQ (for fair comparison). QA-LoRA consistently outperforms both competitors (QLoRA w/ GPTQ and PEQA) in either 0- shot and 5-shot accuracy. The advantage is more significant when the model size small (e.g., 7B and 13B) or the bit width is small (e.g., INT3 or even INT2 is used), demonstrating that QA- LoRA is a strong solution in the scenarios that require computational efficiency. In some cases, the INT4 version of QA-LoRA performs even better than the original version of QLoRA meanwhile the inference speed is much faster (see the next paragraph). We further demonstrate some examples of QA-LoRA in Appendix A, where
can see the qualitative comparison and QA- beyond QLoRA w/ GPTQ QA-LoRA mainly benefits from the quantization aware adaptation; otherwise, the post-training quantization will not be compensated, resulting in unstable results.
Application to large models. As shown in Table 1, in large models (e.g., 33B and 65B), QA- LoRA still achieves improvement over the baseline, QLoRA with GPTQ. Upon these results, discuss the need for QA-LORA in large models in the following aspects. (1) QA-LoRA reduces the computational costs of fine-tuning large models, e.g., using QA-LoRA, only and 2 V100 GPUs are needed for fine-tuning the 33B and 65B models. (2) When larger models are used, there can be increasing needs for low-bit (e.g., INT3 and INT2) quantization, especially when the large models are to be deployed to edge devices. QA-LoRA shows significant advantages in such scenarios.
thttps:/ h/tpps:////r //g thub. htt../c////endccs/tteeeet

---

Published as a conference paper at ICLR 2024
Table 2: The numbers of learnable parameters and time costs of QLoRA and QA-LoRA during the fine-tuning stage. All results are reported on Alpaca with one Tesla-VI00 GPU (the 65B model uses Iwo chips). The number of fine-tuning steps is 10K.
LLaMA-7B
LLaMA-13B
LLaMA-33B
LLaMA-65B
Method
#Params
Time
#Params
Time
#Params
Timea)
#Params
Timeg)
OLoAA
160M
40.0
250M
73.1
488M
148.6
800M
284.5
QA-LoRA
89M
21.5
140M
29.5
272M
51.2
447M
100.5
QA-LoRA
178M
21.8
280M
30.1
544M
52.2
894M
103.5
Table 3: 0-shot commonsense QA accuracy (%) with respect different quantization bit widths.
Method
#Bits
HellaSwag
PIQA
WinoGrande
ARC-e
ARC-c
BoolQ
OBQA
Avg.
LLaMA-7B
16
56.3
78.2
67.1
67.3
38.2
72.9
28.4
58.3
QLoRA
4+16
61.8
78.1
68.4
75.8
73.7
32.8
62.0
LLaMA-7B GPTQ
54.5
76.5
66.9
66.1
36.9
70.9
27.4
57.0
QoRAA w/ GPTQ
57.4
77.6
66.2
70.9
41.8
73.5
31.2
59.8
QA-LoRA
4
58.6
78.0
66.9
71.2
43.9
79.9
34.0
61.8
QLORA w/ GPTQ
52.2
75.2
64.1
65.8
37.2
70.4
27.2
56.0
QA-LoRA
57.6
76.2
66.5
70.2
43.1
76.3
30.6
60.1
QLaRA w/ GPTQ
31.9
58.2
52.4
32.3
20.7
60.6
14.6
38.7
QA-LoRA
2
49.8
70.2
58.5
55.4
33.9
73.7
32.8
53.7
Table 4: 0-shot and 5-shot MMLUU accuracy (%) based on the LLaMA2 model family.
MMLU (0-shot)
MMLU U (5-shot)
Method
Data
#Bits
Hums.
STEM Social Other Avg.
Hums.
STEM
Social Other Avg.
LLaMA2-7B QA-LoRA
16
38.9
32.9
46.6
44.9
40.7
43.0
36.4
51.4
52.2
45.5
Alpaca
4
41.1
35.4
50.2
50.1
43.9
42.1
34.4
49.1
50.3
43.9
QA-LoRA
FLAN v2
4
47.4
39.5
58.9
57.3
50.5
48.4
41.4
59.4
58.6
51.7
LLaMA2-13B
16
48.1
42.7
60.5
59.5
52.31
53.3
44.1
63.3
61.0
55.3
OA-LoRA
Alpaca
48.2
41.7
60.4
58.7
51.91
48.0
43.0
59.7
57.4
51.7
QA-LoRA
FLAN
4
50.7
44.1
63.8
62.0
54.8
52.9
44.8
65.9
64.0
56.6
The efficiency of QA-LoRA. A clear advantage of QA-LoRA lies in its computational efficiency. Table 2 compares QA-LoRA to QLoRA in terms of the learnable parameters and training time during the fine-tuning stage.
The reason behind the fewer amounts of parameters, compared to QLoRA, lies the reduction of the dimensionality of A. Compared LoRA and QLoRA where A has Din D parameters, QA-LoRA reduces the number to L x Dint where L is the group size and I. & Din- This reduces the number of parameters in QA-LoRA by around 1/2 (originally, A and B have similar numbers of parameters). QA-LoRA achieves higher fine-tuning accuracy with fewer parameters.
Regarding the time cost of fine-tuning, amount of LoRA parameters is much smaller than that of the LLM itself (e.g., 89M or 160M vs. 7B). To verify this point, we double Din which also doubles the number of parameters, surpassing that of QLoRA. but QA-LoRA is still much faster than QLoRA (see the table below). The signif- icant advantage of QA-LoRA in training time mainly comes from the use of INT4 quantization. Compared to NF4 quantization used by QLoRA, INT4 operators have been optimized by CUDA and are much faster in execution. Additionally, during the inference stage, QA-LoRA also more than 50% faster than QLoRA because the fine-tuned model (after weight integration) is still in INT4, unlike QLoRA that converts back to FP16.
not largely impacted by the parameters because the
Commonsense QA results. We also evaluate QA-LORA for 0-shot commonsense QA based on LLaMA-7B. Results are summarized in Table 3. Similar to the MMLU results, the 4-bit QA-LoRA is comparable with the mixed-preciiion QLoRA and outperforms the post-quantized QLoRA by an average of 2.0%. The advantage becomes more significant in low-bit scenarios, e.g., the 2-bit OA-LoRA reports remarkable accuracy gain of 15.0% over the 2-bit post-quantized QLoRA.
8

---

Published as a conference paper at ICLR 2024
On LLaMA2 models. We further validate the effectiveness of method on LLaMA2 (Touvron et al., 2023b). As shown in Table 4, we fine-tune the 7B and 13B models of LLaMA2 and test them on MMLU. Compared the original FP16 models, the INT4 models fine-tuned with FLAN v2 are consistently better, while those with Alpaca report slightly lower accuracy. These experiments validate that QA-LoRA is generalized other pre-trained model families.
Table 5: 0-shot and 5-shot MMLU accuracy (%) on different fine-tuning datasets.
Self-instruct
Longform
Chip2
Alpaca
Flan v2
Base Model
Method
#Bits
O-shot 5-shol O-shot 5-shot 0-shol 5-shot
Osshol 5-shol
O-shot
5-shol
OLoRA QLoRA w/ GPTQ
4+16
36.4
32.1
34.5
38.8
44.5
LLaMA-7B
35.4
29.3
33.6
36.0
41.4
QA-LORA
4
32.5
34.4
29.3
33.6
30.4
32.2
38.3
39.4
45,9
47.0
OLoRA LLaMA-13B 13B QLoRA w/ GPTQ
4+16
39.0
43.2
41.6
48.4
51.9
38.4
42.8
41.3
48.0
51.7
QA-LoRA
4
44.4
46.1
39.9
43.3
42.4
45.8
47.9
49.2
51.1
52.4
4.3 ABLATIVE STUDIES
Impact of the quantization group size. We investigate different settings of L, the hyper-paramette that controls the numbers parameters for both quantization and low-rank, adaptation. Results are reported in Table 6 (see Appendix B), where group size (i.e., Din/ L is displayed instead of L). Re- call that larger T. (corresponding to smaller group size) implies a larger number of parameters, Le., smaller quantization loss, and a larger number of adaptation parameters. Meanwhile, it also requires a larger number of storage and computation, though negligible as long as L > One can observe that larger L (e.g., group size is 32) often leads to higher accuracy, and the advantage be- comes more significant when the quantization bit width small, implying that a larger quantization loss needs to be compensated by larger number of parameters.
Impact of Dint We diagnose the performance with respect to Dan in Table (see Appendix B). We find that the MMLU accuracy is not largely impacted by the value of Dint unless it is too small.
Impact of fine-tuning datasets. We also eval- uate QA-LoRA instruct (Wang et al., 2022), Longform (Köksal al., 2023), and Chip2 (LAION, 2023). Results are summarized in Table 5. Compared to Alpaca and FLAN v2, these datasets are relatively small, and thus the fine-tuned models report bit weaker ac- curacy on MMLUU Note that, with I.L.MA-13B as the foundation model, QA-LoRA consistently out- performs QLoRA with mixed precision, meanwhile being much faster in the inference stage.
more datasets such as Self-
55
QA-LORA (2-bit)
50
QA-L (4-bit)
45
40
35
30
Impact of the size fine-tuning datasets. Lastly, evaluate QA-LoRA on different subsets of FLAN v2. The dataset size varies from 160K, 240K, 320K, 400K, and 480K. LLaMA-7B is used the founda- tion model. As shown in Figure 3, low-bit quanti- zation asks for more data, yet 320K is sufficient for both the INT2 and INT4 variants of QA-LoRA.
25
160k
240k
320k
100k
480k
Figure 3: 5-shot MMLU accuracy (%) of QA-LoRA when the LLaMA-7B model is fine-tuned on subsets of FLAN v2 with dif- ferent sizes.
5 CONCLUSION
In this paper, we propose QA-LORA as an efficient method that introduces quantization awareness into the low-rank adaptation of LLMs. At the core of QA-LoRA lies the group-wist wise operations for both quantization and low-rank adaptation, and the key insight comes from balancing the numbers of parameters of both sides. QA-LoRA is easily implemented, generalized across various foundation models and language understanding tasks, and computationally efficient in both fine-tuning and inference stages. Extensive experiments LLaMA validate the effectiveness of QA-LoRA.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
This paper is built upon pre-trained large language models (e.g. LLaMA and LLaMA2) and existing datasets for instruct fine-tuning (e-g., Alpaca and FLAN v2). We do not introduce any new data and thus do not involve human annotation. This paper has no additional ethical concerns beyond large corpus of research in LLMs.
REFERENCES
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty-Fourth AAAI Conference on Artificial Intelligence, 2020.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Bce Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2.03.1 12712, 2023.
Yupeng Chang, Xu Wang. Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang. Yidong Wang, et al. A survey on evaluation of large language models. arXiv preprint arXiv:2307.033 003000.031999 109, 2023.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 2022.
Christopher Clark, Kenton Lee, Ming-Was Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint (t0::/905.0044. 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the al2 reasoning challenge. arXiv preprint arXiv: 10X3::880330557. 2018
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. LLM.int80) 8-bit matrix multiplication for transformers at scale. Advances in Neural Information Processing Systems, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized 1lms. arXiv preprint arXiv:2305.14314 2023a.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized repre- sentation for near-lossless I1m weight compression. arXiv preprint ar arXiv:2306.03072 2023b.
Jacob Devlin, Ming-Was Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep
bidirectional transformers for language understanding. In North American Chapter of the Associ- ation Computational Linguistics, 2019. URL https://///..c...tt..ttt /api..pp./// semant /
II:
Steven K Esser, Jeffrey L. McKinstry, Deepika Bablani, Rathinakumar Appuswamy, and Dharmen- dra Modha. Learned step size quantization. arXiv preprint arXiv:1902.08153. 2019.
Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. International Conference Learning Representations, 2023.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training
compression for generative pretrained transformers. In International Conference on Learning Representations, 2023.
10

---

Published as a conference paper at ICLR 2024
Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, et al. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 2021.
Muhammad Usman Hadi, Rizwan Qureshi, Abbas Shah, Muhammad Irfan, Anas Zafar, Muham- mad Bilal Shaikh, Naveed Akhtar, Jia Wu, Seyedali Mirjalili, et al. Large language models: A comprehensive survey of its applications, challenges, limitations, and future prospects. TechRxiv, 2023.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpattrck, and Graham Neubig. Towards unified view of parameter-efficies transfer learning. In International Conference on Learning Representations, 2022.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-eeffcient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2021.
Zhiqiang Hu, Yihuai Lan, Lei Wang, Wanyu Xu, Ee-Peng Lim, Roy Ka-Wei Lee, Lidong Bing. and Soujanya Poria. LIm-adapters: An adapter family for parameter-efficien fine-tuning of large language models. arXiv preprint arXiv:2304.00 01933, 2023.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory- fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint arXiv:2305. 14152, 2023.
Abdullatif Köksal, Timo Schick, Anna Korhonen, and Hinrich Schütze. Longform: Optimizing struction tuning for long text generation with corpus extraction. arXiv preprint 08840300000 2023.
LAION Open-iistruuctoongeereee
dataset.
httpp::/ggtttb..cmmm htw.ttthhu..ccmm/////0.0. AI/
Open- InwiInnstttttn wtwssttnetionnnneeeeaaaaeeee
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv
preprint arXiv:2101.00// 2021.
Lin, Jiaming Tang, Haotian Tang, Shang Yang, Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Ilm compression and acceleration. arXiv preprint arXiv:2306.00978, 2023.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. arXiv preprint 2rX 2021.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Lim-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. 77888, 2023.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Tay, Denny Zhou, Quoc V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint arXiv:2301. 1. 13688, 2023.
Xinyin Ma. Gongfan Fang, and Xinchao Wang. LIm-pruner: On the structural pruning of large language models. arXiv preprint arXiv:2305.11622 2023
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, and Sayak Paul. Peft: State-of-thh-ant parameter-efficient fine-tuning methods. https: //tpt/biihh com/ huggingface/ppe 2022.
11

---

Published as a conference paper at ICLR 2024
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? a new dataset for open book question answering. In EMNLP, 2018.
OpenAl. Gpt-4 technical report. arXiv preprint arXiv:2303.0877,, 2023.
Gunho Park, Baeseong Park, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. nuqmm: Quantized matmul for efficient inference of large-scale generative language models. arXiv preprint arXiv:2206.095577 2022.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. arXiv preprint arXiv:1907.1064 2019.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: 176b- parameter open-access multilingual language model. arXiv preprint arXiv:22 1.021100100000 2022.
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815-8822 2020.
Mingjie Sun, Zhuang Liu, Anna Bair, and Zico Kolter. simple and effective pruning approach for large language models. arXiv preprint arXiv:2306. 1695, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-folloiing llama model. 2023.
https:/ ////gttthht github..... coo/tatsu--ae
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13977 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2tt.:2330000288. 2023b.
Mojtaba Valipour, Mehdi Rezagholizadch, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558, 2022.
Qichao Wang, Tian Bian, Yian Yin, Tingyang Xu, Hong Cheng, Helen M Meng, Zibin Zheng, Liang Chen, and Bingzhe Wu. Language agents for detecting implicit stereotypes in text-to-image models at scale. arXiv preprint arXiv:2310.1177. 11778 2023.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212. 2tt:::22122.000000 10560, 2022.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yo- gatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. In Transactions on Machine Learning Research, 2022a.
Xiuying Wei, Yunchen Zhang, Xiangguo Zhang, Ruihao Gong, Shanghang Zhang, Qi Zhang, Feng- wei Yu, and Xianglong Liu. Outlier suppression: Pushing the limit of low-bit transformer lan- guage models. In Advances in Neural Information Processing Systems, 2022b.
Lilian Weng. Large transformer model inference optimization. LiI' Jan 2023. URL https: intioopiiiiiiiiiii imizatiiiooo.i//i/o//mom
//lilianweng ottpaa...o/pptttta222--------eee
Mitchell Wortsman, Tim Dettmers, Luke Zettlemoyer, Ari Morcos, Ali Farhadi, and Ludwig Schmidt. Stable and tow-iieciio training for large-scale vision- language models. arXiv preprint arXiv:2304. 13013, 2023.
12

---

Published as a conference paper at ICLR 2024
Xiaoxia Wu, Zhewei Yao, and Yuxiong He. Zeroquant-fp: A leap forward in Ilms post-training w4a8 quantization using floating-poin point formats. arXiv preprint arXiv:2307.09780 2023.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, 2023.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. In Advances in Neural Information Processing Systems, 2022.
Junchi Yu, Ran He, and Zhitao Ying. Thought propagation: An analogical approach to complex reasoning with large language models. In The Twelfth International Conference on Learning Representations, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? CoRR, (bss/1905007830, 2019. URL httpttttpaxii...ooo abs/ 905. 07830.
Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. Glm-130b: An open bilingual pre-trained model. International Conference on Learning Representations, 2023.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christo- pher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettle- moyer. Opt: Open pre-trained transformer language models, 2022. URL https://arxx: org/abs 2205.222222 01068.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du Chen Yang. Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian Yun Nie, and Ji- Rong Wen. A survey of large language models. arXiv preprint arXiv: 18223, 2023a. URL http::/aat http///avvvv/////////eee//..ev.a org/ abs/ 23032333322 0.0.0..1.2....
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. A survey of large language models. arXiv preprint arXvv:2303. 18223, 2023b.
Aojun Zhou, Ke Wang. Zimu Lu, Weikang Shi, Sichun Luo, Zipeng Qin, Shaoqing Lu, Anya Jia, Linqi Song. Mingjee Zhan, et al. Solving challenging math word problems using gpt-4 code interpreter with code-based self-verification. arXiv preprint arXiv:2308.07921 2023.
Xunyu Zhu, Jian Li, Yong Liu, Can Ma, and Weiping Wang. survey on model compression for large language models. arXiv preprint 230.::23008.033333 2023.
13

---

Published as a conference paper at ICLR 2024
A QUALITATIVE STUDIES
In this section, we show a few examples of dialog. We compare our method, QA-LoRA, to the direct competitor, QLoRA, under different quantization bit widths. All QLoRA models are post- processed with GPTQ as described in the main text. We highlight inaccurate answers in blue and totally unacceptable answers in red.
Overall, QLoRA with GPTQ shows unstable behaviors across different cases. For example, in the second case, it crashes in 4-bit and 2-bit quantization but works well in 3-bit quantization. This mainly owes to the uncontrollable quantization loss in post-processigg and such loss cannot be amended by any of the subsequent stages.
Instruction: Suggest five ways to budget your expenses.
QLoRA (4-bit): I. Write down all of your current expenses. Estimate your future expenses.
3. Cul down non-essential spending. 4. Rale your expenses according their importance.
5. Categorize your expenses to make it easier to.
Ours (4-bit): Plan a monthly budget and stick to it. 2. Track your spending. 3. Narrow down your wants and priorities. 4. Cut expenses by shopping around. 5. Eat at home as much
as possible.
QLoRA (3-bit): Gather and document your financial information (income, expenses, and assets) 2. Estimate your cash flow (income, expenses, and assets) 3. Allocate your cash flow to the needed/wwanted category. Monitor your spending
Ours (3-bit): To budget expenses, should: (I) keep track of spending; (2) have spending plan; (3) set weekly goals; (4) only use cash; and (5) try to overspend one area to cover an- other.
QLoRA (2-bit): the maximum number of times 2026 #03289-012-052112 0222500032815-0455 2291-253222.
03289-
Ours (2-bit): Plan monthly budget and stick it. 2. Track your spending. 3. Narrow down your wants and priorities. 4. Cut expenses by shopping around. 5. Eat at home as much
as possible.
Instruction: How do you respond to negative comment on social media? QLoRA (4-bit): I'm sorry you feel that way.
Ours (4-bit): Negative comments social media can be hard to respond to, but I would first try to understand where the person is coming from and why they are saying what they are saying. I would then strive reply in way that does not further escalate the discussion, but instead responds with a mature.
OLoRA (3-bit): I would first thank the person for taking the time to share their thoughts. would then say that I value their opinion and that I here to listen and learn. would then to explain why I feel the way I do and invite them to an open discussion. Ours (3-bit): I response to negative comment on social media by thanking the person for their feedback, clarifying if they are basing their comment off of a bad experience, and inviting them to contact me directly so we can work through any issues or concerns they may have.
QLoRA (2-bit): Demostimeta for the Purts of the Their work of for the parts of Their work of For the parts of3.4.2004 G Mach Rack 2002-200 2004 2.
Ours (2-bit): Negative comments on social media can hard to respond to, but I would first try to understand where the person is coming from and why they are saying what they are saying. I would then strive to reply in a way that does not further escalate the discussion, but instead responds with mature.
Instruction: Categorize the following fruits according to color. Input: Apple, Banana, Avo- cado, Peach.
QLoRA (4-bit): of the same color: Red (Apple, Banana), Green (Avocado, Peach) Other: Yel- low (Peach).
14

---

Published as a conference paper at ICLR 2024
Ours Appll Red. Banana Yellow. Avocado Green. Peach Pink.
QLoRA (3-bit): is yellow is green is black is white.
Ours (3-bit): Apple (Red)- Banana (Yellow)- Avocado (Green)- Peach (Pink).
QLoRA (2-bit); Apple, mash up, and avocado. Ours (2-bit); Apple Red. Banana Yellow. Avocado Green, Peach Pink.
Instruction:
Generate an HTML page that contains the text \ "Welcome to my page QLoRA (4-bit): >n'n<html>\n <head>>> in <title>Welcome Welcome to my page</tirle>> (1 </head> \ <body>> n <hI>Welcomee to my page</hl>\n </body> (nnoodyy>n</mmttt >n<//ml>>>>> An.
Ours (4-bit): Welcome to my page. <n<him>>>>>>> /n <n<oody> <n<p> ><p>>elcooee Welcome to my
page. </p> "nn>>////oyy>>>> /r////od/>>>////mmmt>> /n//hmm>>>n
QLORA (3-bit): <himl>> in <head>>n n <title>My Page</title>> n <Shead>> in <body>> i </p>>/n <p>Welcome Welcome to my p/ge!</p>>y n </body>\n</hmmll An/hhml>>>>>>>//>>//>>///>>>>>> 'n package orc.michael. Ours (3-bit): <HTML>> nn <P> Welcome to my page<//>>> <P>Lorem Ipsum is simply An Note that this response may also contain other
placeholder text. eex.<//>>> /<</HML>>>>>>> <ot,</P>></HTML>>>>>> elements such as stylesheets and script, if they were defined in the task.
QLORA (2-bit): The page has the text \ "Welcome to my page' as it. n228. A reint: a 28/313473-020-520 The m3114733-2202
Ours (2-bit): Welcome t0 my page. aoge.'n<html>>> hecimm/>>>noooo///ppp2>0 Welcome to my
page. page.</p>>> http:>/w//oyy>/////ee
Instruction: Create dialogue between two people having conversation about going to the beach.
QLoRA (4-bit): of course, I'm in! what time were you thinking of meeting up? was thinking of going around noon. sounds good, I'll see you then! Ours (4-bit): (beach conversations A and B). B: Going to the beach sounds like a blast! A: It does, isn't it? B: Yep! A: so what are we doing? B: Planning full itinerary.
QLoRA (3-bit): "Person 1: "I'm excited to go the beach! The sun, sand, and water all so soothing. can't I wait to get there and dive in!" Person 2: "Me too! I can't wait to experience that feeling of being".
Ours (3-bit): "So which beach should we go Fiji or Bora Bora?" "Fiji has more coral reefs, Bora Bora has more beaches, right?"'mn "Yeah, Bora Bora also has the world's most"
QLoRA (2-bit): Such as user 31 4 0-5 Gigs Sector (22. (22.12.2018) [P]/f/ Beyond there is lot of and.
Ours (2-bit): (beach conversations A and B). B: Going to the beach sounds like a blast! A: It does, isn'l II't it? B: Yep! A: so what are we doing? B: Planning full itinerary.
B ADDITIONAL EXPERIMENTAL RESULTS
We plot some important results in Table Figures 4-7 to ease the readers to compare the our model with the baseline, QLoRA. Note that Figure is identical to Figure 1.
We also provide the diagnosis on the L (group size) and Dine (the hidden dimensionality of adapta- tion) hyper-parameters in Tables 6 and 7.
15

---

Published as a conference paper at ICLR 2024
QU-MMA clab 16-10)
OF oRA dl-res
QA--ual A (4-N1)
Q1 ORA (168))
QA- ARA (3-N11
Or ARA
QA-ToRA (2-0))
GU.55 aC.
4811/7.11
34
41,6
45.9
441
39.4
30
26.3
10.0
11.3
(a) QLoRA vs QA-LoRAin 4-bil
(6) QLORA QA-LoRA 3-bit
(c) QLuRAvs QA-LoRA QA-LLRRAAA
2-bil
Figure 4: The comparison of O-shot MMLU accuracy (%) with different quantization bit widths based on the LLaMA model family on the Alpaca dataset. Full results are provided Table 1.
QLoRA (446) t640
QUaRkA (4-kz)
QA-LOAA (803))
QUaRA bii
QA-ToRA (3-03)
QLaRA (24k))
QA-LaRA (200))
60
60
60
25.0
50
(1s 2012
40
35.9
31A
30
10
10
20
(5) QLoRA vs La
(b) QLORA A QA- 3-bit
(c) QLoRA vs QA-LoRA 2-bit
Figure 5: The comparison of 5-shot MMLU accuracy (%) with different quantization bit widths based on the LLaMA model family on the Alpaca dataset. Full results are provided in Table 1.
Qf aRA cobia 16h1))
60
41.54122
60
51A
51A
50
31.1
40.0
50
37.4
40
40
10
LL-744-78
LLAMa-dd
LEaMa-18
LLaMt--30 LLaMA-65
11aM6-00
(4) QLoRA QA-LoRA 4-bit
(b) QLoRA QA-LORA 3-bit
(c) QLoRA LA-LRAA in
Figure 6: The comparison of 0-shot MMLU accuracy (%) with different quantization bit widths based on the LLaMA model family on the FLAN v2 dataset. Full results are provided in Table 1.
QLARA (4:6) 16380
QLARA (4-
QA-tuiB (4-ii)
OLaRA (34b2)
QA-LaRA (1-b1))
QLARA (2460)
QA-TaRA (2-kr)
$3.4
54.
(52.4
30
31.5
46.1
58.9
30
30
(a) QLoRA Qn-LoRA 4-bil
(b) QLoRA QA-LoRA
(c) QLeRA QA-LoRA
Figure 7: The comparison of 5-shot MMLU accuracy (%) with different quantization bit widths based on the LLaMA model family on the FLAN v2 dataset. Full results are provided in Table 1.
16

---

Published as a conference paper at ICLR 2024
Table 6: O-shot and 5-shot MMLU accuracy (%) with respect different group settings.
MMLU (0-shot)
MMLU (5-shot)
Hums.
STEM Social Other
Avg. Hums.
STEM Social Other
Avg.
Base Model
Group Size #Bits
(1)
(t)
(t)
(f)
(?)
(1)
(t)
(1)
(f)
(1)
128
37.3
31.8
39.3
43.7
38.0,
36.5
32.1
41.7
44.0
38.4
64
4
37.5
30.6
41.3
45.4
38.61
36.5
32.6
43.4
45.0
39.1
32
4
38.1
31.1
41.6
46.9
39.4"
36.1
31.9
42.0
44.5
38.4
LLaMA-7B
128
2
24.0
26.7
24.8
25.2
25.01
25.0
29.0
27.9
26.1
26.7
64
2
25.1
26.9
24.7
27.0
25.8
25.0
27.2
25.2
27.3
26.0
32
2
26.4
25.5
25.6
28.7
26.5,
27.3
26.1
26.1
30.3
27.5
128
43.4
39.6
55.5
53.9
47.61
46.5
38.0
55.8
54.5
48.6
43.4
39.3
55.8
53.6
47.6
47.8
39.3
55.7
54.8
49.3
44.3
38.0
55.1
55.5
47.9
48.4
38.3
54.9
55.2
49.2
LLaMA- -3B
128
28.5
28.4
30.6
29.8
29.2
29.2
30.6
32.8
32.4
31.0
30.7
33.7
32.3
37.0
38.3
34.3
2
35.7
33.3
40.9
42.0
37.8
35.6
30.6
39.9
41.7
36.9
Table 7: 0-shot and 5-shot MMLU accuracy (%) by fine-tuning LLaMA-7B on the Alpaca dataset, with respect to different Ditt settings.
MMLU (0-shot)
MMLU (5-shot)
Hums.
STEM Social Other
Avg.
Hums. (1)
STEM Social Other
Avg.
Base Model Dist
#Bits
(f)
(t)
(1)
(1)
(t)
(t)
(t)
(f)
(f)
1
35.6
31.5
39.0
43.6
37.3
35.1
31.4
43.1
44.0
38.1
2
36.5
31.7
41.4
45.8
38.7
35.1
30.7
43.4
44.8
38.2
4
37.3
32.5
41.0
45.7
39.0
36.6
32.3
44.8
44.3
39.3
8
36.1
31.8
44.6
44.5
39.0)
38.2
32.0
42.1
46.5
39.6
16
37.2
32.3
41.3
45.5
39.0,
36.8
31.8
45.2
45.4
39.5
32
4
37.6
32.3
41.6
45.9
39.31
36.0
32.1
45.0
45.2
39.3
64
37.7
31.9
41.7
45.1
39.0)
36.4
31.8
45.0
44.8
39.2
128
37.2
32.4
41.9
45.8
39.21
36.3
32.1
44.9
44.8
39.3
LLaMA-7B
27.4
24.4
25.4
27.4
26.3
27.1
25.4
26.0
28.2
26.7
2
25.9
25.7
24.4
28.0
26.01
26.4
24.7
26.6
28.8
26.6
4
26.7
25.7
25.3
28.6
26.6
26.2
25.0
26.6
29.3
26.7
8
25.4
25.0
25.4
27.3
25.8,
26.0
24.9
27.0
29.3
26.8
16
25.5
24.7
28.7
25.91
26.5
24.7
26.0
29.5
26.7
32
26.2
24.6
25.4
28.0
25.91
25.7
25.0
26.3
29.3
26.5
64
26.4
23.7
24.0
26.4
25.3
27.5
26.3
26.4
28.5
27.2
128
26.1
24.2
25.3
27.5
25.8
27.2
25.3
26.0
28.6
26.8
17

---

Published as a conference paper at ICLR 2024
C MODEL SIZE
We report the sizes of the final models QLoRA and QA-LoRA Table 8. Please note that there are two ways for post-processing in QLoRA, ie., the unmerged (W and 8. A. are stored individually, which saves memory but the inference is slow) and merged (s AB is added to W, which is faster in inference but requires large memory because the matrix must be stored in FP16). QA-LoRA enjoys both low memory usage and a fast inference speed. A side note: In 33B and 65B models, setting L 32 in QA-LoRA results in slightly larger model sizes compared QLoRA, but one can set L = 128 which causes negligible accuracy drop.
Note that the final model size of QA-LoRA is exactly the size of W (or equivalently, W) because AB is merged into W after adaptation. Take the 7B model with L = 32 as an example. The baseline, the unmerged version of QLoRA, is sized 4.6G, in which W is sized 4.0G and A and B combined is sized 0.6G. QA-LoRA increases the first amount 4.3G and eliminates the second amount.
Table 8: The sizes (in GB) of the final models of QLoRA and QA-LoRA.
QLoRA (unmerged)
QLoRA (merged)
QA-LoRA
QA-LoRA
Model
(B = 32)
(B 4, 128)
LLaMA-77 7B
4.6
13.5
4.3
3.7
LLaMA- 13B
8.1
24.4
8.1
6.9
LLaMA-33B
18.9
55.5
20.0
17.5
LLaMA-65B
36.1
122.3
39.0
34.7
18


Reviews:
[{"review_id": "ARK6gYos5J", "reviewer": "Reviewer_KGp3", "summary": "This paper introduces a modification to the QLoRA method, designed to facilitate the training of large language models with limited computational resources. QLoRA initially quantizes neural network weights to NF4 format and subsequently optimizes LoRA matrix weights in FP16. This process increases inference latency, as everything is converted to FP16 during inference. The proposed alternative method outlined in this paper ensures the appropriate quantization of LoRA weights without necessitating a reversion to FP16 during inference. This enhancement involves just a few lines of code, offering a more efficient solution. While the paper lacks specific results and comprehensive explanations, it presents a promising direction for optimizing large language model training within constrained computational budgets.", "questions": "1. In Algorithm 1, the function `merge_with_quantization` is defined but never used. \n2. What is the degree of freedom of quantisation and adaptation - These seem to be new terms that are added in this paper and not used in the literature. These have to be defined, before claiming that they are increased or managed by the proposed method\n3. Page 6 is just results - Page 6 is just results without much to interpret. These are a bunch of numbers. Please consider presenting this table in a better manner. Can this be presented as a graph for readers? Just numbers are hard to read\n4. In Table 2, you indicate that the number of parameters in the method are lesser than QLoRA — almost by 2x. Why is this the case? This seems like a unfair comparison to QLoRA. What is the time taken in hours for fine-tuning with similar number of parameters.\n5. Section 3.3 explains the method and the reasoning behind on why the rank degenerates to 1. But the explanation is not comprehensive.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "**Addresses a Significant Issue** - QLoRA's potential is realized through its ability to quantize LoRA weights, effectively resolving the disparities observed between fine-tuning and inference in QLoRA.\n\n**Streamlined Implementation** - The authors highlight the method's simplicity, emphasizing that it necessitates a mere two lines of code modification to yield impressive enhancements.\n\n**Thorough Assessment** - The evaluation is meticulous, with the authors examining a spectrum of competitive methods and diverse model architectures to demonstrate the method's advantages comprehensively.", "weaknesses": "Reasoning behind the method - Wy should all the c_ij as defined in the paper be equal is not clear — which is the main motivation for the group-wise quantisation. I would be willing to improve the scores with better explanation on the explanation of the method (See the questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6cRoqRWDQO", "reviewer": "Reviewer_YexA", "summary": "The paper proposes a Quantization-Aware Low-Rank Adaptation (QA-LoRA) for efficient fine-tuning of LLM. This work comes improves Q-LORA algorithm by introducing group-wise operators which lift the need for post-training quantization. QA-LoRA implementation is simple and generic. It benefits from a balance between the number of parameters required for adaption and quantization. The experiments show that fine-tuning and inference stages are computationally efficient thanks to the use of INT4. The memory footprint of QA-LoRA is lower than QLoRA. In terms task accuracy, QA-LoRA is better than Q-LoRA with post-training quantization (GPTQ).", "questions": "- I wonder if larger model where the need of this technique is crucial, e.g., 30B-60B, could be discussed.\n- Figure 3 legend should have QA-LoRA instead of A-LoRA", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This work solves a limitation of previous parameter-efficient tuning of LLMs by eliminating the need for a separate post-training quantization which drops model accuracy\n- QA-LoRA further enhances memory efficiency of SOTA while preserving accuracy\n- The experiments are convincing as they cover a wide range of scenarios", "weaknesses": "QA-LoRA introduce a hyper-parameter (L: Group size). This requires additional optimization and It is unclear if it can be selected without tuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0JpsSERa7Y", "reviewer": "Reviewer_2aMS", "summary": "This manuscript proposes QA-LoRA, a LoRA based parameter efficient LLM finetuning scheme with quantization. QA-LoRA extends QLoRA to be able to add low-rank matrices with pre-trained weights in low-bit tensors directly, without the need to PTQ on low-rank matrices. To guarantee that the summation of low-rank matrices and pre-trained weights are still within the same quantization range, the authors relax the requirement of each row being the same into groups, through group-wise quantization. This improves the accuracy and efficiency during inference. During evaluation, the authors experimented with a series of LLaMA and LLaMA2 models with different sizes. Results showed that the proposed models can achieve superior performance than LoRA and QLoRA.", "questions": "See those in weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "* The paper organization, presentation, and references are good.\n* The proposed method has enough novelty.", "weaknesses": "* Parameter offset in experiments: The proposed method incorporates group-wise/sub-channel qunatization, which includes an additional number of parameters for scales. Also, the proposed QA-LoRA reduces the size of low-rank matrices. However, these parameter offsets are not reflected in the results, which could be misleading to the audiences. It would be more informative to add the actual model size (or estimated) in MB/GB for each of the models.\n* In the ablation study, only group size is examined. It would be worthwhile to experiment with the D_int as well, as it is also part of the tradeoff between model size and accuracy. It would be interesting to see what is the lowest D_int in this setup, compared to vanilla LoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)