LoftQ: LoRA-Fine-Tuning-aware Quantization for Large Language Models
Authors: Yixiao Li
Yifan Yu
Chen Liang
Nikos Karampatziakis
Pengcheng He
Weizhu Chen
Tuo Zhao
Venue: ICLR 2024 oral (2024)
Keywords: quantization
compression
large language models
NLP
machine learning
low rank
Primary area: optimization

Abstract: Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning (Dettmers et al., 2023). In this work we focus on the scenario where quantization and LoRA fine- tuning are applied together on a pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ (LoRA-Fine-Tuning-aware Quantization), a novel quantization framework that simultaneously quantizes an LLM and finds a proper low-rank initialization for LoRA fine-tuning. Such an initialization alleviates the discrep- ancy between the quantized and full-precision model and significantly improves the generalization in downstream tasks. We evaluate our method on natural lan- guage understanding, question answering, summarization, and natural language generation tasks. Experiments show that our method is highly effective and out- performs existing quantization methods, especially in the challenging 2-bit and 2/4-bit mixed precision regimes. We will release our code.

Published as a conference paper at ICLR 2024
LOFTQ: LORA-FINE-TUNING-AWAAN QUANTIZA-
TION FOR LARGE LANGUAGE MODELS
Yixiao Li
Yifan Yu
Chen Liang'
Pengcheng He²
Nikos Karampatziakis
Weizhu Chen?
Tuo Zhao
ABSTRACT
Quantization is an indispensable technique for serving Large Language Models (LLMs) and has recently found its way into LoRA fine-tuning (Dettmers al.) )023). In this work we focus on the scenario where quantization and LoRA fine- tuning applied together on pre-trained model. In such cases it is common to observe a consistent gap in the performance on downstream tasks between full fine-tuning and quantization plus LoRA fine-tuning approach. In response, we propose LoftQ LLoRA-Fine-Tune Tuning-aware Quantization), a novel quantiza- tion framework that simultancously quantizes an LLM and finds proper low- rank initialization for LoRA fine-tuning. Such an initialization alleviates the dis- crepancy between the quantized and full-precision model and significantly im- proves generalization in downstream tasks. We evaluate our method on nat- ural language understanding, question answering. summarization, and natural language generation tasks. Experiments show that our method is highly ef- fective and outperforms existing quantization methods, especially in the chal- lenging 2-bit and 2/4-bit mixed precision regimes. The code is available on https: /ttp//gggttt.tt//// cow com/yx111122//Loooeo
INTRODUCTION
The advent of Pre-trained Language Models (PLMs) has marked transformative shift in the field of Natural Language Processing (NLP), offering versatile solutions across various applications (He) al. 2021b; Lewis et al. 2019 Touvron al. 2023). They have showcased unparalleled profi- ciency in executing a variety of language tasks, including Natural Language Understanding (NLU) and Natural Language Generation (NLG). These models typically have millions even billions of parameters, necessitating substantial computational and memory requirements. However, the exten- sive computational and memory demands of these models pose significant challenges, especially for deployments where resources are often constrained and need to be shared among many users.
To mitigate the extensive storage requirements of pre-trained models, quantization serves piv- otal compression technique (Zafrir et al. 2019 Shen et al. 2020 Bai et al. 2022 Dettmers et al.) 2022). converting high-precision numerical values into discrete set of values. Typically, model parameters, originally stored in 16-bit float format, are transformed into 4-bit integer format through quantization, resulting in a substantial 75% reduction in storage overhead. Additionally, to facilitate the adaptation of quantized pre-trained models to downstream tasks efficiently. Rank Adaptation (LoRA) is a viable approach (Hu et al. (021). This technique parameter-effiiient fine-tuning method traditionally applied to high-precision pre-trained models. It is based on the hypothesis that the differences between fully fine-tuned weights and pre-trained weights exhibit low-rank properties. This allows these differences be represented using low-rank matrices. As result, the original pre-trained weights remain unaltered, with adaptations confined solely to these low-rank matrices, enabling effective task adaptation.
Equal contribution
Li, Yu, Liang and Zhao are affiliated with Georgia Institute of Technology.
Correspondence to
vixx xixia aolll @egatcchh..................ch.e. edu 42 9egatech.hh...........ooog..hc.e edu and tour watrzhheeegeech...oooooo "He Karamparziakisand and Chen affiliated with Microsoft Azure.

---

Published as a conference paper at ICLR 2024
When quantizing pre-trained models, practitioners often concentrate primarily on the quantization technique, inadvertently neglecting the importance of subsequent LoRA fine-tuning (Dettmers et al.
2023
Diao et al. 2023). For example, QLoRA inherits the fixup initialization (Zhang et al. 2019) used in LoRA, which (Dettmers et al. 2023) attaches zero initialized low-rank adapters (see Section the quantized pre-trained model. The inevitable discrepancy introduced by quantization dur-
2.3
ing the approximation of the original high-precision numbers, a scenario particularly pronounced in low-bit situations such as the 2-bit regime, can adversely impact the initialization of LoRA fine- tuning. As illustrated in Figure 1a the quantized pre-trained model obtained by QLoRA exhibits severe degradation below the 3-bit level. This deviation in initialization often results an inferior fine-tuning performance. As illustrated in Figure Tb the fine-tuning performance drops as the quan- tization bit decreases when applying QLoRA. Moreover, noteworthy that QLoRA fails below the 3-bit level.
In this paper, we introduce a novel quantization framework, called LoRA-Fine Sin---Fiiinnn-awwee Quantization (LofiQ). It is designed specifically for pre-trained models that require quantization and LoRA fine-tuning. This framework actively integrates low-rank approximation, working in tan- dem with quantization jointly approximate the original high-precision pre-trained weights. This synergy significantly enhances alignment with the original pre-trained weights as illustrated in Fig- Z Consequently, our method provides an advantageous initialization point for subsequent LoRA fine-tuning, leading to improvements in downstream tasks.
12
215
Log of Perplexity
Log of Perplexity
6-
4-
3203 64
2.49
55
16
2.5
2.25
2.5
2.25
2
Number of Bits
Number of Bits
(a) Pre-trained LLAMA-2-13h on WikiText-2
(b) Fine-tuned ILAMA-2-13h on WikiText-2
Figure 1: QLoRA performance with different bits. Left: QLoRA initialization of ILAAMA2-133
on WikiText-2. Right: Apply QLoRA to LLAMA-2-13b on WikiText-2 language modeling task. Smaller perplexity indicates better performance.
We evaluate our quantization framework by conducting extensive experiments on downstream tasks, such NLJU, question answering, summarization, and NLG. Experiments show that LoftQ consis- tently outperforms QLoRA across all precision levels. For instance, with 4-bit quantization, we achieve a 1.1 and 0.8 gain Rouge-1 for XSum (Narayan et al. 2018) and CNN//ailyMail (Her- mann al.) (2155), respectively. LoftQ excels particularly in low-bit scenarios and works effectively with different quantization methods. For example, we achieve over an 8% gain on MNLI (Wang et al. 2019) and more than 10% on SQuADv1.1 (Rajpurkar et al. (0166) with both 2-bit NormalFloat and the 2-bit uniform quantization. We have not seen our approach performs worse than QLoRA.
2
BACKGROUND
2.1 TRANSFORMER MODELS
transformer model contains a sequence of layers, where each layer consists of two sub-layers: a multi-head self-attention (MHA) and fully connected feed forward network (FFN) (Vaswani et al.) 2017). Given the input X E gaxd, where n is the sequence length and d is the hidden dimension of the model, MHA computes the h attention heads in parallel:
MHA(X) Concat Concat(head,, I, ..., heads) W.:
where head; =Softmax XW, (XW,,)) )T /dd))WW, for i= i,
where Wg:, W We, E Rdddd are query. key, and value matrices, W. E Rddd is the output matrix, and dy d/h. FFN comprises two linear transformations and activation function, and is defined as FFN(X) = o oXW, + b, W,, + b2, where W,, E Rdddm, W,, E Rd xd and a-)) is the activation function. A residual connection is used and followed by layer normalization.
2

---

Published as a conference paper at ICLR 2024
14
LoftQ QLoRA
60
LoftQ
QooRA
Discrepancy
Discrepanc
8
40
6
20
Unifomm
NormalFlos
Unitom HermalFlost
Uniform
Normalfoott Uniferm NomalFibat
201
(a) Spectral norm the initialization difference
(b) Frobenius norm of the initialization difference
Initialization discrepancy between the LoRA initialization and the original pre-trained weight matrix, described by the spectral norm and Frobenius norm of the difference. The weight matrix in the above figures is randomly selected in BART-large. The initialization is obtained by QLoRA and LoftQ, with Uniform and NormalFloat quantization methods applied at both 2-bit and 4-bit levels. LoftQ successfully mitigates the discrepancy, especially the 2-bit level.
Figure 2:
2.2 QUANTIZATION
Quantization. Given figh-precision number, e.g., such as 32-bit floating point number, HP E R, N bit quantization encodes it to integer X E {0, 1,.., 2 1}. This process can be expressed as
I round ((2 1)F (XHP)):
where F(:): R [0, 1] is normalization function. Uniform quantization assumes F(X) (X
Xmin) //X m. Dettmers et al. (203)) proposes 4-bit NormalFloat Quantization (NF4).
assumes X (N00,0') a²) and hence [(X) ((X/o), where of-) is the cumulative distribution
function of the standard normal distribution.
Dequantization. lookup table T, where
- -i= 0, 2 11
(2)
2N
used decode the integer INT to its simulated high figh-precision counterpart X RR. Therefore, the dequantization can be expressed as
X T[X!NT]
Simulated Quantization for Matrices. While it is possible to perform multiplication directly be- common apply simulated quantization for matrices (Bai et al. 2020 Shen et al.) 2020). There, quantized weight matrices are stored as encoded integers in memory, and are temporarily dequantized simulated high- high-precision matrices by the lookup table when engaged in multiplication operations. In simulated quantization, it is only necessary to an- alyze the map from a high-precision matrix to a simulated high-precision matrix. We denote this end-to-end process by (-):): Ra + Rmxx where Ry [T[i] E R < <
tween quantized representations.
2.3 LOW-RANK ADAPTATION
LoRA (Hu et al., 2021) updates two small weight matrices A and B that are attached to frozen pre-trained weight matrix W. Hence, a linear transformation, XW, is reformulated as
= XW +XAB
(4)
where E Raxd, W [dd xds A E xr B E xd and & min d2). Initially,
A~N(00 B B=0,
(5)
align to the pre-trained weights. During the fine-tuning, W is fixed while A and B updated by some SGD-type optimization method.
It is worth noting that if low-rank adapters A and B attached to quantized backbone Q yv (W) and are initialized by (5). the starting weight Q + ABT weight W due to the discrepancy introduced by the quantization.
T is no longer equal to the pre-trained
3

---

Published as a conference paper at ICLR 2024
3 METHOD
We propose LoRA-Finee LeeAA--i--TTuning--ee Tuning-aware Quantization (LoftQ), a quantization framework for LLMs. alternatively applies quantization and low-rank approximation to approximate original pre-trained weights. This quantization framework provides promising initialization for LoRA fine-tuning, which alleviates the quantization discrepancy in QLoRA and improves generalization in downstream tasks significantly.
3.1 LORA-AWARE QUANTIZATION
We use an N-bit quantized weight Q E IN xda and low-rank approximations A E Rdix,, B E Rdax" to approximate the original high-precision pre-trained weight W E Rd. xd2 as the initializa- tion of LoRA fine-tuning. Specifically, before fine-tuning, initialize the network by minimizing the following objective:
min I - - AB 1F Q.A.B
where 1 denotes the Frobenious norm. This objective in (6) takes LoRA fine-tuning into consid- eration by jointly optimizing the initial values of the quantized backbone Q and low-rank adapters A, B. Contrarily, practitioners typically convert the pre-trained weight W into quantized weight Q outright, neglecting the subsequent LoRA fine-tuning process. This oversight leads to notable performance degradation in downstream tasks arising from the quantization discrepancy.
3.2 ALTERNATING OPTIMIZATION
We solve the minimization problem in (6) by alternating between quantization and singular value decomposition (SVD). To begin with, we set Ag, and Bg equal to O.
Quantization. At the I-th step, we quantize the difference between the original pre-trained weight matrix W and the low-rank approximation 1 from the previous step to obtain the quantized weight matrix Qe by
Qr = 4N (W- - A,
where 9N (-) maps a high-precision weight matrix to a quantized matrix.
We remark that algorithm is compatible with different quantization functions gx((). We apply NF4 and the uniform quantization in Section 4 as examples. We also remark that Q: is not an exact solution of the minimization in (6), given the fixed A,- B,T,, 1 but it is efficient approximation.
SVD. After obtaining the t-th quantized weight Q., SVD is applied to the residual of the quantization denoted by R2 W Q by
d
Rs = Σ
(8)
where d = min(d1, d2), Te.1 2 022 2 2 Ft.d are the singular values of R., u4,'s and v2, i's are the associated left and right singular vectors of R, We then obtain rank-r approximation of R, by A, B, where
4 Vor,14411 (or,
B, [ Vo Uorl
(9)
We summarize our method in Algorithm T It is worth noting that T= = 1 is special case where Q is the exact quantized weight obtained by QLoRA, and low-rank approximations A1, B₁ obtained by the SVD of the quantization residual W - Q1 T = 1 is sufficient to mitigate the quantization discrepancy, and alternating optimization helps find closer initialization the pre-trained weight W. which further improves the performance (see Section 3
We remark that the computational cost of LoftQ is negligible because it is applied to individual weight matrices and can be executed in parallel. We also remark one can apply LoftQ only once to a pre-trained model and reuse the initialization obtained by LoftQ for different downstream tasks.
3.3 APPLYING TO LORA FINE-TUNING
We store the Qr E Ed xda obtained by LoftQ using integer matrix M by I and lookup table T by (2). We initialize the backbone with the integer matrix M and initialize the low-rank adapters with Ar, Br obtained by LoftQ.
4

---

Published as a conference paper at ICLR 2024
Algorithm LofiQ
input Pre-trained weight W, target rank r, --bit quantization function gv(+), alternating step T
1: Initialize Au < 0, Bb < 0
2: for t= 1 to T do
3: Obtain quantized weight Q < qN QN(W - A,-1B,11)
4: Obtain low-rank approximation Ac, Br SVD(W Qt) by
5: end for
output Qr, Ar, Br
During LoRA fine-tuning, we freeze the integer weight M and optimize the low-rank adapters with an efficient optimization algorithm, e.g., AdamW (Loshchilov & Hutter [017]. In forward propa- gation, the integer weight M is temporarily dequantized to the simulated high-precision weight Qr by its lookup table, as described in (3). In back propagation, gradients and optimizer state are only related low-rank adapters A, B, which reduces considerable training cost.
EXPERIMENTS
We evaluate our method on NLU and NL. tasks. We apply LoftQ for quantizing DeBERTaV3 (He et al. 2021b), BART-large (Lewis et al. 2019). and LLAMA-2 series (Touvron et al. 2023).
Implementation Details. Following the prior works of LoRA variants (Zhang et al. 2023 He et al., 2021a), we freeze all the backbone weight matrices and add low-rank adapters to weight matrices in MHA and FFN of all layers. We quantize the weight matrices that are attached by low- rank adapters. All the quantized models and adapters used in this paper available on hhtps: //huggingf face co/ Loff@] Our implementation is based on publicly available Huggingface Transformers code-base (Paszke et al. 2019). All the experiments are conducted NVIDIA A100
GPUs.
Quantization Methods. We apply quantization methods to demonstrate LoftQ is compatible with different quantization functions:
Uniform quantization is a classic quantization method. It uniformly divides a continuous interval into 2 categories and stores local maximum absolute value for dequantization.
NF4 and its 2-bit variant NF2 are quantization methods used in QLoRA (Dettmers et al., 2023). They assume that the high-precision values are drawn from a Gaussian distribution and map these values to discrete slots that have equal probability.
We perform 2-bit and 4-bit quantization on all models, achieving compression ratios of 25-30% and 15-20% at the 4-bit and 2-bit levels, respectively. The compression ratios and trainable parameter ratios for all models are detailed in the Appendix
Baselines. We compare LoftQ with the following baseline methods:
Full fine-tuning is the common approach for adapting a pre-trained model to down- stream tasks. The model is initialized with pre-trained weights and all parameters are up- dated through an SGD-type optimization method.
Full precision LoRA (LoRA) is lightweight method for task adaptation, where stores the backbone using 16-bit numbers and optimizes the low-rank adaptors only. The adaptors are applied the same matrices in LoftQ.
QLoRA similar LoRA except the backbone is quantized into low-bit regime. The low- rank adapters are initialized using (5) and are applied to the same matrices as in LoftQ.
4.1 ENCODER- ONLY MODEL: DEBERTAV3
Models and Datasets. We quantize the DeBERTa (He al. 202 with LoftQ, then fine- tune and evaluate the model on the General Language Understanding Evaluation (GLUE) bench- mark (Wang et al. 2019). SQuADv1.1 (Rajpurkar et al. 2016), and ANLI (Nee al. 2019). The specific tasks GLUE are given in Appendix C Following previous works (Zhang et al. (023), we exclude WNLI in the experiments.
5

---

Published as a conference paper at ICLR 2024
Implementation Details. We select the learning rates from f1x10-5 5x 1xx100..1 We quantize the entire backbone. Given that GLUE, SQuADv1.1, and ANLI are relatively easy NLU tasks, we also quantize the embedding layer for higher compression efficiency. We apply the NormalFloat and the uniform quantization for LoftQ and LoRRA at both 2-bit and 4-bit levels. We rank 16 and 32 for low-rank adapters. More implementation details, such the training epochs and batch sizes, are presented in Appendix D.2
Main Results. Table D and Table 2 summarize the results for 2-bit quantization on the GLUE, SQuADv1.1, and ANLI datasets, by NF2 and the uniform quantization, respectively. Our method consistently outperforms QLoRA on all settings with respect to different ranks, quantization meth- ods, and datasets. When using the uniform quantization (Table 2) our method achieves 88.0% aceuracy on MNLI-m, surpassing the QLoRA baseline by 8%. For tasks like SST and SQuADv1.1, our method even approaches the full fine-tuning performance at 2-bit level. The 4-bit quantization experiment results are presented in Appendix D.1 as both LoftQ and QLoRA achieve performance close to full fine-tuning.
Table 1: Results with 2-bit LoftQ of DeBERTaV3-ball models on GLUE development set, SQuADvI.1 development set, ANLI test set using NF2 quantization. We report the median over four seeds. N.A. indicates the model does not converge. The best results on each dataset are shown in bold.
Rank
Method
MNLI
QNLI
RTE
SST
MRPC
CoLA Matt
QQP Acc
STSB Corr
SQuAD EM/FI
ANLI
m mm
Acc
Acc
Acc
Acc
Acc
Full
90.5/90.6
94.0
95.3
89.5/93.3
69.2
92.4/89.8
91.6/91.1
88.5/92.8
LoRA
90.4/90.5
94.6
95.1
89. 9/9.6
92.0/89.4
91.7/91.1
87.3/9..1
60.2
QLoRA
75.4/75.6
82.4
55.9
86.5
73 83.8/82.8
N.A.
8/82.3
83.0/82.8
61.5 71.2
N.A.
16
LoftQ
84.7/85.1
86.6
61.4
90.2
83.8/88.6
37.4
90.3/86,9
87.1/86.9
81,5/88.6
47.1
QLoRA
78. 78.5/7887
80.4
56.7
86.9
73.8/82.7
N.A.
87.1/82.7
83.6/83.3
64.6/73.8
N.A.
LoftQ
86.0/86.1
89.9
61.7
92.0
83.6/87.2
47.5
91.0/87.9
87.5/87.0
82,9/89.8
49.0
Table 2: Results with 2-bit LoftQ of DeBERTaV3-bas. models on GLUE development set, SQuADv1.1 development using Uniform quantization. We report the median over four seeds. N.A. indicates the model does not converge. The best results on each task are shown in bold.
Rank
Method
MNLI
QNLI
RTE
SST
MRPC Acc
CoLA Matt
QQP
STSB
SQUAD Em/FI
m/ mm
Acc
Ace
Acc
Acc
P/S Corr
Full FT
90,5/90.6
94.0
82.0
95.3
89.5/93.3
69.2
92.4/89.8
91.6/91.1
88.5/92.8
16
LoRA
90.4/90.5
94.6
85.1
95.1
89.9/93.6
69.9
92.0/89.4
91.7/91.1
87.3/93.1
QLoRA
76.5/76.3
83.8
56.7
86.6
75.7/84.7
N.A.
87.1/82.6
83.5/83.4
69. 69.5/77.6
16
LoftQ 87.3/8.11
90.6
61.1
94.0
87.0/90.6
59.1
90.9/58.0
87.9/87.6
84.4/91.2
QLoRA
79.9/79.5
83.7
57.8
86.9
76.5/84.5
N.A.
88.6/84.7
84.1/84.0
71.6/80.2
32
LoftQ
88.0/88.1
92.2
63.2
94.7
87.5/91.2
60.5
91.3/88.3
89,5/89.2
85.2/91.6
Our method is also more stable compared to QLoRA in the low-bit regime. For instance, while QLoRA fails to converge on CoLAA for both quantization methods and ranks, LoftQ converges in all cases and achieves score of 60.5 using uniform quantization at rank 32. LoftQ stands out in its ability to consistently attain robust and improved performance by effectively preserving the starting point of pre-trained weights.
4.2
ENCODER-DECODEE MODEL: BART
We quantize BART-large model (Lewis et al. 2020) with LoftQ, then fine- tune and evaluate the model on two commonly used summarization datasets: XSum (Narayan al. 2018) and CNW/DailyMail(Herman et al. 2015).
Models and Datasets.
Implementation Details. We apply LoftQ to weight matrices in MHA and FFN of both encoder and decoder layers. We report ROUGE 1/2/L scores, which are the metrics for summarization tasks (Lin) 2004]. We conduct quantization experiments in both 2-bit and 4-bit scenarios. We experiment with both NormalFloat and the uniform quantization in both 2-bit and 4-bit scenarios. In each precision, we choose rank equal to and 16 for a fair comparison with the full precision LoRA baseline (Zhang al. 2023). Please see Appendix F for detailed configurations.

---

Published as a conference paper at ICLR 2024
Main Results. Table 3 summarizes 4-bit quantization experiment results the XSum and CNN/DailyMail test sets. Our method consistently outperforms QLoRA at both ranks on both datasets. It even surpasses full precision LoRA at both ranks Xsum. We will discuss this un- expected results in Section The 2-bit quantization results are shown in Table Our observation is consistent with the NLU experiments, that LoftQ demonstrates the convergence to reasonable results, while QLoRA does not converge. This indicates our method is robuster by narrowing the initialization gap.
Table Results with 4-bit LoftQ BART-large on XSum and CNN//aily We report ROUGE- 1/2/L. Lead-3 means choosing the first 3 sentences as the summary. N.A. indicates the model does not converge. Full FT: full fine-tuning. We report the median over five seeds.
Quantization
Rank
Method
XSum
CNN/DailyMail
Lead-3
16.30/L660011.95
40.42/17..2/36.66
Full FT
45.14/22.27/37..55
44.16/21.28/40.99
Full Precision
LoRA
13.40/20.20/35.1
44.72/21 58/41.84
16
LoRA
43.95/20.72/35.66 45.03/21.844 154/42.15
QLoRA
42.91/19.72/34.88 13.10/20.220 22/40.2222000
LoftQ
44.08/20.72/35.89
43.81/20.99/40.88
NF4
QLoRA
43.29/20.05/35.16 15.42220.62/441
16
LoftQ 44.51/21.14/6.188 43.96/21.06/40.66
QLoRA
41.84/18.71/33.74
N.A.
8
LoftQ 43.86/20.51/35.69
43.73/20.91/40.77
Uniform
QLoRA
42.45/19.36/34.1
16
LoftQ
44.29/20.90/6.000
43.87/20.99/40.92
Table Results with 2-bit LoftQ BART-large on XSum and CNN/DailyMail using NF2 quanti- zation. N.A. indicates the model does not converge. We report ROUGE-1/2/L, the higher the better. We report the median over five seeds.
Rank
Method
XSum
CNN/DailyMail
QLoRA
N.A.
N.A.
LoftQ
39.63/16.65/31.62
42.24/19.444 9.4/19944/2904
QLoRA
N.A.
N.A.
16
LoftQ
40.81/17.85/32.80
12.52/19.81/99.51
4.3 DECODER-ONLY MODEL: LLAMA-2
Models and Datasets. We quantize LLAMA-2-7b and LLAMA-2-13b (Tourron et al. LoftQ. We then fine-tune and evaluate the models NLG datasets: GSM8K (Cobbe et al.) 2021) and WikiText-2 (Merity et al. 2016). Please see Appendix for more details about the datasets.
2023) with
Implementation Details. Similarly, we apply LoftQ to weight matrices in MHA and FFN of all layers. In WikiText-2 evaluation, report perplexity. In case of overfitting, we apply weight decay to low-rank adapters for all settings. In GSM8K evaluation, we extract numerical answers in the generated solutions and then calculate the accuracy using those numerical answers. We conduct experiments with both NF2 and NF4. Please see Appendix F for detailed configurations.
Main Results. Table 5 presents a summary of our experiments on LLAMA-2-7b and LLAMA-2- 13b using b-ti 4-bit, and mixed- sixeed-precision NormalFloat quantization methods on WikiTexx-2 and GSM8K datasets. WikiText-2, our method consistently outperforms QLoRA across all quantiza- tion precision settings on both models. When dealing with the challenging 2-bit precision. where QLoRA fails to converge, LoftQ manages to achieve perplexity of 7.85. In GSM8K, our method achieves better par performance compared to QLoRA across different model sizes and quanti- zation precision levels. For example, our method achieves 26.5% accuracy using 2- precision of LLAMA-2-7b, where QLoRA does not converge.
7

---

Published as a conference paper at ICLR 2024
To provide customized trade-off between the performance and precision, also explore mixed- precision (equivalent to bits) quantization where matrices in the first half layers are quantized using 4 bits, and the rest matrices remain 2 bits. We witness remarkable 4.1% accuracy boost on the GSM8K dataset using LLAMA-2-77 and a 4.7% boost using LLAMA-2-1 1--b. This result underscores the potential of LoftQ for complex mixed-precision quantization scenarios.
Table 5: Results of LoftQ using NormalFloat for LLAMA-2 series on WikiText-2 and GSM8K. 3/2.5/2.25-bir indicates mixed-precision quantization: 4-bit precision for the first 16/8/4 layers and 2-bit precision for the rest of layers. We report the perplexity (the smaller the better) for WikiText-2 and accuracy for GSM8K. The rank of low-rank adapters is 64. N.A. indicates the model does not converge. We report the median over five random seeds.
LLAMA-2-76
LLAMA-2-13h
Method
Bit
WikiText-22
GSM8K1
Wik
GSM8K1
LoRA
5.08
38.5
5.12
48.8
QLoRA
4
5.70
38.2
5.22
48.8
LoftQ
5.24
38.0
5.16
QLoRA
3
5.73
32.1
5.22
40.7
LoftQ
5.63
36.2
5.13
45.4
QLoRA
2.5
N.A.
N.A.
19.39
N.A.
LoftQ
2.5
5.78
31.1
5.22
QLoRA
2.25
N.A.
N.A.
N.A.
N.A.
LoftQ
2.25
6.13
27.5
5.45
QLoRA
N.A
N.A.
N.A.
N.A.
LofiQ
2
7,85
26.5
7.69
33.4
4.4 ANALYSIS
Effectiveness of Alternating Optimization. We conduct experiments with different alternating step T to verify the effectiveness of the alternating optimization and to find the best value T as hyperparameter for different models. Across all tasks and models, observed that alternating optimization yields substantial improvements even with a minimal alternating step. This suggests that it rapidly narrows the discrepancy between quantized weights and pre-trained weights, making our method easy to apply. For example, LoftQ achieves 21.14 Rouge-2 score on XSum using only I step. Interestingly, we noticed that increasing the alternating step beyond certain point tends to result in diminishing returns. We suspect this phenomenon occurs because, as the gap becomes smaller, becomes more challenging for alternating optimization consistently minimize the gup at each step. This challenge emerges because of the inherent errors introduced by the quantization method. Nevertheless, results from Figure indicate our method is not sensitive the alternating step T and is able to consistently enhance downstream fine-tuning performance.
27.0
88.0
25.2
21.14
20.83
22.5
Accimar
20.0
20.05
79.9
26.0
0
10
10
15.0
5
10
Alternating Step
Alterating Step
Alternating Step
(a) MNLI
(b) GSM8k
(c) XSum
Figure 3: Comparison of different alternating step T used in LoftQ. = indicates we use QLoRA
method that initializes low-rank adapters by (5). T 1, 5, 10 indicates we use different T for LoftQ
described in Algorithm Left: Uniform 2-bit BeBERTaV3-basse Middle: NF2 2-bit LLAMA-2-
13b. Right: NF4 Barge
8

---

Published as a conference paper at ICLR 2024
5 DISCUSSION
Start with quantization or SVD in the alternating optimization? An alternative algorithm to the alternating optimization is that we first obtain the low-rank approximation A,, B, and then obtain the quantized weight Q: by switching Line and Line 4 in Algorithm We note this is a valid alterna- tive method as both still jointly minimize the objective in (6). of this alternative method. It is noteworthy that the alternative method still outperforms QLoRA significantly, even though it is worse than the primary version. This observation underscores potential for performance improvement by achieving closer approximation of pre-trained weights within the low low-preciis preciiiiiionn regime.
the performance
LoftQ better than Full- precisieciionn LoRA? ? We find LoftQ outperforms full precision LoRA in XSum and GSM8K (see Table and Table 5) Beside the overfitting caused by lack of regularization, anonther possible explanation for this unexpected phenomenon is that the initial low-rank adapters obtained by LoftQ are non-zero while they are all zero in full precision LoRA as described in (5). Such zero initialization could make the fine-tuning unstable, and therefore it performs worse than LoftQ. We leave the study of the robustness of LoftQ as future work.
Table 6: Results of 2-bit uniformly quantized DeBERTaV3-base on part of GLUE. LoftQ(SVD First) indicates the alternative LoftQ that swiches Line 3 and Line 4 in Algorithm median over four random seeds. The best results on each task are shown in bold.
|
We report the
MNLI
QNLI
SST2
Method
Rank
m /
Acc
Acc
Full FT
90.5/90.6
94.0
95.3
QLoRA
32
79.9/79.5
83.8
86.6
LoftQ(SVD First)
32
87.8/87.7
84.9
89.7
LoftQ(Quantiziion First)
32
88.0/88.1
92.2
94.7
RELATED WORK
Quantization-Aware Training (QAT) often used to obtain quantized models that are adapted in downstream tasks (Peri et al.) 2020 Liu et al. (003)). It involves quantization and full model fine-tuning at the same time. However, QAT requires massive training cost, such as the gradient and optimization state. Moreover, it is difficult to compute the gradient of quantized weights. Our method, with the help of LoRA, sidesteps the aforementioned issues, providing a light approach for downstream task adaptation.
Post- Dost-Trainin Quantization (PTQ) is category of popular quantization frameworks (Frantar et al, 2022 Xiao et al. 2023), which also be used for task adaptation. It calibrates the high-precision model with a small subset the training dataset. Therefore, the subsequent quantization guided by the training dataset, providing task-specific quantized models. Besides, it does not involve any gradient backpropagation, so is cost-efficient. However, it usually results in lower accuracy com- pared QAT.
CONCLUSION
We propose LoftQ, quantization framework for LLMs, which alternatively applies quantization and low-rank approximation to the original high-precision pre-trained weights, to obtain an ini- tialization for the subsequent LoRA fine-tuning. Experiments on natural language understanding. question answering, summarization, and natural language generation show that our framework markably surpasses existing methods, e.g., QLoRA, for quantizing encoder-only, pecoder-decoder and decoder-only models. We have not observed QLoRA. Moreover, our quantization framework demonstrates effectiveness and robustness particu- larly in low-bit quantization regimes, e.g., the 2-bit level.
method exhibiting
performance

---

Published as a conference paper at ICLR 2024
REFERENCES
Haoli Bai, Wei Zhang, Lu Hou, Lifeng Shang, Jing Jin, Xin Jiang, Qun Liu, Michael Lyu, and Irwin King. Binarybert: Pushing the limit bert quantization. arXiv preprint arXiv:2012.15701, 2020.
Haoli Bai, Lu Hou, Lifeng Shang, Xin Jiang, Irwin King, and Michael R Lyu. Towards efficient post- training quantization of pre-trained language models, Advances in Neural Information Processing Systems, 35:1440-1441, 2022.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. The second pascal recognising textual entailment challenge. 2006.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. The fifth pascal recognizing textual entailment challenge. In TAC, 2009.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Lopez-Gazpio, and Lucia Specia. SemEval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the Ith International Workshop on Semantic Evaluation (SemEval-20017 pp. 1-14, Vancouver, Canada, August 2017. Association for Computational Linguistics. doi: 10. N0.18633/v1/17-222
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. /4168, 2021.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Machine Learning Chullenges Workshop. 2007.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm. int8 O: 8-bit matrix multiplication for transformers at scale. arXiv preprint arXiv:2208.0733: 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305. 143 2023.
Shizhe Diao, Rui Pan, Hanze Dong, Ka Shun Shum, Jipeng Zhang, Wei Xiong, and Tong Zhang- Lmflow: An extensible toolkit for finetuning and inference of large foundation models. arXiv preprint wrXiv:2306.12222 2023.
William B. Dolan and Chris Brockett. Automatically constructing a corpus sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh, Gptq: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210. 17323, 2022.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. The third PASCAL recognizing textual entailment challenge. In Proceedings of the ACI-PASCAL. Workshop Textual Entail- ment and Paraphrasing, pp. 1-9, Prague, June 2007. Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. Towards unified view of parameter-efficient transfer learning. arXiv preprint arXiv:21 2021a.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with gradient-disentanlled embedding sharing. arXiv preprint 1rX1v:2111.09543, 2021b.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. Advances in neural information processing systems, 28, 2015.
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.0900 2021.
Hector Levesque, Emnest Davis, and Leora Morgenstem. The winograd schema challenge. In Thir- teenth international conference on the principles of knowledge representation and reasoning, 2012.
10

---

Published as a conference paper at ICLR 2024
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequeeee pre- training for natural language generation, translation, and comprehension. arXiv preprint arXiv: 1910. 13461, 2019.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. BART: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7871 1-7880, Online, July 2020. Association for Computational Linguisties. doi: 10. 10.18653/v1/2020.. aal.0aaac--mm maalmmmmi.............
Yixiao Li, Yifan Yu, Qingru Zhang, Chen Liang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Losparse: Structured compression of large language models based on low-rank and sparse ap- proximation. arXiv preprint arXiv:2306. 1222, 2023.
Chin-Yes Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics.
Zechun Liu, Barlas Oguz, Changsheng Zhao, Ernie Chang, Pierre Stock, Yashar Mehdad, Yangyang Shi, Raghuraman Krishnamoorthi, and Vikas Chandra. Llm-qat: Data-free quantization aware training for large language models. arXiv preprint arXiv:2305. /7888, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. ArXiv, abs/1 0t//1808.00 008.08745, 2018
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adver- sarial nli: A new benchmark for natural language understanding. ArXiv, (os/1990.11499) 2019. URL https://apil http://app.aaa.a....mmaemmm www.attiscchoooo..eeee..eeciccte http://a......eeerrrro--0000231.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto,o Martin Raison, Alykhan Tejani, Sasank Chilamkurthy. Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library. In Advances in Neural Information Processing Systems 32, pp. 8024-8035. Curran Associates, Inc., 2019.
Dheeraj Peri, Jhalak Patel, and Josh Park. Deploying quantization-aware trained networks using tensorrt. In GPU Technology Conference, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383 2392,, Austin, Texas, November 2016. Association for Computational Linguistics. doi: 10.18653/w1/D1--112
Sheng Shen, Zhen Dong, Jiayu Ye, Linjian Ma, Zhewei Yao, Amir Gholami, Michael W Mahoney, and Kurt Keutzer. Q-bert: Hessian based ultra low precision quantization of bert. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 8815 5-211, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1644 Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics.
11

---

Published as a conference paper at ICLR 2024
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov. Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXtv:2337.099288 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions the Association for Computational Linguistics, 7:625 641, 2019. doi: 10.1162/
mecla_002990
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122 New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10.18663/1//118-11000
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. Smoothquant: Accurate and efficient post-training quantization for large language models. In International Conference on Machine Learning, pp. 38087-38099, PMLR, 2023.
Ofir Zafrir, Guy Boudoukh, Peter Izsak, and Moshe Wasserblat. Q8bert: Quantized 8bit bert. In 2019 Fifth Workshop Energy Efficient Machine Learning and Cognitive Computing-Neurllle Edition (EMC2-NIPS), pp. 36-39, IEEE, 2019.
Hongyi Zhang, Yann N Dauphin, and Tengyu Ma. Fixup initialization: Residual learning without normalization. arXiv preprint 1901:19901009330 2019.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. arXiv preprint arXiv:2303. 10512 2023.
12

---

Published as a conference paper at ICLR 2024
A MODEL COMPRESSION RATIO AND MEMORY FOOTPRINT
We report the compression ratio after applying LoftQ in Table 7 It is defined as
compression ration backhone size LoRA adapter size
pre-trained size
We also measure the GPU memory cost during training. Given that GPU memory varies by models, tasks, sequence lengths, batch sizes, etc. We report LLAMA-2 on GSM8K as an example in Table
8
Table 7: Compression ratios of backbones.
Compression ratio (%)
Trainable ratio ( (%)
Quantization method
Model
Rank
Bits
DeBERTaV3-bazz
15.6
3.1
16
2
Uniform
DeBERTaV3-bul
18.8
6.3
32
2
Uniform
DeBERTaV3-base
17.2
3.1
16
2
NF2
DeBERTaV3.. base
20.4
6.3
32
2
NF2
BART-large
15.3
1.2
8
4
NF2
BART-large
16.7
2.5
16
4
NF2
BART-large
27.8
1.2
8
4
NF4
BART-large
29.0
2.5
16
4
NF4
BART-large
26.2
1.2
8
Uniform
BART-large
27.5
2.5
16
4
Uniform
LLAMA-2-7b
16.6
2.4
64
Nf2
LLAMA-2-7b
29.0
2.4
64
4
Nf4
LLAMA-2-13b
16.0
64
2
Nf2
LLAMA-2-136
28.5
1.9
64
4
Nf4
Table 8: GPU memory footprint
Model
Dataset
Seq length
Batch size
GPU Mem
ILLAMA--77b
GSM8K
384
15GB
LLAMA-2-13b
GSMSK
384
24GB
B QUANTIZATION TIME
We report the execution time of LoftQ applying to single weight matrix in Table The time is tested on Intel(R) Xeon(R) CPU ES-2650 @ 2.20GHz.
Table 9: Execution time of LoftQ applying to different weight matrices.
Model
Size
Step
Quantization method
Time
DeBERTaV3-BE ba-
768 x 768
5
Uniform
1s
Hargel
1024 1024
5
NF4
1s
LLAMA-2-77
4096 x 4096
5
NF4
21s
LLAMA-2-13b
5120 x 5120
5
NF4
43s
C GLUE DATASET STATISTICS
We present the dataset statistics of GLUE Wang et al. (2019) in the following table.
GLUE includes two single-sentence classification tasks: SST-2 (Socher et al. 2013 and CoLA WWrrstadt et al. 2019), and three similarity and paraphrase tasks: MRPC (Dolan & Brockett 2005
STS-B (Cer et 2017). 7), and QQP. GLUE also includes four natural language inference tasks in
GLUE: MNLI (Williams al. 2018). QNLI Raajurkar al. 2016). RTE (Dagan et al. 2007 Bar-
Haim et al. 2006; Giampiccolo et al. 2007 Bentivogli et al. 2009). and WNLI (Levesque et al.
2012).
13

---

Published as a conference paper at ICLR 2024
Corpus
Task
#Train
#Dev
#Test
#L.abel
Metries
Single- Sentence Classification (GLUE)
CoLA
Acceptability
8.5k
1k
lk
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLI
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Accuracy/F1
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy/F1
QNLI
QA/NLI
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS-B
Similarity
7k
1.5k
1.4k
Pearson/Spearman corr
Table 10: Summary of the GLUE benchmark.
D NATURAL LANGUAGE UNDERSTANDING
D.I GLUE WITH 4-BIT
We show the 4-bits results in the Table 11 Both methods can achieve performance close to full- finetuning.
Table 11: Results with 4-bit LoftQ of DeBERTaV3-bass models GLUE development set using NF4 quantization. We report the median over four seeds. Results with N.A. indicate the model does converge. The best results on each dataset are shown in bold
MNLI
SST-2
QNLI
ANLI
Method
Rank
m/ mm
Acc
Acc
Acc
Full FT
90.5/90.6
95.3
94.0
59.8
QLoRA
32
89.9/89.9
95.3
94.2
59.4
LoftQ
32
89.9/90.0
95.3
94.1
59.9
D.2 TRAINING DETAILS
Implementation Details. The implementation of LoftQ is based on publicly available Huggingface
Paszke et al. 2019) case-base
Hyper-parameter Details. We select the learning rate of (1 x 10 10- 5 5 10 10-4 5 10 +)), and use the selected learning rate for both uniform quantization experiments and nf2 quantization experiments. We use batch size 32 for all GLUE tasks and ANLI We use batch size of 16 for SQuADv1.1. We LoftQ of iterations for all GLUE tasks.
Table 12 summarizes the detailed hyperparameters for each task used in training DeBERTaV3-bass using uniform quantization. Table [3] summarizes the detailed hyperparameters for each task used in training DeBERTaV3: b3-base using nf2 quantization.
Table 12: dyper-parameen setup of LoftQ for GLUE benchmark for training DeBERTaV3- using Uniform quantization.
papper-parameter
ONLI
MRPC
00P
STS-B
SQuADvL1
ANLI
# epochs
5
20
10
60
60
12
I coming rite
1x 10
5x
10
5x
10
1x100 10
5x 10
5x 10
5 10
x 10
6x 10-6
x 10
14

---

Published as a conference paper at ICLR 2024
Table 13: using NF2 quantization.
setup of LoftQ for GLUE benchmark for training DeBERTaV3-
Hyper prpee-parameter
P0P
STS-B
SQuADvi.
A opochs
5
20
10
60
10
10
60
60
10
12 10
Leuminng
10
5x
10
10
5x 10
5x 10
5 10
1x 10
10
E SUMMARIZATION
E.I TRAINING DETAILS
We choose Adam as the optimizer and try learning rate from X 10 10 x 10-5,7 10 10 10-1,3 10-4, 4 x 10-1). We show the optimal learning rate for different settings in Table 14
5.2 x
We use LoftQ of iteration for all BART-large experiments. Table [4]and Table 15 summarize the learning rate and other syperr-prrmeters for CNNiDaillMaiil and XSum.
Table 14: Hyper-parameter setup of LoftQ BART-large CNN/DailyMail
NF4
4-bit Uniform
NF2
Hyperparameter
rank8
ra
rank8
rank16
rank8
rank16
Learning rate
2c-4
2c-4
2c-4
3e-4
2e-4
2e-4
Epoch
15
15
15
15
15
15
Batch size
64
64
64
64
64
64
Table 15: Hyper-parameter setup of LoftQ BART-large XSum
NF4
4-bit Uniform
NF2
Hyperparameter
rank8
rank16
rank8
rank16
rank8
rank16
Learning rate
2e-4
2e-4
2c-4
2c-4
2e-4
2e-4
Epoch
25
25
25
25
25
25
Batch size
32
32
32
32
32
32
F NATURAL LANGUAGE GENERATION
We set the batch size as 32 for WikiText-2 and 16 for GSMSK. We train 2 epochs on WikiText-2 and 6 epochs GSM8K. We select learning rate from 10-5 10-5
10-5 x 10-4 x
10-4, 4 x 10-1). Specific settings are summarized Table 16 and Table 17
G COMPARISON TO PRUNING
Pruning is also widely used compression method. Here we compare LoftQ with the state-of-the art pruning method Li et al. (2023). We show the comparison Table [8| We can see our method significantly outperforms the pruning methods on DeBERTaV3-base model. We also remark that LoftQ can consistently reduce the memory of both training and storage. contrast, pruning requires training the entire full-precision matrix, which implies that it can not achieve any memory savings during the training stage.
H EXTENSION TO CONVOLUTIONAL LAYERS
Low-rank adapters can also be applied to convolutional layers. Given an input feature map X E gxxwxx and 02 2D convolutional kernels K; xdxd , 1, 2, 2,,,.., C22 the output of the convolutional layer is
Y stack(XX & Ki,.., X Koz);
(10)
where Y Rhxwxez and denotes the 2D convolution operation.
15

---

Published as a conference paper at ICLR 2024
Table 16: Hyper-parameter setup of LoftQ LLAiM-2-series on GSM8K
Model
Hyperparameter
NF4
NF2
Mixed- dixee-precision
LLAMA-2-7b
learning rate
10
10-4
10
LLAMA-2-13b
learning rate
x 10
x 10
3 x 10
Table 17: Hyper-parameter setup of LoftQ LLAMA-2-series on WikiText-2
Model
Hyperparameter
NF4
NF2
Mixed-precision
LLAMA-2-77
learning rate
3 x 10
3 x 10
3 10"
LLAMA-2-13b
learning rate
x 10 x 10
3 x 10
Table 18: Results of LoftQ using 2-bits uniform quantization compared with LoSparse with DeBERTaV3-bass models on some of GLUE development sets. Here Ratio is the proportion of total remaining weights. Results with N.A. indicate the model does not converge.
MNLI
SST-2 Acc
QNLI
Method
Ratio
m / mm
Acc
Full FT
100%
90.5 90.6
95.3
94.0
83.3/82.9
87.6
90.4
LoSparse
20%
84.5/83.8
91.7
88.6
15.6%
87.3/87.1
94.0
90.6
LofiQ
18.8%
88.0/88.1
94.7
92.4
We can reformulate Equation (0) into matrix multiplication as
Y
Z x H
where E Rfwxeid H E Rexe, by extending and flattening the input X together with con- catenating and flattening kernels. We first extend a vector æi, Re by its neighbor vectors within the kernel window:
J. Concat(x
x (++)).
Now, X becomes X' E df We then flatten X' into Z E TR For kernels, we first concatenate (Ki, Koz} into H1' E (exxeexxxx, We then flatten H' into H.
Note that H can be approximated by low-rank matrix
U
where U E R V E Radxx,, < (an by SVD. Therefore, the original convolution layer can be approximated
f = (UVT) xXU
(11)
(12)
= -
(13)
Note that Z x V can be restored into convolution operation where we have r kernels D, Re xdxd i 1, 2, ..... and M x T can also be restored into a convolution operation where have 02 kemnels Uj E gr i 1.2,..... C2.
16


Reviews:
[{"review_id": "0TKkZiKsvi", "reviewer": "Reviewer_tN6u", "summary": "This work proposes better initialization for LoRA adaptors A and B, and the Quantization of pre-trained weights W_{pt} in a setup where two things are desired:\n1) downstream fine-tuning\n2) quantization of W_{pt}. \n\nThe authors propose an iterative method to find better initializations for these matrices. Through rigorous experiments the work shows that the proposed initialization is better than the vanilla initialization proposed in QLoRA.\nThe authors conduct experiments with almost-extreme quantization (2 bit) to show efficacy of their approach, where the traditional methods (QLoRA) even fail to train. \nThe work also attempts to analyze the impact of number iterations (of the proposed iterative method) and the experiments are conducted well.", "questions": "1) For the XSUM and GSM8k tasks, LoftQ gets better accuracy than full-precision LoRA. I wonder how the FP LoRA was tuned? Maybe 4 bit quantization does implicit regularization, and FP LoRA  was not regularized well enough? This would especially make a difference if the tasks are low dimensional. In other words, if a high capacity LLAMA 13B model is fine-tuned LoRA style on GSM8k, how did the authors ensure that the model was not overfitted?\n\n2) It would be nice to analyze number of epochs, and training steps required for baseline full precision LoRA and LoftQ. \n\n3) LoRA's original motivation stems from \"training efficiency\" while maintaining the inference cost the same as the base model. Conversely quantization's main motivation is inference efficiency. Keeping training efficiency aside, a good baseline maybe quantization aware fine-tuning (i.e. no LoRA), to establish upper bound on accuracy for LoftQ. \n\n4) It wasn't very fully clear but are the LoRA adaptors, A and B, quantized as well in LoftQ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* This work presents well motivated initialization method for LoRA + Quantization\n* Through extensive experimentation on several architectures and benchmarks, this work clearly elucidates pitfalls of QLoRA and effectiveness of the proposed method", "weaknesses": "None, but a few clarifying questions stated below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u93H33DD7O", "reviewer": "Reviewer_5E8o", "summary": "This paper introduces a method (LoftQ) to initialize the quantized weights in a transformer based model for future LoRA-based fine-tuning. Different from initializations for Quantized Lora used in prior methods, such as fixup or zero-init, LoftQ initalizes the quantized matrix weights and lora weights together to minimize the Frobenious norm of the difference between the floating point weights and the quantized weights. The initialization process is iterative where the quantized matrix is obtained through a standarized quantization process and the lora quantized weights are obtained from a SVD decomposition.\n\nExperiments on encoder models (classification), encoder-decoder models (summarization), and decoder models (math reasoning, language modeling) are conducted and results are in favor of the LoftQ initialization.", "questions": "1. It would be nice to show the memory footprint for 2-bit quantized models during training. \n2. Would the quantized lora initialization in turn help full quantized fine-tuning?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The lack of a proper initialization of quanitzed lora methods intuitively makes sense, the authors identified this problem and proposed a simple but working solution to address this problem. I appreciate this simplicity.\n2. The experiments are well conducted over quite a few domains/datasets, models, and quantization schemas.\n3. The paper is well written.", "weaknesses": "1. It might be better to put higher priority and conduct more experiments on decoder-based (or encoder decoder) models for generative tasks. It seems that quantized lora (whether with or without intialization) lacks too much in classification tasks with encoders, to the extent that pratictionars probably won't want to train quantized lora models on these tasks. \n2. Otherwise, I find this paper well rounded without significant weaknesses.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v1ta4wc4KY", "reviewer": "Reviewer_BzhB", "summary": "This paper introduces a new approach for weight quantisation and parameter-efficient fine-tuning via low-rank adapters termed LoftQ. LoftQ is inspired by QLoRA and aims to improve over it by providing a better quantisation and better initialisation for the low-rank adapter weight matrices.  \n\nFor background: LoRA makes the assumption that the difference between pre-trained and fine-tuned weights can be approximated by a low-rank matrix, i.e. $W_{ft*} = W_{pt} + AB^T$. \n\nThe core contribution of this work relies on the observation that QLoRA quantises $W_{pt}$ but still relies on the default LoRA initialisation which assumes a non-quantised matrix $W_{pt}$.\n\nTo address this shortcoming, the authors propose an iterative LoRA-aware quantisation which jointly improves the quantisation of $W_{pt}$, making it more similar to the pre-trained weight, and the initialisation of $A$ and $B$ (as the authors note, QLoRA is a special case of their proposed algorithm). \n\nThe authors compare their proposed approach to QLoRA and full fine-tuning across several models and datasets, showing that it consistently outperforms QLoRA.\n\nIn addition to their main experiments, the authors provide ablations investigating their proposed approach in more detail.\n\n- Dettmers et al. 2023 - QLoRA: Efficient Finetuning of Quantized LLMs", "questions": "**Questions and suggestions**\n- The result of the LoftQ algorithm is a quantised weight matrix ($Q_T$) as well as the LoRA matrices ($A_T$, $B_T$). An interesting ablation would be to discard $A_T$ and $B_T$ and use the default LoRA initialisation instead. This would tell us more about the importance of initialising $A_T$ and $B_T$ differently.\n- One of the findings in the QLoRA paper is that it is crucial to add LoRA adapters to every linear layer of the model (Figure 2 in the QLoRA paper). It could be interesting to run a similar ablation with your method. Given your improved initialisation, maybe it is sufficient to add LoRA adapters to fewer layers.\n- It could be interesting to study the difference in initialisation of the low-rank matrices more. Does your work provide insights into what makes a good LoRA initialisation and could these insights be potentially applied to non-quantised LoRA as well? \n\n\n**Typos and writing suggestions**\n\n- Introduction, second paragraph: \"It is predicated on the hypothesis ...\" \n    - You might want to use \"based\" instead of predicated\n- Discussion, LoftQ better than full precision LoRA: \"Such zero initialisation could cause the fine-tuning unstable\"\n    - This sentence needs rewriting", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The core contribution of this work is well motivated and grounded in the shortcomings of an existing widely used approach.\n- The authors provide sufficient experimental results to demonstrate the usefulness of their approach\n- The authors provide ablation studies, investigating important details of their approach\n- The paper is well written, the structure is clear and easy to follow", "weaknesses": "I couldn't identify serious weaknesses of this work but I have some suggestions and questions for the authors. See below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (oral)