LQ-LoRA: Low-rank plus Quantized Matrix Decomposition for Efficient Language Model Finetuning
Authors: Han Guo
Philip Greengard
Eric Xing
Yoon Kim
Venue: ICLR 2024 poster (2024)
Keywords: Low-rank plus Quantized Matrix Decomposition
Efficient Language Model Finetuning
Primary area: general machine learning (i.e., none of the above)
TL;DR: Using matrix decomposition to enable more aggressive quantization before LoRA fine-tuning.

Abstract: We propose a simple approach for memory-efficient adaptation of pretrained language models. Our approach uses an iterative  algorithm  to decompose each  pretrained  matrix into a high-precision low-rank component  and a memory-efficient quantized component. During finetuning, the quantized component remains fixed and only the  low-rank component is updated. We present an integer linear programming formulation of the quantization component which enables dynamic  configuration of quantization parameters (e.g., bit-width, block size) for each matrix given an overall target memory budget.  We further explore a data-aware version of the algorithm which uses an approximation of the Fisher information matrix to weight the  reconstruction objective during matrix decomposition. Experiments on finetuning RoBERTa and LLaMA-2 (7B and 70B) demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines and enables aggressive quantization to sub-3 bits with only minor performance degradations. When finetuned on a language modeling calibration dataset, LQ-LoRA can also be used for model compression; in this setting our 2.75-bit LLaMA-2-70B model (which has 2.85 bits on average when including the low-rank components and requires 27GB of GPU memory) performs respectably compared to the 16-bit baseline.

Published as a conference paper at ICLR 2024
LQ-LORA: LOW-RANK PLUS QUANTIZED MATRIX DECOMPOSITION FOR EFFICIENT LANGUAGE MODEL FINETUNING
Han Guo
Philip Greengard
Eric Xing 1o
Yoon Kim
CCamegie Mellon University, *Columbia University Mooaaedd bin Zayed University of Artificial Intelligence, Petuum Inc. Massaacuseett Institute of Technology hanguo@cs. edu, pg21 88coll@bia.ede edu, epxing@cs. cmu. edu, yoonkimêmit monnkim@mit.edi
ABSTRACT
We propose a simple approach for memory-efficiet adaptation of pretrained lan- guage models. Our approach uses iterative algorithm to decompose each pre- trained matrix into a high-precision low-rank component and a memory-effciiet quantized component. During finetuning, the quantized component remains fixed and only the low-rank component is updated. We present an integer linear pro- gramming formulation of the quantization component which enables dynamic configuration of quantization parameters (e.g., bit-width, block size) for each matrix given an overall target memory budget. We further explore a data-aware version of the algorithm which uses an approximation of the Fisher information matrix to weight the reconstruction objective during matrix decomposition. Ex- periments on finetuning RoBERTa and LLaMA-2 (7B and 70B) demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outper- forms strong QLoRA and GPTQ-LoRA baselines and enables aggressive quanti- zation to sub-3 bits with only minor performance degradations. When finetuned language modeling calibration dataset, LQ-LoRA can also be used for model compression; in this setting our 2.75-bit -LaMAA2-7700 model (which has 2.85 bits average when including the low-rank components and requires 27GB of GPU memory) performs respectably compared to the 16-bit baseline."
INTRODUCTION
Despite the increased availability of large language models (LLMs) and their pretrained parameters (Zhang al., 2022; Scao et al., 2022; Touvron et al., 2023a;b), their sheer size makes them expensive to adapt to new datasets via full finetuning. This is particularly unideal since a small amount of supervised finetuning learning interactive agents that can follow general instructions (Wang et al., 2023; Taori et al., 2023; Team, 2023; Zhou et al., 2023), and moreover, LLMs finetuned via reinforcement learning with human feedback (Ouyang et al., 2022) represent some of the most capable AI systems that exist today (OpenAI, 2023; Bubeck et al., 2023). Improving the memory-effiiienc of LLM finetuning thus remains a key step in widening the scope of problems to which LLMs can be practically applied.
instruction following data has been shown to be effecive approach for
One promising framework for memory-efficient LLM adaptation is through parameter-efficcen fine- tuning methods, which typically learn a smaller finetunable extension to the base pretrained model (see Ding et al. (2023) for survey). These methods can reduce the amount of memory required for finetuning the pretrained parameters remain fixed thus reducing the need to allocate memory for storing gradients and optimizer states for these parameters while the number of new parameters to be optimized is fraction of the fixed parameters. Of the many existing parameter-efficient finetun- ing methods, low-rank adaptation (LoRA; Hu et al., 2022) has emerged as a popular technique for efficient LLM adaptation. In LoRA, the pretrained model's 's weight matrix W is reparameterized as W + L2 and only L1 and L2 are finetuned. Recent works have improved the memory effeciecy of LoRA further by applying it quantized pretrained model, i.e., using the reparameterization ((W) + L, L2 where q(-) is some quantization function (Dettmers et al., 2023a; Chai et al., 2023).
OOr code and models are available at https: htt.hb/.comm/aannus9////111..... This work was com- pleted while Han Guo was visiting student MIT.

---

Published as a conference paper at ICLR 2024
In LoRA, Lg is initialized to 0 ensure that the model output is the same the pretrained model at the beginning of finetuning (i.e., X(W + و2Lي) = XW). However, if the pretrained ma- trices are quantized to the extent where there is substantial quantization error (which has been empirically found to occur at sub-4-bit regimes), zero initialization may not be optimal since ((W) + L2 * W. In this paper, we exploit the fact that LoRA only performs low-rank up- dates to the quantized model to derive an initialization scheme that takes the quantization error into account. We use an iterative algorithm similar to those used in the robust PCA literature (Wright et al., 2009; Candès et al., 2011; Zhou & Tao, 2011) to decompose W such that W ~ Q + L, Lg Here Q is the quantized component which remains fixed and L is the low-rank component. Dur- ing adaptation only L₁ and L2 (which captures the high-variance subspaces of W) are finetuned. Instead of applying the same quantization configuration to all layers, we use integer linear program- ming to find mixed quantization strategy that allows for the assignment of different configurations (bits, block size, etc.) to each matrix given an overall target bit rate. Finally, we explore a data-aware version of the algorithm which modifies the decomposition objective with an approximation of the Fisher information matrix obtained from calibration samples.
We apply LQ-LoRA to adapt RoBERTa (Liu et al., 2019) and LLaMA-2 (Touvron et al., 2023b) models and find that can meaningfully improve upon strong QLoRA (Dettmers et al., 2023a) and GPTQ-LoRA (Frantar al., 2022: Chai et al., 2023) baselines while enabling users to flexibly set a target memory budget. L--LLRA can also be applied on standard language modeling datasets to serve as weight-only post-training quantization (PTQ) method. In this setting we find that we are able to compress LLaMA-2-70BB to 2.85 bits with only small perplexity degradation.
2
BACKGROUND
2.1 LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS
Low-rank adaptation of large language models (LoRA; Hu et al., 2022) has emerged as a simple but effective approach for redueing the memory footprint during LLM finetuning. Given matrix W E Radk of a pretrained linear layer, LoRA initializes two matrices L1 E RRdx L2 E Brek with r < min(d, k), where L1 is initialized to Gaussian noise and L2 is initialized to 0 (in order to ensure that L2 = O at the start of training). LoRA then reparameterizes the linear layer as X(W + L2) (here X is the previous layer's activation), and only finetunes L₁ and L2 during language model adaptation. (The bias vector is omitted for brevity.) )) LoRA is more memory-efficiint than full finetuning as there no need allocate GPU memory for the gradients and the associated optimizer states (e.g., the momentum and variance statistics in Adam (Kingma & Ba, 2015)) for W.
2.2 WEIGHT QUANTIZATION OF LARGE LANGUAGE MODELS
Standard round-to-nearest (RTN) quantization, which quantizes/dcquaatiies a block of weights as max( [n]) and bit size b. has ~ X clamp (L ; 2-- - 1) with scaling factor = 2t 1-1
been shown to be effective for quantizing pretrained LLM's weights to 8-bits (Yao et al., 2022). However, (sub) 4-bit guantization has been empirically found to be difficult with RTN, and recent methods generally employ data-aware strategy which uses calibration samples to obtain better weight quantization (Frantar et al., 2022; Dettmers et al., 2022; Xiao et al., 2022; Kim et al., 2023b; Lin et al., 2023; Dettmers al., 2023b; Shao et al., 2023, inter alia).
Our approach relies on the recently proposed NormalFloat (NF) quantization scheme (Dettmers et al., 2023a), which exploits the fact that the distribution of the weights of trained model is ap- proximately Gaussian. Following the presentation from Yoshida (2023), NF quantization calculates
20-1 evenly-spaced values from [6, 1. and 2 + evenly-spaced values from [], 1 8). where
5 ( + 22 This results in 2" probability values .p.....,Ppi) where P1 = 8, P2--1 1 and
2 30
P2- 1 8. These probabilities are converted into quantiles [g1..... .2-] where qi = -- (pt) is the Gaussian quantile for p., and these quantiles are normalized to [-1, 1] by q.
_h_ Then, given 9
a block of weights u = [ui...., u.] and the absmax value s = max(lul) for that block, the weights
U in this block are quantized to the nearest quantile cj. i.e., Cj = arg min.g g %
dk blocks, and hence storing the absmax values s for each block For a d x k matrix there are B could become substantial with small block sizes. Dettmers et al. (2023a) thus employ a double quantization strategy where the set of absmax values [8 [s1,..., z. for given matrix are quantized again via RTN. Based on this quantization scheme, Dettmers et al. (2023a) propose QLoRA, which
2

---

Published as a conference paper at ICLR 2024
performs NF quantization 4 bits the pretrained LLM, and learns low-rank updates. QLoRA has been found to be competitive with full finetuning across a number of benchmarks, and thus serves as the main baseline of the present work.
3 METHOD: LQ-LORA
Our approach relies simple factorization scheme which decomposes each pretrained matrix into a low-rank matrix plus a quantized matrix ($3.1), where only the low-rank component is adapted during finetuning. In $3.2 explore mixed quantization strategy via integer linear programming to allow for dynamic quantization across layers given a target average bit rate. We further consider data-aware version of LQ-LoRA by using the empirical Fisher information matrix weight the reconstruction objective during matrix factorization ($3.3).
3.1 LOW-RANK PLUS QUANTIZED MATRIX DECOMPOSITION
As noted in $2.1, LoRA reparameterizes a pretrained matrix as W as + L and initializes L from Gaussian and L2 to 0 before finetuning. While this ensures that the model output is exactly the same as before reparameterization the start of finetuning, may present an issue when working with quantized version of W since we could have | Quantize(W) [ >> O when quantizing to low bits. This initialization moreover does not take into account W's structure when deciding on which subspaces to adapt. We approach this problem from the perspective of matrix factorization where we are interested factorizing the original matrix into an easily quantizable component and a low-rank component that captures high- high-variance directions,
arg min |W - (Q + L2),), where Q E Qddk L E Rdxr L2 E R L2
(1)
Here Q C Rdxk is the set of matrices that are losslessly NF-quantizable to b-bits. This opti- mization problem is similar to the one faced in robust principal components analysis (RPCA; Wright et al., 2009; Candès et al., 2011), which aims to decompose matrix W into L + S where L is low- rank and S sparse. Following iterative algorithms which have been shown to be effective for RCPA (Lin et al., 2010; Zhou & Tao, 201 1), approximately solve Eq. via alternating between optimizing L, L2, and Q:
L (t) L + SVD(W - Q(--1),)).
|| - (Q((-1) L)]lF, (Q)(-))D)]lp,
= arg min rank(L)
Q ( Quantize - -L (')
arg min NW-QQu 1 ( Qeqex
where Q()) initialized O. Unlike (some) RPCA algorithms for which theoretical convergence
guarantees can be obtained (Ma & Aybat, 2018), the above algorithm is heuristic, We thus employ
simple stopping criterion where keep track of the error IW (QQ() LL ("I("))] r and terminate the algorithm if the error increases. The iterative decomposition algorithm is shown in Algorithm 2.
Each step of the algorithm (i.c., randomized SVD followed by quantization) takes a few seconds on modern GPU for 4096 4096 matrix.
Preliminary experiments. In Figure 1 (left) we show the decomposition error I| -Q + L, L2)]l r for few layers of I.LaMA--7BB as function of the number of steps. We find that our algorithm, while heuristic, is empirically effective. Figure (center) we show the quantization error for 3-bit NF quantization for all matrices, while in Figure (right) we show the corresponding error for LQ decomposition. For both approaches we find that the value and output projection ma- trices become harder to quantize deeper layers, while the key and query matrices become easier; however, our LQ decomposition is able to improve upon vanilla quantization for all layers.
II practice we use randomized SVD instead of full SVD, which significantly reduced runtime for the SVD portion of the algorithm without much deterioration in performance.
Despite the simplity of our approach, we are aware of prior work on low-rank plus quantized ma- trix decomposition, except for a recent preprint which proposes to perform SVD on the residuals E = W - Quantize( W) to correct for crrors after quantization (Yao et al., 2023). This approach can be seen as performing a single step the iterative algorithm with the initialization Q = (uunntie(WW). our exper- iments did observe significant differences in performance when we initialized (0)) Quantize(W).
3

---

Published as a conference paper at ICLR 2024
Error with Ferations
I|W - quantze
LG: W- (quantizelW) ال4ييي
apereet
18-
ayeraed:
ayara#22:
syecceld: 5,0
36-
syerem24: 5,p1d]
sywrem22: 5,51
syero#22: 5.
70-
20-
27-
p402 pm
Sowny (2))
40. prd
0
20 800
160
30
15
20
10 15 2
16
Bumbbrr Tlcrations
Layer Index
Laper Index
Figure 1: (Left) The decomposition error || |W - layers of LLaMA-2-78 zation for all layers. (Right) IQ decomposition error for 3-bit quantization with rank 64. I.Q decomposition results in less quantization error.
(Q L2)l1 for the query projection matrices for different a function of the number of LQ steps. (Center) Quantization error for NF-3 quanti-
3.2 MIXED-CONFIGURATTION QUANTIZATION VIA AN INTEGER LINEAR PROGRAM
LQ-LoRA uses the NormalFloat (NF) quantization scheme from Dettmers al. (2023a) quantize the residual Q at each time step. NF-quantization has several parameters that affect the overall com- pression rate such as the number of quantile bins, number of blocks, and bits for double quantization. In this paper we work with slightly different variant which quantizes a matrix A via the following:
A, s -uanuizaneeeee (A, bo, Bo), 8, v = Quantize-INTI (s, b1, B1), V = cast (v, by)
Concretely, we first apply NF-quantization with size by and bucket size Bo to obtain the quantized matrix A and the absmax values for each block s = [si..... dk (see $2.2). These absmax values Bo
are further quantized to bits via uniform integer quantization with bucket size B1 to obtain the quantized vector s, along with the absmax values for s, i.e., v = [Vi.... v. + Finally, we cast
to by bits to obtain 6j. Dequantization, which is needed on the fly for finetuning and inference, simply reverses this process.
This quantization scheme requires storing S, v to represent A. We can thus quantify the storage cost (number of bits) for storing A given configuration (bo. bi, b2, Bo, B₁)
b2
storage A c) sizeof(A)
bo +
+
Bo
Bo B
The original NF-4 double quantization is a special case with CNF4 (4, fp32, 64, 256) and storago A, CNF4) = 4.127 sizcoo i.e., NF-4 requires on average 4.127 bits per parameter.
Dynamic quantization configurations. Prior works on quantizing LLMs have generally focused on applying the same quantization strategy each matrix, which cannot adapt users' varying resource constraints and moreover may be suboptimal given that some matrices may be harder to quantize than others. We explore mixeedprecisson quantization strategy based on integer linear programming (Yao et al., 2021; Tang et al.. 2022; Kundu et al., 2022), which allows for the allocation of different configurations to each matrix given a user-defined target target bit rate.
Let (bg, by.b, Bg, B2) be the configuration parameters and further let C be the set of possible configurations which specified by the user (see Table 5 for the settings we consider in this work).
Letting (W()) [eEN be the set of N matrices in LM, our goal is to find an assignment matrix X E (0, 1) Nx(cl that minimizes the Frobenius norm between the matrices before and after low-rank plus quantized decomposition, while respecting target memory budget. One way to approach this
$ 'Lc., given U1 = absmax( (($1 ....... , sB,)) for a group of size B₁ we have 8, = clamp (1 1;0, 221-2)
"This approach deviates from the original approach in that use integer quantization on opposed to FP8 (which did not affect results), and we lower precision (which led to negligible increase in error).
4

---

Published as a conference paper at ICLR 2024
Algorithm LQQLoRA (Section 3)
Algorithm 3 GetConfig (Section 3.2)
Input: KEINy: Parameters
Input: (W[])eE[]] JE(NN]
Parameters
[F HEENN: Fisher information (optional)
F Jee[N]= Fisher information (optional)
C: List of quantization configurations
C: List of quantization configurations
r: LoRA rank
r: Target rank
Bq: Quantization budget
By: Quantization budget
# get quantization configurations (Section 3.2). ()) <- Get GetConfig(( (W()) C,,r, Bo)
E,S S.S- zeros(N, [C]) # initialize error and storage
for < to .V do
for + to N do
for c E C de
matrix decomposition (Section 3.1).
Q (9)
Li
(1) L2 ..-LQ((W)) ,c,)
Q ( L : LQ(W()) FF() ((())))F))) c08 (r)
Eli,c f e
S + storage( (W
Output: (Q() _1 LL HE(N)
# get optimal configuration given budget with ILP.
(c hE(N] <- ILLPSolveeS, Bo)
Algorithm 2 LQ (Section 3.1)
Output: {o he(M
Input: W: Input weight matrix
F: Fisher information (optional)
Algorithm Factorize (Section 3.3)
c: Quantization configuration
Input: Input matrix
7: Target rank
F: SVD weighting matrix (optional)
Initialize Q < and 60 f- 00
r: Target rank
for / 4- 1 to T do
if F is None then
L1, L2 + Factorize( W - F, Q - Quantize (W- L2
(randomized) SVD with target rank U. E. + SVD(A,r)
F is None then
L UVE La + VVV
f - IW (Q+L, P
else # weighted SVD (Section 3.3).
else
Daw <- RowAAveaaee (F)
# weighted error (Section 3.3).
Deal +- ColAverage F)
4 vF @ (W- (Q+LI
U V + SVD(D A
if e: > 6t-1 then break
D L v VV D
Output: Q, -L
Output: L
optimization problem is through the following integer linear program,
min Xx
Σ
error (A (6), ()))) X(i,c].
VE(N)
CEC
subject to
Σ Σ
storage (A ()) X(i, c] S budget,
(EIN] CEC
Σ E(i,c] = Vi [N].
DEC
Here error (W(), .) = [Q algorithm from Sec. 3.1 where the Quantize function uses configuration c. To approximately solve this ILP we pre-compute the errors for all matrices and quantization configurations (C²) = 39) and use an off-the-shelf solver," as shown in Algorithm 3. The pre-computation is one-time process and takes a few hours when parallelized across four A100 GPUs for LLaMA-2-7B, Once the (ap- proximately) optimal configuration cf is found, we apply the decomposition on W(1) one more time with c( to obtain the final matrices Q (() L for i E [N] (see Algorithm 1).
the reconstruction error after running the iterative
Implementation. We use www.orchh.issptchh...cc to duck-type torch. Tensor and overload PoTorc operations to perform just-in-time dequantization. We then PyTorch's compiler to compile the bits-unpacking, dequantization, other linear algebra operations. For batch size > 1, this PyTorch- based implementation (followed by compilation) tions such as bi tsandbytes." 8 Further details and speed comparisons are given in Appendix B.
fast some custom CUDA implementa-
3.3 DATA-AWARE MATRIX DECOMPOSITION VIA FISHER- WEIGHTED SVD
The decomposition objective considered in $3.1 is data-agnostic insofar as it treats each entry of W as equally important for reconstruction during factorization. Following recent works which demon-
'Here we overload to refer both its tuple representation E C and its index representation E =[C]).
7 httpps:/ ///ww. gurobi, com/ hhttps://gi http://giittcccccccmeeettttteeee b tsadderrtrerttdsdttteeeeeeeeee
5

---

Published as a conference paper at ICLR 2024
strate the importance of using calibration data for quantizating LLMs (Frantar et al., 2022; Lin et al., 2023; Kim et al., 2023b), we next consider a data-aware version of the approach by using a diagonal approximation of the Fisher information matrix weight the reconstruction objective. The (diag- onal of the) empirical Fisher information matrix measures how sensitive the model's output to a perturbation of each parameter, and has previously been exploited improve low-rank compression (Hsu et al., 2022) and quantization (Kim et al., 2023b) of pretrained language models, When applied to the LQ decomposition algorithm from $3.1, this results in the following weighted SVD problem: given E := W Q and weighting matrix F, we must find matrices L1 E IR
Rdd
L
E R that
form the best rank-r approximation, L1, L2 arg L, VF (E ( L2)
where @ is
the Hadamard product. Unliked its unweighted counterpart, this problem is in general intractable (and in fact NP-hard; Razenshteyn et al., 2016) and is typically addressed through approximate methods (Srebro & Jaakkola, 2003; Li al., 2016; Tuzhilina & Hastie. 2021). However, if we sume that either rows or columns of the weight matrix F have identical values, the above problem can be solved exactly by standard SVD,
U.E.V < SVD(DD row AD L1 f- I-D
nOW
L VEV Pol
(4)
where D row and Dool are diagonal matrices that consist of row- and column- meann of VF. Please see Algorithm 4 and SA for details. While the homogenous row/column assumption clearly does
not hold for F, we found this approach to work well in practice." We note that this approximation a simple extension of Hsu et al. (2022) who use D but not Dew in their weighted SVD (we found that using both the row- and column-averages performed slightly better).
Discussion. This data-aware version of LQ-LoRA requires backpropagating through the pre- trained LM to obtain the Fisher matrices. We note however, that we compute {F hegvv based on some generic text data to obtain the LQ-LoRA initializations, and use the same initialization for different downstream tasks. This makes the data-aware approach practical, as the Fisher computaton and the matrix decomposition needs to performed only once (as in the non-data-aware version).
4 EMPIRICAL STUDY
We conduct experiments with LQ-LoRA across three settings: (1) continual language modeling on C4 training data, (2) instruction tuning on the OpenAssistant dataset (Köpf et al.. 2023), (3) and finetuning on GLUE (Wang et al., 2018). For (1) and (2) we work with LLLMA-22 models (Touvron et al., 2023b), while for (3) we use RoBERTa-Large (Liu et al., 2019). Our setup closely follows the setup from Dettmers al. (2023a). The weshht-weighted version of LQ-LoRA uses randomly sampled sequences from the C4 training set, where for RoBERTa-Large we employ the masked language modeling objective (also on C4) to obtain the Fisher matrix.
Baselines. Our main baselines include QLoRA (Dettmers et al.. 2023a) and GPTQ-LoRA. Both approaches perform PTQ on the pretrained model before learning low-rank updates to the quantized model for adaptation; QLoRA uses NF-quantization, while GPTQ-LORA uses approximate second- order information to solve for arg wee! [XW- X W |F (Frantar & Alistarh, 2022; Frantar
al., 2022). As the original papers were applied on top LLaMA-1 (Touvron et al., 2023a), for fair comparison we reimplement these baselines on top of LLaMA-2. We follow Dettmers et al. (2023a) use rank 64 for our main experiments, and ablate on the rank in our analysis section.
Evaluation. To evaluate models trained C4, we use three metries: perplexity on C4 validation, perplexity on WikiText-2 (Merity et al., 2016), and 5-shot MMI.U accuracy (Hendrycks et al., 2021). For instruction-uus involves asking GPT-4 to make pairwise comparisons between its outputs and those of GPT-3.5 (with the possibility of a tie) over 80 curated questions. We chose this evaluation scheme over the 10-point rating system, following the recommended setup from Dettmers et al. (2023a)." For the GLUE benchmark, we show the average metries across all tasks.
models," 10 we
Vicuna-style automatic evaluation (Team, 2023). This
In preliminary experiments we also explored a version of data-aware L.Q-LoRA where we approximately minimized |X (Q + L2 L2))[ using activations X from calibration data, instead of the Fisher infor- mation matrix. However we found this to underperform the Fisher approach.
We did not include GPTQ-L.RA in instruction-tuning experiments because the training was unstable. However we do not use El.O-style rating system (which would require evaluations across all possible pairs) due to the large numher of models involved.

---

Published as a conference paper at ICLR 2024
C4 (7B)
C4 (70B)
WikiText (7B)
WikiText (70B)
203
onn
MMLU (7B)
MMLU (70B)
OpenAssistant (7B)
OpenAssistant (70B)
64-
01
02-
03
0.1-
230 2.5 2U5 236
325 230 275 4.9
239
25 0.0 235 260
320
Figure LQ-LoRA L models with rank = 64. C4/Wikipedi//MMLU results are based on finetuning on C4, Vicuna eval is based on finetuning on the OpenAssistant dataset. QLoRA (Dettmers et al., 2023a) and GPTQ-LoRA (Chai et al., 2023) are hased on our own reimplementations. Dense refers to unquantized models (no training) except for instruction tuning experiments. In the latter case, dense refers to full finetuning (7B model only, OOM for 70B).
Training details. Unless specified otherwise, learning rate of x 10-5, with a few exceptions. For continual language modeling, we train on partition of the C4 data for half epoch, using sequence length of 1024 for both training and evaluation. To estimate the Fisher, we use 10000 samples from C4 with a sequence length of 1024. For the GLUE tasks, we use similar setup, but with masked language modeling objectives on C4. For instruction tuning, we use the hyperparameters suggested by Dettmers et al. (2023a) (except LoRA dropout). For GLUE fine-tuning, we follow the learning rate and number of epochs recommended by Hu et al. (2022) for the QLoRA baseline. However, we only fine-tune the model for 5 epochs for MNLI and QQP due to their sizes.
rank of 64, no LoRA dropout, and a default
Method
Bits
GLUE
4.1 RESULTS
Full FT QLoRA 3-bit
88.5
Figure 2 shows the results of language modeling and instruc- tion tuning on LLaMA-2 across different model sizes and met- rics. The full numeric results in Table 6 of Appendix C. In gen- cral we find that LQ-LoRA almost always outperforms QLoRA and GPTO-LoRA at (near) similar bit budgets. For example, 3.5 bit (Fisher) LQ-LoRA is generally comparable to NF-4- bit QLoRA (which requires 4.127 bits/param); similarly, 2.75- bit LQ-LoRA is competitive with NF-3-bit QLoRA (which re- quires 3.127 bits/param). These comparisons highlight the util- ity of the mixeddquantization scheme since these mixed strate- gies would not even have been found without the IIP. should be noted, however, that as approach the 2.5-bit range, perfor- mance begins to degrade significantly. At the smaller 7B scale, the Fisher-weighted version of LQ-LoRA outperforms the weighted version by a significant margin at all target bit widths. However, this discrepancy shrinks at the 70B scale.
3.127
86.1
QLoRA (ILP)
2.5
75.4
2.75
80.7
3.0
85.5
3.25
86.1
LQ-LoRA
2.5
85.7
2.75
87.1
3.0
87.3
3.25
88.1
LQ-LORA (Fisher)
2.5
87.3
2.75
86.4
3.0
87.3
3.25
88.3
Table 1: Performance GLUE with RoBERTa-Large.
Table shows GLUE benchmark w/ RoBERTa-Laree, where we observe similar trends: LQ-LoRA outperforms QLoRA at similar bit-wi,ths, and Fisher-weighted LQ-LoRA is effective at 2.5 bits.
4.2 LQ-LORA FOR MODEL COMPRESSION
Results. Table 2 shows the results on C4 and WikiText, where we follow prior PTQ works (Frantar et al., 2022: Shao et al., 2023) and measure performance through C4 and WikiText-2 perplexity a specific subset of data. LQ-LoRA with 2.75 bits results in an average bits/param 2.95 bits and 2.85 bits for the 7B and 70B models respectively, when taking into account the LoRA components

---

Published as a conference paper at ICLR 2024
Table 2: LQ-LoRA comparison against other sub-4-bit PTQ methods. While we only experiment with LQ- LoRA LLaMA-2 (bottom), show other PTQ LLaMA-I (Table 7) as well calibrate results, as most prior works have focused on LLaMA-1. "Effective bits" takes into account the extra storage needed to store quantization parameters (c.g. scaling factors). In LQ-LoRA this includes the LoRA components, which are themselves quantized to 8 bits. For other methods, we take results corresponding to setting with 3-bit quantization and group- 128 (if possible, otherwise the closest one). The effective bits for LQ-LoRA dependent on model size, hence we show the effective bits both settings. 'Results from Shao et al. (2023).
Method
Effective Bits (7B, 65B/70B)
C4
WikiText 65B/70B
7B
65B/70B
7B
LaMM--2 Uncompressed
6,97
5.52
5.47
3.31
RTN (3-bits, g128)
3.15
8.40
6.02
6.66
3,97
GPTQ (3-bits, g128) (Frantar et ul., 2022)
3.15
7.89
5.85
6.29
3.85
AWQ (3-bits, g128) (Lin et al., 2023)f
3,15
7.84
6.24
OmniQuant (3-bits, g128) (Shao et al., 2023)1
3.15
7.75
5,85
6.03
3.78
OmniQuant (2-bits, g64) (Shao al., 2023)T
2.28
12.72
7.88
9.62
6.11
I.Q-LoRA 72-bbss 64-rank, Fisher)
2.95, 2.85
7.60
5.88
5.67
3.65
Table 3: Performance on HuggingFace's Open LLM benchmark with LLaMA-2. We use the same LQQ-LoRA setup in Table 2 (2. 75,bits, 64-rank, Fisher).
Method
Size
ARC
HellaSwag
MMLU
TruthfulQA
Winogrande
GSMSK
Average
Uncompressed (16 bits)
7B
53.2
78.6
39.0
46.6
73.6
14.9
51.0
LQ-LoRA (2.95 bits)
7B
49.8
75.9
39.3
43.0
72.4
7.4
48.0
Uncompressed (16 bits)
70B
67.2
87.3
44.8
69.6
83.7
53.7
67.7
LQ-LoRA (2.85 bits)
70B
65.8
86.2
44.5
66.9
83.2
45.6
65.3
("Effective bits" in Table 2). We find that this generally outperforms other sub-4-bit PTQ methods which also use calibration data to quantize the pretrained models.
Given the promising results with LQ-LoRA continual language modeling, next experiment with whether larger-scale language modeling can improve results further and enable the use of LQ- LoRA viable technique for model compression. Specifically, bits, 64-rank) and fine-tune it on a larger calibration dataset of two C4 partitions and -ikiText-2, using sequence length of 2048. We further quantize the low-rank components themselves using NF-8 configuration after training."
take LQ-LoRA (Fisher, 2.75-
In Table 3, we evaluate the zero/few-ssot capabilities using the Eleuther AI Language Model Evalu- ation Harness (Gao et al., 2023), a unified framework to test generative language models on large number of different evaluation tasks. Specifically, we follow HuggingFace 's the Open L.M Leader- board 1 and evaluate models on 6 key benchmarks: ARC (Clark et al., 201 8), HellaSwag (Zellers et al., 2019), MMLU (Hendrycks et al., 2020), TruthfulQA (Lin et al., 2022), Winogrande (Sak- aguchi et al., 2021), and GSM8k (Cobbe et al., 2021). We observe that there is nontrivial degrada- tion on some benchmarks (GSMSK, ARC), indicating that perplexity degradations are not always commensurate with downstream zero/few-shot perforn
LLaMA-2 (708)
4.3 ANALYSIS
confducoonigurration quantization. We show the allocations of quantization configuration, measured by the average bits per parameter for given ma- trix, in Figure 5. Each plot displays the decisions of the ILP for 2.75 target bit rate. ILP able to al- locate different configurations to different matrices, and this decision is indeed different between Fisher- weighted and non-Fisher-wwiihtee variants.
Figure 3: Storage (GB) broken down by quan- tized, LoRA, and other parameters. LLaMA-2 7B and 70B with 16-bits requires 14GB and 139GB.
LoRA ranks. We investigate the effect of LoRA rank with the LLaMA-2-76 model in Table fixing the quantization configuration to NI-3 (i.e., 3.727 bits/param). QLoRA is insensitive the LoRA rank. However, LQ-LoRA is able to make "better" use of the additional ranks to minimize the error initialization, leading to improved performance.
Memory requirements. In Figure 3 show the memory required for storing the model for dif- ferent bit rates, broken down by the non-quantized component, the quantized component, and the
12 NF-8 replaces the first-level quantization of the original NE-4 with 8-bits. 13 httpps:// /ttpsss///ggggggaaeeeeeee ace. hpppcseeeuggiingggeeeee aceet/ /open 11m leaderboard
8

---

Published as a conference paper at ICLR 2024
Table 4: C4 and WikiText perplexity as function of LoRA ranks. For both QLoRA and LQ-LoRA used
fixed NF-3 configuration for all layers, which incurs an average cost of bits/param. The rightmost column
shows the sum of errors across all matrices in LLaMA-2-7b, which corresponds to |W Quantize( (W)ll
for QLoRA and W
(Q + L, for LQ-LoRA.
Method
LoRA rank
C4
WikiText
Error
QLoRA 3-bit
32
8.21
6.75
(3.127 bits/param)
64
8.21
6.76
9.83 x 10*
128
8.21
6.76
LQ-LoRA 3-bit (3.127 bits/param)
32
8.02
6.61
7.99
10
64
7.93
6.51
7.12
10
128
7.84
6,46
5.98
10
LoRA parameters. Quantization into sub-3-bits greatly decreases the memory required for running the model. At sub-3 bits, it becomes possible to run the 70B model on a single GPU with 40GBs. Finetuning requires more memory due to memory required for the activations and LoRA gradi- ents/optimizer states. However, we are able to run full forward/backward passes on the sub-3-bit 70B models single 80GB GPU with batch size 2 and sequence length 2048.
DISCUSSION AND LIMITATIONS
Our simple iterative algorithm was found be empirically effective but is ultimately heuristic, optimization algorithms could be
and it would be interesting to see if more theoretically-princt derived. And while we focused on NF-quantization to enable comparison against QLoRA, applying LQ decomposition on top of other quantization approaches could result in further gains. It is also possible to extend the ILP-based mixed-precision approach to mixed-precision quantization and mixed-rank decomposition to enable the assignment of different ranks to each matrix.
We also discuss some negative results as well as limitations of LQ-LoRA. We found that re- factorizing the matrix periodically (e.g., after every K gradient steps) did not yield improvements. Insofar as our initialization of L1 and L2 could orient the model to adapt itself in ways that may not be optimal for the task at hand, also tried hybrid approach where half of the adaptable low-rank component comes from LQ-LoRA, and the other half comes from standard LoRA initialization, but did not find this to improve results. Our approach also heavily relies on the fact that adaptation will occur through low-rank updates, and thus is not generally applicable to other parameter-efficient finetuning methods.
RELATED WORK
Parameter-efficient finetuning. Our work is closely related to parameter-efffiient finetuning, such Adapters (Houlsby al., 2019; Mahabadi et al., 2021), prompt tuning (Li & Liang, 2021; Lester ct al., 2021), and other methods which update subparts 2021; Zaken al., 2022; Sung et al., 2021; Hu al., 2022). Recent work also combines parameter- efficient finetuning methods with quantization (Kwon et al., 2022; Dettmers et al., 2023a).
the parameter vector (Guo et al.,
Low-rank plus sparse/quantizel matrix decomposition. Decomposing data matrix into a low- rank matrix plus sparse matrix (also known as robust PCA) is well-studied from both theoretical and applied perspectives (Lin et al., 2010: Zhou & Tao, 2011: 2013; Liu et al., 2013; Aravkin al., 2014; Hintermuller & Wu, 2014; Yi et al., 2016; Zhang & Yang, 2017, inter alia). Within deep learning robust PCA has previously been applied to compress smaller models with fewer than 100M parameters (Chen & Ranftl, 2018; Cai et al., 2021). Saha et al. (2023) uses sketching techniques to obtain a quantized, low-rank approximation of pretrained matrix. Recent contemporaneous work (Li et al., 2023) also performs low-rank plus quantized decomposition for LLM adaptation.
LLM compression. While there has been much work on low-rank compression of smaller LLMs with fewer than IB parameters (Chen et al., 2021; Tukan et al., 2021; 1; Tahaei et al., 2021), 1), low-rank approaches for IB+ LLMs remain underexplored, possibly because singular values of the pretrained matrices of LLMs have been found to decay slowly (Chen et al., 2021). Existing approaches for LLM compression have thus generally focused on quantization. Much recent work has focused on data-aware quantization strategies (Dettmers et al., 2022; Xiao et al., 2022; Dettmers et al., 2023b; Frantar et al., 2022; Kim al., 2023b; Lin et al., 2023).
CONCLUSION
This work proposes a simple extension of LoRA which factorizes the pretrained matrices into low- rank and quantized components, where the quantization component employ dynamic con- figuration strategy. We observed this low-rank plus quantized decomposition approach to yield meaningful improvements over strong baselines.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Minyoung Huh, Isha Puri, Hongyi Wang, Lirui Wang, and the members of the Hyundai 42dot research team for helpful comments and discussions. We are also grateful to Mengzhao Chen for clarification questions regarding OmniQuant. Eric Xing and Han Guo acknowledge the support of Microsoft PhD Fellowship, NGA HM04762010002, NSF ISS1955532, NIGMS R01GM1 NSF IIS2 I 239522, NSF BCS2040381, NSF IIS23 I 1990, SRC AIHW 2024AH3210, and DARPA ECOLE PRR0011239006 This study was supported by funds from Hyundai Motor Group, MIT- IBM Watson AI, and the MLA @CSAIL initiative.
REFERENCES
A. Aravkin, S. Becker, V. Cevher, and P. Olsen. A variational approach to stable principal component pursuit. In Conference on Uncertainty in Artificial Intelligence (UAI), July 2014.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of Artificial General Intelligence: Early experiments with GPT-4. arXiv:2303. 12712 2023.
HanQin Cai, Jialin Liu, and Wotao Yin. Learned Robust PCA: A Scalable Deep Unfolding Approach for High-Dimensional Outlier Detection. arXiv:2 10.00669, 2021.
Emmanuel J Candès, Xiaodong Li, Yi Ma, and John Wright. Robust principal component analysis? Journal of the ACM (JACM), 58(3):1-37, 2011.
Yuji Chai, John Gkountouras, Glenn G Ko, David Brooks, and Gu-Yeon Wei. Inr2. 1: Towards fine-tunable quantized large language models with error correction through low-rank adaptation. arXiv preprint arXiv:2306.0811 2023.
Jiahao Chen and Rene Ranftl. Deep robust pea using convolutional autoencoders. 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2836-2840. IEEE, 2018.
Patrick Chen, Hsiang-Fu Yu, Inderjit Dhillon, and Cho-Jui Hsieh. DRONE: Data-aware Low-rank Compression for Large NLP Models. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 19321-29334. Curran Associates, Inc., 2021.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint TXXXvv8803.05455 2018
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rXiv:2110.14168, 2021.
Tim Detimers, Mike Lewis, Younes Belkada, and Luke Zettlemoye.. LLM.int80: 8-bit Matrix Multiplication for Transformers at Scale. arXiv:2208 07339, 2022.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized lms. arXiv preprint arXiv:2305. 2023a.
Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashk- boos, Alexander Borzunoy, Torsten Hoefler, and Dan Alistarh. Spqr: A aparse-quantized repre- sentation for near-lossless Ilm weight compression. arXiv preprint arxiv:2306.03300 2023b.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, Jing Yi, Weilin Zhao, Xiaozhi Wang, Zhiyuan Liu, Hai-Tao Zheng, Jianfei Chen, Yang Liu, Jie Tang, Juanzi Li, and Maosong Sun. Pframeter-effiiient fine- tuning of large-scale pre-trei language models. Nature Machine Intelligence, 2023.
10

---

Published as a conference paper at ICLR 2024
Elias Frantar and Dan Alistarh. Optimal brain compression: A framework for accurate post-training quantization and pruning. Advances in Neural Information Processing Systems, 10:4475-4480 2022.
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, and Dan Alistarh. GPTQ: Accurate post-training compression for generative pretrained transformers. arXiv preprint arXiv:2210.17323, 2022.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Fos- ter, Laurence Golding, Jeffrey Hsu, Alain Le Noac"h, Haonan Li, Kyle McDonell, Niklas Muen- nighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 12 2023. URL https: /ttp:::://zeodoooo httrg//ecoods/11555500
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of ACL, 2021.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. International Conference on Learning Representations, 2020.
Dan Hendryeks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In International Conference on Learning Representutions, 2021. URL / httpp//erreiiieeeeefruummmm/// httt/forumiddddkjjjjpp 3GmQ
Michael Hintermuller and Tao Wu. Robust Principal Component Pursuit via Inexact Alternating Minimization on Matrix Manifolds. Journal of Mathematical Imaging. 51:361-377, 2014.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficiee transfer learning for nlp. In International Conference on Machine Learning, pp. 1790-2799, PMLR, 2019.
Yen-Chang Hsu, Ting Hua, Sungen Chang, Qian Lou, Yilin Shen, and Hongxia Jin. Language model compression with weighted low-rank factorization. In International Conference on Learning Representations, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Proceedings of ICLR, 2022.
Jeonghoon Kim, Jung Hyun Lee, Sungdong Kim, Joonsuk Park, Kang Min Yoo, Se Jung Kwon, and Dongsoo Lee. Memory-efficient fine-tuning of compressed large language models via sub-4-bit integer quantization. arXiv preprint MiXi:23355.11522 2023a.
Sehoon Kim, Coleman Hooper, Amir Gholami, Zhen Dong, Xiuyu Li, Sheng Shen, Michael W Mahoney, and Kurt Keutzer. Squeezellm: Dense-and-spares quantization. arXiv preprint 2Tt:22306..0622, 2023b.
Diederik P. Kingma and Jimmy Ba. Adam: A Method for Stochastic Optimization. In Proceedings of ICLR, 2015.
Souvik Kundu, Shikai Wang. Qirui Sun, Peter A Beerel, and Massoud Pedram. Bmpy: bit-gradient sensitivity-drivee mixed-preeision precision quantization of dnns from scratch. In 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE), pp. 588-591. IEEE, 2022.
Se Jung Kwon, Jeonghoon Kim, Jeongin Bae, Kang Min Yoo, Jin-Hwa Kim, Baeseong Park, Byeongwook Kim, Jung-Woo Ha. Nako Sung, and Dongsoo Lee. AlphaTuning: Quantization- aware efrfmeter-efficient adaptation of large-scale pre-trained language models. Findings Abu Dhabi,
of the Association for Computational Linguistics: EMNLP 2022, pp. United Arab Emirates, December 2022. Association for Computational Linguistics. URL https: //aclanthology, org/2022. eedlinng-eenl..... 240.

---

Published as a conference paper at ICLR 2024
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rut Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnv Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. AsenAssistant Conversations Democratizing Large Language Model Align- ment. arXiv:2304.07323 2023.
Changhun Lee, Jungyu Jin, Taesu Kim, Hyungjun Kim, and Eunhyeok Park. Owq: Lessons learned from activation outliers for weight quantization in large language models. arXiv preprint arXiv:2306.00222. 2023.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cexsing, pp. 3045-3059 Online and Punta Cana, Dominican Republic, November 2021. Associa- tion for Computational Linguistics. URL.https://acllnthoo.
org/2021. emnlp-main. 243.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of ACL, August 2021.
Yixiao Li, Yifan Yu, Chen Liang. Pengcheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-finettuning-wwar quantization for large language models. arXiv preprint arXiv:2310.08B6 2023.
Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank ap- proximation via alternating minimization. In International Conference Machine Learning, pp- 2358-2367. PMLR, 2016.
Lin, Jiaming Tang, Haotian Tang. Shang Yang. Xingyu Dang, and Song Han. Awq: Activation-aware weight quantization for Il compression and acceleration. arXiv preprint 0700002306.00977, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3214-3252, 2022.
Zhouchen Lin, Minming Chen, and Yi Ma. The Augmented Lagrange Multiplier Method for Exact Recovery of Corrupted Low-Rank Matrices. arXiv: (00v::0000555. 2010.
Guangean Liu, Zhouchen Lin, Shuicheng Yan, Sun, Yong Yu, and Ma. Robust recovery of subspace structures by low-rank representation. IEEE Transactions Pattern Analysis and Machine Intelligence, 35(1):171-184, 2013.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 1907. 1699,, 2019.
Shiqian Ma and Necdet Serhat Aybat. Efficient Optimization Algorithms for Robust Principal Com- ponent Analysis and Its Variants. arXiv:1806.0344 2018.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghami, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypemeeworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguisties and the I 1th International Joint Conference on Natural Language Processing (Volume I: Long Papers), pp. 565-576, 2021.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016.
OpenAl. GPT-4 Technical Report. arXiv:2303.08774, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, et al. Training language models to follow in- structions with human feedback. 2rXii:2203..21155 2022.
Ilya Razenshteyn, Zhao Song, and David Woodruff. Weighted low rank approximations with provable guarantees. In Proceedings of the forty-eighth annual ACM symposium on Theory of Computing, pp. 250-263, 2016.
12

---

Published as a conference paper at ICLR 2024
Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Min- jia Zhang, Dong Li, and Yuxiong He. (ZeRO-Offload): Democratizing Selllion-Scale model training. In 2021 USENIX Annual Technical Conference (USENIX ATC 21), pp. 551-564, 2021.
Rajarshi Saha, Varun Srivastava, and Mert Pilanci. Matrix compression via randomized low rank and low precision factorization, arXiv preprint arXiv:23 10.222200.110288 11028, 2023.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9):99-106, 2021.
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, et al. BLOOM: A Por-mmerre Open-Access Multilingual Language Model. arXiv:2211.05 2022.
Wenqi Shao, Mengzhao Chen, Zhaoyang Zhang, Peng Xu, Lirui Zhao, Zhiqian Li, Kaipeng Zhang, Peng Gao, Yu Qiao, and Ping Luo. Omniquant: Omnidirectionally calibrated quantization for large language models. arXiv preprint (XXXI:23308.113777 2023.
Nathan Srebro and Tommi Jaakkola. Weighted low-rank approximations. In Proceedings of the 20th international conference on machine learning (ICMI,-03), pp. 720-727, 2003.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 24193-24205. Curran Associates, Inc., 2021. URL https: / wwwporeeeiigg.nuurriii. httpscccppppp///////// /221://22221ffll (b26535548887955866b83 567 56738cc51-Paper... pdf.
Marzieh S. Tahaei, Ella Charlaix, Vahid Partovi Nia, Ali Ghodsi, and Mehdi Rezagholizadeh. Kro- neckerBERT: Learning Kronecker Decomposition for Pre-trained Language Models via Knowl- edge Distillation. wrXiv:2109.06243. 2021.
Chen Tang, Kai Ouyang. Zhi Wang. Yifei Zhu, Wen Ji, Yaowei Wang, and Wenwu Zhu. Mixed- precision neural network quantization via learned layer-wise importance. In European Conference Computer Vision, pp. 259-275. Springer, 2022.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: instruction-followinng llama model. https://gt thub. tttp:/ttttttttaaaaaaee
2023.
The Vicuna Team. Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Qual- ity. 2023. https://mmsst http///ssss.oggbogg/o//oo wwwpoo/ggg/o/---//////// 03- vicuna/..
Hugo Touvron, Thibaur Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and Efficient Foundation Language Models. arXiv:2302. 13971, 2023a.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov. Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.09288 099888 2023b.
Murad Tukan, Alaa Maalouf, Matan Weksler, and Dan Feldman. No fine-tuning, no cry: Robust svd for compressing deep networks. Sensors, (16):5559,, August 2021. ISSN 1424-3210. doi: 111399///1115599
Elena Tuzhilina and Trevor Hastie. Weighted low rank matrix approximation and acceleration. arXiv preprint arXiv:2109. 1057, 2021.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self gelf-geeerated instructions. In Proceedings of the 61st Anmual Meeting of the Association for Computational Linguistics (Volume Toronto, Canada, July 2023.
I: Long Papers), pp.
13

---

Published as a conference paper at ICLR 2024
John Wright, Arvind Ganesh, Shankar Rao, Yigang Peng, and Yi Ma. Robust principal component analysis: Exact recovery of corrupted low-rank matrices via convex optimization. In Advances in Neural Information Processing Systems, volume 22, pp. 2080-2088, 2009.
Guangxuan Xiao, Ji Lin, Mickael Seznec, Hao Wu, Julien Demouth, and Song Han. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10433. 2022.
Zhewei Yao, Zhen Dong, Zhangcheng Zheng, Amir Gholami, Jiali Yu, Eric Tan, Leyuan Wang. Qi- jing Huang, Yida Wang, Michael Mahoney, al. Hawq-v3: Dyadic neural network quantization. In International Conference on Machine Learning, pp. 11875-1 1886. PMLR, 2021.
Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers. arXiv:2206.0186/, 2022.
Zhewei Yao, Xiaoxia Wu, Cheng Li, Stephen Youn, and Yuxiong He. Zeroquant-v2: Exploring post-training quantization Ilms from comprehensive study low rank compensation. arXiv preprint arXiv:2303.08332. 2023.
Xinyang Yi, Dohyung Park. Yudong Chen, and Constantine Caramanis. Fast Algorithms for Robust PCA via Gradient Descent. arXiv:1605.07760 2016. v1, last revised v2.
Davis Yoshida. Nf4 isn't information theoretically optimal (and that's good). arXiv preprint arXiv:23066.06960 2023.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficie fine-tuning for mansformer-basee masked hunuuagemodell In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Short Pupers), pp. 1-9, 2022.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791-4800 2019.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, et al. Opt: Open pre-trained transformer language models. arXiv:2205.01068 2022.
Teng Zhang and Yi Yang. Robust PCA by Manifold Optimization. arXiv: 2017.
Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206 2023.
Tianyi Zhou and Dacheng Tao. Godec: Randomized low-rank & sparse matrix decomposition in noisy case. Proceedings of the 28th International Conference on Machine Learning (ICML-I1), 2011.
Tianyi Zhou and Dacheng Tao. Greedy bilateral sketch, completion & smoothing. In Carlos M. Carvalho and Pradeep Ravikumar (eds.), Proceedings of the Sixteenth International Conference on Artificial Intelligence and Statistics, volume 31, 2013.
14

---

Published as a conference paper at ICLR 2024
A DATA-AWARE MATRIX DECOMPOSITION VIA FISHER-WEIGHTED WEIGHTED SVD
The decomposition objective considered in $3.1 is data-agnostic insofar as it treats each entry of W as equally important for reconstruction during factorization. Following recent works which demonstrate the importance of using calibration data for quantizating LLMs (Frantar et al., 2022; Lin al., 2023; Kim et al., 2023b), we next consider a data-aware version of the approach by using diagonal approximation of the Fisher information matrix to weight the reconstruc- tion objective. The (diagonal of the) empirical Fisher information matrix for W is given by F Rd where each entry of the matrix is the averaged square of the derivative over D sam-
ples, i.c., F., - - (ow., PLM PLM (x(dt))² Intuitively, this metric measures how sensitive the model''s output is to perturbation of each parameter, and has previously been exploited to im- prove low-rank compression (Hsu et al., 2022) and quantization (Kim et al., 2023b) of pretrained language models. We similarly
to weight the decomposition objective,
VF - (Q +,,
(5)
where O is the Hadamard produet. When applied to the LQ decomposition algorithm from $3.1, this results in the following weighted SVD problem, where given E == w Q and weighting matrix RR* that form the best rank-r approximation,
F, we must find matrices L E Rdxr ,L
L1, L2 arg min
VF ( (E- L,
L1
Unliked its unweighted counterpart, this problem is in general intractable (and in fact NP-hard; Razenshteyn al., 2016) and is typically addressed through approximate methods (Srebro & Jaakkola, 2003; Li et al., 2016; Tuzhilina & Hastie, 2021). However, if we assume that either rows or columns of the weight matrix F have identical values, we have following identity,
L1, L2 mig L.L
|F (E L
-arg min ||row LE-L, Deall L1
where Drow a diagonal matrix consists of row-means of vF, and Deol is a diagonal matrix con- sisting of the column-meant means of /F, i.e.,
Drow = diag (avg(( / F......... avg( FF.)) Dew diag ([ava(VF.,) (vg((FF..).
F.
In this case the above problem can be solved exactly by standard SVD,
V + SVD(D ADaa). L + DVL
L VE
row
(See Algorithm 4.) While the homogenous row/column assumption clearly does not hold for F, we found this approach to work well in practice." We note that this approximation is simple extension of Hsu et al. (2022) who use D Drow but not Deel in their weighted SVD (we found that using both the row- and column- aveunaas performed slightly better).
Discussion.
This data-aware version of LQ-LORA requires being able to backpropagate through the pretrained LM in order to obtain the Fisher matrices [ which, in some sense, goes against the setting targeted by memory-efficiet adaptation methods wherein full finetuning is not considered possible. This is valid point, and hence we study both version of LQQ-LRR in our empirical study. We note however, that we compute ( based on some generic text data
to obtain the LQ-LORA initializations (Q(9, ( L ( JiE(M)- and use the same initialization for different downstream tasks. This makes the data-aware approach practical, as the Fisher computaton and the matrix decomposition needs to performed only once (as in the non-data-aware version).
B IMPLEMENTATION DETAILS
Here discuss implementation details for efficiently implementing LQ-LoRA.
15

---

Published as a conference paper at ICLR 2024
100 matmuls:: [1. 4096] 4096)
100 maimuus: [16, 4096] (4096, 4096)
100 matrills: (4096, 4096) (4096, 4096
203
64M 0.41444)
260
4.6
404
6W00
(a) Run-times on A100 GPU
100 matmuls: [1, /4096 4096]
100 matmuul:: [16, 4096] x (4096,
100 matmuls: (4096, 4096) (4996, 4096)
359
10
265
305
461
308
36:00 4:31 37
365
20
485
80.
09:68
802 050 0181
201
308
461
(b) Run-times A6000 GPU
Figure A100/A6000 GPU runtime to perform 100 matrix matrix multiplications fp32 between input data and quantized matrices (which involves on the fly dequantization). bitsandbytes tes (bnb) (Dettmers et al., 2023a) has separate implementations for training and for inference (matrix-vector multiplications, leftmost figure). We use the same quantization configuration as NE-4 and vary the first-level bits (2, 3, 4, 8) for consistent comparisons.
Search Space.
Please see Table 5.
PyTorch-based mixed-quantizaaton. Weight-only quantization techniques typically require packing sub- 8-bit matrices into natively-supported data types (e.g.. int8), and then unpacking float-point format during dequantization. As such, existing implementations often require custom CUDA extensions that dependent particular quantization configuration, making it difficult to extend to quaetizaantiitiionn strategies. Our implementa- tion is based entirely on PyTorch for fast experimentation and implementation of dynamic quantization strategies. functionality to
Configuration grid
bo
{2,3, 4}
bi
[2,3, 3. 4)
b2
(bf16, fp16, fp32)
Bo
{16, 32, 64)
B1
{16, 64, 256}
Table 5: Scarch space C for the NF quanti- zation configurations.
We use PyTorch's duck-type tensor, which redefines behaviors under PyTorch operations such addition and matrix multiplication. We then PyTorch's ' (full-graph) compiler to compile the following operations: (1) bits-unpacking, (2) dequantization, (3) linear algebra operations such add and matmul, and (4) transpose and casting (for bf 16 training). For LoRA finetuning, we observed this Py PyTocc-based implementation (followed by compilation) be as fast QLoRA's bi bitsandbytes implementation, which relies heavily CUDA extensions that are tailored for 4-bit NF quantization.
Tensor,
LoRA optimizer offloading. We also optionally work with CPU-based optimizer (Ren et al., 2)21), which extends the pageable optimizer proposed in Dettmers et (2023a). This implementa- tion takes advantage of the fact that in LoRA, only small portion of parameters needs to be trained, which makes data movement between CPU and GPU, as well as computation on CPU, relatively
"In preliminary experiments we also explored version of data-aware L.Q-LoRA where we approximately mnnmmized |X(W (Q + L, L2))[ using activations X from calibration data, instead of the Fisher infor- mation matrix. However found this underperform the Fisher approach. 15 "ptttps://ddv--eeee http:///vvvdiuuu.....e py pytorch. org/t/whatt wwwpwhht---wwwere and-why-ise iiny-is--i wwwttooddcpptttheeeeee dispatch/557 "nntpps:/githull httm://mmeeteer//iiiiiiie tsandbytes
16

---

Published as a conference paper at ICLR 2024
Table 6: LQ-LoRA LLaMA-2 models with rank = 64. C4Wiikppcdi/MMMLU results are based on finctuning C4. Vicuna eval is based finetuning the OpenAssistant dataset. QLoRA (Dettmers et al., 2023a) and GPTQ-LoRA based on our own reimplementations.
Method
Bits per param
C4 (PPL.
WikiText (PPL.
MMLU (acc.)
Vieuna
Eval
70B
7B
70B
7B
70B
7B
70B
7B
Dense (no training)
6.50
8.22
3.68
6.10
0.70
0.46
Dense (full finctuning)
OOM
0.41
QLoRA 3-bit
3.127
6.23
8.21
4.12
6.76
0.68
0.43
0.46
0.33
QLoRA 4-bit
4.127
6.01
7.61
3.78
6.25
0.70
0.46
0.47
0.41
GPTQ-LoRA 3-bit
3.148
6.34
8.48
4.33
7.09
0.67
0.39
GPTQ-LoRA 4-bit
4.156
6.03
7.68
3.82
6.29
0.69
0.45
QLoRA ILP
2.50
2223.2
2996.3
3319.4
4084.3
0.23
0.23
0.00
0.00
2.75
2193.9
2736.5
3292.6
3932.2
0.23
0.27
0.00
0.00
3.00
1781.5
1969.3
2587.0
3091.0
0.23
0.23
0.44
0.33
3,25
6,15
8.04
3,99
6,66
0.69
0,44
0.50
0.41
3,50
6.10
7.91
3,93
6.51
0.69
0.45
0.47
0.36
3.75
6.06
3.85
6.39
0.69
0.44
0.55
0.35
4.00
6.02
7.65
3.80
6.29
0.70
0.45
0.49
0.49
LQ-LoRA
2.50
6.83
10.00
8.44
0.62
0.31
0.57
0.23
2.75
6.42
8.95
4.44
7.55
0.66
0.31
0.56
0.38
3,00
6.18
8.09
4.08
6.73
0.68
0,41
0,59
0.47
3.25
6.10
7.83
3,95
6.44
0.69
0.44
0.56
0.56
3.50
6.06
7.75
3.88
6.39
0.69
0.46
0.55
0.45
3.75
6.02
6.27
0.69
0.45
0.65
0.40
4.00
5.99
7.57
3.77
6.23
0.69
0.46
0.66
0.44
LQ-LoRA (Fisher)
2.50
6.72
7.42
0.67
0.39
0.59
0.45
2.75
6.35
8.25
4.32
6.78
0.67
0.43
0.56
0.44
3.00
6.14
7.88
4.02
6.48
0.68
0.44
0.65
0.51
3.25
6.08
7.76
3,92
6.40
0.69
0.46
0.54
0.49
3.50
6.04
6.31
0.45
0.62
0.49
3.75
6.01
7.57
3.80
6.24
0.69
0.47
0.59
0.47
4.00
5.98
7.53
3.76
6.20
0.70
0.46
0.66
0.51
manageable. We retain copy of trainable parameters on CPU, offload gradients from GPU to CPU before executing optimizer step on the parameter copy on CPU, and copy them back into GPU. We overlap the per-matrix optimizer step and CPU to GPU movement through async copy. On the largest billion parameter model, we noticed 14% memory saving with only marginal (<2%) increase in training speed with this strategy.
Runtimes. Figure displays the run-times of matrix multiplications between FP32 input data and quantized matrices. We dequantize the matrix just-in-time before executing the ma matmul, and hence the runtime is lower-bounded by FP32 matmu (dense). We enable TF32 in PyTorrch to utilize Tensor Cores, and collect CUDA time through PyTorch Profiler. Notably, the dequantization overhead is relatively small for reasonably-sine matrices.
C
FULL RESULTS
Table shows the full numeric results of LQ-LoRA vs. QLoRA and GPT-LoRA. Table shows extended PTQ results including those on LLaMA-1 (top) from prior works. Figure/ shows the allocations of quantization configuration, measured by the average bits per parameter for given matrix.
17

---

Published as a conference paper at ICLR 2024
Table 7: LQ-LoRA comparison against other sub-4-bit PTQ methods. While only experiment with LQ- LoRA on LLaMA-2 (bottom), show other PTQ results on LLaMA-1 (top) well calibrate our results, as most prior works have focused on LLaMA-1. "Effective bits" takes into account the extra storage needed to store quantization parameters (c.g., scaling factors). In LQ-LoRA this includes the LoRA components, which are themselves quantized to 8 bits. For other methods, we take results corresponding to setting with 3-bit quantization and group-size 128 (if possible, otherwise the closest one). The effective bits for SpQR and LQ- LoRA are dependent model size, and hence we show effective bits for both settings. 'Results from Shao et al. (2023). We show 3.t-bits instead 3.01-bits with group size 128 because the latter performed worse.
Method
Effective Bits (7B, 65B/70B)
C4
WikiText 65B/70B
7B
6SB/70B
7B
LLMMA-1 Uncompressed"
16
7.08
5.62
5.68
3,53
SpQR (Dettmers et al., 2023b)
3,94, 3.90
7.28
5.70
5.87
3.68
RTN (3-bits, g128)
3.15
8.62
6.10
7.01
4.24
GPTQ (3-bits, g128) (Frantar et al., 2022)!
3.15
7.85
6,00
6.55
4.17
AWQ (3-bits, g128) (Lin al., 2023yf
3.15
7.92
5.94
6,46
3.99
PFQA (3-bits, g128) (Kim et al., 2023a)
3.15
5.91
OWQ (3-bits) (Lee al., 2023yf
3.1
8.15
6.16
6.39
4.08
SqueezeLLLM (3-bits, 0.45% (Kim al., 2023b)
3.24
7.56
6.13
SqueezeLLLM (3-bits) (Kim et al., 2023b)
3.02
7.75
6.32
OmniQuant (3-bits, g128) (Shao al., 2023)t
3.15
7.75
5.93
6.15
3.94
OmniQuant (2-bits, g64) (Shao al., 2023)
2.28
11.78
7.60
8.90
5.65
IRFC (3-bits, g128) (Chai al., 2023)
3.35
8.24
5.52
LREC (2-bits, g128) (Chai et al., 2023)
2.24
12.52
8.74
LLMMA-2 Uncompressed
16
6.97
5.52
5.47
3.31
RTN (3-bits, g128)
3.15
8.40
6.02
6.66
3.97
GPTQ (3-bits, g128) (Frantar et al., 2022yf
3,15
7.89
5.85
6.29
3.85
AWQ (3-bits, g128) (Lin et al., 2023yf OmniQuant (3-bits, g g128) (Shao al. 2023)
3.15
7.84
6.24
3.15
7.75
5.85
6.03
3.78
OmniQuant (2-bits, g64) (Shao al., 2023)
2.28
12.72
7.88
9.62
6.11
LQ-LoRA (2. 75-bits, 64-rank, Fisher)
2.95,
7.60
5.88
5.67
3.65
slukk
Lalt
Figure 5: Visualization of the bits/param allocated by ILP broken down by matrix type. The y-axis is the bits/param, while x-axis indicates layer number. We show the allocation for QLoRA, LQ-LoRA and Fisher- weighted LQ- LQ-LoRA for target bit rate of 2.75 bits.
18


Reviews:
[{"review_id": "mN0VWTVpgl", "reviewer": "Reviewer_qudp", "summary": "This work proposes a new initialization scheme for doing fine tuning of Large Language Models (LLMs) that have been subjected to Post-Training Quantization (PTQ). The authors motivate their problem by first discussing (along with appropriate references) that the conventional initialization scheme for LoRA, in which the first low-rank adapter is initialized as a Gaussian matrix, and the second low-rank factor is initialized to a zero-matrix, is suboptimal when fine-tuning a PTQ model. The initialization scheme proposed in this paper considers a low-rank + quantized decomposition of the LLM weight matrices. Subsequently, the low-rank factors are used as initializations for fine-tuning.\n\nMost of the paper discusses how to obtain this quantized + low-rank decomposition of the matrix. They do so using an alternating minimization algorithm, wherein the the low-rank component is obtained by computing the SVD of the error residual from the quantized matrix, and the quantized component is obtained by quantizing the error residual from just the low-rank component. This alternating algorithm is a heuristic, and it is terminated when the objective function value, i.e., the Frobenius norm error between the original matrix and its quantized + low-rank decomposition starts diverging (or is small enough).\n\nIn addition to this, the work also considers a dynamic bit allocation strategy across different layers, and formulate this problem as an Integer Linear Program. This is a constrained optimization problem, which minimizes the Frobenius norm reconstruction error subject to a total target bit rate. Moreover, they also propose a data-aware quantization strategy, wherein instead of treating each parameter weight equivalently, their sensitivity with respect to the loss function is evaluated using Fisher matrix, and an alternative objective function is minimized instead.\n\nThe authors convincingly do extensive numerical evaluations on several tasks, and identify that the predominant regime where their initialization provides benefits is where aggressive compression is required (eg., sub 4-bit quantization bit requirement per parameter).", "questions": "I have a few questions:\n\n1. The authors mention: \"LoRA obviates the need to allocate memory for storing gradients and optimizer states\" -- shouldn't it be stated as LoRA does not require us to store gradients for all parameters of the LLM, but only for the low-rank adapters, for which the number of parameters can be a fraction of the total number of LLM parameters?\n\n2. Why did the authors choose NormalFloat instead of instead of a simple RTN quantization scheme? In principle, it seems that LQ-LoRA can be extended with any quantization scheme, if I'm not mistaken? And RTN has benefits over NF, such as no Gaussian modeling assumptions on the weights?\n\n3. Where does the value $\\delta = \\frac{1}{2}\\left(\\frac{1}{30} + \\frac{1}{32}\\right)$ come from? Was it proposed in the NF paper? Does this value remain this same if more flexible quantization resolutions are considered (as is done in this paper in the ILP, but not probably in the NF paper)?\n\n4. Page 3: How is NF quantization **lossless**?\n\n5. In Fig. 1 (center), the weight matrices are just quantized using NF 3-bits, whereas on the right figure, a low-rank + quantized decomposition is obtained, where the quantization is again 3-bit. The caption says \"LQ decomposition results in less quantization error\". Is this a fair comparison in terms of the total memory requirement? Isn't it obvious that the right figure will have low error than center, since in the center, the residual from quantization error is approximated by a low-rank factorization in high-precision, whereas residual is not considered in the center?\n\n6. In Alg. 1 pseudocode -- Does $B$ denote the quantization budget or the total number of blocks (as used in the main text)?\n\n7. Page 6: What does it mean by: \"weight matrix $\\bf F$ has homogeneous rows or columns\"? Please clarify in the main text.\n\n8. Is Table 4 the data-aware or the agnostic variant? Also in Table 4: LQ-LoRA with rank $64$ on C4 has ppl $7.93$. unless I'm mistaken, shouldn't there be a corresponding $7.93$ value in Table 2 as well?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The work provides a new initialization strategy for fine-tuning LLMs that have been subjected to post-training quantization. Conventional LoRA initialization schemes fail are suboptimal for aggressive quantization regimes, and it is this regime where LQ-LoRA proposed in this work provides an advantage.\n\nThe simplicity of this approach is appealing, and it can be readily used with existing quantization schemes in addition to the NormalFloat (NF) quantization scheme utilized in this paper. Proposed ILP formulation of dynamic bit allocation and the data aware variant are also quite interesting. The comprehensive numerical evaluations are also quite descriptive, and clearly identifies where LQ-LoRA performs better, and where it does not.", "weaknesses": "I have some concerns in mind (which are not drawbacks of the paper), but it would be nice if the authors addressed and/or discussed them:\n\n1. One of the contributions of this work is the ILP formulation for dynamic bit allocation across layers. This dynamic configuration of quantization parameters (e.g., bit-width, block size) subject to a total target memory budget is quite interesting. Hardware-wise, the proposed strategy necessitates mixed-precision compute (i.e., different bits for different layers). Even without the ILP, the $Q + L_1L_2$ decomposition requires handling $Q$ is low-precision format, whereas $L_1$ and $L_2$ is high (original)-precision format (eg., $16$-bit). Moreover, the ILP formulation outputs bit-budget allocation at quite fine resolutions like $2$ or $3$-bits. My concern is that such low precision is not easily available as current hardware primitives, i.e.,we can find a $4$ bit GPU, but can we find a $2$ bit GPU? I understand that the simulations are done in PyTorch that provide the flexibility of finer precisions (the authors also mention this on Page 5, \"Implementation\"). It would be worth discussing that this is a significant bottleneck in the deployment of this scheme for actual benefits in hardware. Please note again that I do not see this as a significant drawback of this paper, but it is important that the authors acknowledge this.\n\n2. There is a recent work on joint low-rank and quantized decomposition of a matrix:\n\n\"Matrix Compression via Randomized Low Rank and Low Precision Factorization\" (Saha, Srivastava & Pilanci) (https://arxiv.org/abs/2310.11028)\n\nThis work derives Frobenius norm error bounds for the low-rank decomposition of a matrix, in which the low-rank factors are also quantized. LQ-LoRA considers the low-rank factors to be in high-precision. This work is complementary in the sense that the the low-rank factors of the LQ-LoRA decomposition can also be quantized (in case, the hardware is limited to low-precision only). This will also help in circumventing the mixed-precision hardware issue mentioned in point 1 above (i.e., now $W$, $L_1$, and $L_2$ -- all three can be in the same precision). The analyses techniques proposed in this work can also be used to upper bound the Frobenius norm error of LQ-LoRA in order to make it more theoretically principled.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KiRVKvlyeg", "reviewer": "Reviewer_bc89", "summary": "This paper proposes LQ-LoRA, a method for fine-tuning LLMs in a memory-efficient manner. Each weight matrix is decomposed into a low-rank component and a quantized component. The paper makes three contributions relative to the previously proposed QLoRA paper (Dettmers et. al, 2023):\n1) An iterative algorithm for initializing the quantized and low-rank components for approximating a weight matrix, to minimize the Frobenius approximation error.\n2) An integer linear program for assigning the best quantization configuration to each weight matrix, under a specified total memory budget.\n3) A data-aware quantization strategy, which assigns more weight during the matrix approximation to parameters that are more \"important\" according to the Fisher information matrix.\n\nThe paper shows that these methods yield meaningful improvements over baseline quantization methods, across experiments on (1) language modeling on C4 (Llama-2 model), (2) instruction tuning on OpenAssistant (Llama-2 model), and (3) fine-tuning on GLUE (RoBERTa-Large).", "questions": "- Is the only difference between QLoRA+ILP, and LQ-LoRA, the initialization?\n- Does the ILP budget, as well as the \"bits per param\" column, also consider the low-rank components?\n\nSuggestions:\n- Can you normalize the y-axes in Figure 1 to be relative error $||X - (Q + L1*L2)||_F / ||X||_F$, to make it easier to interpret?\n- Perhaps a Table version of Figure 3 would be helpful to better see how much memory is taken by the low-rank vs. quantized components. Discussing this issue earlier on and more prominently would be helpful for giving readers a better intuitive understanding of what components of the system take the most memory.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Compressing a LLM by replacing each weight matrix into a quantized component (frozen) and low-rank component (which can be fine-tuned) is a great idea for attaining more memory efficient version(s) of a model.\n- The proposed iterative initialization method (equation 2) is a natural and simple way to initialize the low-rank (16-bit) and quantized components for each weight matrix, that effectively reduces the approximation error of the method.\n- The proposed methods give meaningful improvements over baselines (Table 2), across several tasks and model sizes.", "weaknesses": "- The idea of decomposing a weight matrix into a low-rank component and a quantized component had already been proposed in QLoRA (Dettmers et. al, 2023).\n- A few ablations / baselines could be added, to make clearer where the gains of the method come from. For example:\n1) How important is the ILP to LQ-LoRA? Can you show the performance of LQ-LoRA without the ILP?\n2) Can you show the performance of the regular LoRA method (no quantization), and also quantization (at different bit-rates) without LoRA, in Table 2?\n- I found it unusual that while the quantization configurations were optimized extensively (chosen via ILP), the rank of the low-rank components was kept fixed at 64 for the vast majority of experiments (except for Table 4). Perhaps the rank could also be chosen with the ILP?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Bm88GSraZd", "reviewer": "Reviewer_CSBT", "summary": "This paper proposes LQ-LoRA, a memory-efficient LLM adaptation method that decomposes each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. The algorithm is adapted from QLoRA and applied modification to solve the problem that zero initialization of the low-rank matrix may not be optimal when the fixed matrix is quantized. The method decomposes the matrix by an iterative algorithm and updates only the low-rank matrix weights during fine-tuning. Results showed that the proposed method outperforms QLoRA and LREC with similar bit compression rates.", "questions": "Included in the weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed method decomposes the pretrained matrix into a quantizable fixed matrix and low-rank matrix that is already optimized before fine-tuning starts, which contributes to improved accuracy.\n-\tThe paper shows that LQ-LoRA can be used as a mixed quantization strategy, and also proposes a data-aware version of the algorithm, which enables users to flexibly set a target memory budget.\n-\tResults show that the proposed method can be generalized to different model families by showing outperforming results with RoBERTa and LLaMA.", "weaknesses": "- The authors have introduced a method that employs an iterative algorithm for initialization. Can they provide insights regarding the computational latency associated with their approach?\n\n- The authors assert the efficiency of LQ-LoRA based on empirical evidence, yet lack theoretical backing. To strengthen the credibility of the algorithm, a comparison might be beneficial, especially with methods that initialize the Q(W) + L1L2 matrix in a manner that closely mirrors the original pretrained matrix W. Consider, for instance, the use of GPTQ as a compensatory mechanism.\n\n- It appears that this paper serves as an expanded or refined rendition of the Q-LoRA paper. As such, it seemingly inherits the same limitation, notably the inference overhead, given that this approach must fail to integrate the LoRA layer into an existing linear layer. \n\n- Similarly, I would like to raise a query about the paper's novelty. While this method undeniably enhances the current approach (Q-LoRA), from a PEFT perspective, there could be superior methods, particularly concerning inference challenges. On the topic of novelty, I await the insights of fellow reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNxDn8kZuD", "reviewer": "Reviewer_4gD2", "summary": "The authors of the paper propose an iterative method to decompose a pre-trained weight matrix W into a quantized component $Q$ and a low-rank component $L_1L_2$, which encourages $Q + L_1L_2$ to approximate $W$ as much as possible. They also present dynamic configuration quantization via integer linear programming and a data-aware decomposition method by employing the Fisher information matrix.", "questions": "None", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors of the paper the first time tackles the issue that $Q + L_1L_2$ might not be equal to its original pre-trained weight matrix $W$. Moreover, they try to assign different configurations to each pre-trained weight matrix by utilizing integer linear programming.", "weaknesses": "It seems necessary that LQ-LoRA should be compared to the case when BRECQ, OPTQ, and/or FlexRound is used to quantize pre-trained weight matrices with $L_2$ initialized to zero. The reason why I bring this up is that if BRECQ, OPTQ, and/or FlexRound is employed for $q(\\cdot)$, $X(q(W) + L_1L_2)$ would be approximately equal to $XW$ because all their objectives are designed to minimize the difference between $ Xq(W)$ and $XW$. Then, the need for the proposed data-aware decomposition method would be marginal. Although the authors mention that they stick to NF quantization, integer quantization can be also surely used in QLoRA.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)