LoRA: Low-Rank Adaptation of Large Language Models
Authors: Edward J Hu
yelong shen
Phillip Wallis
Zeyuan Allen-Zhu
Yuanzhi Li
Shean Wang
Lu Wang
Weizhu Chen
Venue: ICLR 2022 Poster (2022)
Keywords: Transfer learning
Adaptation
Transformer
Fine-tuning
Low-rank
RoBERTa
DeBERTa
GPT-2
GPT-3

Abstract: An important paradigm of natural language processing consists of large-scale pre-training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible.
Using GPT-3 175B as an example -- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expensive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared to GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by a factor of 10,000 and the GPU memory requirement by a factor of 3. LoRA performs on-par or better than fine-tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite having fewer trainable parameters, a higher training throughput, and, unlike adapters, no additional inference latency. We also provide an empirical investigation into rank-deficiency in language model adaptation, which sheds light on the efficacy of LoRA. We release a package that facilitates the integration of LoRA with PyTorch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 at https://github.com/microsoft/LoRA.

Published as a conference paper at ICLR 2022
LORA: LOW-RANK ADAPTATION OF LARGE LAN- GUAGE MODELS
Edward Hu*
Yelong Shen'
Phillip Wallis Lu Wang
Zeyuan Allen-Zhu Weizhu Chen
Yuanzhi Li
Shean Wang
Microsoft Corporation
edward. hu@mi la quebec {yeshe, phwallis, zeyuana, swang, luw, yuanzhil@andee cmu edu
com
ABSTRACT
An important paradigm of natural language processing consists of large-scale pre- training on general domain data and adaptation to particular tasks or domains. As we pre-train larger models, full fine-tuning, which retrains all model parameters, becomes less feasible. Using GPT-3 175B as an example- deploying independent instances of fine-tuned models, each with 175B parameters, is prohibitively expen- sive. We propose Low-Rank Adaptation, or LoRA, which freezes the pre-trained model weights and injects trainable rank decomposition matrices into each layer of the Transformer architecture, greatly reducing the number of trainable parameters for downstream tasks. Compared GPT-3 175B fine-tuned with Adam, LoRA can reduce the number of trainable parameters by factor of 10,000 and the GPU memory requirement by a factor of LoRA performs on-pa par or better than fine- tuning in model quality on RoBERTa, DeBERTa, GPT-2, and GPT-3, despite hav- ing fewer trainable parameters, higher training throughput, and, unlike adapters, additional inference latency. We also provide empirical investigation into -ank-deficinncy in language model adaptation, which sheds light on the efficacy LoRA. We release package that facilitates the integration of LoRA with Py Torch models and provide our implementations and model checkpoints for RoBERTa, DeBERTa, and GPT-2 https: thtba:///ggttuoooooo//cc com/m/ croooft..oo......
INTRODUCTION
Many applications in natural language processing rely on adapt- ing one large-scale, pre-trained language model to multiple down- stream applications. Such adaptation is usually done via finttunnng which updates all the parameters of the pre-trained model. The ma- jor downside of fine-tuning is that the new model contains many parameters as in the original model. As larger models are trained "inconvenience" for GPT-2 (Radford et al., b) or RoBERTa large (Liu et al. 2019) to critical deployment challenge for GPT-3 (Brown al. 175 billion trainable parameters !
h
Pretrained Weights
8
every few months, this changes from
W Baxd
N00.
2020 with
d
Many sought to mitigate this by adapting only some parameters or learning external modules for new tasks. This way, we only need to store and load a small number of task-specific parameters in ad- dition to the pre-trained model for each task, greatly boosting the operational efficiency when deployed. However, existing techniques
Figure I: Our reparametriza-
tion. We only train and B.
often introduce inference latency (Houlsby et al. 2019 Rebutfi al.) 2017) by extending model
depth or reduce the model's usable sequence length (Li & Liang 2021 Lester al. 2021
Ham-
"qqual contribution.
While GPT-3 175B achieves non-trivial performance with few-shot learning, fine-tuning boosts its perfor-
mance signifccanlly as shown in Appendix

---

Published as a conference paper at ICLR 2022
More importantly, these method often fail to match the tinettuing baselines, posing a trade-off between efficiency and model quality.
bardzumyan et al. 2020 Liu et al. 2021 (Section 3).
We take inspiration from Li al. 20) Aghajanyan et al. (2020) which show that the learned wwer-parametrized models in fact reside on a low intrinsic dimension. We hypothesize that the change in weights during model adaptation also has low "intrinsic rank", leading our proposed Low-Rank Adaptation (LoRA) approach. LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers' change during adaptation instead, while keeping the pre-trained weights frozen, as shown in Figure I Using GPT-3 175B as an example, we show that very low rank (i.e., r in Fiiuure Icaan be one two) suffices even when the full rank (i.e., d) is as high as 12,288, making LoRA both storage- and compute- -ffpuee-effiieen.
LoRA possesses several key advantages.
A pre-trained model can be shared and used to build many small LoRA modules for dif- ferent tasks. We can freeze the shared model and efficiently switch tasks by replacing the matrices A and B in Figure reducing the storage requirement and twit-switching over- head significantly.
LoRA makes training more efficient and lowers the hardware barrier to entry by up times when using adaptive optimizers since we do not need to calculate the gradients or maintain the optimizer states for most parameters. Instead, we only optimize the injected, much smaller low-rank matrices.
Our simple linear design allows us to merge the trainable matrices with the frozen weights when deployed, introducing inference latency compared fully fine-tuned model, by construction.
LoRA is orthogonal to many prior methods and can be combined with many of them, such as prefix-tuning. We provide an example in Appendix H
Terminologies and Conventions We make frequent references to the Transformer architecture and use the conventional terminologies for its dimensions. We call the input and output di- mension size of a Transformer layer dmodee-- We use W,, Wa. W., and W, to refer to the query/key/value/oupuut projection matrices in the self-attention module. W Wo refers trained weight matrix and AW its accumulated gradient update during adaptation. We use r to denote the rank of LoRA module. We follow the conventions set out by (Vaswani et al. 2017 Brown et al. (020) and use Adam (Loshchilov & Hutter 2019 Kingma & Ba (017)) for model optimization and use a Transformer MLP feedforward dimension djfn dmodel.
pre-
PROBLEM STATEMENT
While our proposal is agnostic to training objective, we focus on conditional language modeling as our motivating use case. Below is a brief description of the language modeling problem and, in particular, the maximization of conditional probabilities given a spse-fe prompt.
Suppose we are given pre-trained autoregressive language model Po (ylæz) parametrized by T. For instance, (y(pz) can be a generic multi-task learner such as GPT (Radford et al. b. Brown al. 2020) based on the Transformer architecture (Vaswani et al.] 2017). Consider adapting this pre-trained model to downstream conditional text generation tasks, such as summarization, machine reading comprehension (MRC), and natural language SQL (NL2SQL). Each downstream task is represented by a training dataset of context-targed pairs: Z = {(T,, y4) b=)),=, where both x; and yi sequences of tokens. For example, in NL22QL,, x; is natural language query and yı its corresponding SQL. command; for summarization, x; is the content of an article and yi its summary.
During full fine-tuning, the model is initialized to pre-trained weights t and updated to A by repeatedly following the gradient to maximize the conditional language modeling objective:
max +
Σ
log (Po()))2,,
(1)
(#.)
2

---

Published as a conference paper at ICLR 2022
One of the main drawbacks for full fine-tuning is that for each downstream task, we learn different set of parameters A& whose dimension | equals Poo.- Thus, if the pre-trained model is large (such as GPT-3 with p x 175 Billion). storing and deploying many independent instances of fine-tuned models can be challenging, if at all feasible.
In this paper, adopt a more parameter-efficient approach, where the task-specific parameter increment A = (+) is further encoded by a much smaller-sized set of parameters + with |0| < | The task of finding A thus becomes optimizing over O:
M log (V)
Σ (2
In the subsequent sections, propose to low-rank representation to encode A that is both compute- and memory-efficient. When the pre-trained model is GPT-3 175B, the number of train- able parameters [0] can be as small 0.01% of |
3 TREN'T EXISTING SOLUTIONS GOOD ENOUGH?
The problem we set out to tackle is by no means new. Since the inception of transfer learning, dozens of works have sought make model adaptation more parameter- and compute-effcient. See Sec- tion 6ffor survey of some of the well-known works. Using language modeling as an example, there are two prominent strategies when comes to efficient adaptations: adding adapter layers (Houlsby) et al. [0199] Rebuffi et al. 2017 Pfeiffer et al. [021 Rücklé et al. (2020) optimizing some forms of the input layer activations (Li & Liang [2021; Lester et al. 2021] Hambardzumyan et al., 2020) Liu et al. 2021). However, both strategies have their limitations, especially in a large-scale and latency-sensitin production scenario.
Adapter Layers Introduce Inference Lateney There many variants of adapters. We focus on the original design by Houusby et al. (2019) which has two adapter layers per Transformer block and more recent by Lin et al. (2020) which has only per block but with an additional LayerNorm (Ba et al. 2016). While one can reduce the overall latency by pruning layers or exploit- ing multi-task settings (Ricklé al. 2020 Pfeiffer al.) 202I). there is no direct ways bypass the extra compute in adapter layers. This seems like non-issue since adapter layers are designed have few parameters (sometimes <1% of the original model) by having small bottleneck di- mension, which limits the FLOPs they can add. However, large neural networks rely on hardware parallelism to keep the latency low, and adapter layers have to be processed sequentially. This makes difference in the online inference setting where the batch size is typically as small as one. In generic scenario without model parallelism, such as running inference on GPT-2 (Radford et al. b) noticeable increase in latency when using adapters, even with
medium single GPU, very small bottleneck dimension Table
Batch Size
Sequence Length
512
256
128
(
0.5M
11M
11M
The-Tune/LoRA
1449,440.8
338.0=10.6
19.8+22.7
Ad
1482.0+ (+22.%%)
355.8:10.5 (45.0%)
23.9422.1 (+20.7%)
Adapter#
1492.2+1.0 (+3.0% 366.3+00. (+8.4%)
25.8+-2.2 (+30.3%)
Table I: Infernece latency of a single forward pass in GPT-2 medium measured in milliseconds, av- eraged over 100 trials. We use an NVIDIA Quadro RTX8000. "IQ]" denotes the number of trainable parameters in adapter layers. Adapter' and Adapter" are two variants of adapter tuning. which we describe in Sectiin 5.1 The inference latency introduced by adapter layers can be significant in an online, short-sequence-llnnthh scenario. See the full study in Apppenddx C
This problem gets when need to shard the model as done in Shoeybi et al. (2020): Lep-
ikhin et al. (2020), because the additional depth requires more synchronous GPU operations such as Al Reduce and Broadcast, unless we store the adapter parameters redundantly many times.
3

---

Published as a conference paper at ICLR 2022
Directly Optimizing the Prompt is Hard The other direction, exemplified by prefix tuning (Li & Liang 202I), faces a different challenge. We observe that prefix tuning is difficult to optimize and that its performance changes man-ooooonically in trainable parameters, confirming similar observations in the original paper. More fundamentally, reserving a part of the sequence length for adaptation necessarily reduces the sequence length available process downstream task, which we suspect makes tuning the prompt underperform other methods, We defer the study on task performance to Section 5
OUR METHOD
We describe the simple design of LoRA and its practical benefits. The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in experiments as the motivating use case.
4.1 LOW-RANK-PARAL PARAMETRIZED UPDATE MATRICES
A neural network contains many dense layers which perform matrix multiplication. The weight ma- trices these layers typically have full ranks. When adapting specific task, Aghajanyan et al. (020) shows that the pre-trained language models have a low "instrisic dimension" and can still learn efficiently despite a random projection
smaller subspace. Inspired by this, we hypothe- size the updates to the weights also have a low "intrinsic rank" during adaptation. For pre-trained weight matrix Wo Rddx we constrain its update by representing the latter with a low-rank de- composition Wo + AW Wo + BA, where B E gd A E Rr* and the rank < min (d, k). During training. Wo is frozen and does not receive gradient updates, while and B contain trainable parameters. Note both Wo and AW B A multiplied with the same input, and their respective output vectors are summed coordinate wise. For Wox, our modified forward pass yields:
h
Woz + AWr x Wox + BAr
33)
We illustrate our reparametrization in Figure We use random Gaussian initialization for A and zero for B. so AW = BA is zero at the beginning of training. We then scale AW* by 2, where a constant in When optimizing with Adam, tuning a is roughly the as tuning the learning rate we scale the initialization appropriately. As a result, we simply set a to the first r we try and do not tune it. This scaling helps reduce the need retune hyperparameters when we vary
r (Yang& & Hu 2021
A Generalization of Full Fine-tuning. A more general form of fine-tuning allows the training of subset the pre-trained parameters. LoRA takes step further and does not require the accumulated gradient update to weight matrices to have full ranks during adaptation. This means that when applying LoRA to all weight matrices and training all biase we roughly recover the expressiveness of full fine-tuning by setting the LoRA rank to the rank of the pre-trained weight matrices. In other words, as we increase the number of trainable parameters 3l training LoRA roughly converges to training the original model, while adapter-basse based methods converges an MLP and masex methods to a model that cannot take long input sequences.
No Additional Inference Latency. When deployed in production, we can explicitly compute and store W = Wo + BA and perform inference as usual. Note that both Wo and BA are in d When we need to switch to another downstream task, we can recover Wo by subtracting BA and then adding different B' , quick operation with very little memory overhead, Critically, this guarantees that we do not introduce any additional latency during inference compared to a fine-tuned model by construction.
4.2 APPLYING LORA TO TRANSFORMER
In principle, we can apply LoRA to any subset of weight matrices in neural network to reduce the number of trainable parameters. In the Transformer architecture, there are four weight matrices in
Thhe represent negligible number of parameters compared to weights. Inevitable when adapting hard tasks.
4

---

Published as a conference paper at ICLR 2022
the self- self-attention module (Wq: We. W., Wo) and two in the MLP module. We treat Wg (or Wt. W;) as a single matrix of dimension dmodet x dmodel. even though the output dimension is usually sliced into attention heads. We limit our study to only adapting the attention weights for downstream tasks and freeze the MLP modules (so they are not trained in downstream tasks) both for simplicity and parameter-efficiency. We further study the effect on adapting different types of attention weight matrices in a Transformer in Section B.I We leave the empirical investigation of adapting the MLP layers, LayerNorm layers, and biases to a future work.
Practical Benefits and Limitations. The most significant benefit comes from the reduction in memory and storage usage. For a large Transformer trained with Adam, we reduce that VRAM usage by up to 2/3 if r < dmmdde as we do not need to store the optimizer states for the frozen parameters. On GPT-3 175B, reduce the VRAM consumption during training from 1.2TB to 350GB. With r = 4 and only the query and value projection matrices being adapted, the check- point size is reduced by roughly 10,000x (from 350GB to 35MB This allows to train with significantly fewer GPUs and avoid I/O bottlenecks. Another benefit is that we can switch between tasks while deployed at a much lower cost by only swapping the LoRA weights
opposed to all the parameters. This allows for the creation of many customized models that can be swapped in and out on the fly on machines that store the pre-trained weights in VRAM. We also observe a 25% speedup during training on GPT-3 175B compared to full fine-tuning 51 as we do not need to calculate the gradient for the vast majority of the parameters. See Appendix for how to calculate gradient when using LoRA.
LoRA also has its limitations. For example, it not straightforward to batch inputs to different tasks with different A and B in a single forward pass, if one chooses to absorb A and B into W to eliminate additional inference latency. Though it is possible to not merge the weights and dynamically choose the LoRA modules to use for samples in a batch for scenarios where latency is not critical.
5 EMPIRICAL EXPERIMENTS
We evaluate the downstream task performance of LoRA on RoBERTa (Liu et al. 2019). De- BERTa (He et al. 2021), and GPT-2 (Radford et al. b). before scaling up to GPT-3 175B (Brown) et al. 2020). Our experiments cover a wide range of tasks, from natural language understanding (NLU) to generation (NLG). Specifically,
evaluate on the GLUE (Wang et al. 2019) benchmark for RoBERTa and DeBERTa. We follow the setup of Li & Liang 2021) on GPT-2 for a direct com- parison and add WikiSQI. (Zhong al. 2017) (NI. to SQL queries) and SAMSum (Gliwa al., 2019) (conversation summarization) for large-scale experiments on GPT-3. See Appendix D for more details on the datasets we use. We use NVIDIA Tesla V100 for all experiments.
5.1 BASELINES
To compare with other baselines broadly, we replicate the setups used by prior work and reuse their reported numbers whenever possible. This, however, means that some baselines might only appear in certain experiments. We point out that for most tasks, our goal is not to compete with SOTA because 1) for tasks such as WikiSQL, SOTA uses specialized SQL-aware decoders, and: 2) SOTA makes frequent use of techniques such as ensembling and adversarial training, which might confound our study on adaptation. Thus, we focus on a scenario where we are given a powerful pre-trained autoregressive language model and use as few task-specific modifications as possible. See Appendix E for how we count the number of trainable parameters for different baselines.
Fine-Tuning (FT) is common approach for adaptation. During fine-tuning, the model is initialized to the pre-trained weights and biases, and all model parameters undergo gradient updates.A simple variant is to update only some layers while freezing others. We include one such baseline reported in prior work LL & Liang 2021) on GPT-2, which adapts just the last two layers (FT2).
WWe still need the 350GB model during deployment; however, storing 100 adapted models only requires 350GB 35MB 100 354GB as opposed to 100 350GB as 35TB. 'For GPT-3 175B, the training throughput for full fine-tuning is 32.5 tokens/s per V100 GPU; with the same number of weight shards for model parallelism, the throughput is 43.1 tokens/s per V100 GPU for LoRA.
5

---

Published as a conference paper at ICLR 2022
Model Method
# Trainable Parameters
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
RoBree (FT)*
125.0M
87.6
94.8
90.2
63.6
92.8
91.9
78.7
91.2
86.4
RoB
(BitFit)**
0.IM
84.7
93.7
92.7
62.0
91.8
84.0
81.5
90.8
85.2
RoBrse (Adpt"ye
0.3M
87.1
94.221
88.5
60.8
93.1..1
90.2
71.5
89.7
84.4
RoB (Adpt"y
0.9M
94.723
88.4
62.619
93.0=a
90.6
25.9
90.3
85.4
RoBree
(LoRA)
0.3M
87
95.11.2
89.7
63.4
93.32.a
90.8
86.61.7
91.5
87.2
RaBe (FT)*
355.0M
90.2
96.4
90.9
68.0
94.7
92.2
86.6
92.4
88.9
RaBa (LoRA)
0.8M
90.6
96.2..
90.9
68.2
94.9
91.62.1
87.4
92.6
89.0
RoBaagg (Adpt")
3.0M
90.2...
9..1
90.2+.>
68.3
94.8
91.9
83.8
92.1
88.4
RoBuug (Adpf"f
0.8M
90.5
96.64.2 89.7+11
67.8
94.8..3
91.7+ 80.1
91.9+.4
87.9
RoBurge (Adpt")
6.0M
89.9
96.2+3 88.7429
66.5
94.7
92.1 83.4
91.0+117
87.8
RoBuung (Adpt")) )
0.8M
90.3
96.34.5 87.7
66.3
94.7
2.2 91.5...1 72.9429
91.5
86.4
RoBiagg (LoRA)f
0.8M
90.6
96.22
90.2+10
68.2219
94.8= 91.6 85.2
92.32.5
88.6
DeBXXL (FT)* DeBxXL (LoRA)
1500.0M
91.8
97.2
92.0
72.0
96.0
92.7
93.9
92,9
91.1
4.7M
91.9
96.94.2 92.6+
72.4
96.0#.1 92.9 94.9+
93.0
91.3
Table 2: RoBERTatuse RoBERTanagee and DeBERTaxxI. with different adaptation methods the GLUE benchmark. We report the overall (matched and mismatched) accuracy for MNLI, Matthew's correlation for CoLA, Pearson correlation for STS-B, and accuracy for other tasks. Higher is better for all metrics. indicates numbers published in prior works. f indicates runs configured in a setup similar Houlsby al. (2019) for a fair comparison.
Bias-only or BitFit a baseline where we only train the bias vectors while freezing everything else. Contemporarily, this baseline has also been studied by BitFit (Zaken al., 2021).
Prefix-embedding tuning (PreEmbed) inserts special tokens among the input tokens. These spe- cial tokens have trainable word embeddings and are generally not in the model's vocabulary. Where to place such tokens can have an impact on performance. We focus on "prefixing", which prepends such tokens to the prompt. and "infixing", which appends to the prompt; both are discussed in Li & Liang (202I).
Prefix-layer tuning (PreLayer) is an extension prefix-embedding tuning. Instead of just learning the word embeddings (or equivalently, the activations after the embedding layer) for some special tokens, learn the activations after every Transformer layer. The activations computed from pre- vious layers are simply replaced by trainable ones.
Adapter tuning as proposed in Houlsby al. (219)) inserts adapter layers between the self- attention module (and the MIP module) and the subsequent residual connection. There are two fully connected layers with biases in an adapter layer with nonlinearity in between. We call this original design Adapter". Recently, Lin et al. (2200) proposed a more efficient design with the adapter layer applied only after the MLP module and after LayerNorm. We call it Adapter'. This is very similar to another deign proposed in Pfeiffer et al. (2021). which call Adapter". We also include another baseline call DropterDrop (Rücklé et al. 2020) which drops some adapter layers for greater efficiency (Adapter"). We cite numbers from prior works whenever possible to maximize the number of baselines we compare with; they are in rows with an asterisk (*) in the first column.
LoRA adds trainable pairs of rank decomposition matrices in parallel to existing weight matrices. As mentioned in Section 4.2 we only apply LoRA W,, and W, in most experiments for simplicity.
5.2 ROBERTA BASE/LARGE
RoBERTa (Liu et al. 2019) optimized the pre-training recipe originally proposed in BERT (Devlin) et al. 2019a) and boosted the latter's task performance without introducing many more trainable parameters. While RoBERTa has been overtaken by much larger models on NLP leaderboards such as the GLUE benchmark (Wang et al., 2019) in recent years, it remains competitive and popular pre-trained model for its size among practitioners. We take the pre- traitra RoBERTa base (125M) and RoBERTa large (355M) from the HuggingFace Transformers library (Wolf al. 2020

---

Published as a conference paper at ICLR 2022
Model Method
Trainable Parameters
E2E NLG Challenge
BLEU
NIST
MET
ROUGE-L
CIDEr
GPT-2 M (FT)*
354.92M
68.2
8.62
46.2
71.0
2.47
GPT-2 M (Adapter")e
0.37M
66.3
8.41
45.0
69.8
2.40
GPT-2 M (Adapter")e
11.09M
68.9
8.71
46.1
71.3
2.47
GPT-2 M (Adapter*)
11.09M
67.3
8.50+.07
46.0+.2
70.7
2.44
GPT-2 M (FT
25.19M
68.1
8.59
46.0
70.8
2.41
GPT-2 M (PreLayer)*
0.35M
69.7
8.81
46.1
71.4
2.49
GPT-2 M (LoRA)
0.35M
70.4
8.85
46.8
71.8
2.53
GPT-2 I. (FT)*
774.03M
68.5
8.78
46.0
69.9
2.45
GPT-2 L (Adapter*)
0.88M
69.1
8.68
46.3
71.4
2.49
GPT-2 L (Adapter')
23.00M
68.9
8.70+.04
46.1
71.3
2.45 +.02
GPT-2 L (PreLayer)*
0.77M
70.3
8.85
46.2
71.7
2.47
GPT-2 (LoRA)
0.77M
70.4
1.89
46.8
72.0
2.47 1.02
Table 3: GPT-2 medium (M) and large (L) with different adaptation methods on the E2E NLG Challenge. For metrics, higher is better. LoRA outperforms several baselines with comparable or fewer trainable parameters. Confidence intervals are shown for experiments numbers published in prior works.
'indicates
and evaluate the performance of different efficient adaptation approaches on tasks from the GLUE benchmark. We also replicate Houlsby al. (2119) and Pfeiffer et al. (2021) according to their setup. To ensure a fair comparison, we make two crucial changes to how we evaluate LoRA when comparing with adapters. First, use the same batch size for all tasks and of 128 to match the adapter baselines. Second, we initialize the model to the pre- pre-trained model for MRPC, RTE, and STS-B, following this more restricted setup from HHoulsby et al. (2019) are labeled with t- The result is presented in Table (Top Three Sections). See Section F. for details on the hyperparameters used.
sequence length
model already adapted to MNLI like the fine-tuning baseline. Runs
5.3 DEBERTA XXL
DeBERTa (He et al. 2021 is more recent variant of BERT that is trained much larger scale and performs very competitively on benchmarks such as GLUE (Wang et al. 2019) and Su- perGLUE (Wang et al. 2020). We evaluate if LoRA can still match the performance of fully fine-tuned DeBERTa XXL (I.5B) on GLUE. The result is presented in Table 2 (Bottom Section). See [Section F22 for details the hyperparameters used.
5.4 GPT-2 MEDIUM/LARGE
Having shown that LoRA can be a competitive alternative full fine-tuning on NLU, we hope to answer if LoRA still prevails on NLG models, such GPT-2 medium and large (Radford et al.) (). We keep our setup as close as possible Li & Liang (202T) for a direct comparison. Due to space constraint, we only present our result on F2E NLG Challenge (Table 3) in this section. See (Section I.] for results on WebNLG (Gardent et al. 2017) and DART (Nan et al. 2020). We
include list of the hyperparameters used in Section F.3
5.5 SCALING TO GPT-3 175B
As final stress test for LoRA, scale up to GPT-3 with 175 billion parameters. Due to the high training cost, we only report the typical standard deviation for given task over random seeds, as opposed to providing one for every entry. See Section F.4 for details the hyperparameters used.
LoRA matches or exceeds the fine-tuning baseline on all three datasets. Note that not all methods benefit monotonically from having
As shown in Table 4
trainable parameters, shown in Fig- ure 2] We observe a significant performance drop when we use more than 256 special tokens for prefix-embedding tuning
than 32 special tokens for prefix-layer tuning. This corroborates

---

Published as a conference paper at ICLR 2022
Trainable Parameters
WikiSQL
MNLI-m
SAMSum
Model&Method
Acc. (%)
Acc. (%)
R1/R2/RI.
GPT-3 (FT)
175,255.8M
73.8
89.5
52.0/28.0/44 0/44.5
GPT-3 (BitFit)
14.2M
71.3
91.0
51.3/27 41.3357.443.55
GPT-3 (PreEmbed)
3.2M
63.1
88.6
2..3/24.2/40.
GPT-3 (PreLayer)
20.2M
70.1
89.5
50.8/27.3/433
GPT-3 (Adapter")
7.1M
71.9
89.8
53.0/28, 9/44,8
GPT-3 (Adapter")
40.1M
73.2
91.5
53.2/29.0/44
GPT-3 (LoRA)
4.7M
73.4
91.7
53.8/29.8/45.9
GPT-3 (LoRA)
37.7M
74.0
91.6
53.4/29.2/45.1
Table 4: Performance of different adaptation methods on GPT-3 175B. We report the logical form validation accuracy on WikiSQL, validation accuracy on MultiNL.-matched, and Rouge-1/2/L. on SAMSum. LoRA performs better than prior approaches, including full fine-tuning. The results on WikiSQL have fluctuation around +0.5%, MNLI-m around +0.1%, and SAMSum around +0.2/10.2//0.11 for the three metries.
similar observations in Li & Liang (021). While a thorough investigation into this phenomenon out-of-scope for this work, suspect that having more special tokens causes the input distri- bution to shift further away from the pre-training data distribution. Separately. we investigate the performance of different adaptation approaches the low-data regime in /Section 1.3
WikisQL
MultiNLL-matched
0.75
0.92
Validation Accuracy
0.70
0.90
Method
Fine-Tune
0.88
PrefixEmbed
0.60
PrefixLayer
X
Adapter(H)
0.55
0.84
10
10
11
Jogio Trainable Parameters
logro Trainable Parameters
Figure 2: GPT-3 175B validation accuracy vs. number of trainable parameters of several adaptation methods WikiSQL and MNLI-matched. LoRA exhibits better scalability and task performance. See Section 1.2 for more details on the plotted data points.
6 RELATED WORKS
Transformer Language Models. Transformer Vaswani et 2017) esquence-to-sequee architecture that makes heavy use of self-attention. Radford al. (a) applied to autoregressive lan- guage modeling by using a stack Transformer decoders. Since then, Transformer-basee language models have dominated NLP, achieving the saate-of-the-arr in many tasks. A new paradigm emerged with BERT (Devlin et al. 2019b) and GPT-2 (Radford et al. b) both are large Transformer lan- guage models trained training on general domain data provides significant performance gain compared to training on task-specific data directly. Training larger Transformers generally results in better performance and remains an active research direction. GPT-3 (Brown et al. (020)) is the largest single Transformer language model trained to-date with 175B parameters.
large amount of text where fine-tuning
task-specific data after pre-
Prompt Engineering and Fine-Tuning. While GPT-3 175B can adapt its behavior with just a few additional training examples, the result depends heavily the input prompt (Brown et al. 2020). This necessitates an empirical art composing and formatting the prompt to maximize model's performance desired task, which is known as prompt engineering or prompt hacking.
8

---

Published as a conference paper at ICLR 2022
Fine-tuning retrains model pre- tre-trained general domains to specific task Devlin al. (201 bb) Radford et al. (a). Variants of it include learning just a subset of the parameters Devlin et al. (001b))
Collobert & Weston (2008), yet practitioners often retrain all of them to maximize the downstream performance. However, the enormity of GPT-3 175B makes it challenging to perform fine-tuning in the usual way due to the large checkpoint produces and the high hardware barrier entry since it. has the same memory footprint as pre-training.
Crrameter-Efficienn Adaptation. Many have proposed inserting adapter layers between existing
layers in a neural network (Houlsby et al. 2019 Rebufti et al. 2017 Lin et al. 2020). Our method uses a similar bottleneck structure to impose a low-rank constraint on the weight updates. The key functional difference is that our learned weights can be merged with the main weights during inference, thus not introducing any latency, which is not the case for the adapter layers (Section 3). A comtenporary extension of adapter is COMPACTER (Mahabadi parametrizes the adapter layers using Kronecker products with some predetermined weight sharing scheme. Similarly, combining LoRA with other tensor product- based methods could potentially improve its parameter efficiency, which we leave to future work. More recently, many proposed optimizing the input word embeddings in lieu of fine-tuning, akin to a continuous and differentiable generalization of prompt engineering (Li & Liang] 202T] 2020 Liu et al. 202T). We include comparisons with & Liang (2021) in our experiment section. However, this line of works can only scale up by using more special tokens in the prompt, which take up available sequence length for task tokens when positional embeddings are learned.
al. 202T), which essentially
[02T]Lester Lester et al. [202] Hambardzumyan et al.)
Low-Rank Structures Deep Learning. Low-rank structure is very common in machine learn- ing. A lot of machine learning problems have certain intrinsic low-rank structure (Li et al.) 2016 Cai et al. 2010 Li et al., 2018b; Grasedyck et al. 2013). Moreover. it is known that for many deep learning tasks, heavily over-parametrized neural network will cxhibit low-rank properties af- ter training (Oymak et al. 2019). Some prior works even explicitly impose the low-rank constraint
when training the original neural network (Sainath et al. [0033] Povey et al.] 2018 Zhang et al.,
2014 Jaderberg et al. 2014| Zhao et al., | Khodak et al. 2021] Denil et al. [2014) however, the best of our knowledge, none of these works considers low-rank update to a frozen model for adaptation to downstream tasks. In theory literature, it is known that neural networks outper- form other classical learning methods, including the corresponding (finite-widdie neural tangent kernels (Allen-Zhu et al. 2019 Li & Liang 2018) when the underlying concept class has certain low-rank structure (Ghorbani et al. 2020 Allen-Zhu Li, 2019 Allen-Zhu Li [020a). Another theory result in AhllenZhu & Li (020b) suggests that low-rank adaptations can be useful for adver- sarial training. In sum, we believe that our proposed low-rank adaptation update is well-motivated by the literature.
7 CONCLUSION AND FUTURE WORK
Fine-tuning enormous language models is prohibitively expensive in terms of the hardware required and the storage/switching cost for hosting independent instances for different tasks. We propose LoRA, an efficient adaptation strategy that neither introduces inference latency nor reduces input sequence length while retaining high model quality. Importantly, it allows for quick switcswit when deployed as a service by sharing the vast majority of the model parameters. While we focused on Transformer language models, the proposed principles are generally applicable to any neural networks with dense layers.
There are many directions for future works. 1) LoRA can be combined with other efficient adapta- tion methods, potentially providing orthogonal improvement. 2) The mechanism behind fine-tuning or LoRA is far from clear - how are features learned during pre-training transformed to do well on downstream tasks? We believe that LoRA makes it more tractable to answer this than full fine- tuning. 3) We mostly depend on heuristics to select the weight matrices to apply LoRA to. Are there more principled ways to do it? 4) Finally, the maffdeefcy of A AW suggests that W could be rank-deficient as well, which inspire future works.

---

Published as a conference paper at ICLR 2022
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tume Tuning. arXiv:0012.13255 [cs], December 2020. URL
http://arxiv. org/ abs 2012. 13255
NewrlPS, 2019. Full version available at http:///aaaav...rrg///a/1//3555/ 10337 Zeyuan Allen-Zhu and Yuanzhi Li. What Can ResNet Learn Efficiently, Going Beyond Kernels? In
Zeyuan Allen-Zhu and Yuanzhi Li. Backward feature correction: How deep learning performs deep learning. arXiv preprint SBt::2001.044133 2020a.
Zeyuan Allen-Zhu and Yuanzhi Li. Feature purification: How adversarial training performs robust deep learning. arXiv preprint arXiv:2005.10190 2020b.
Zeyuan Allen-Zhu Yuanzhi Li, and Zhao Song. convergence theory for deep learning via over-
parameterization. In ICML, 2019, Full version available at atttsp:://anavvv.orr//bbb//////
03962
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization, 2016.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language Models are Few-Shot Learners. arXiv:2005. 14166 [cs], July 2020. URL Att::/tt:/ax.. org/ 2005.1416.......
Jian-Feng Cai, Emmanuel Candès, and Zuowei Shen. A singular value thresholding algorithm for matrix completion. SIAM Journal on optimization, 20(4)) 2010.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity multilingual and crosslingual focused evaluation. Proceedings of the Ith International Workshop on Semantic Evaluation (eemEvall2011), 2017. doi: 10.18653/
v1/s17-2001. URL //tt::://dxx doi erg/ 10. 1tt.1/16////111122220
Ronan Collobert and Jason Weston. unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of the 25th international conference
on Machine learning, ICML '08, pp. 160-167, New York, NY, USA, July 2008. Association doi: 191114551900156139999111 URL
for Computing Machinery. ISBN 978-1-60558 205-4.
htttps://do..rrg htt.:///.....1111111111111..1...
Misha Denil, Babak Shakibi, Laurent Dinh, Mare' Aurelio Ranzato, and Nando de Freitas. Predicting parameters in deep learning, 2014.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, 2019a.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep
Bidirectional Transformers for Language Understanding. arXiv:1810.04805 [cs], May 2019b. URL http::::/aaxx..oo//ppppot
11tes//111... 04805 arXiv: 1810.04805.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases.
In Proceedings of the Third International Workshop on Paraphrasing (/WP2005), 2005. URI.
https://aclanthol..o / Saa0////05550000
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura
The webnlg
challenge: Generating text from rdf data. In Proceedings of the 10th International Conference on Natural Language Generation, pp. 124-133, 2017.
Behrooz Ghorbani, Song Mei, Theodor Misiakiewicz, and Andrea Montanari. When do neural networks outperform kernel methods? arXiv preprint arXiv:2006. 13409, 2020.
10

---

Published as a conference paper at ICLR 2022
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. Samsum corpus: A human- annotated dialogue dataset for abstractive summarization. CoRR, abs/1911.12237, 2019. URL
http://arxiv. org/ abs abs/1911 12237
Lars Grasedyck, Daniel Kressner, and Christine Tobler. A literature survey of low-rank tensor approximation techniques. CAMM-Mitteilune 36(1):53-78, 2013.
Jihun Ham and Daniel D. Lee. Grassmann discriminant anallys:s a unifying view on subspace-based learning. In ICML, pp. 376-383, 2008. URL https:tt////mm //tp::://doi.. http:///ooo..g//.1111111100/1//1 1ttp//10.111/1139900 1390204
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. WARP: Word-level Adversarial
ReProgramming. (kKi:2/01.00122 [cs], December 2020. URL http:///arxv..org/aba. 2101. 00121
arXiv: 2101.00121.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention, 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe,
Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. arXiv:1902.0075I [cs, stat], June 2019. URL
00751
Max Jaderberg, Andrea Vedaldi, and Andrew Zisserman. Speeding up convolutional neural networks with low rank expansions. arXiv preprint 4rXiv:1405.3866 2014.
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolò Fusi. Initialization and regularization of factorized neural layers, 2021.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu. Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with conditional computation and automatic sharding, 2020.
Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficieen Prompt Tuning. 2eXiv:2/04.. 08691 [cs], April 2021. arXiv: 2104.08691.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the Intrinsic Di- mension of Objective Landscapes. arXiv:/804.088: 08838 (cs, star), April 2018a. URL htt.::
08838 arXiv: 1804.08838.
Xiang Lisa Li and Percy Liang. Prefix Tunnnn Optimizing Continuous Prompts for Generation. arXiv:2101.00190 [cs], January 2021.
wwgg/bs//211..00000 201. 90
Yuanzhi Li and Yingyu Liang. Learning overparameterizze neural networks via stochastic gradient descent on structured data. Advances in Neural Information Processing Systems, 2018
Yuanzhi Li, Yingyu Liang, and Andrej Risteski. Recovery guarantee of weighted low-rank ap- proximation via alternating minimization. International Conference on Machine Learning, pp. 2358-2367 PMLR, 2016.
Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning The- ory, pp. 4-. PMLR, 2018b.
Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efficien transfer learning. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 441-459 Online, November 2020. Association for Computational Linguistics. doi: 18.136/3/v //2/20 1t2020.findinn.....e 2ww....iningseemnll...... org/2020. f findi emm 1p.41
URL https: https////aaathoogg...
11

---

Published as a conference paper at ICLR 2022
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. GPT Understands, Too. arXiv:2103. 103. [cs], March 2021. URL http:://axxi...o// abs/ 2103. arXiv: 2103.10385.
Yinhan Liu, Myle Out Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. 2019.
Ilya Loshchilov and Frank Hutter. arXiv:1711.05101, 2017.
Decoupled weight decay regularization. arXiv preprint
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers, 2021.
Linyong Nan, Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, al. Dart: Open-domain structured data record to text generation. arXiv preprint 2007022007.0281, 2020.
Jekaterina Novikova, Ondfej Dušek, and Verena Rieser. The e2e dataset: New challenges for end- to-end generation. arXiv preprint arXiv: 170.0094469 09254, 2017.
Samet Oymak, Zalan Fabian, Mingehen Li, and Mahdi Soltanolkotabi. Generalization guaran- tees for neural networks via harnessing the low-rank structure of the jacobian. arXiv preprint 1oXiv:1906.05392, 2019.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning. 2021.
Daniel Povey, Gaofeng Cheng, Yiming Wang, Ke Li, Hainan Xu, Mahsa Yarmohammadi, and San- jeev Khudanpur. Semi- Semi-orthogonal low-rank matrix factorization for deep neural networks. In Interspeech, pp. 3743-3747, 2018.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving Language Under- standing by Generative Pre-Training. pp. 12, a.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language Models are Unsupervised Multitask Learners, pp. 24, b.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerabl quesiions for squad. CoRR, 006.0002..2..... 2018. /ttt/ttp://arxix http:///wwwi...g///////000.00../ 03822
Sylvestre-Alvise Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. arXiv: / 77755080455 [cs, stat), November 2017. URL http:://axxi...rrrr///// arXiv: 1705.08045.
abs/1705. 08045
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfciffer, Nils Reimers, and Iryna Gurevych. Adapterdrop: On the efficiency of adapters in transformers, 2020.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low- rank matrix factorization for deep neural network training with high-dimensional output targets. In 2013 IEEE international conference on acoustics, speech and signal processing, pp. 6655- 6659. IEEE, 2013.
Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Ims Training multi-billion parameter language models using model par- allelism, 2020.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1642, Seattle, Washington, USA, October 2013. Association for Computa- tional Linguistics. URL httphhtttt:/atte Utt//htgee///ecltttoooooorrrgrgg
12

---

Published as a conference paper at ICLR 2022
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proceedings of the 31st In- ternational Conference on Neural Information Processing Systems, pp. 6000-6010, 2017.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: stickier benchmark for general-purpont language understanding systems, 2020.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. arXiv preprint arXiv: 1805. 12471, 2018.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technolo- gies, Volume I (Long Papers), pp. 1112-1 1122, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: 10. 10.18653/v1/N18-I11 1101.
URL htthttt::/ www aclweb.
org/ hets/anthllooyy/ieee 1101
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gug- ger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. As- sociation for Computational Linguistics. URL httht:/:: www www.:.ccleeb.erg/aatnoo..oo
22200. www.p- demos.
Greg Yang and Edward Hu,
Feature Learning in Infinite-Width Neural Networks.
arXiv:2011. 2.2212201. arXiv: 2011.14522.
14522 [cond-mar], May 2021. httl:/tpp///arx..ooo////////////
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2021.
Yu Zhang, Ekapol Chuangsuwanich, and James Glass. Extracting deep neural network bottleneck features using low-rank matrix factorization. In 2014 IEEE international conference on acoustics, speech and signal processing (ICASSP), pp. 185 189. IEEE, 2014.
Yong Zhao, Jinyu Li, and Yifan Gong. Low-rank plus diagonal adaptation for deep neural networks. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5005-5009. IEEE, 2016.
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Generating structured queries from natural language using reinforcement learning. CoRR, abs/1709.0010/ 2017. URL http://
arxiv. wttp://g/aabs.....oo. 00103
13


Reviews:
[{"review_id": "xdraVP9ExCF", "reviewer": "Reviewer_TD94", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- Simple and efficient methods to utilize pre-trained models such as GPT-3 for model adaptation\n- Good performance and comparison with fine-tuning baselines. \n\nWeaknesses:\n- The idea is incremental. The additional parameters have a similar intuition as adapters.\n- Missing comparison to the SOTA models.\n- The training efficiency remains unclear. It would be useful if the authors could report the time during training and compare the latency to the baseline models.\n", "summary_of_the_paper": "The paper proposes a low-rank adaptation that compresses the pre-trained model weights by applying matrix decomposition (e.g., SVD) into large pre-trained models. The factorized pre-trained weights are frozen, and the small adaptation parameters are trainable during the finetuning stage. This method is very useful in the practical setting. They showed the effectiveness of applying this method in various downstream tasks by adding small adaptation parameters.", "summary_of_the_review": "Justification:\n\nI would say that the idea is relevant for achieving efficient pre-trained models that will be useful for many practical applications where latency is crucial. The paper is also well-written, with numerous experiments to support the claim. Therefore, I think the claim is supported with empirical results; however, the paper's novelty is limited. The time efficiency still needs to be supported by training and inference time information. The approach is incremental, very similar to Adapters; adding small trainable parameters to the model.\n\nTypographical errors:\n- comtenporary => contemporary\n\nReferences related to Matrix decomposition by design (please consider to add):\n- Compressing Pre-trained Language Models by Matrix Decomposition\n- Greenformer: Factorization toolkit for efficient deep neural networks\n- Factorization Tricks For LSTM Networks\n- Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer\n\nQuestions:\n- What is the optimal rank for the factorized matrices? I was wondering to know if you have findings on choosing the optimal rank in the experiment.\n\nI would be happy to increase my score if the authors address my concerns.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Qx_SNb6d8xu", "reviewer": "Reviewer_jHjA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n- The idea of using this parallel low-ranked path to perform efficient fine-tuning is novel (to the best of my knowledge) and interesting. To elaborate, LoRA allows to simply add the low-ranked representation to the original weights, so to avoid further computation during inference, and this is possible since the model is trained in this way (by construction), check Eq 3. This is definitely a clever trick that allows not only an efficient inference but also efficient training since the optimizer (e.g., Adam) only need to store the gradients of the low-ranked parameters rather than the whole model (check the \"Practical Benefits and Limitations.\" paragraph, Page 5). \n- A large set of experiments (both in NLG and NLU) strongly supports the claim made by the authors. \n- The paper is clearly motivated and covers a large amount of previous work. I especially enjoyed Section 3. \n\nMinor weaknesses\n- The training efficiency of LoRA is a bit unclear. In \"Practical Benefits and Limitations.\" the authors mention a 2/3 reduction in VRAM by not storing the gradients of the previous steps in Adam, which is true but it is important to mention that the gradient of the original weights is still needed (although not stored) at the current optimization step to compute the gradient for the low-ranked weights (if I understood correctly your method). Anyhow, this step is needed also by other efficient fine-tuning methods (e.g. Adapter and Prompt-Tuning) and can be very costly in very large models (e.g., GPT-3). I would encourage the authors to provide a discussion about the training efficiency.\n- The authors did not report any baselines outside of the efficient fine-tuning setting. This is important for the reader to understand how far an efficient model is compared to the current SOTA. By no means, the reader, and me, expect new SOTA, but it is good to have these references in the table. ", "summary_of_the_paper": "In this paper, the authors propose a LoRA a lightweight fine-tuning method for language models (LMs). LoRA achieves similar or better performance than full-model fine-tuning while adding only 0.01-0.05% of the original parameters and freezing the original weight of the LM, it is efficient in training and it does not add further computational cost during inference, as required by other methods. \n\nThe central idea is to only learn a parallel set of low-ranked parameters (Figure 1) rather than fine-tuning the entire model. To elaborate, the authors propose to 1) freeze the original LM, 2) create a parallel set of low-ranked parameters ($W=BA$ where $B\\in\\mathcal{R}^{d\\times r}$ and $A\\in\\mathcal{R}^{r\\times k}$) for every parameter in the original model, 3) training the model by only updating the low-ranked parameters where each hidden state is computed as $W_0 x + BA x$ (Eq. 3, $W_0\\in\\mathcal{R}^{d\\times k}$ are the original frozen parameter of the model), and 4) test the model using the new set of parameters computed as $W_0 + BA$ (thus no additional computation at inference time). \n\nThe authors evaluate LoRA on both NLU (GLUE) and NLG (text-to-SQL, Dialogue Summary, E2E-NLG) tasks by comparing many models (Roberta, DeBerta, GPT-*) under different training settings and baselines such as full-finetuning, Bit-fit (only training bias vectors), prefix tuning methods, and Adapter methods. The results show that LoRA outperforms other efficient fine-tuning methods and often full-model finetuning, by only using a tiny number of parameters compared to the original model (Table 3/4). Moreover, LoRA does not add any inference overhead, as shown in Table 1, compared to other methods (e.g, Adapter), which required further computation. \n ", "summary_of_the_review": "The authors present an efficient fine-tuning method for LMs that freeze the original model and tune only an additional 0.01% of parameters. This method is efficient both in training and inference, and achieve comparable or better performance than full-model fine-tuning. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "HVwpvt4s3u2", "reviewer": "Reviewer_Vqdv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think the paper presents an interesting idea and has the potential to be impactful. But its execution can be significantly improved:\n- The paper overclaims the benefit of low-rank finetuning without enough experiments to back it up.  LoRA only finetunes W_q and W_v. One missing baseline is a full-rank finetuning model, which only updates W_q and W_v. The memory/parameter efficiency of LoRA is exaggerated.\n- Thorough proof reading is needed. Please see below for a long, but by no means exhaustive list of typos.-\n- One may argue that LoRA can take more training steps to converge due to low-rank updates. So it would be great to compare learning curves and training time overhead.\n\nDetails:\n- What does VRAM stand for? If it is GPU memory, then the 2/3 saving of training memory is quite surprising to me—LoRA still needs to build the forward and backward computation graphs for the whole model. Could the authors clarify?\n- A (probably more general) variant of LoRA would be, instead of learning A and B, one forces the gradients to be of low-rank. So accumulating multiple low-rank steps could still yield a higher-rank update to the parameters. Have the authors considered this?\n\nTypos, writing suggestions:\n- “Reduce something by 10K/3 times” is strange. Please consider changing it to reduce by a factor of 10K, or reduce by 2/3\n- Please P(X), P(X = x), and p(x).\n- Whenever the paper talks about autoregressive language models, the math is never a language model.\n- A non-issue -> not an issue\n- Less performant -> underperform\n- These layers have full-rank -> these layer have full ranks.\n- An inevitability -> Inevitable", "summary_of_the_paper": "This paper presents LoRA, a method aiming to improve the efficiency of fine-tuning large language models. Specifically, it fixes the underlying pretrained model, and learn low-rank (by construction) parameter matrices to update the model. In this way, the amount of parameters to finetune is significantly reduced if the rank is small. Compared to other finetuning methods, LoRA performs similarly or better, with improved memory and parameter efficiency.\n", "summary_of_the_review": "Se main review", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-7fUFarlyFj", "reviewer": "Reviewer_i5pN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper is well motivated and easy to follow. Although the idea of using low-rank approximation of neural networks is not new, this work is the first to propose adding low-rank matrices as the task-specific parameters in the pretrained models. The benefits of this approach are obvious. It introduces no additional computation in single-task inference compared to adapters and prompt-based methods, and is much more training efficient than full model finetuning. Since the method works for any parameter matrix, it can be generally applicable to most of the neural network models. Experiments also demonstrate its efficacy on several pretrained language models and on both NLU and NLG tasks.\n\nOne limitation of this method (also mentioned in the paper) is how the model works for multi-task serving setting, which is possibly the most common scenario in deploying large-scale pretrained models. Since LoRA is able to meet this requirement by not merging the pretrained weights (W_0) and low-rank weights (BA), it will be interesting if the author can report the comparison results of inference latency among (no-merging) LoRA and other methods. Moreover, with regard to training efficiency, it is better to compare with adapters and prompt/prefix-tuning with the same amount of trainable parameters. Another line of work on mixture of experts (MoE) also shows promising results on mult-task/domain training (e.g., DEMix Layers: https://arxiv.org/abs/2108.05036), where we can selectively train expert modules for specific downstream task (while freezing the rest parameters). It will be interesting to see how RoLA compares with it.\n", "summary_of_the_paper": "This paper proposes a low-rank adaptation method LoRA for application of pretrained models for downstream tasks. Each downstream task is associated with a set of trainable low-rank matrices that can be added to the pretrained parameters. Since the amount of low-rank parameters are orders of magnitude smaller than that of the pretrained parameters, the finetuning cost as well as the inference latency of LoRA is lower than existing methods including model finetuning, adpaters, and prompt tuning. Experiments on several pretrained models (RoBERTa, DeBERTa, GPT-2, GPT-3) show that RoLA achieves comparable or even better performances than existing methods.\nThe proposed method is simple and effective in adapting pretrained models to downstream tasks, which could become one of the major solutions among the finetuning, adapters and promp-tuning. The experiments are comprehensive, showing that LoRA works well in different types of pretrained models and tasks. \n", "summary_of_the_review": "This work proposes to add parameters of low-rank matrices into pretrained model, which are tuned for specific downstream task. The method is effective under different types of pretrained models and tasks, and benefits the deployment of large-scale models. My overall recommendation to this paper is accept.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a new method for fine-tuning large language models, which is lightweight since it only adds a small amount of parameters, while keeping the original parameters frozen. The main idea is to add a low rank matrix which is learned during fine-tuning to the original weight matrices of the model, which are frozen. The reviewers agreed that the method is simple, original and well motivated. Moreover, it compares well compared to other fine-tuning baselines, such as adaptors or full fine-tuning. For these reasons, I recommend to accept this work to the ICLR conference.
Decision: Accept (Poster)