Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents LoRA, a method aiming to improve the efficiency of fine-tuning large language models. Specifically, it fixes the underlying pretrained model, and learn low-rank (by construction) parameter matrices to update the model. In this way, the amount of parameters to finetune is significantly reduced if the rank is small. Compared to other finetuning methods, LoRA performs similarly or better, with improved memory and parameter efficiency.

main_review: I think the paper presents an interesting idea and has the potential to be impactful. But its execution can be significantly improved:
- The paper overclaims the benefit of low-rank finetuning without enough experiments to back it up.  LoRA only finetunes W_q and W_v. One missing baseline is a full-rank finetuning model, which only updates W_q and W_v. The memory/parameter efficiency of LoRA is exaggerated.
- Thorough proof reading is needed. Please see below for a long, but by no means exhaustive list of typos.-
- One may argue that LoRA can take more training steps to converge due to low-rank updates. So it would be great to compare learning curves and training time overhead.

Details:
- What does VRAM stand for? If it is GPU memory, then the 2/3 saving of training memory is quite surprising to me—LoRA still needs to build the forward and backward computation graphs for the whole model. Could the authors clarify?
- A (probably more general) variant of LoRA would be, instead of learning A and B, one forces the gradients to be of low-rank. So accumulating multiple low-rank steps could still yield a higher-rank update to the parameters. Have the authors considered this?

Typos, writing suggestions:
- “Reduce something by 10K/3 times” is strange. Please consider changing it to reduce by a factor of 10K, or reduce by 2/3
- Please P(X), P(X = x), and p(x).
- Whenever the paper talks about autoregressive language models, the math is never a language model.
- A non-issue -> not an issue
- Less performant -> underperform
- These layers have full-rank -> these layer have full ranks.
- An inevitability -> Inevitable
summary_of_the_review: Se main review
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a low-rank adaptation method LoRA for application of pretrained models for downstream tasks. Each downstream task is associated with a set of trainable low-rank matrices that can be added to the pretrained parameters. Since the amount of low-rank parameters are orders of magnitude smaller than that of the pretrained parameters, the finetuning cost as well as the inference latency of LoRA is lower than existing methods including model finetuning, adpaters, and prompt tuning. Experiments on several pretrained models (RoBERTa, DeBERTa, GPT-2, GPT-3) show that RoLA achieves comparable or even better performances than existing methods.
The proposed method is simple and effective in adapting pretrained models to downstream tasks, which could become one of the major solutions among the finetuning, adapters and promp-tuning. The experiments are comprehensive, showing that LoRA works well in different types of pretrained models and tasks. 

main_review: This paper is well motivated and easy to follow. Although the idea of using low-rank approximation of neural networks is not new, this work is the first to propose adding low-rank matrices as the task-specific parameters in the pretrained models. The benefits of this approach are obvious. It introduces no additional computation in single-task inference compared to adapters and prompt-based methods, and is much more training efficient than full model finetuning. Since the method works for any parameter matrix, it can be generally applicable to most of the neural network models. Experiments also demonstrate its efficacy on several pretrained language models and on both NLU and NLG tasks.

One limitation of this method (also mentioned in the paper) is how the model works for multi-task serving setting, which is possibly the most common scenario in deploying large-scale pretrained models. Since LoRA is able to meet this requirement by not merging the pretrained weights (W_0) and low-rank weights (BA), it will be interesting if the author can report the comparison results of inference latency among (no-merging) LoRA and other methods. Moreover, with regard to training efficiency, it is better to compare with adapters and prompt/prefix-tuning with the same amount of trainable parameters. Another line of work on mixture of experts (MoE) also shows promising results on mult-task/domain training (e.g., DEMix Layers: https://arxiv.org/abs/2108.05036), where we can selectively train expert modules for specific downstream task (while freezing the rest parameters). It will be interesting to see how RoLA compares with it.

summary_of_the_review: This work proposes to add parameters of low-rank matrices into pretrained model, which are tuned for specific downstream task. The method is effective under different types of pretrained models and tasks, and benefits the deployment of large-scale models. My overall recommendation to this paper is accept.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a low-rank adaptation that compresses the pre-trained model weights by applying matrix decomposition (e.g., SVD) into large pre-trained models. The factorized pre-trained weights are frozen, and the small adaptation parameters are trainable during the finetuning stage. This method is very useful in the practical setting. They showed the effectiveness of applying this method in various downstream tasks by adding small adaptation parameters.
main_review: Strengths:
- Simple and efficient methods to utilize pre-trained models such as GPT-3 for model adaptation
- Good performance and comparison with fine-tuning baselines. 

Weaknesses:
- The idea is incremental. The additional parameters have a similar intuition as adapters.
- Missing comparison to the SOTA models.
- The training efficiency remains unclear. It would be useful if the authors could report the time during training and compare the latency to the baseline models.

summary_of_the_review: Justification:

I would say that the idea is relevant for achieving efficient pre-trained models that will be useful for many practical applications where latency is crucial. The paper is also well-written, with numerous experiments to support the claim. Therefore, I think the claim is supported with empirical results; however, the paper's novelty is limited. The time efficiency still needs to be supported by training and inference time information. The approach is incremental, very similar to Adapters; adding small trainable parameters to the model.

Typographical errors:
- comtenporary => contemporary

References related to Matrix decomposition by design (please consider to add):
- Compressing Pre-trained Language Models by Matrix Decomposition
- Greenformer: Factorization toolkit for efficient deep neural networks
- Factorization Tricks For LSTM Networks
- Lightweight and Efficient End-to-End Speech Recognition Using Low-Rank Transformer

Questions:
- What is the optimal rank for the factorized matrices? I was wondering to know if you have findings on choosing the optimal rank in the experiment.

I would be happy to increase my score if the authors address my concerns.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In this paper, the authors propose a LoRA a lightweight fine-tuning method for language models (LMs). LoRA achieves similar or better performance than full-model fine-tuning while adding only 0.01-0.05% of the original parameters and freezing the original weight of the LM, it is efficient in training and it does not add further computational cost during inference, as required by other methods. 

The central idea is to only learn a parallel set of low-ranked parameters (Figure 1) rather than fine-tuning the entire model. To elaborate, the authors propose to 1) freeze the original LM, 2) create a parallel set of low-ranked parameters ($W=BA$ where $B\in\mathcal{R}^{d\times r}$ and $A\in\mathcal{R}^{r\times k}$) for every parameter in the original model, 3) training the model by only updating the low-ranked parameters where each hidden state is computed as $W_0 x + BA x$ (Eq. 3, $W_0\in\mathcal{R}^{d\times k}$ are the original frozen parameter of the model), and 4) test the model using the new set of parameters computed as $W_0 + BA$ (thus no additional computation at inference time). 

The authors evaluate LoRA on both NLU (GLUE) and NLG (text-to-SQL, Dialogue Summary, E2E-NLG) tasks by comparing many models (Roberta, DeBerta, GPT-*) under different training settings and baselines such as full-finetuning, Bit-fit (only training bias vectors), prefix tuning methods, and Adapter methods. The results show that LoRA outperforms other efficient fine-tuning methods and often full-model finetuning, by only using a tiny number of parameters compared to the original model (Table 3/4). Moreover, LoRA does not add any inference overhead, as shown in Table 1, compared to other methods (e.g, Adapter), which required further computation. 
 
main_review: Strengths
- The idea of using this parallel low-ranked path to perform efficient fine-tuning is novel (to the best of my knowledge) and interesting. To elaborate, LoRA allows to simply add the low-ranked representation to the original weights, so to avoid further computation during inference, and this is possible since the model is trained in this way (by construction), check Eq 3. This is definitely a clever trick that allows not only an efficient inference but also efficient training since the optimizer (e.g., Adam) only need to store the gradients of the low-ranked parameters rather than the whole model (check the "Practical Benefits and Limitations." paragraph, Page 5). 
- A large set of experiments (both in NLG and NLU) strongly supports the claim made by the authors. 
- The paper is clearly motivated and covers a large amount of previous work. I especially enjoyed Section 3. 

Minor weaknesses
- The training efficiency of LoRA is a bit unclear. In "Practical Benefits and Limitations." the authors mention a 2/3 reduction in VRAM by not storing the gradients of the previous steps in Adam, which is true but it is important to mention that the gradient of the original weights is still needed (although not stored) at the current optimization step to compute the gradient for the low-ranked weights (if I understood correctly your method). Anyhow, this step is needed also by other efficient fine-tuning methods (e.g. Adapter and Prompt-Tuning) and can be very costly in very large models (e.g., GPT-3). I would encourage the authors to provide a discussion about the training efficiency.
- The authors did not report any baselines outside of the efficient fine-tuning setting. This is important for the reader to understand how far an efficient model is compared to the current SOTA. By no means, the reader, and me, expect new SOTA, but it is good to have these references in the table. 
summary_of_the_review: The authors present an efficient fine-tuning method for LMs that freeze the original model and tune only an additional 0.01% of parameters. This method is efficient both in training and inference, and achieve comparable or better performance than full-model fine-tuning.
review_title: null

Meta-review: This paper introduces a new method for fine-tuning large language models, which is lightweight since it only adds a small amount of parameters, while keeping the original parameters frozen. The main idea is to add a low rank matrix which is learned during fine-tuning to the original weight matrices of the model, which are frozen. The reviewers agreed that the method is simple, original and well motivated. Moreover, it compares well compared to other fine-tuning baselines, such as adaptors or full fine-tuning. For these reasons, I recommend to accept this work to the ICLR conference.
Decision: Accept (Poster)