Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

Authors: Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, Xiaolin Huang
Primary area: optimization
Keywords: low-rank adaption, flat minima, efficient training

Abstract:
Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computational and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tuning (PEFT) method, provides an efficient way to fine-tune models by optimizing only a low-rank matrix. Despite recent progress made in improving LoRA's performance, the connection between the LoRA optimization space and the original full parameter space is often overlooked. A solution that appears flat in the LoRA space may exist sharp directions in the full parameter space, potentially harming generalization performance. In this paper, we propose Flat-LoRA, an efficient approach that seeks a low-rank adaptation located in a flat region of the full parameter space. Instead of relying on the well-established sharpness-aware minimization approach, which can incur significant computational and memory burdens, we utilize random weight perturbation with a Bayesian expectation loss objective to maintain training efficiency and design a refined perturbation generation strategy for improved performance. Experiments on natural language processing and image classification tasks with various architectures demonstrate the effectiveness of our approach.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730559259543

thread_type:
official_review

<thread_object>
creator: reviewer_ngbn

<content>
summary:
This paper discusses how to improve the generalization of LoRA, with good writing, innovative ideas, and credible experimental performance. However, I think the article overlooks one point: LoRA only fine-tunes a very small number of parameters, which is unlikely to overfit on a specific task and can inherently maintain good generalizability. If I am wrong, please convince me.

strengths:
1. The paper introduces Flat-LoRA, a novel method that improves upon traditional Low-Rank Adaptation (LoRA) techniques by targeting a flat region of the loss landscape. This aims to enhance generalization performance by avoiding sharp minima that can degrade model performance on unseen data.

2. Flat-LoRA addresses the computational and memory cost issues associated with fine-tuning large-scale models. By optimizing only a low-rank matrix and using random weight perturbations, it achieves parameter-efficient fine-tuning without additional memory costs during inference.

3. The paper not only proposes a new technique but also details the underlying mathematical foundations. It discusses the optimization objective, perturbation generation, and integrates a Bayesian expected loss objective to maintain training efficiency.

weaknesses:
1. There is no significant improvement, the limited enhancement on the GLUE benchmark does not prove the lack of generalizability of methods like LoRA.  I need to know if your method is effective on more datasets. More datasets and models should be compared. For example: the SuperGLUE benchmark, SQuAD, XSum, CNN/Dailymail, and some LoRA training on Stable Diffusion.

2. There is a lack of extensive comparisons, such as with methods like DyLoRA[1] and AdaLoRA[2]. 

3. More relevant articles, such as AWP[3], should be cited, which is an effective method to improve generalization.

[1] Valipour M, Rezagholizadeh M, Kobyzev I, et al. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation[J]. arXiv preprint arXiv:2210.07558, 2022.
[2] Zhang Q, Chen M, Bukharin A, et al. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning[J]. arXiv preprint arXiv:2303.10512, 2023.
[3] Wu D, Xia S T, Wang Y. Adversarial weight perturbation helps robust generalization[J]. Advances in neural information processing systems, 2020, 33: 2958-2969.

questions:
1. How does it perform on the SuperGLUE benchmark, SQuAD, XSum, and CNN/Dailymail?
2. How does it perform on Stable Diffusion?
3. How does it compare between AdaLoRA and DyLoRA?
4. How is the performance on Llama3 with the alpaca dataset?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614958100

thread_type:
official_review

<thread_object>
creator: reviewer_pg6m

<content>
summary:
This paper introduces Flat-LORA, which adds noise to the training process: W + AB + ε, where ε is stored using a seed. The authors attempt to achieve SAM-like effects through this method.

Advantages:
1. The paper is written clearly
2. The core idea seems reasonable

Disadvantages:
1. The paper lacks mathematical rigor in several places, particularly in equations 8 and 9
2. Insufficient experimental validation:
   - Should test on larger models (e.g., LLAMA 13B or 70B)
   - Should evaluate on SuperGLUE
3. Lacks necessary ablation studies, particularly regarding σ^2

Technical Issues:
1. Equations 8 and 9 have fundamental problems:
   - var(X) should be a covariance matrix
   - var(W'_{i,:}X) should be a scalar
   - These dimensions are inconsistent and cannot be equated
2. In Equation 7, n should be sqrt(n), as large n values would result in negligibly small epsilon values added to the weight matrix

Additional Concerns:
1. No memory usage results are reported
2. Table 3 lacks full-tuning baseline results
3. this paper should report results on more diverse datasets
4. More comprehensive ablation studies on σ^2 are needed

strengths:
Advantages:

The paper is written clearly
The core idea seems reasonable

weaknesses:
Disadvantages:

The paper lacks mathematical rigor in several places, particularly in equations 8 and 9
Insufficient experimental validation:
Should test on larger models (e.g., LLAMA 13B or 70B)
Should evaluate on SuperGLUE
Lacks necessary ablation studies, particularly regarding σ^2
Technical Issues:

Equations 8 and 9 have fundamental problems:
var(X) should be a covariance matrix
var(W'_{i,:}X) should be a scalar
These dimensions are inconsistent and cannot be equated
In Equation 7, n should be sqrt(n), as large n values would result in negligibly small epsilon values added to the weight matrix
Additional Concerns:

No memory usage results are reported
Table 3 lacks full-tuning baseline results
this paper should report results on more diverse datasets
More comprehensive ablation studies on σ^2 are needed

questions:
no
</content>
</thread_object>

</thread>
<thread>
creation date:
1730651319984

thread_type:
official_review

<thread_object>
creator: reviewer_t8en

<content>
summary:
Based on the consideration of local optimal values, this paper proposes that the flat optimal value learned by LoRA is not necessarily flat at full rank. Corresponding mathematical explanations are provided. Following this, using the idea of SAM, a method is designed to add full-rank noise perturbations to search for a global optimal solution.

strengths:
1. The writing is well done, with motivations and insights clearly explained in an intuitive manner, accompanied by reasonable mathematical assumptions to introduce the design of full-rank noise perturbations.
2. The design for storing random seeds for memory efficiency and integrating into mixed-precision training is very clever, saving additional overhead.

weaknesses:
1. On line 125, the sentence "the LoRA matrices. better initialization strategy for LoRA parameters." should probably use a comma instead of a period.
2. The work on adding full-rank noise has been done in Noisetune ([https://arxiv.org/abs/2202.12024](https://arxiv.org/abs/2202.12024)) and LORASC ([https://arxiv.org/abs/2407.01491](https://arxiv.org/abs/2407.01491)), especially in LoRASC, where the exact same approach is used, adding a full rank noise to each LoRA optimization process. 
3. Overall, the core technical implementation of this paper is to add random perturbations at each step of LoRA training. While the approach is elegant, it’s not particularly novel, and there doesn’t seem to be very convincing experimental results. The proposed Flat-LoRA series offers limited improvement across various LoRA variants and tasks, with most gains being around a few tenths of a percentage in accuracy (except for significant improvements in gsm8k and Human-Eval). It would be helpful to include tasks like apaca and other SFT tasks. I’m particularly interested in the practical significance of this work—flat local optima should theoretically bring stronger generalization, such as supporting OOD (e.g., Alpaca, instruct-eval, etc.) and robustness (e.g., image-R, image-C) evaluations, proving that a flatter local optimum could enhance out-of-domain generalization. This would be valuable since even sharp in-domain optima can perform well, and this might be why performance improvements are modest.

questions:
What do you think is the essential difference or advantage between adding perturbations to weights and adding perturbations to data samples? Is adding perturbations to data samples simpler?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730701414941

thread_type:
official_review

<thread_object>
creator: reviewer_uxap

<content>
summary:
This paper introduces Flat-LoRA, a novel extension to the Low-Rank Adaptation (LoRA) framework, designed to optimize model fine-tuning via discovering solutions within a flatter loss landscape in the full parameter space. Different from traditional LoRA, which may result in sharp solutions that impact generalization negatively, Flat-LoRA incorporates random weight perturbations and a Bayesian expectation loss objective to maintain efficiency. The approach seeks to combine parameter efficiency with enhanced generalization capabilities across both NLP and CV tasks, demonstrating improvements over conventional LoRA in various experimental setups.

strengths:
1. The idea of optimizing to reach a flat landscape in the full parameter space while maintaining the advantages of parameter efficiency is innovative and well-justified.
2. The methodology is generally clearly articulated, and lots of experiments show considerable improvements over existing LoRA-based methods, which validates the efficacy of Flat-LoRA proposed.
3. The explanation of the Bayesian expectation loss objective function and the perturbation generation strategy is very thorough and definitely contributes to the transparency of the proposed method.
4. This work is equipped with practical implications, as fine-tuning LLMs efficiently is increasingly important in current ML field.

weaknesses:
1. Althougth the paper emphasizes the computational overhead and the minimal memory, the perturbation generation and its integration into the mixed-precision training could be simplified or clarified furthermore.
2. The method concentrates mostly on linear layers in the transformer-based models. Despite the fact that the authors acknowledge this as a limitation, extending such approach to other parameter categories would make the method more versatile.
3.: Although the comparisons with approaches such as SAM are very insightful, deeper analysis with more recent variations of sharpness-aware algorithms could strengthen the study and contribution.

questions:
Could the perturbation generation strategy be optimized or adapted to incorporate other noise categories (e.g., adversarial perturbations)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732546286696

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Looking forward to the feedback

comment:
Dear AC and PC Members,

First and foremost, we would like to express our sincere gratitude for your time and efforts in reviewing our manuscript. As the time is coming to the discussion ending, we are eager to engage in further discussions and do our best to address the reviewers' concerns, which might help you with better decisions. Below, we summarize the reviewers' comments and our corresponding responses:

1. **Limited improvement on GLUE:** we use a stronger baseline compared to previous works (e.g., LoRA-GA) for more rigor evaluation. Additionally, experiments on more challenging SuperGLUE demonstrates greater improvements of Flat-LoRA.
2. **Limited experiments scope:** we have substantially extended our evaluation scopes with larger models (e.g., LLama-2-13B) and various datasets (e.g., Alpaca, SuperGLUE, etc.). We also discuss the effectiveness of flatness across different tasks.
3. **Lack of mathematical rigor:** we have revised the corresponding part to clarify and address potential misunderstandings. We also correct and justify the scaling factor $1/\sqrt{n}$ for noise generation.
4. **Unlikely overfitting for LoRA:** we visualize how very low-rank may appear sharp in the full parameter space and show that flatness can help even in underfitting regime. We also show that the noise injection approach in Flat-LoRA can be readily incorporated into full fine-tuning.
5. **Lack of comparisons, ablations, citations:** we have included these experiments and references accordingly as suggested by the reviewers.


The necessary updates have been integrated into the revised PDF file, with changes highlighted in blue. We sincerely hope the reviewers could provide further feedback, and we look forward to continuing discussions with you.

Best regards,

The Anonymous Author(s) of Paper8654
</content>
</thread_object>

</thread>
<thread>
creation date:
1732772031050

thread_type:
official_comment

<thread_object>
creator: area_chair_honr

<content>
comment:
Dear Reviewers,

If you have not responded to author's rebuttal, please kindly do so as soon as possible. The deadline is Dec 2, but the authors can potentially further clarify questions if you respond earlier. Thanks!

Best, AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
Summary: Flat-LoRA adds random weight perturbations into LoRA optimization to locate flat minima in the loss landscape. This improves performance and robustness in vision and language models.

Strengths: novel idea considering full parameter space in LoRA; practical method; clear presentation.

Weaknesses: marginal performance gains; missing comparisons with baselines; limited datasets for evaluation; 

Reasons for the decision: while Flat-LoRA introduces a novel idea of optimizing flat minima, its marginal performance improvements, limited baselines, and datasets in evaluation do not justify acceptance. In general the paper does not receive enough support from the reviewers.