Flat-LoRA: Low-Rank Adaption over a Flat Loss Landscape

## Abstract

Abstract Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computational and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tuning (PEFT) method, provides an efficient way to fine-tune models by optimizing only a low-rank matrix. Despite recent progress made in improving LoRA’s performance, the connection between the LoRA optimization space and the original full parameter space is often overlooked. A solution that appears flat in the LoRA space may exist sharp directions in the full parameter space, potentially harming generalization performance.
In this paper, we propose Flat-LoRA, an efficient approach that seeks a low-rank adaptation located in a flat region of the full parameter space.
Instead of relying on the well-established sharpness-aware minimization approach, which can incur significant computational and memory burdens, we utilize random weight perturbation with a Bayesian expectation loss objective to maintain training efficiency and design a refined perturbation generation strategy for improved performance.
Experiments on natural language processing and image classification tasks with various architectures demonstrate the effectiveness of our approach.

## 1 Introduction

Pre-training followed by fine-tuning is a widely adopted training pipeline among modern machine learning practitioners for achieving state-of-the-art (SOTA) performance (Girshick et al., ; Kolesnikov et al., ; Wortsman et al., ; Yu et al., ), leveraging the versatile knowledge within the pre-trained models. However, the enormous size of these pre-trained models makes fine-tuning all parameters for downstream tasks resource-intensive, making it impractical to store optimizer states or multiple model weights when dealing with multiple tasks. Recently, Low-Rank Adaptation (LoRA) (Hu et al., ) has been proposed to address this resource challenge. In LoRA fine-tuning, only a low-rank matrix is optimized and then added to the pre-trained weights after training, incurring no additional computational or memory costs during inference. This approach significantly reduces the number of trainable parameters, thereby lowering the training cost as well as storage cost when dealing with different tasks.

Many works have been proposed to enhance the performance of LoRA by introducing more dedicated budgets for rank allocation (Zhang et al., ), decomposing optimization for direction and magnitude updates (Liu et al., ), or designing better initialization strategy for LoRA parameters (Meng et al., ; Wang et al., ), etc. These studies demonstrate the significant potential for improving LoRA performance.
However, the connection between the LoRA optimization space and the original full parameter space is often overlooked. Essentially, LoRA restricts training to a much lower-dimensional subspace, and its performance depends on the properties of the solutions within this subspace in relation to the full parameter space, as the merged weights are ultimately used during inference.
As illustrated in Figure 1, a flat minima in the LoRA space (blue) may exhibit sharp direction (red) in the view of the full parameter space, which may degenerate the generation performance.

Figure: Figure 1: Illustration of LoRA optimization. LoRA constrains training in a lower-dimensional subspace (blue). A flat minima in LoRA subspace (blue curve) may exhibit sharp direction in full parameter space (red curve).
Refer to caption: x1.png

It is widely believed that a flatter loss landscape can lead to better generalization performance (Hochreiter & Schmidhuber, , ). This idea has given rise to a well-established training strategy called Sharpness-Aware Minimization (SAM), which has shown great generalization improvement in training neural networks. Applying SAM to large language models (LLMs)’ training together with LoRA is certainly promising, but there are several issues should be discussed. First, unlike the existing attempts that flatten the landscape in a LoRA subspace (Li et al., ), which is not aware of the sharpness outside the LoRA space,
we pursue a solution that aligns with a flatter loss landscape in the full weight space. Second, the original SAM doubles the training time cost, which is impractical for fine-tuning large models.
Additionally, to capture the sharpness in the full parameter space, we need to calculate the gradients of the full weight parameters, which contradicts the principles of parameter-efficient fine-tuning (PEFT). To cope with these challenges, we propose using random weight perturbations to maintain time and memory efficiency and design effective generation strategies to improve generalization performance.

Our main contribution can be summarized as follows:

- •
We propose Flat-LoRA that firstly aims to optimize the sharpness of the loss landscape within the full parameter space where the low-rank adaptation resides. It incurs minimal additional computational and memory costs and can be easily integrated with existing techniques to enhance LoRA performance, delivering consistent improvements.
- •
We propose to use expected Bayesian loss to optimize the sharpness for keeping the training efficiency and design effective generation strategy to generate random weight perturbation to enhance the generalization performance, making it easy for practical usage.
- •
Experiments on natural language processing and computer vision tasks with various architectures to demonstrate that our approach can achieve state-of-the-art performance.

## 2 Related Work

Flat minima and generalization.
The connection between the flatness of local minima and generalization has received much attention (Hochreiter & Schmidhuber, ; Chaudhari et al., ; Keskar et al., ; Dinh et al., ; Izmailov et al., ; Li et al., ; Zhou et al., ; Li et al., ; Li & Giannakis, ; Li et al., , ).
Recently, many works have tried to improve the model generalization by seeking flat minima (Tsuzuku et al., ; Zheng et al., ; Bisla et al., ; Li et al., ). For example, Chaudhari et al. () proposed Entropy-SGD to search for flat regions by minimizing local entropy.
Wen et al. () designed SmoothOut framework to smooth out the sharp minima.
Notably, Sharpness-Aware Minimization (SAM) (Foret et al., ) established a generic training scheme for seeking flat minima by formulating the optimization as a min-max problem and encouraged parameters sitting in neighborhoods with uniformly low loss, achieving state-of-the-art generalization improvements across various tasks.
However, it requires twice the training time as regular training, limiting its application to large model training.
Another promising branch of methods recovers flat minima by minimizing the expected Bayesian training loss under random weight perturbation (RWP) (Bisla et al., ), which is efficient as no additional gradient step is required. Li et al. () further enhance generalization performance by introducing an adaptive random perturbation generation strategy and a mixed loss objective. However, when applying these approaches to PEFT training, we must be mindful of the additional memory and time costs they may introduce.

Low-rank adaption. Recent works have indicated that the intrinsic dimension for optimizing deep neural networks (DNNs) may be much lower than the number of parameters (Li et al., ; Gur-Ari et al., ; Wu et al., ). Li et al. () demonstrated that the training trajectory of DNNs can be low-dimensional and propose subspace optimization to enhance training efficiency and robustness (Li et al., ).
Low-Rank Adaptation (LoRA) was proposed to model the weight changes for each layer during fine-tuning, aiming to reduce training costs. It effectively decreases the number of trainable parameters, thereby lowering the memory burden for training and storage. This approach is currently mainstream because it avoids adding any overhead during inference while often demonstrating strong performance. Many works have been proposed to enhance the performance of LoRA by introducing more dedicated budgets for rank allocation (Zhang et al., ), decomposing optimization for direction and magnitude updates (Liu et al., ), designing better initialization strategy for LoRA parameters (Meng et al., ; Wang et al., ), or better aligning each gradient step to the full fine-tuning  (Wang & Liang, ), etc.

## 3 Method

In this section, we first give a brief review on the low-rank adaption (LoRA). We then introduce our LoRA optimization objective considering the flatness of the landscape. We finally describe our random perturbation generation strategy for effectively improving the generalization performance.

### 3.1 LoRA: Low-Rank Adaption

Based on the finding that DNNs’ optimization happens in a subspace with much smaller dimensions than the number of parameters (Li et al., , ), LoRA utilizes low-rank matrices to model the weight change for each layers’ weights $\mathbf{W}\in\mathbb{R}^{n\times m}$ during the fine-tuning as $\Delta\mathbf{W}=s\cdot\mathbf{B}\mathbf{A}$, where $s$ is a scaling factor, $\mathbf{B}\in\mathbb{R}^{n\times r}$ and $\mathbf{A}\in\mathbb{R}^{r\times m}$ with the rank $r\ll\{n,m\}$ to achieve parameter efficiency.

For the original output $\mathbf{h}=\mathbf{W}\mathbf{x}$, the modified forward pass is

$$ $\displaystyle\mathbf{h}=\mathbf{W}\mathbf{x}+\Delta\mathbf{W}\mathbf{x}=( \mathbf{W}+s\cdot\mathbf{B}\mathbf{A})\mathbf{x}.$ (1) $$

During the initialization of LoRA, matrix $\mathbf{A}$ is commonly initialized with Kaiming distribution (He et al., ) and matrix $\mathbf{B}$ is set to zeros. During the training, only the low-rank matrices $\mathbf{A}$ and $\mathbf{B}$ are optimized with the pre-trained weight $\mathbf{W}$ being frozen. During the inference, the low-rank matrices $\Delta\mathbf{W}$ are merged to the pre-trained weight $\mathbf{W}$, and in this way there is no additional computational or memory costs.

### 3.2 LoRA with a Flat Landscape

Despite recent works have made great efforts for improving LoRA performance, most works focus solely on finding a solution performing well within the LoRA solution space, specifically the rank $r$ matrix space $\mathcal{M}_{r}=\{\Delta\mathbf{W}\in\mathbf{R}^{m\times n}|\mathrm{rank}(
\Delta\mathbf{W})=r\}$. For example, Li et al. () consider applying SAM to LoRA parameters and study the properties of LoRA parameters with SAM:

$$ $\displaystyle\min_{\mathbf{A},\mathbf{B}}~{}~{}\max_{\|(\mathbf{\epsilon}_{ \mathbf{A}},\mathbf{\epsilon}_{\mathbf{B}})\|\leq\rho}~{}~{}L\left(\mathbf{W}+ s\cdot(\mathbf{B}+\mathbf{\epsilon}_{\mathbf{B}})(\mathbf{A}+\mathbf{\epsilon} _{\mathbf{A}})\right).$ (2) $$

However, solely considering the properties of the optimization subspace defined by LoRA parameters may present several limitations.
During inference, the low-rank matrices $\Delta\mathbf{W}$ are merged into the pre-trained weights $\mathbf{W}$.
A solution that performs well within the LoRA subspace may reside in a sharp region of the full parameter space, as illustrated in Figure 1, potentially harming overall generalization performance.
Therefore, it is crucial to consider the loss landscape of $L(\mathbf{W}+\Delta\mathbf{W})$, where $L(\cdot)$ denotes the loss objective. In other words, we need to find a low rank adaption $\Delta\mathbf{W}$ that positions the merged weights in a flat region of the full parameter space.

Following the well-established sharpness-aware minimization (SAM) objective (Foret et al., ), we can formulate the flat loss objective as follows:

$$ $\displaystyle\min_{\mathbf{A},\mathbf{B}}~{}~{}\max_{\|\bm{\epsilon}\|\leq\rho }~{}~{}L(\mathbf{W}+s\cdot\mathbf{B}\mathbf{A}+\bm{\epsilon}).$ (3) $$

However, directly applying SAM to optimize the sharpness of the merged weight space has several disadvantages: 1) it doubles the training cost, which is less desirable with large models, and 2) it requires storing an additional copy of weights for perturbation, which contradicts the principle of parameter-efficient fine-tuning. To achieve a flatter loss landscape while maintaining time and memory efficiency, we propose relaxing the maximization problem in Eq. (3) to an expectation, resulting in the following Bayesian expected loss objective:

$$ $\displaystyle\min_{\mathbf{A},\mathbf{B}}~{}~{}\mathop{\mathbb{E}}_{\bm{ \epsilon}\sim\mathcal{N}(0,\sigma^{2}\mathbf{I})}~{}~{}L(\mathbf{W}+s\cdot \mathbf{B}\mathbf{A}+\bm{\epsilon}),$ (4) $$

where $\sigma$ controls the variance of the noise, which we will describe in the next section. This expected loss can be seen as applying a smoothing filter over the loss landscape within the full parameter space, and optimizing it can help recover flatter minima. (Bisla et al., ).
For each optimization step, we would sample a noise $\mathbf{\epsilon}$ and calculate the perturbed gradient to optimize the low-rank matrices $\mathbf{A}$ and $\mathbf{B}$. Note that the noise is generated based on the model weights, thus incurring no additional gradient steps as SAM does.

### 3.3 Effective Random Perturbation Generation

We then describe how to effectively generate random weight perturbation, which is the core for improving the generalization performance. Let $\mathbf{W}^{\prime}=\mathbf{W}+s\cdot\mathbf{B}\mathbf{A}$. In this paper, we have experimented with various schemes for noise generation and hope to find the most effective one for random weight perturbation generation that could benefit the community. For the merged weight $\mathbf{W}^{\prime}\in\mathbb{R}^{m\times n}$ that represents a linear layer with input dimension $n$ and output dimension $m$, our design considers the following two perspectives:

- •
Filter structure: we aim to generate the weight noise by filter (Bisla et al., ). There contains $m$ filters $\mathbf{W}^{\prime}=(\mathbf{W}^{\prime}_{1,:},\mathbf{W}^{\prime}_{2,:},
\cdots,\mathbf{W}^{\prime}_{m,:})$ that process the input $\mathbf{X}\in\mathbb{R}^{n}$. Elements within a filter of larger norm should receive a larger strength of perturbation.
- •
Input dimension: we hope that the variance introduced to the forward pass by the added random weight perturbation is independent of the input dimension. Given an input dimension $n$, the magnitude of noise added to each element should be scaled by a factor of $1/{n}$.

Finally, our new weight noise generation scheme is formulated as follows:

$$ $\displaystyle\bm{\epsilon}\sim\mathcal{N}\left(0,\frac{\sigma^{2}}{{n}}\mathrm {diag}(\|\mathbf{W}^{\prime}_{1,:}\|_{2}^{2},\|\mathbf{W}^{\prime}_{2,:}\|_{2} ^{2},\cdots,\|\mathbf{W}^{\prime}_{m,:}\|_{2}^{2})\mathbf{I}_{m\times n}\right),$ (5) $$

where $\mathbf{I}_{m\times n}$ denotes a matrix of size $m\times n$ with all ones. Here $\sigma$ is the hyper-parameter that needs to be chosen for controlling the perturbation strength.

#### Analysis on forward variance.

We then interpret adding random weight perturbation as variance injection during the forward propagation.
Consider the hypothesis that the variance of the input $\mathbf{X}$ is $\mathrm{var}(\mathbf{X})$.
Then we have:

$$ $\displaystyle\mathrm{var}(\mathbf{W}^{\prime}_{i,:}\mathbf{X})$ $\displaystyle=\|\mathbf{W}^{\prime}_{i,:}\|_{2}^{2}\cdot\mathrm{var}(\mathbf{X }),$ (6) $\displaystyle\mathrm{var}\left((\mathbf{W}^{\prime}+\bm{\epsilon})_{i,:} \mathbf{X}\right)$ $\displaystyle=(1+\sigma^{2})\|\mathbf{W}^{\prime}_{i,:}\|_{2}^{2}\cdot\mathrm{ var}(\mathbf{X}).$ (7) $$

Thus, by injecting weight perturbations, we introduce variance with a rate of $\sigma^{2}$ into the forward pass. It is important to note that since we introduce a magnitude of
$1/n$ for noise generation (see Equ. (5)), this variance ratio is independent of the input dimension. Additionally, we note that this variance will not increase exponentially during the forward propagation of the network due to the presence of layer normalization.

#### Storing random seed for memory efficiency.

Memory is an important factor to consider to PEFT training.
To optimize Eqn. (4), we first generate random perturbation $\bm{\epsilon}$ and then perform gradient descent with $\nabla L(\mathbf{W}+s\cdot\mathbf{B}\mathbf{A}+\bm{\epsilon})$. Thus, we need to store the weight perturbation for recovering the weight after obtaining the perturbed gradient. When model is large, storing a copy of weight perturbation is prohibitive. Luckily, for random weight perturbation, we only need to store the seed for random generator and corresponding norms for each filter $\|\mathbf{W}^{\prime}_{1,:}\|_{2}^{2},\|\mathbf{W}^{\prime}_{2,:}\|_{2}^{2},
\cdots,\|\mathbf{W}^{\prime}_{m,:}\|_{2}^{2}$, allowing us to recover the random perturbation $\bm{\epsilon}$ when necessary. This approach incurs minimal additional memory and offers significant advantages over SAM, which requires calculating the full gradient, thereby necessitating a hard copy of the perturbation that cannot be reduced.

#### Integration to mixed precision training.

Finally, we note that the noise injection step can be seamlessly integrated into mixed-precision training even without the need to store the filter norms, which is commonly used in large-scale training. Specifically, in mixed-precision training, we maintain two copies of the weights: the full-precision FP32 weights and the half-precision BF16 weights in memory. We can inject noise during the half-precision auto-cast step, thus eliminating the need to store a copy of the weight perturbation or the filter norms.

## 4 Experiments

In this section, we evaluate the performance of Flat-LoRA on various benchmark datasets. We first conduct experiments on Natural Language Understanding tasks using a subset of GLUE datasets (Wang et al., ) with T5-base model (Raffel et al., ). We then experiment over image classification tasks with CLIP ViT-B/32 model (Radford et al., ). We finally give ablation studies and discussions on our method. The code will be released.

### 4.1 Experiments on Natural Language Understanding

Setting.
We finetune the T5-Base model on several datasets from GLUE benchmark, including MNLI, SST, CoLA, QNLI, and MRPC. Performance is evaluated on the development set using accuracy as the primary metric. We use lora with rank 8 and 16 with lora alpha 16. We finetune the models with 10 epochs with a cosine learning rate schedule, except for MNLI and QNLI we use 1 epochs. We use learning rate of 0.0005 for LoRA fine-tuning and 0.0001 for full fine-tuning with weight decay 0.1. The random perturbation strength $\sigma$ is set to 0.05 with an cosine increasing strategy. Mean and standard deviations are calculated over 3 independent trials.

Results.
As shown in Table 1, Flat-LoRA consistently outperforms LoRA for ranks 8 and 16, achieving average performance gains of 0.38% and 0.56%, respectively. In some cases, the performance of LoRA deteriorates when increasing the rank from 8 to 16, as seen with the MRPC dataset, which is relatively small and susceptible to overfitting. Flat-LoRA effectively addresses the overfitting issue and achieves greater improvements with increasing LoRA rank, demonstrating the advantages of our flat loss objective.

**Table 1: Results (%) on fine-tuning T5-base with Full Fine-tuning and LoRA variants on a subset of GLUE datasets.**
| Method | MNLI | SST2 | CoLA | QNLI | MRPC | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Full FT | 86.19±0.04 | 94.15±0.09 | 82.84±0.12 | 93.10±0.04 | 89.22±0.23 | 89.10 |
| LoRA (r=8) | 86.24±0.02 | 94.55±0.07 | 82.20±0.22 | 93.06±0.03 | 88.97±0.42 | 89.00 |
| Flat-LoRA (r=8) | 86.20±0.04 | 94.75±0.20 | 83.19±0.38 | 93.16±0.09 | 89.59±0.37 | 89.38 |
| LoRA (r=16) | 86.49±0.06 | 94.52±0.21 | 82.89±0.44 | 92.97±0.05 | 88.89±0.64 | 89.15 |
| Flat-LoRA (r=16) | 86.51±0.01 | 94.84±0.02 | 84.08±0.31 | 93.28±0.03 | 89.83±0.64 | 89.71 |

### 4.2 Experiments on Image Classification

Setting. We finetune the CLIP-ViT-B/32 model on five image classification tasks, including CIFAR-10/100 (Krizhevsky & Hinton, ), Cars (Krause et al., ), SVHN (Netzer et al., ), and DTD (Cimpoi et al., ).
We resize all input image to a size of 224x224 and freeze the classification head. We try LoRA with rank 8 and 16 and finetune the models with 10 epochs with a cosine annealing schedule. The learning rate is set to 0.0005 for LoRA and $1\times 10^{-5}$ for full fine-tuning with weight decay 0.1. The random perturbation strength $\sigma$ is set to 0.15 with an cosine increasing strategy. Mean and standard deviations are calculated over 3 independent trials.

Results.
We measure the performance with classification accuracy and report the results in Table 2. We observe that Flat-LoRA consistently outperforms LoRA with ranks 8 and 16, showing average improvements of 0.56% and 0.74%, respectively. Notably, Flat-LoRA with rank 8 surpasses both LoRA with rank 16 and full fine-tuning by 0.28%. These results confirm the effectiveness of our flat loss objective on improving LoRA performance.

**Table 2: Results (%) on fine-tuning CLIP ViT-B/32 with full fine-tuning and LoRA variants on image classification datasets.**
| Method | CIFAR-10 | CIFAR-100 | Cars | SVHN | DTD | Avg. |
| --- | --- | --- | --- | --- | --- | --- |
| Full FT | 97.99±0.01 | 89.06±0.11 | 73.30±0.43 | 97.44±0.03 | 76.80±0.25 | 86.92 |
| LoRA (r=8) | 97.90±0.02 | 87.74±0.13 | 73.22±0.53 | 97.49±0.08 | 76.86±0.34 | 86.64 |
| Flat-LoRA (r=8) | 98.09±0.04 | 88.64±0.23 | 74.17±0.71 | 97.59±0.04 | 77.51±0.28 | 87.20 |
| LoRA (r=16) | 97.99±0.03 | 88.12±0.23 | 73.80±0.42 | 97.56±0.08 | 77.34±0.32 | 86.92 |
| Flat-LoRA (r=16) | 98.21±0.04 | 89.27±0.07 | 74.89±0.52 | 97.71±0.10 | 78.24±0.44 | 87.66 |

Figure: (a) MRPC with T5-Base
Refer to caption: x2.png

### 4.3 Ablation Study and Discussion

#### Comparison with Other Methods

We then compare our approach with other recently proposed methods for improving LoRA, including initialization-based methods such as PiSSA and LoRA-GA, as well as optimization-based methods like DoRA and LoRA+. Our experiments are conducted on the CoLA and MRPC datasets using the T5-base model with lora rank 8. The results are presented in Table 3.
We observe that Flat-LoRA consistently outperforms previous methods by 0.53%. Furthermore, our flat loss objective can be easily integrated with earlier approaches to yield consistent improvements by 0.31% to 0.93%. This highlights the effectiveness of considering the sharpness of the full parameter space.

Note that here we adopt a stronger training baseline, including employing a larger learning rate and longer training epochs, which achieves significantly better performance than the results reported in previous work (Wang et al., ).
In fact, CoLA and MRPC are two datasets that achieve that most significant improvement by LoRA-GA reported in the original paper (Wang et al., ).
Under our experimental settings, LoRA-GA does not demonstrate advantages over vanilla LoRA and may even perform worse. This may be because LoRA-GA provides a smart initialization strategy by maximizing gradient alignment with full parameter training, allowing for quicker convergence to a good local optimum (e.g., achieved in just one epoch as noted in Wang et al. ()). However, this approach may not be optimal for reaching a better global optimum.

**Table 3: Comparison with other methods on GLUE subsets with T5-Base.**
| Method | CoLA | MRPC |
| --- | --- | --- |
| LoRA Hu et al. () | 82.20±0.22 | 88.03±0.14 |
| PiSSA (Meng et al., ) | 82.44±0.14 | 88.96±0.44 |
| LoRA-GA (Wang et al., ) | 81.83±0.21 | 87.58±0.41 |
| DoRA Liu et al. () | 83.16±0.15 | 89.46±0.37 |
| LoRA+ Hayou et al. () | 81.65±0.34 | 89.30±0.47 |
| Flat-LoRA (ours) | 83.19±0.38 | 89.59±0.37 |
| Flat-PiSSA (ours) | 83.35±0.48 | 89.89±0.71 |
| Flat-LoRA-GA (ours) | 82.23±0.34 | 88.15±0.54 |
| Flat-DoRA (ours) | 83.56±0.27 | 89.99±0.47 |
| Flat-LoRA+ (ours) | 82.56±0.23 | 89.61±0.44 |

#### Results under different lora ranks.

Following the settings in Section 4.1 and 4.2,
we test the performance of Flat-LoRA under different lora ranks. The results are shown in Figure 2. We observe that Flat-LoRA consistently outperforms LoRA across different LoRA ranks. Even with low rank 1, which is typically underfit and the performance is not the best, Flat-LoRA still brings significant performance improvement over LoRA, by +1.10% on MRPC and +1.15% on CIFAR-100. This highlights the importance of considering the sharpness of the full parameter space. Additionally, as the LoRA rank increases, LoRA’s performance can degrade due to overfitting, such as on MRPC with T5-Base. Flat-LoRA helps mitigate overfitting by finding flatter minima that generalize better.
Thus, we conclude that Flat-LoRA can improve the LoRA performance not only in low-rank scenarios where little full parameter space information is considered, but also in high-rank setting for overcoming the overfitting.

## 5 Conclusion

In this paper, we introduce Flat-LoRA, an efficient low-rank adaptation approach that aims to optimize the sharpness of the loss landscape within the full parameter space that LoRA situates in. Deviating from standard sharpness-aware approach that incurs significant computation and memory burdens, we employ a Bayesian expectation loss objective minima and utilize designed random weight perturbations to pursuit flat minima, maintaining the training speed and memory efficiency characteristic of parameter-efficient fine-tuning. Flat-LoRA achieves state-of-the-art performance in LoRA fine-tuning and can be easily integrated with previous methods for consistent improvements. Extensive experiments on natural language processing and computer vision tasks demonstrate the effectiveness of our approach.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Could the perturbation generation strategy be optimized or adapted to incorporate other noise categories (e.g., adversarial perturbations)?
soundness: 3
strengths: 1. The idea of optimizing to reach a flat landscape in the full parameter space while maintaining the advantages of parameter efficiency is innovative and well-justified.
2. The methodology is generally clearly articulated, and lots of experiments show considerable improvements over existing LoRA-based methods, which validates the efficacy of Flat-LoRA proposed.
3. The explanation of the Bayesian expectation loss objective function and the perturbation generation strategy is very thorough and definitely contributes to the transparency of the proposed method.
4. This work is equipped with practical implications, as fine-tuning LLMs efficiently is increasingly important in current ML field.
summary: This paper introduces Flat-LoRA, a novel extension to the Low-Rank Adaptation (LoRA) framework, designed to optimize model fine-tuning via discovering solutions within a flatter loss landscape in the full parameter space. Different from traditional LoRA, which may result in sharp solutions that impact generalization negatively, Flat-LoRA incorporates random weight perturbations and a Bayesian expectation loss objective to maintain efficiency. The approach seeks to combine parameter efficiency with enhanced generalization capabilities across both NLP and CV tasks, demonstrating improvements over conventional LoRA in various experimental setups.
weaknesses: 1. Althougth the paper emphasizes the computational overhead and the minimal memory, the perturbation generation and its integration into the mixed-precision training could be simplified or clarified furthermore.
2. The method concentrates mostly on linear layers in the transformer-based models. Despite the fact that the authors acknowledge this as a limitation, extending such approach to other parameter categories would make the method more versatile.
3.: Although the comparisons with approaches such as SAM are very insightful, deeper analysis with more recent variations of sharpness-aware algorithms could strengthen the study and contribution.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: What do you think is the essential difference or advantage between adding perturbations to weights and adding perturbations to data samples? Is adding perturbations to data samples simpler?
soundness: 3
strengths: 1. The writing is well done, with motivations and insights clearly explained in an intuitive manner, accompanied by reasonable mathematical assumptions to introduce the design of full-rank noise perturbations.
2. The design for storing random seeds for memory efficiency and integrating into mixed-precision training is very clever, saving additional overhead.
summary: Based on the consideration of local optimal values, this paper proposes that the flat optimal value learned by LoRA is not necessarily flat at full rank. Corresponding mathematical explanations are provided. Following this, using the idea of SAM, a method is designed to add full-rank noise perturbations to search for a global optimal solution.
weaknesses: 1. On line 125, the sentence "the LoRA matrices. better initialization strategy for LoRA parameters." should probably use a comma instead of a period.
2. The work on adding full-rank noise has been done in Noisetune ([https://arxiv.org/abs/2202.12024](https://arxiv.org/abs/2202.12024)) and LORASC ([https://arxiv.org/abs/2407.01491](https://arxiv.org/abs/2407.01491)), especially in LoRASC, where the exact same approach is used, adding a full rank noise to each LoRA optimization process. 
3. Overall, the core technical implementation of this paper is to add random perturbations at each step of LoRA training. While the approach is elegant, it’s not particularly novel, and there doesn’t seem to be very convincing experimental results. The proposed Flat-LoRA series offers limited improvement across various LoRA variants and tasks, with most gains being around a few tenths of a percentage in accuracy (except for significant improvements in gsm8k and Human-Eval). It would be helpful to include tasks like apaca and other SFT tasks. I’m particularly interested in the practical significance of this work—flat local optima should theoretically bring stronger generalization, such as supporting OOD (e.g., Alpaca, instruct-eval, etc.) and robustness (e.g., image-R, image-C) evaluations, proving that a flatter local optimum could enhance out-of-domain generalization. This would be valuable since even sharp in-domain optima can perform well, and this might be why performance improvements are modest.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: no
soundness: 2
strengths: Advantages:

The paper is written clearly
The core idea seems reasonable
summary: This paper introduces Flat-LORA, which adds noise to the training process: W + AB + ε, where ε is stored using a seed. The authors attempt to achieve SAM-like effects through this method.

Advantages:
1. The paper is written clearly
2. The core idea seems reasonable

Disadvantages:
1. The paper lacks mathematical rigor in several places, particularly in equations 8 and 9
2. Insufficient experimental validation:
   - Should test on larger models (e.g., LLAMA 13B or 70B)
   - Should evaluate on SuperGLUE
3. Lacks necessary ablation studies, particularly regarding σ^2

Technical Issues:
1. Equations 8 and 9 have fundamental problems:
   - var(X) should be a covariance matrix
   - var(W'_{i,:}X) should be a scalar
   - These dimensions are inconsistent and cannot be equated
2. In Equation 7, n should be sqrt(n), as large n values would result in negligibly small epsilon values added to the weight matrix

Additional Concerns:
1. No memory usage results are reported
2. Table 3 lacks full-tuning baseline results
3. this paper should report results on more diverse datasets
4. More comprehensive ablation studies on σ^2 are needed
weaknesses: Disadvantages:

The paper lacks mathematical rigor in several places, particularly in equations 8 and 9
Insufficient experimental validation:
Should test on larger models (e.g., LLAMA 13B or 70B)
Should evaluate on SuperGLUE
Lacks necessary ablation studies, particularly regarding σ^2
Technical Issues:

Equations 8 and 9 have fundamental problems:
var(X) should be a covariance matrix
var(W'_{i,:}X) should be a scalar
These dimensions are inconsistent and cannot be equated
In Equation 7, n should be sqrt(n), as large n values would result in negligibly small epsilon values added to the weight matrix
Additional Concerns:

No memory usage results are reported
Table 3 lacks full-tuning baseline results
this paper should report results on more diverse datasets
More comprehensive ablation studies on σ^2 are needed

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. How does it perform on the SuperGLUE benchmark, SQuAD, XSum, and CNN/Dailymail?
2. How does it perform on Stable Diffusion?
3. How does it compare between AdaLoRA and DyLoRA?
4. How is the performance on Llama3 with the alpaca dataset?
soundness: 3
strengths: 1. The paper introduces Flat-LoRA, a novel method that improves upon traditional Low-Rank Adaptation (LoRA) techniques by targeting a flat region of the loss landscape. This aims to enhance generalization performance by avoiding sharp minima that can degrade model performance on unseen data.

2. Flat-LoRA addresses the computational and memory cost issues associated with fine-tuning large-scale models. By optimizing only a low-rank matrix and using random weight perturbations, it achieves parameter-efficient fine-tuning without additional memory costs during inference.

3. The paper not only proposes a new technique but also details the underlying mathematical foundations. It discusses the optimization objective, perturbation generation, and integrates a Bayesian expected loss objective to maintain training efficiency.
summary: This paper discusses how to improve the generalization of LoRA, with good writing, innovative ideas, and credible experimental performance. However, I think the article overlooks one point: LoRA only fine-tunes a very small number of parameters, which is unlikely to overfit on a specific task and can inherently maintain good generalizability. If I am wrong, please convince me.
weaknesses: 1. There is no significant improvement, the limited enhancement on the GLUE benchmark does not prove the lack of generalizability of methods like LoRA.  I need to know if your method is effective on more datasets. More datasets and models should be compared. For example: the SuperGLUE benchmark, SQuAD, XSum, CNN/Dailymail, and some LoRA training on Stable Diffusion.

2. There is a lack of extensive comparisons, such as with methods like DyLoRA[1] and AdaLoRA[2]. 

3. More relevant articles, such as AWP[3], should be cited, which is an effective method to improve generalization.

[1] Valipour M, Rezagholizadeh M, Kobyzev I, et al. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation[J]. arXiv preprint arXiv:2210.07558, 2022.
[2] Zhang Q, Chen M, Bukharin A, et al. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning[J]. arXiv preprint arXiv:2303.10512, 2023.
[3] Wu D, Xia S T, Wang Y. Adversarial weight perturbation helps robust generalization[J]. Advances in neural information processing systems, 2020, 33: 2958-2969.

Meta-review: Summary: Flat-LoRA adds random weight perturbations into LoRA optimization to locate flat minima in the loss landscape. This improves performance and robustness in vision and language models.

Strengths: novel idea considering full parameter space in LoRA; practical method; clear presentation.

Weaknesses: marginal performance gains; missing comparisons with baselines; limited datasets for evaluation; 

Reasons for the decision: while Flat-LoRA introduces a novel idea of optimizing flat minima, its marginal performance improvements, limited baselines, and datasets in evaluation do not justify acceptance. In general the paper does not receive enough support from the reviewers.

Decision: Reject