Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Authors: Tao Li
Zhengbao He
Yujun Li
Yasheng Wang
Lifeng Shang
Xiaolin Huang
Venue: ICML 2025 poster (2025)
Keywords: Low-rank adaptation
generalization
efficient training
Primary area: optimization
TL;DR: We propose Flat-LoRA that aims to efficiently optimize the sharpness of the loss landscape in the full parameter space for low-rank adaptation.

Abstract: Fine-tuning large-scale pre-trained models is prohibitively expensive in terms of computation and memory costs. Low-Rank Adaptation (LoRA), a popular Parameter-Efficient Fine-Tuning (PEFT) method, offers an efficient solution by optimizing only low-rank matrices. Despite recent progress in improving LoRA's performance, the relationship between the LoRA optimization space and the full parameter space is often overlooked. 
A solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, potentially compromising generalization.
We introduce Flat-LoRA, which aims to identify a low-rank adaptation situated in a flat region of the full parameter space.
Instead of adopting the well-established sharpness-aware minimization approach, which incurs significant computation and memory overheads, we employ a Bayesian expectation loss objective to preserve training efficiency. 
Further, we design a refined strategy for generating random perturbations to enhance performance and carefully manage memory overhead using random seeds.
Experiments across diverse tasks—including mathematical reasoning, coding abilities, dialogue generation, instruction following, and text-to-image generation—demonstrate that Flat-LoRA improves both in-domain and out-of-domain generalization.
Code is available at https://github.com/nblt/Flat-LoRA.

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Tao Li Zhengbao He
Yujun Li
Yasheng Wang? Lifeng Shang?
Xiaolin Huang'
Abstract
Fine-tuning large-scale pre-trained models is pro- hibitively expensive terms of computation and memory costs. Low-Rank Adaptation (LoRA), a Fine-Tuuing (PEFT) method, offers efficient solution by optimizing only low-rank matrices. Despite recent progress in improving LoRA's performance, the relation- ship between the LoRA optimization space and the full parameter space is often overlooked. solution that appears flat in the loss landscape of the LoRA space may still exhibit sharp directions in the full parameter space, potentially compro- mising generalization. We introduce Flat-LoRA. which aims to identify low-rank adaptation situ- ated in flat region of the full parameter space. In- stead of adopting the well-established sharpness- aware minimization approach, which incurs sig- nificant computation and memory overheads, we employ Bayesian expectation loss objective preserve training efficiency. Further, design refined strategy for generating random perturba- tions to enhance performance and carefully man- age memory overhead using random seeds. Exper- iments across diverse tasks-inchs including mathemat- ical reasoning, coding abilities, dialogue genera- tion, instruction following, and text-to-image gen- derfion-demonstraatt that Flat-LoRA improves both in-domain and out-of-domain generalization. Code is available at https: //gttub//////t//g///// com/ LoRA.
popular Murameter-Effiiiee
23
36
Figure Illustration of LoRA optimization space. LoRA constrains optimization a lower-dimensional space (blue). A flat minimum in LoRA space (blue curve) may exhibit sharp directions in the full parameter space (red curve).
of-the-art performance by leveraging the versatile capabili- ties of pre-trained models (Girshick et al., 2014; Kolesnikov et al., 2020; Radford et al., 2021; Li et al., 2024c). However, the enormous size these models makes fine-tuning all parameters resource- intersvvve. Recently, Low-Rank Adapta- tion (LoRA) (Hu al., 2022) has been proposed to address this challenge. LoRA fine-tunes only low-rank matrices, which can be merged with the pre-trained weights after training, incurring no extra overhead during inference. This approach significantly reduces trainable parameters, thereby lowering both training and storage requirements.
Many methods have been proposed to enhance LoRA per- formance, such as adaptive rank allocation (Zhang al., 2023a), decomposition of optimization into direction and magnitude (Liu et al., 2024), and improved initialization strategies (Meng et al., 2024; Wang et al., 2024). Despite the promising potential these methods offer, the connection between the LoRA optimization space and the original full parameter space often overlooked. Essentially, LoRA constrains optimization
1. Introduction
Pre-traine tre-training followed by fine-tuning has become the domi- nant paradigm in modern machine lcarning, achieving state-
much lower-dimensional space, and its performance depends on how solutions in this re- stricted space relate the full parameter space since the merged weights are ultimately used during inference. As illustrated in Figure 1, flat minimum in the LoRA space may contain sharp directions in the view the full param- eter space, potentially leading to performance degradation. Figure 6 further demonstrates this phenomenon, revealing
'Equal contribution. This work was conducted when Tao was an intern at Huawei Noah's S Ark Lab. 'Department of Automa- tion. Shanghai Jiao Tong University, Shanghai, China HHuawei Noah's Ark Lab. Correspondence Xiaolin Huang <xiaolin- @ huang@-juu.cdu.ccm jdd..cdu......
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
We find that low-rank adaptation may exhibit sharper loss landscapes in the full parameter space, prompting us to propose Flat-LoRA to mitigate this sharpness.
Ranfor Scerl and Filer Norra
We employ Bayesian expected loss with designed ran- dom weight perturbations to pursue flat minima, seam- lessly integrating with existing methods while main- taining computational and memory efficiency.
Pretr sined
Pretraised
Wrights
Weights
Generate
A
Extensive experiments across various natural language processing and computer vision tasks demonstrate that Flat-LoRA significantly improves both in-domain and out-of-domain generalization.
LoRA
Plat-LoRA
Illustration of LoRA (Left) and Flat-LoRA (Right). By introducing designed random weight perturba- tions during fine-tuning, Flat-LoRA identifies a low-rank solution that is flat in the loss landscape of the full parameter space. Unlike SAM, it eliminates the need for additional gradient steps and remains memory-efficient by storing only the random seed and small number of filter norms (less than l/r of the LoRA parameters for rank r).
Figure 2:
2. Related Work
2.1. Flat Mínima and Generalization
The connection between the flatness of local minima and generalization has received much attention (Hochreiter & Schmidhuber, 1997: Chaudhari et al., 2017; Keskar et al., 2017; Dinh et al., 2017; Izmailov et al., 2018; Li et al., 2018b; Wu et al., 2020; Kwon et al.. 2021; Zhuang et al., 2022: Li et al., 2024e). Recently, many works have tried to improve generalization by seeking flat minima (Tsuzuku et al., 2020; Zheng et al., 2021; Bisla et al., 2022). For example, Chaudhari et al. (2017) propose Entropy-SGD to search for flat regions by minimizing local entropy. Wen et al. (2018) design SmoothOut framework to smooth out the sharp minima. Notably, Sharpness-Aware Minimization (SAM) (Foret et al., 2020) establishes a generic training scheme for seeking flat minima by formulating a min-max problem and encouraging parameters sitting in neighbor- hoods with uniformly low loss, achieving state-of-the-an generalization improvements across various tasks. However, SAM doubles the training time compared to regular training, limiting its applicability to large-scale training.
the sharpness of loss landscape for the minima found by LoRA when examined in the full parameter space.
Flat minima in the loss landscape are widely believed to improve generalization and increase robustness to distri- bution shifts between training and test data (Hochreiter & Schmidhuber, 1994; 1997). This understanding has inspired Sharpness-Await Minimization (SAM) (Foret al., 2020), which is formulated as a min-max problem and has achieved state-of-the-art generalization. While integrating SAM with LoRA (referred to as LoRA-SAM (Li et al., 2024a)) for large model fine-tuning is promising, there are several is- sues that should be discussed. First, LoRA-SAM can only optimize the sharpness of the loss landscape in restricted space (Section 3.2), which may not effectively improve gen- eralization. Second, SAM requires an additional gradient step, doubling the training cost and rendering it impracti- cal for large models. Lastly, computing sharpness in the full parameter space necessitates calculating gradients and storing perturbations for all weights, which contradicts the fine-tuning.
Another branch of methods for recovering flat minima in- volves minimizing the expected Bayesian training loss under random weight perturbation (RWP), which efficient and doesn't require additional gradient step (Bisla et al., 2022). Wang & Mao (2021) propose Gaussian model perturbation as a regularization scheme for improving SGD training, but it remains inefficient for multiple noise sampling. Bisla et al. (2022) connect the smoothness of loss objective to generalization and adopt filter-wise random Gaussian per- turbation generation to recover flat minima and improve generalization. Li al. (2022c; 2024d) further enhance the generalization performance of RWP by introducing an adap- tive perturbation generation strategy and a mixed loss ob- jective. Wu et al. (2022); Li et al. (2024b) demonstrate that injecting small random noise before or during fine-tuning can improve generalization. However, when applying to parameter-efficien fine-tuning, we must be mindful of the additional memory costs they may introduce.
principles of parameter-effictt
To address these challenges, we propose employing the Bayesian expectation loss objective (Duchi et al., 2012; Bisla et al., 2022) to smooth the loss landscape, thereby achieving flat minima in the full parameter space. Our approach, termed Flat-LoRA, leverages efficient random weight perturbations that can be stored as random seeds. In contrast to SAM, which requires additional gradient steps and maintaining an extra copy of model weights, Flat-LoRA ensures both time and memory efficiency. Moreover, we introduce refined perturbation generation strategies that con- sider weight magnitude and model width scaling, resulting in improved generalization performance.
Our main contributions can be summarized as follows:
2

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
2.2. Low-rank Adaptation and Variants
B E Rmxr and A E Rex* with rank r min n) for parameter efficiency. We omit the scaling factor 8 a/r here for simplicity, as it can be merged into A and B. For the original output h = Wz, the modified forward pass is
Recent studies have shown that the intrinsic dimension re- quired for optimizing deep neural networks (DNNs) can be significantly lower than the total number of parame- ters (Li et al., 2018:t 8a; Gur-Ari et al., 2018). Notably, Li et al. (2022a) demonstrate the low-dimensional properties of DNNs' training dynamics, which have been leveraged to mitigate overfitting issues in adversarial training (Li et al., 2022b). Low-Rank Adaptation (LoRA) (Hu et al., 2022) is proposed to model the weight changes for each layer during fine-tuning. It effectively decreases the number of trainable parameters, thereby lowering the memory burden for train- ing and storage. This approach is currently the mainstream because it avoids adding overhead during inference while demonstrating strong performance (Wang et al., 2023).
h = Wx + AW = (W + BA)x.
(1)
At initialization, matrix A is commonly drawn from Kaim- ing distribution (He al., 2015) and matrix B is set to zeros. During the training, only the low-rat low-rank matrices A and B are optimized with the pre-trained weight W being frozen. Dur- ing the inference, the low-rank matrices AW are merged to the pre-trained weight W, and there are no additional computation or memory costs.
3.2. LoRA with a Flat Landscape
Many works have been proposed to enhance the perfor- mance of LoRA. AdALoRA (Zhang et al., 2023a) dynami- cally prunes insignificant weights during fine-tuning through singular value decomposition (SVD), enabling allocating more rank to important areas under a fixed parameter budget. DoRA (Liu et al., 2024) improves optimization performance by decomposing weight updates into their direction and mag- nitude components. LoRA+ (Hayou al., 2024) proposes to use different learning rates for the two matrices in LoRA to improve convergence. PiSSA (Meng et al., 2024) pro- poses to use the SVD decomposition of the original matrix W to initialize the LoRA matrices, which provides a better initialization for LoRA parameters. LoRA-GA (Wang et al., 2024) proposes to align the gradient of LoRA to that of full fine-tuning at initialization. LoRA- (Wang& & Liang, 2025) further proposes to align each gradient step to the full fine-tuning. Li et al. (2024a) develop resfurte-efffiient SAM variant, called Balancednnes-AAarr Regularization (BAR), tailored for optimization. In this paper, we improve LoRA by optimiz- ing the sharpness of the loss landscape in the full parameter space, and our approach orthogonal to previous works.
Despite recent efforts to improve LoRA performance, most studies focus solely on finding solutions performing well in the LoRA optimization space, specifically the rank-r matrix space M, (AW E R==xa rrank(AW) S r} (focusing single LoRA module). Following the well-established sharpness-aware minimization (SAM) objective (Foret et al., 2020), natural approach is to apply SAM to LoRA param- eters (LoRA-SAM) (Li et al., 2024a) with:
min max L LW + (B +BB+sB)(a (B+ (2) A,B ((
where L(-) denotes the loss objective for a specific task, EB m ,EA R2 are the adversarial weight pertur- bations over low-rank matrices, ((EA,)))) denotes the total norm of weight perturbations (typically using the fg-norm), and is the neighborhood radius.
However, focusing solely on the properties of the optimiza- tion space defined by LoRA parameters may have limita- tions. During inference, the low-rank adaptation AW is merged into the pre-trained weights W. A solution that performs well within the LoRA space may be situated in a sharp region of the full parameter space, as illustrated in Fig- ure 1, which could potentially harm overall generalization. To be more clear, employing first-order Taylor expansion for approximation to solve the inner maximum problem in Eqn. (2) (Foret et al., 2020), the equivalent weight perturba- tion applied to W by Eqn. (2) is
problems, such as LoRA
3. Method
In this section, we first briefly review Low-Rank Adapta- tion (LoRA). Then, we introduce our LoRA optimization objective that considers the landscape flatness of the full pa- rameter space. Finally, we describe our random perturbation generation strategy for effectively improving generalization.
ew = BEA + A + SBEA
BB (V W L) + (V LWA
(3)
3.1. LoRA: Low-Rank Adaptation
+ o² (Vw A B (VwL),
Based on the finding that DNNs' optimization happens in a subspace with much smaller dimension than the number of parameters (Li et al., 201 8a; 2022a), LoRA utilizes low- rank matrices to model the change for each layer's weights W E Rmxn during fine-tuning as AW- = BA, where
where Vw L is the gradient w.r.t. full parameter weights W and = p/vl B (V w L) + [[V L.AT I.. is (
scaling factor, with || denoting the Frobenius norm.
Notably, when B is initialized as zero as defaulted in Hu et al. (2022), B will remain small during the training (Hao
3

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
al., 2024) and Eqn. (3) roughly becomes:
and improving generalization. Let W W + BA denote the merged weight matrix W E R for linear layer
Ew R (Vw A.
(4)
with input dimension and output dimension m. Our design considers the following two key aspects:
We also empirically validate this in Appendix B. Eqn. (4) indicates that LoRA-SAM only optimizes sharpness within the column space spanned by A, which constitutes a small subspace of the full parameter space. As demonstrated in Table 6, applying SAM's sharpness optimization exclusively to LoRA parameters compromises generalization improve- ments compared to applying it to the full parameter space.
Filter structure: We aim to generate the weight per- turbation by filter (Bisla et al., 2022). There m filters W (W W input x E I Elements within a filter of a larger norm should receive a larger strength of perturbation.
W m,.) that process the
Therefore, it is crucial to consider the loss landscape in the full parameter space and identify a low-rank adaptation that positions the merged weights a flat region. To achieve this goal, we propose the following flat loss objective:
Input dimension: To ensure that the variance intro- duced during the forward pass by random weight per- turbation is independent of the input dimension, we scale the variance of noise added to each element by a factor of 1 //n, where is the input dimension.
min max L(W + BA +A++Ew EW), A,B 1ewwles
Our random weight generation scheme is formulated as:
where EW E Rmx" is the adversarial perturbation over the full parameters. However, directly applying SAM to optimize the sharpness of the full weight space has several disadvantages: 1) it doubles the training cost, which is less desirable for large models, and 2) it requires storing an additional copy of weights for restoring perturbation, which contradicts the principle of parameter--fffcient fine-tuning.
(Ew)is ~ N (0, [1
where a is a hyper-parameter that controls the perturbation strength. Figure illustrates the comparison between LoRA and Flat-LoRA.
We then analyze the effects of introducing random weight perturbation on the activation. Given an input x E R", and under the hypothesis that æ is a random vector where each element has the same variance var[æ;] and expectation E[x;;, we have:
To achieve a flatter loss landscape while maintaining time and memory efficiency, we propose relaxing the maximiza- tion problem in Eq. (5) to an expectation, resulting in the following Bayesian expected loss objective:
min A,B
Blewhes-W(0oot) L(W + BA+ (6)
var WorW [. = IW2.112 var(ax].
(8)
where o² denotes the noise variance, which will be further discussed in Section 3.3. This expected loss smooths the loss function in the full parameter space, as shown in the following lemma, promoting convergence to flatter minima.
After injecting random weight perturbation e, we have:
var [ww' + EW),,
Lemma 3.1 (Bisla et al.). Assume the loss function L(W) a-Lipschitz continuous and 8-smooth w.r.t. W under Ez- norm. The smoothed function functionn
|IW5 var[x;] + Σ var [ew
E(sw)s=~((0.0t) L(W (EW)
o²
[[W var[z;]] + n
|| 11 (var[æ:] + B (xv])
min : -s)-mmothh w.r.t. W.
= (1+ + o² >I[ 112 var[z;] + o² WW EE [z.].
To optimize Eqn. (6), we sample a noise matrix Ew for each optimization step and compute the perturbed gradient to optimize the low-rank matrices A and B. Note that the noise perturbation, generated based on merged model weights, climinates the need for additional gradient steps required by SAM. In practice, we recommend gradually increasing the perturbation strength to progressively recover flatter minima for better performance.
The injection of random weight perturbations Ew increases the forward activation variance by a factor of 1 + a², along with bias term determined by [[[r]]. This amplified vari- ance facilitates escape from sharp local minima. By in- corporating a scaling factor 1/n in the noise generation process, the variance increase becomes independent of input dimension n, as formalized in the following:
Proposition 3.2. For input & E E. with identical variance and mean across elements, injecting random weight pertur- bations according to Eqn. (7) increases the output variance independently of the input dimension n.
3.3. Effective Random Perturbation Generation
In this section, we introduce our approach for generating random weight perturbations aimed optimizing sharpness

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Additionally, we note that this variance would not increase exponentially during the forward propagation of the network due to the existence of layer normalization.
performance gains of 0.48% and 0.57%, respectively. The improvements are particularly notable on smaller datasets, such as CoLA and MRPC, with gains of 1.19% and 0.94%, respectively, at rank 16. This is because smaller datasets are more prone to overfitring, and Flat-LoRA effectively mitigates this issue, leading to greater performance improve- ments compared to LoRA.
Storing random seed for memory efficiency. Mem- ory cost is crucial for parameter-efficien fine-tuning. Optimizing Eqn. (6) requires generating random pertur- bation Ew and computing gradient WW + BA + Ew). While storing the full weight perturbation for large models would be prohibitive, it is sufficient store only the seed for the random generator and filter norms [IIW,,11. [W2 12 ... ||",, 113. This allows for the re- construction of Ew when needed. This approach requires minimal memory overhead (i.e.. ((m)), in contrast to SAM, which requires storing a full perturbation copy (O(m x n)) when optimizing sharpness in the full parameter space.
4.2. Image Classification
Setting. We fine-tune the CLIP VIT-B/32 model on five image classification tasks, including CIFAR- 10/100 (Krizhevsky & Hinton, 2009), Cars (Krause et al., 2013), SVHN (Netzer al., 2011), 1), and DTD (Cimpoi et al., 2014). We resize all input images to a size of 224 x 224 and freeze the classification head. We experiment with LoRA using ranks of 8 and 16 and fine-tune the models for 10 epochs under a cosine annealing schedule. The learning rate is set to 0.0005 for LoRA and Flat-LoRA and 0.0001 for full fine-tuning, with a weight decay of 0.1. The per- turbation strength is set to 0.15 for Flat-LoRA with cosine-increasing strategy. The mean and standard devia- tion are calculated over 3 independent trials.
Simple approach for mixed precision training. Mixed- precision training, common in large-scale applications, en- ables memory-efficient integration of perturbation injection during precision casting. Since this training mode main- tains both FP32 and FP/BF16 weight copies, we can in- ject perturbations during the half-precision auto-cast step before forward propagation, eliminating the need store perturbations filter norms. However, proach-storing
primary ap- storing perturbations via filter norms and random seeds-remaing more versatile as it functions independently of mixed-precision training.
Results. We measure the performance with classification accuracy and report the results in Table 2. Again, we observe that Flat-L LoRA significantly outperforms LoRA at both ranks 8 and 16, achieving averaged improvements of 0.56% and 0.74%, respectively. Notably, Flat-LoRA with rank 8 surpasses both LoRA with rank 16 and full fine-tuning by 0.28%. These results confirm the effectiveness of optimizing the loss landscape's sharpness in the full parameter space.
4. Experiments
In this section, we evaluate the performance of Flat-LoRA on diverse tasks: natural language understanding, image classification, dialogue generation, mathematical reason- ing, coding abilities, and text-to-image generation. We then demonstrate its enhanced out-of-domain generalization abil- ity, followed by ablation studies and discussions. The code provided in supplementary materials.
4.3. Large Language Model
Setting. To evaluate the scalability of Flat-LoRA, we further conduct experiments on large language models. Specifically, we fine-tune Llama 2-7B (Touvron et al., 2023) on three tasks: chat, muth, and code, following Wang et al. (2024). We use learning rate of 5e-4 and employ a cosine learning rate scheduler with warmup ratio of 0.03. The LoRA rank is set to 8 with LoRA alpha 16, and the training epoch is set to 2. The backbone uses BF16 precision, with the parame- ters of LoRA modules set to FP32 precision. For chat task, we fine-tune the model on WizardLM (Xu et al., 2023) and test on the MT-Bench dataset (Zheng et al., 2023). For math task, we fine-tune the model on MetaMathQA (Yu et al., 2024) and evaluate on GSM8K evaluation set (Cobbe et al., 2021). For code task, we fine-tune the model on Code- Feedback (Zheng al., 2024) and evaluate it on HumanEval (Chen et al., 2021). Training uses 52K for chat and 100K samples for math and code tasks. The random perturbation strength o is set to 0.05 with a cosine-increasiee strategy.
4.1. Natural Language Understanding
Setting, We fine-tune the T5-Base model on several datasets from the GLUE benchmark, including MNLI. SST, CoLA, QNLI, and MRPC, following Wang et al. (2024). Perfor- mance is evaluated the development set using accuracy as the primary metric. We use LoRA with rank 8 and LoRA alpha 16. We fine-tune the models with 10 epochs using a cosine learning rate schedule; except for MNLI and QNLI, we use epoch. We use learning rate of 0.0005 for LoRA fine-tuning and 0.0001 for full fine-tuning. The random per- turbation strength o is set to 0.05 with a cosine-increasing strategy. Mean and standard deviation are calculated over 3 independent trials.
Results. As shown in Table 1, Flat-LoRA consistently outperforms LoRA at ranks 8 and 16, achieving average
Results. We measure the performance of the chat task by the first-turn score with GPT-4, the math task by accuracy,
5

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
Table I: Results (%) on fine-tuning T5-base with a subset of GLUE datasets.
Method
MNLI
SST2
CoLA
QNLI
MRPC
Avg.
Full
#6.19
94.15
82.84+0.12
93.10=0.01
89.22.0.23
89.10
LoRA (r 8)
86.24
94.25+0.07
82.87+0.22
93.06=0.03
88.5640.37
88.99
Flat-LoRA (r 8)
86.2040.04
94.75
83.61 10.38
93.16
89.59
89,47
LoRA (r 16)
86.49 10.06
94.5210.21
82.89 10.44
92.97 1..05
88.89 10.44
89.15
Flat-LoRA (r 16)
40.01
94.84=0.02
84.08
93.28
89.83+0.34
89.72
Table 2: Results (%) on fine-tuning CLIP ViT-B/32 with image classification datasets.
Method
CIFAR-10
CIFAR-100
Cars
SVHN
DTD
Avg.
Full FT
97.99
89.06
13.30
97.4410.00
76.80
86.92
LoRA (r 8)
97.90
87.74
73.22
97.49
10.08
76.86.10.4
86.64
Flat-LoRA (r 8)
98.09
88.6440.23
74.17 10.71
97.59 10.04
77.51
87.20
LORA (r 16)
97.99 10.03
88. 10.23
73.80.1042
97.5610.08
77.3410.32
86.92
Flat-LoRA (r 16)
40.04
89.27
74.89 #0.52
97.71
78.24
87.66
and the code task by PASS@1 metric. From the results in Table 3, we observe that Flat-LoRA significantly enhances LoRA's performance, achieving an improvement of +0.20 on the MT-Bench dataset, 18% on the GSM8K dataset, and +3.08% on the Human-Eval dataset. Notably, these gains are substantially larger than those observed smaller models, such T5-base and CLIP VIT-B/32, highlight- ing the significance of pursuing flat minima for large-scale models. Moreover, the baselines we adopted are consid- erably stronger than those reported in previous works; for instance, we achieve 57.47% (ours) versus 42.08% (Wang al. (2024)) for LoRA on the GSM8K dataset. Despite these stronger baselines. Flat-LoRA continues deliver significant accuracy improvements over the standard LoRA, demonstrating its effectiveness in enhancing generalization.
constant learning rate of 0.0001. The batch size is set to 1. The LoRA rank and alpha are set to 4. The random perturbation strength a is set to 0.1 for Flat-LoRA. Other hyperparameters are set to default values.
Results. As shown in Figure Flat-LoRA exhibits better personalization than LoRA while maintaining better gen- eration ability. For instance, in the second column, the image generated by Flat-LoRA includes a distinctive square behind the bird, aligning more closely with the "icon" fea- ture present in the training images (top row). Furthermore, Flat-LoRA more effectively preserves the concept of eyes, whereas, in columns 1, and 5, the birds generated by LoRA missing eyes.
Table 3: Results on fine-tuning Llama 2-7B.
3 Icon 120 boai)
Method
MT-Bench
GSM8K
Human-Eval
Full FT
5.30
59.36
35.3 12.13
LoRA (r=8) 8)
5.96+0.08
57.47-0.11
24.85
LoRA
LORA (r
6.1640.05
60.65-0.63
27.93
Maa-LoRA
4.4. Text-to-Image Generation
Setting, We fine-tune the SDXL model (Podell et al., 2023) with the pipeline of Dreambooth (Ruiz et al., 2023) and the scripts implemented by HuggingFace. The finetuning dataset, 3D Icons contains 23 training images, all of which have a square. We fine-tune the model for 500 steps with
Prompt: TOK icon of flying bird, in the style of TOX
Figure 3: Images generated by SDXL fine-tuned with LoRA and Flat-LoRA on 3D icon datasets. Each column uses the same seeds for fair comparison.
htttpas//huggingeee wwt.e/////aaatteeeeeeeee w/wptt////nnnnnnnnommo icon
6

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Table 5: Comparison with other LoRA variants. The ex- periments are conducted on the GLUE subsets using the T5-Base model.
+1.38
LoRA Flai-LoRA
+2.27
70
Perfomance
+3.04
60
Dataset
+3.56
Method
80
CoI.A
MRPC
40
Baseline Methods
LoRA (Hu al., 2022)
82.87 1:022
88.56.4..3
30
Corruption Level
PISSA (Meng et al., 2024)
83.1840.24
88.96 05.9444.44
LoRA-GA (Wang et al., 2024)
83.13
88.7320.48
DoRA (Liu et al., 2024)
83.1640.1 15
89 89,46.10.37
Figure 4: Performance comparison LoRA and Flat-LoRA across different corruption levels of CIFAR-100-C. The model is fine-tuned on CIFAR-100 with CLIP ViT-B/32.
AdaLoRA (Zhang al., 2023b)
82.58 H0.56
48.79
LoRA+ (Hayou al., 2024)
82.65 +0.22
89.30
Our Methods
Flat-LoRA
83.61 +0.38
89.59
Flat-PiSSA
8
89.8 49.89
4.5. Out-of-Domain Generalization
Flat-LoRA-GA
83,41
89,20
Flat-DoRA
83.5610.27
89.99 10.47
Flat minima have been shown to better accommodate dis- tributional shifts between training and test data, thereby improving out-of-domain generalization. This property is particularly critical for pretrained vision and language mod- els, which are designed for a wide range of applications. Below, we explore this property of Flat-LoRA in detail.
Alat-AdaLooRA
83.13
89.23
Flat-LoRRA+
83.5610.46
89.61 10.41
4.6. Integration with Other Methods
In this section, we compare Flat-LoRA with recently pro- posed LoRA variants, including PiSSA, LoRA-GA, DoRA, AdaLoRA, and LoRA+. Experiments are conducted on the CoLA and MRPC datasets using the T5-base model with LoRA rank 8. The results are presented in Table 5. We observe that Flat-LoRA consistently outperforms previous methods on both datasets by 0.53% and 0.13%, respectively. Furthermore, the flat loss objective can be seamlessly in- tegrated with the previous approaches yield consistent improvements on both datasets by 0.91% and 0.93%, re- spectively. Note that these improvements are achieved at minimal additional cost, as shown in Table 7. This high- lights the scalability of our approach and the effectiveness of considering the sharpness of the full parameter space.
Corruption datasets. We focus on image classification tasks evaluate the robustness of Flat-LoRA under data distribution shifts. Specifically, we fine-tune CLIP VIT- B/32 on CIFAR- 100 and evaluate the model on corrupted CIFAR-100-C (Hendrycks & Dietterich, 2019). The results across varying levels of corruption severity are presented in Figure 4. Flat-LoRA consistently outperforms LoRA, with performance gains increasing as corruption severity rises, from +1.38% at level +3.56% at level 5. These results demonstrate that the flatter minima identified by Flat-LoRA enhance out-of-domain generalization compared to LoRA.
Instruction following. We fine-tune the Ilama 2-13B model on the Alpaca dataset (Taori et al., 2023), which simulates real-world variability and prepares the model to handle unseen or shifted distributions at test time. The model is evaluated on InstructEval (Chia al., 2023), instruction-following benchmark, using the official code provided by Chia et al. (2023). The experimental setup follows Ren et al. (2024). From the results in Table 4, we observe that Flat-LoRA consistently outperforms LoRA. No- tably, improvements on DROP and Human--Evl are more pronounced (+0.71% and +1.83% respectively).
4.7. Ablation Studies and Discussion
LoRA Fly-Lo8
Pyn-1oR
y
80.0
20.0
03.5-
17.5
17.0
87.5
Raat
Rant
Table 4: Results on the Llama 2-13B model on the Alpaca datasets and evaluate the performance using the InstructEval metrics.
tasks. We fine-tune
(a) MRPC with TS-Base
(b) CIFAR-100 with ViT-B/32
Figure 5: Performance comparison across different LoRA ranks. Keeping the LoRA alpha fixed 16, we vary the LoRA ranks among {1, 4, 16, 64). The results are averaged over three independent trials.
Method
MMLU
DROP
BBH
Human-Eval
Full
o2.36_0.45
38.23
35.38 15.44
LoRA = 8)
51.22
37. 37.26+0.63
34.77 13.01
Results under different LoRA ranks. Following the set- tings in Section 4.1 and 4.2, we evaluate the performance
Flat-LoRA (r 8)
318880.5
38.18
35.22
15.24
7

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
Table 6: Comparison with SAM on the GLUE subsets using the T5-Base model.
Method
Flat Space
CoLA
MRPC
Additional Memory
Training time
LoRA
40.59
88.56
1x
LoRA+SAM
A, B
83.3 #0.48
88. 88.98
O( + x
2x
LoRA+SAM
W
83.67 10.39
89.26 89.26.10.2
O(m x n))
2x
Flat-LoRA
A, B
83.19 70
88.81 10.51
O(m+ +r)
1x
Flat-LoRA
W
83.61 +0.38
$9.59
((m)
1x
of Flat-LoRA under different LoRA ranks. The results are shown in Figure 5. We observe that Flat-LoRA consistently outperforms LoRA across different LoRA ranks by +1.10% on MRPC and +1.15% on CIFAR-100. Even at LoRA rank which is typically underfitting, Flat-LoRA still delivers significant performance boost over LoRA. This highlights the importance of considering the sharpness of the full pa- rameter space. Additionally, as the LoRA rank increases, we observe that LoRA's performance can degrade due to overfitting, particularly on MRPC, which is a small dataset with 3.7k data points. Flat-LoRA effectively mitigates this overfitting issue by identifying flatter minima that generalize better. Thus, conclude that Flat-LoRA enhances LoRA fine-tuning performance not only in underfitting scenarios, where the rank is low and limited information from the full parameter space is explored, but also in high LoRA rank situations, where the risk of overfitting is more pronounced.
with Section 4.3, and we use micro-batch size of 2, running on an NVIDIA GeForce RTX 4090 GPU. Flat-LoRA is im- plemented based on our default random seed approach. We observe that Flat-LoRA adds minimal overhead compared to LoRA- only 0.12GB of extra memory and minutes of training time. These results highlight that Flat-LoRA can be conveniently integrated into LoRA training with little additional overhead.
Table 7: Comparison of memory and time usage
Method
Memory
Training Time
GSM8K (%)
LoRA
23.49GB
7h 22min
57.47
Flat-LoRA
23.61GB
7h 33min
60.6540.63
Comparison with SAM. We compare Flat-LoRA to SAM integrated with LoRA across different flat spaces: applying SAM's sharpness optimization to the full parameter space (W) and to the LoRA parameters (A, B). Following the setup described in Section 4.1, we evaluate perturbation radii over {0.001, 0.003, 0.005, 0.01, 0.05, 0.1, 0.2, 0.5), finding that p = 0.05 yields optimal performance when applied to the full parameter space (W), while p = 0.003 is optimal for the LoRA parameters (A, B). From the results in Table observe that applying SAM to the full param- eter space (W) consistently outperforms its application to the LoRA parameters (A, B), achieving improvements of +0.36% on CoLA and +0.28% on MRPC. However, SAM over W incurs an additional memory overhead of O(m x n) to store adversarial weight perturbations, rendering it im- practical for parameter-efficien training. By contrast, Flat- LoRA achieves performance comparable to, or better than, SAM applied to W, while requiring only O(m) additional memory. Furthermore, Flat-LoRA preserves the training efficiency of vanilla LoRA (1 x), whereas SAM-based ap- proaches double the training time (2x) due to the need for additional gradient computations.
Landscape visualization. In Figure 6, we visualize the surfaces of the loss landscape for LoRA and Flat-LoRA at ranks I and 16. Following the technique proposed by Li et (2018b), we plot the loss surface along random "filter- normalized" directions in the full parameter space (W). For both LoRA and Flat-LoRA, the merged weights used for visualization. The results demonstrate that Flat-LoRA consistently achieves significantly flatter loss landscape compared to LoRA at both ranks. Notably, when the LoRA rank is lower, the corresponding loss landscape tends to be sharper, highlighting the importance of optimizing the sharpness in the full parameter space.
5. Conclusion
We present Flat-LoRA, an efficient low-rank adaptation method that optimizes the sharpness of the loss landscape in the full parameter space. Unlike conventional sharpness- aware minimization approaches that impose heavy com- putation and memory overhead, we employ the Bayesian expectation loss objective to pursue flat minima and design refined generation schemes for random weight perturba- tions while maintaining efficiency. Extensive experiments across natural language processing and computer vision demonstrate Flat-LoRA's effectiveness in improving both in-domain and -ft-ain generalization.
Memory and time costs. In Table 7, we report the memory and time usage for fine-tuning MetaMathQA datasets using the Llama 2-7B model. The training settings are the same
8

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
G., al. Evaluating large language models trained code. arXiv preprint arXiv:2107.03374, 2021.
Chia, Y. K., Hong, P., Bing, I.., and Poria, S. Instructeval: Towards holistic evaluation of instruction-tunee large lan- guage models. arXiv preprint arXiv:2306.0477 047577 2023.
94
2A
022
63
-64
-44
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
50
0.
-04
-2A
(a) LoRA (r 1)
(b) Flat-LoRA (r 1)
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, Nakano, R., et al. Training verifiers solve math word problems. arXiv preprint arXiv:2110.114 14168..1...,,..... 2021.
Dinh, L.,, Pascanu, R., Bengio, S., and Bengio, Y. Sharp minima can generalize for deep nets. In International Conference on Machine Learning (ICML), 2017.
-22
-0.2
0.2
.02
-22
0.0
0.2
0.2
(c) LoRA (r 16)
(d) LoRA 16)
Duchi, J. C., Bartlett, P. L., and Wainwright, M. Ran- domized smoothing for stochastic optimization. SIAM Journal Optimization, 22(2):674-701, 2012.
Figure 6: Loss landscape visualization in the full parameter space. The experiments are conducted on CIFAR-100 with CLIP ViT-B/32.
Foret, P., Kleiner, A., Mobahi, H., and Neyshabur, B.
Sharpness-awart minimization for efficiently improving generalization. arXiv preprint arXiv:2010.01412. 2020.
Impact Statement
Girshick, R., Donahue, J., Darrell, T., and Malik, Rich fea- ture hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
This paper presents work whose goal advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Gur-Ari, G., Roberts, D. A., and Dyer, E. Gradient descent happens in tiny subspace. arXiv preprint arXiv: 18Xx:18122.0454, 2018.
Acknowledgment
This work was supported by National Key Research Devel- opment Project (2023YFFI 104202) and National Natural Science Foundation of China (62376155).
Hao, Y., Cao, Y., and Mou, L. Flora: Low-rank adapters are secretly gradient compressors. International Confer- ence on Machine Learning (ICML), 2024.
Hayou, S., Ghosh, N., and Yu, B. Lora+: Efficient low rank
References
adaptation of large models. In International Conference on Machine Learning (ICML), 2024.
Bisla, D., Wang, J., and Choromanska, A. Low-pass fil- tering sgd for recovering flat optima in the deep learn- ing optimization landscape. In International Conference on Artificial Intelligence and Statistics, pp. 8299-8339. PMLR, 2022.
He, K., Zhang, X., Ren, S., and Sun, J. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE inter- national conference on computer vision (ICCV), 2015.
Chaudhari, Choromanska, A., Soatto, S., LeCun, Y., Bal- dassi, C., Borgs, C., Chayes, J.. Sagun, L., and Zecchina, R. Entropy-sgd: Biasing gradient descent into wide val- leys. In International Conference on Learning Represen- tations (ICLR), 2017.
Hendrycks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. International Conference on Learning Represen- tations (ICLR), 2019.
Hochreiter, S. and Schmidhuber, Simplifying neural nets by discovering flat minima. Advances in Neural Information Processing Systems (NeurIPS), 1994.
Chen, M., Tworek. J., Jun, H., Yuan, Q., Pinto, H. P.D.O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman,

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Hochreiter, S. and Schmidhuber, J. Flat minima. Neural
Li, S., Yang, Y., Shen, Y., Wei, F., Lu, Z., Qiu, L.., and Yang, Y. Lorasc: Expressive and generalizable low-rank adaptation for large models via slow cascaded learning. In Findings of the Association for Computational Linguis- tics: EMNLP 2024, pp. 12806-12816, 2024b.
computation, 1997.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., et al. Lora: Low-rank adaptation of large language models. In International Conference Learn- ing Representations (ICLR), 2022.
Li, T., Tan, L.,, Huang, Z., Tao, Q. Liu, Y., and Huang, X. Low dimensional trajectory hypothesis is true: Dnns can be trained in tiny subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 45 (3):3411-3420) 2022a.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. arXiv preprint 1803.00003. 2018.
Li, T., Wu, Y., Chen, S., Fang. K., and Huang, X. Subspace adversarial training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022b.
Keskar, N. S., Mudigere, D., Nocedal, J., Smelyanskiy, M., and Tang, T. P. On large-batch training for deep learn- ing: Generalization gap and sharp minima. Interna- tional Conference on Learning Representations (ICLR), 2017.
Li, T. Yan, W., Lei, Z., Wu, Y., Fang. K., Yang. M., and Huang. X. Efficient generalization improvement guided by random weight perturbation. arXiv preprint arXiv: 11..22211111488 2022c.
Kolesnikov, A., Beyer, L., Zhai, X., Puigcerver, J., Yung, J., Gelly, S., and Houlsby, N. Big transfer (bit): General visual representation learning. In European conference on computer vision (ECCV), 2020.
Li, T., Jiang, W., Liu, F., Huang, X., and Kwok, J. T. Leamn ing scalable model soup on single gpu: An efficient subspace training strategy. In European conference on computer vision (ECCV), 2024c.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object rep- resentations for fine-graid -re-igraied categorization. In Proceed- ings of the IEEE international conference on computer vision workshops (ICCVW), 2013.
Li, T., Tao, Q. Yan, W., Wu, Y., Lei, Z., Fang, K., He, M., and Huang, X. Revisiting random weight perturbation for efficiently improving generalization. Transactions on Machine Learning Research (TMLR), 2024d. ISSN 2835- 8856. URL https: http::://ppeeeeeaetttt /openreviewweeeeeeeeeeeeee net forum? i id-WobgOHpooXX
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report, 2009.
Kwon, J., Kim, J., Park, H., and Choi, I. K. Asam: Adaptive sharpness-aware minimization for scale-invariant learn- ing of deep neural networks. In International Conference on Machine Learning (ICML), 2021.
Li, T., Zhou, P., He, Z., Cheng, X., and Huang, X. Friendly sharpness-aware minimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024e.
Langley, P. Crafting papers on machine learning. In Langley, P. ), Proceedings of the 17th International Conference on Machine Learning (ICML 2000). pp. 1207-1216, Stan- ford, CA, 2000. Morgan Kaufmann.
Liu, S.-y., Wang, C.-Y., Yin, H., Molchanov, P., Wang, Y.C.. F., Cheng, K.-T., and Chen, M.-H. Dora: Weight- decomposed low-rank adaptation. In International Con- ference on Machine Learning (ICML), 2024.
Meng, F., Wang, 7.,, and Zhang, M. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Li, B., Zhang, L., and He, N. Implicit regularization of sharpness-awwen minimization for scale-invariant prob- lems. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised feature learning. Granada. 2011.
Li, C., Farkhoor, H., Liu, R., and Yosinski, J. Mcasuring the intrinsic dimension of objective landscapes. In Interna- tional Conference on Learning Representations (ICLR), 2018a.
Li, H., Xu, Z., Taylor, G., Studer, C., and Goldstein, T. Visualizing the loss landscape of neural nets. In Advances in Neural Information Processing Systems (NeurIPS), 2018b.
Podell, D.,, English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., and Rombach, R. Sdxl: Im- proving latent diffiusion models for high- resolution image synthesis. arXiv preprint arXiv:2307.01955 2023.
10

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning 2021.
Wu, C., Wu, F,, Qi, T., and Huang. Y. Noisytune: A little noise can help you finetune pretrained language models better. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers); pp. 680-685, 2022.
Wu, D., Xia, S.-T., and Wang, Y. Adversarial weight pertur- bation helps robust generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Ren, P. Shi, C., Wu, S., Zhang, M., Ren, Z., de Ri- jke, M., Chen, Z. and Pei, J. Mini-ensemble low- fine-tuning, arXiv
rank adapters for parameter-efficttne preprint ar ArXiv:2402. 17263, 2024.
Xu, C., Sun, Q., Zheng, K., Geng, X., Zhao, P., Feng. J.. Tao, C., and Jiang, D. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304. 12244, 2023.
Ruiz, N., Li, Y., Jampani, V., Pritch, Y., Rubinstein, M., and Aberman, K. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and patlern recognition (CVPR), 2023.
Yang, Y., Li, X., Zhou, Z., Song, S., Wu, J., Nie, L., and Ghanem, B. Corda: Context-oriented decomposi- tion adaptation of large language models for task-aware parameter-efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Taori, R., Gulrajani, I. Zhang, T., Dubois, Y., Li, X., Guestrin, C., Liang, P., and Hashimoto, T. B. Stanford alpaca: An cestructioo-followinn Ilama model, 2023.
Yu, L., Jiang, W., Shi, H., Yu, Liu, Z.., Zhang, Y., Kwok, J. T., Li, Z., Weller, A., and Liu, W. Metamath: Boot- strap your own mathematical questions for large language models. In International Conference on Learning Repre- sentations (ICLR), 2024.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. Llama 2: Open foundation and fine- tuned chat models. arXiv preprint arXiv:2307.09250 2023.
Zhang, Q., Chen, M., Bukharin, A., He, P., Cheng, Y., Chen, W., and Zhao, Adaptive budget allocation for parameter-efficte fine-tuning. In International Confer- ence on Learning Representations (ICLR), 2023a.
Tsuzuku, Y., Sato, I., and Sugiyama, M. Normalized flat minima: Exploring scale invariant definition of flat min- ima for neural networks using pac-bayesiat analysis. In International Conference Machine Learning, pp. 9636- 9647. PMLR, 2020.
Zhang, Q., Chen, M.,, Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W., and Zhao, T. Adalora: Adaptive budget allocation for parameter-efficieet fine-tuning, In International Conference on Learning Representations (ICLR), 2023b.
Wang, S., Yu, L.,, and J. Lora-ga: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Zhao, J., Zhang, Z., Chen, B., Wang, Z., Anandkumar, A., and Tian, Y. Galore: Memory- IIm training by gradient low-rank projection. In International Conference on Machine Learning (ICML), 2024.
Wang, Y., Ivison, H., Dasigi, P., Hessel, J., Khot, T., Chandu, K., Wadden, D... MacMillan, K., Smith, N. A., Beltagy, I., et al. How far can camels go? exploring the state of instruction tuning on open resources. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z.,, Zhuang, Y., Lin, 7.,, Li, 7.,, Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems (NeurIPS), 2023.
Wang, Z. and Liang, J. Lora-pro: Are low-rank adapters properly optimized? In International Conference Learning Representations (ICLR), 2025.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opencodeinterpreeer:: Integrat- ing code generation with execution and refinement. In Findings of the Association for Computational Linguistics ACL, 2024
Wang, Z. and Mao, Y. On the generalization of models trained with sgd: Information-theorede bounds and im- plications. In International Conference on Learning Rep- resentations (ICLR), 2021.
Wen, W., Wang, Y.. Yan, F., Xu, C., Wu, C., Chen, Y., and Li, H. Smoothout: Smoothing out sharp minima to improve generalization in deep learning. arXiv preprint arXiv:1055.077 07898, 2018
Zheng, Y., Zhang, R., and Mao, Y. Regularizing neural net- works via adversarial model perturbation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.

---

Flat-LoRA: Low-Rank Adaptation over a Flat Loss Landscape
Zhuang, J., Gong, B., Yuan, L.,, Cui, Y., Adam, H., Dvornek, N., Tatikonda, S., Duncan, J., and Liu, T. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations (ICLR), 2022.
12

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
A. Training-vs-Test Loss and Generalization Gap Curves
We plot the training-vs-test loss curves and generalization gap on CIFAR-100 and MRPC datasets Figure Al. The results show Flat-LoRA exhibits slightly higher training loss than LoRA, with a smaller generalization gap between training and test accuracies. Thus, we can conclude that the gains of Flat-LoRA are not due to lower training loss but due to better optimization that confers better generalization.
CIFAR-100 with CLIP VIT-B/32
CIFAR-100 with CLIP VIT-B/32
12
0.7
LoRA
Generalization Gap (%)
10
Flat-LoRA
0.6
8
0.5
Lo58
0.4
2
Training (LoRA)
Training loss LORA)
Test loss (LoRA)
-2
Test loss (Flat-LofA)
-4
0.0
2
4
8
Epochs
Epochs
(a) Training/test loss curves on CIFAR-I00
(b) Generalization gap curves on CIFAR- -1000
MRPC with T5-base
MRPC with T5-base
0.25
LoRA
Generalization Gap (%)
Hatt-LoRA
0,20
Loss
0.15
0.10
Training loss (LoRA)
Training loss (Flat-LoRA)
2
0.05
Test loss (LoRA)
Test loss (Flat-LoRA)
0
0.00
2
Epochs
Epochs
(c) Training/test loss MRPC.
(d) Generalization gap curves MRPC.
Figure A1: Training-vs-test loss and generalization gap curves comparison. Flat-LoRA exhibits slightly higher training loss
than LoRA, with smaller generalization gap between training and test accuracies.
B. Validation on the Components of ew
In this section, we validate the approximation of Eqn. (4), i.e., Ew RS EBA == ()V A. We conduct an experiment on the MRPC dataset with T5-base model and record the statistics of || during the training. The results are shown in Figure A2. We observe that ||saAA] |aaw ||
[|ew
0.95 throughout the training. This validates the approximation of Eqn. (4).
C. Extending Perturbation to All Layers
We extend the injection of random weight perturbation to all layers, referred to as "Flat-LoRA (all)". Specifically, we additionally add perturbations to layernorm layers, biases, class embeddings, etc. We generate noise based on the absolute weight [W] From the results in Table A1, we observe that Flat-LoRA (all) indeed improves performance, though the improvement not large as Flat-LoRA (Linear) over LoRA.
13

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
MRPC on T5-base
1.00
EEAA
Tw
0.99
0.98
0.97
Ratio
0.96
0.95
0.94
0.93
200
400
600
800
1000
1200
Steps
Figure A2: Statistics of ||= the actual weight perturbation of LoRA-SAM ew is almost determined by =B.. This indicates that LoRA-SAM primarily optimizes the sharpness within the subspace spanned by A. The experiment is conducted on the MRPC dataset with the TS-base model.
||sw We observe that
|| remains almost above 0.95 throughout training, indicating that [[ew
Table A1: Results on CIFAR-10/100 with CLIP VIT-B/32.
Method
CIFAR-10
CIFAR-100
LoRA
97.90
87.74
Flat-LoRA (linear)
98.0940.04
88.6440.23
Flat-LoRA (all)
98.1340.03
88.7640.19
14

---

Flat-LoRA: Low-Rank Adaptation over Flat Loss Landscape
D. Ablation the Variance Magnitude
To evaluate the impact of the perturbation variance magnitude for Flat-LoRA, we vary o among {0, 0.01, 0.05, 0.10, 0.15, 0.20) and fine-tune CIFAR-100 on CLIP ViT-B/32 and ViT-L/14 well GSM8K on Llama
2-7B and Llama 2-13B. From the results in Table A2 and Table A3, observe that the optimal results are achieved when is 0.05 or 0.10 for both datasets and different network sizes. Hence, suggest 0.05/0.10 for practice usage.
Table A2: Results on CIFAR-100 with different variance magnitude.
0
0.01
0.05
0.10
0.15
0.20
ViT-B/32
87.74
88. 88.14
88.37
88.65
88.64
88.0640.31
VIT-L/14
92.13
40.07 10.07
92.63
93.11 10
92.98 20.1
92.4610.03
Table A3: Results on GSM8k with different variance magnitude.
0
0.01
0.05
0.10
0.15
0.20
LLama 2-7B
57.47
58.45
20.63
60.56
60.08 7.776
58.50
LLama 2-13B
66.76=0.23
67.02+0.67
67.75
68.11
67.66.0097
67.34
E. More Comparisons to LoRA's Varints
In Table A4, we compare Flat-LoRA with more recently proposed LoRA varints, including oBAR/nBAR (Li et al., 2024a), LoRA-Pro (Wang & Liang, 2025), GaLore (Zhao et al., 2024), and CorDA (Yang et al., 2024). The experiments are conducted on the T5-base model with MRPC and CoLA datasets. We can observe that Flat-LoRA achieves competitive or better performance than those state-of-the-art variants.
Table A4: Performance comparison on MRPC and CoLA.
Methods
MRPC
CoLA
oBAR (Li et al., 2024a)
88.58
83.07
nBAR (Li et al., 2024a)
88.63 20.4
82.78
LoRA-Pro (Wang & Liang, 2025)
89.23
83.17 +0.28
GaLore (Zhao al., 2024)
88.90 18.90.1.12 1.12
83. 13.14, 10.57
CorDA (Yang et al., 2024)
89.76
83.3840.47
Flat-LoRA (Ours)
89.59 -0.37
83.61
15


Reviews:
[{"review_id": "IN1iO8pA4I", "reviewer": "Reviewer_aowQ", "summary": "The paper proposes “Flat-LoRA,” a parameter-efficient fine-tuning method that adds random weight perturbations (with an intelligent weight dependent scaling), in order to achieve flatter minima and improve generalization. Experimental results on both vision (CLIP and Stable Diffusion) and language (T5, Llama-2) tasks consistently show that Flat-LoRA outperforms baseline LoRA variants, with particular gains in low-data and out-of-domain scenarios.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. We address your concerns point by point as follows:\n\n---\n**Q1:** The paper does not explicitly show train-vs-test loss curves or a generalization gap. Hence, it is unclear whether the gains are from genuinely improved generalization rather than lower training loss or more favorable optimization dynamics.\n\n**A1:** We plot the train-vs-test loss curves and generalization gap on CIFAR-100 and MRPC datasets in [Figure 1](https://anonymous.4open.science/r/Flat-LoRA/rebuttal.pdf) and present the final training-vs-test loss and generalization gap below. The results show Flat-LoRA exhibits slightly higher training loss than LoRA, with a smaller generalization gap between training and test accuracies. Thus, we conclude that the gains of Flat-LoRA are not due to lower training loss but due to better optimization that confers better generalization.\n\nCIFAR-100 | Final train loss | Final test loss| Generalization gap (%)\n---|---|---|---\nLoRA|0.02±0.01|0.48±0.01|11.65±0.03\nFlat-LoRA|0.05±0.02|0.46±0.00|9.92±0.08\n\nMRPC | Final train loss | Final test loss| Generalization gap (%)\n---|---|---|---\nLoRA|0.03±0.03 | 0.25±0.00 | 10.96±0.18\nFlat-LoRA|0.04±0.03|0.20±0.01 | 9.50±0.22\n\n---\n**Q2:** The limitations of the theoretical treatment in Section 3.2 could be more clearly stated.\n\n**A2:** Thank you for this valuable feedback. We propose to make the following changes in the revision to more clearly state the limitation of LoRA-SAM:\n\n1. We will give a clearer and more rigorous derivation on the actual perturbation of LoRA-SAM in the full parameter space.\n\n2. We will include an experiment to validate the approximation $\\varepsilon_W \\approx \\varepsilon_B A=c(\\nabla_W L)A^\\top A$ (Eqn. (4)) by showing $\\frac{\\|\\varepsilon_BA\\|}{\\|\\varepsilon_W\\|}>0.95$ throughout the training. The demonstration experiment can be found at [Figure 2](https://anonymous.4open.science/r/Flat-LoRA/rebuttal.pdf).", "claims_and_evidence": "Two main claims are supported:\n\n* Flat-LoRA yields flatter solutions in the full parameter space rather than just in the low-rank subspace: The authors visualize the loss landscapes (Figure 6) and demonstrate flatter minima compared to standard LoRA.\n\n* Flat-LoRA significantly improves performance across multiple tasks (e.g., GLUE, text-to-image, code generation) and robustness: Comprehensive experimental results show gains over standard LoRA and other PEFT improvements.\n\nOne is not: \n\n* Flat-LoRA generalizes better: While the flatter loss landscape and improved performance can be indicative of better generalization, the paper does not explicitly show train-vs-test loss curves or a generalization gap. Hence, it is unclear whether the gains are from genuinely improved generalization rather than lower training loss or more favorable optimization dynamics.", "methods_and_evaluation_criteria": "The paper evaluates performance on standard benchmarks (GLUE, text/image tasks, code generation, etc.), making comparisons against strong LoRA baselines and other variants. This breadth of experiments aligns well with the parameter-efficient fine-tuning literature.", "theoretical_claims": "The authors argue that smoothing the loss in the full weight space leads to broader basins and better generalization. While the Bayesian expectation loss formulation is standard, the paper’s main theoretical link is that random perturbation akin to filter-wise noise effectively regularizes the final solution.", "experimental_designs_or_analyses": "Experiments are structured clearly, with appropriate baselines and repeated trials. The paper carefully examines both standard benchmarks and out-of-domain shifts (e.g., CIFAR-100-C corruption, instruction-following tasks). One limitation: results compare final performance but do not show training curves or highlight differences in train versus test loss. Explicit train–test curves or a direct measure of the generalization gap would strengthen the evidence that improved performance primarily stems from better generalization.", "relation_to_broader_scientific_literature": "Flat-LoRA extends the line of work on PEFT techniques by bringing insights from random perturbation based regularization and sharpness-aware minimization to improve generalization. There is not much that is particular to LoRA or PEFT beyond the observation that it can also be prone to sharp minima.", "essential_references_not_discussed": "No crucial references appear missing.", "other_strengths_and_weaknesses": "Strengths:\nThe paper raises an interest distinction which can motivate future works: flatness in the “low-rank” space versus actual flatness once merged with the frozen pre-trained weights. It demonstrates strong empirical results on diverse tasks, with small memory/time overhead.\nWeaknesses:\nThe paper does not explicitly demonstrate that performance gains arise from better train–test generalization. The limitations of the theoretical treatment in Section 3.2 could be more clearly stated.", "other_comments_or_suggestions": "NA", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dtrpzRxkmr", "reviewer": "Reviewer_wt1A", "summary": "This paper proposes Flat-LoRA, a novel approach to improving Low-Rank Adaptation (LoRA) by incorporating a Bayesian expectation loss objective and random weight perturbations to encourage flatter minima in the full parameter space, all while maintaining computational efficiency.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. We address your concerns point by point as follows:\n\n---\n**Q1:** The method can be viewed as a computationally efficient alternative to SAM rather than an entirely novel approach. \n\n**A1:** SAM has been proven to be an effective training strategy for improving generalization capability. However, its extension to large models has not been widely used due to the doubled computational cost and additional memory overhead, especially in parameter-efficient finetuning scenarios (e.g., LoRA). Unlike LoRA-SAM, which naively applies SAM to LoRA, our Flat-LoRA provides a simple and easy-to-implement solution that halves the computational cost and searches for a broader flat solution. As a result, we for the first time enable practical sharpness-aware optimization in the broader full parameter space for the LLM fine-tuning task with little extra memory and computational cost. \n\n---\n**Q2:** The method is sensitive to the variance magnitude ($\\sigma$). The overall gains remain limited. \n\n**A2:** We fully understand your concern. But we would like to point out that the perturbations given by $\\sigma=0.2$ are generally too large. Instead, we suggest $\\sigma=0.05/0.10$. Our paper includes different tasks and networks (e.g., ViT, T5, LLama, SDXL) and shows that $\\sigma=0.05/0.10$ leads to consistent generalization improvements.\n\nTo further address this issue, especially in relation to different sizes of neural networks, we propose an improved perturbation generalization scheme that employs a scaling factor to make $\\sigma$ **independent of the network width** (Proposition 3.2). To validate this, we conduct experiments on the larger ViT-L/14 model. The results demonstrate that the optimal $\\sigma$ can be transferred from ViT-B/32 to ViT-L/14, and in all scenarios, the performance of Flat-LoRA surpasses that of LoRA. We will include these in the revision.\n\nCIFAR-100| LoRA | $\\sigma=0.01$ | $\\sigma=0.05$ | $\\sigma=0.10$ | $\\sigma=0.15$ | $\\sigma=0.20$ |\n---|---|---|---|---|---|---\nViT-B/32 | 87.74±0.13 | 88.14±0.22 | 88.37±0.41 | 88.65±0.35 | 88.64±0.23 | 88.06±0.31 \nViT-L/14 | 92.13±0.17 |  92.33±0.07 |  92.63±0.11 |  93.11±0.13 |  92.98±0.21 | 92.46±0.03\n\nThe limited improvements may be due to our use of a stronger training protocol compared to related works (e.g., LoRA-GA [1]), leaving less room for enhancement. Moreover, Flat-LoRA is a practical plug-in method with minimal extra cost, and its improvements are more significant on larger LLMs.\n\n[1] LoRA-GA: Low-Rank Adaptation with Gradient Approximation, NeurIPS'24\n\n---\n**Q3:** Clarify why Flat-LoRA outperforms LoRA+SAM. Add experiments comparing SAM and its variants with hyperparameter tuning to ensure fair comparisons. \n\n**A3:** Thanks for your insightful question. The advantages of Flat-LoRA over LoRA+SAM are twofold:\n\n(1) SAM and variants require twice the training time, which is impractical for large-scale models. Flat-LoRA addresses this limitation, making it more practical for real-world applications.\n\n(2) LoRA-SAM is to pursue flatness on the LoRA parameters, which only optimize a subspace of sharpness. Flat-LoRA uses perturbation on the full parameters and thus is capable of optimizing the sharpness of a broader space. \n\nSpecifically for the accuracy improvement, we think the direct reason is (2). \n\nTo better clarify the above discussions, we followed your suggestion and conducted a more detailed search for the perturbation radius $\\rho$ across $\\\\{0.001, 0.003, 0.005, 0.01, 0.05, 0.10, 0.20, 0.50, 1, 1.5, 2\\\\}$ for SAM and its variants. The results could be seen in the following table. \n\nT5-base|Flat space|MRPC |CoLA\n---|---|---|---\nLoRA|-| 88.56±0.37 | 82.87±0.59\nLoRA+SAM ($\\rho=0.003$)|A,B|88.98±0.22  | 83.31±0.48\nLoRA+GSAM ($\\rho=0.003$)|A,B|89.03±0.36 | 83.11±0.17\nLoRA+ASAM ($\\rho=0.05$)|A,B|89.12±0.44 | 83.23±0.44\nFlat-LoRA ($\\sigma=0.05$)|W| 89.59±0.37 | 83.61±0.38\n\nFor LoRA+SAM, the best performance is obtained by setting $\\rho=0.003$, which is significantly smaller than its typical value used for training full parameters (usually $\\rho=0.1$). Such a small perturbation makes the difference between LoRA and LoRA-SAM not significant. Frankly, we did not expect that the suitable value for $\\rho$ is very small, so $\\rho=0.003$ is out of the range of previous hyperparameter-tuning. We will update the result of LoRA-SAMs.\n\n**Q4:** Provide ablation studies on the impact of variance magnitude ($\\sigma$) in large language models.\n\n**A4:** We test different $\\sigma$ on LLama-2-7b/13b as below. We observe that $\\sigma=0.05/0.10$ is a good choice for both 7b and 13b models. We will add this in the revision.\n \n  GMS8k | LoRA | $\\sigma=0.01$ |  $\\sigma=0.05$ | $\\sigma=0.10$ | $\\sigma=0.15$ | $\\sigma=0.20$ \n --- | -- | --- | ---| --- | --- |---\n LLama-2-7b | 57.47±0.45 | 58.35±0.42 |  60.65±0.63|60.56±0.48 | 60.08±0.76 | 58.50±0.85\n LLama-2-13b | 66.76±0.23| 67.02±0.67 | 67.75±0.70| 68.11±0.53 | 67.66±0.97 |  67.34±1.17", "claims_and_evidence": "The paper’s claims are well-supported by empirical results, demonstrating improved robustness and efficiency across multiple tasks.", "methods_and_evaluation_criteria": "The authors conducted experiments on datasets spanning multiple domains, including Natural Language Understanding, Image Classification, Large Language Model, and Text-to-Image Generation. These benchmarks are well-matched to the research scope, making them appropriate for evaluating generalization.", "theoretical_claims": "n/a – The paper does not include formal theoretical proofs beyond conceptual justifications.", "experimental_designs_or_analyses": "I have reviewed all the experiments. The experimental design is well-structured and covers various domains and tasks. However, an additional ablation study on perturbation strength ($\\sigma$) in larger models would help assess the generalizability of the approach.", "relation_to_broader_scientific_literature": "The paper provides a simple yet effective method for improving LoRA training by leveraging perturbation-based optimization to encourage flatter minima.", "essential_references_not_discussed": "The paper adequately discusses relevant related works.", "other_strengths_and_weaknesses": "##### Strengths:\n- The paper is well-structured and clearly written, making it easy to follow.\n- Flat-LoRA provides a lightweight modification to LoRA training, leveraging random weight perturbations to encourage flatter minima.\n- The experimental analysis is comprehensive, covering various benchmarks and ablation studies, demonstrating its robustness and broad applicability.\n\n##### Weaknesses:\n- The method can be viewed as a computationally efficient alternative to SAM rather than an entirely novel approach. Specifically, it replaces min-max optimization with Bayesian expected loss, making the core idea an approximate acceleration of SAM. Thus, the performance improvements are incremental rather than groundbreaking.\n- The method is sensitive to the variance magnitude ($\\sigma$). Table C3 shows that when $\\sigma=0.2$, Flat-LoRA performs poorly on CIFAR datasets, suggesting a strong dependence on precise hyperparameter tuning. Additionally, even with optimal hyperparameters, the overall gains remain limited.", "other_comments_or_suggestions": "See the Questions for Authors section.", "questions_for_authors": "Q1. The paper states: \"Applying SAM directly to A, B shows no significant improvement over vanilla LoRA. In contrast, Flat-LoRA achieves comparable or superior performance to SAM.\" However, Flat-LoRA appears to be a simplified version of SAM, as it simplifies min-max optimization into Bayesian expected loss minimization. From a theoretical perspective, SAM should achieve superior performance given the same computational budget. Can you clarify why Flat-LoRA outperforms LoRA+SAM and whether this is due to better stability, improved optimization, or simply reduced computational overhead? It would be beneficial to add experiments comparing SAM and its variants with hyperparameter tuning to ensure fair comparisons. **I consider this the most critical issue—if well-addressed, I would be inclined to support the acceptance of this paper.**\n\nQ2. Could you provide ablation studies on the impact of variance magnitude ($\\sigma$) in large language models (Section 4.3)? Since $\\sigma$ plays a crucial role in the method’s effectiveness, further analysis on scalability and adaptability in larger models would better demonstrate Flat-LoRA’s robustness.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9b0lMDHjqG", "reviewer": "Reviewer_oboy", "summary": "**1. Summary of Contributions**:\nThe paper introduces **Flat-LoRA**, a novel method for low-rank adaptation (LoRA) that aims to find parameter-efficient fine-tuning solutions residing in a flatter region of the full parameter space. The authors identify that standard LoRA, while efficient, might converge to minima that appear flat within its low-dimensional optimization space but are sharp in the context of the original, full parameter space, potentially hindering generalization. To address this, Flat-LoRA employs a **Bayesian expectation loss objective with carefully designed random weight perturbations** applied to the merged weights (pre-trained weights + LoRA adaptations). This approach smooths the loss landscape in the full parameter space, encouraging convergence to flatter minima without incurring the significant computational and memory overheads associated with sharpness-aware minimization (SAM). A key contribution is a **memory-efficient perturbation generation strategy** that relies on storing random seeds and filter norms instead of full perturbation matrices. The paper demonstrates through extensive experiments across diverse tasks—including NLP, computer vision, and generative modeling—that Flat-LoRA improves both in-domain and out-of-domain generalization compared to standard LoRA and sometimes even surpasses full fine-tuning.\n\n*   **Quality of Writing/Presentation**:\n    *   The paper is generally **well-written and clearly structured**. The introduction effectively sets the context and motivates the proposed approach.\n    *   The explanation of LoRA and Flat-LoRA is lucid.\n    *   The figures (Figure 1, 2, 3, 4, 5, 6) are helpful in illustrating the concepts and results.\n    *   The experimental setup and results are detailed, allowing for potential reproducibility.\n    *   The inclusion of ablation studies and comparisons with related work enhances the presentation.\n    *   The appendix provides additional experimental results, further supporting the claims.\n    *   **Minor improvements could be made by ensuring consistent levels of detail in describing the experimental setups across different tasks and models.** For instance, more specific hyperparameter choices could be consistently provided in the main text or referenced clearly in the appendix.\n\n*   **Literature**:\n    *   The paper provides a comprehensive overview of related work on flat minima and generalization as well as low-rank adaptation techniques and their variants.\n    *   It appropriately cites key papers in both areas, including seminal works on flat minima (Hochreiter & Schmidhuber, 1997; Foret et al., 2020) and LoRA (Hu et al., 2022).\n    *   The discussion of the limitations of applying SAM directly to LoRA parameters (Li et al., 2024a) and the full parameter space effectively positions Flat-LoRA within the existing literature.\n    *   The comparison with recent LoRA enhancement methods (AdaLoRA, DoRA, LoRA+, PiSSA, LoRA-GA) demonstrates a strong understanding of the current state-of-the-art.\n\n**3. Pros and Cons**:\n\n*   **Pros**:\n    *   **Improved Generalization**: Flat-LoRA demonstrates significant improvements in both in-domain and out-of-domain generalization across diverse tasks and models.\n    *   **Computational Efficiency**: It maintains the training efficiency of standard LoRA, avoiding the doubled training cost of SAM.\n    *   **Memory Efficiency**: The method is memory-efficient by storing only random seeds and filter norms, unlike SAM which requires storing full perturbation matrices.\n    *   **Seamless Integration**: Flat-LoRA can be easily integrated with existing LoRA variants, leading to further performance gains.\n    *   **Principled Approach**: The use of Bayesian expectation loss provides a theoretical basis for seeking flatter minima in the full parameter space.\n    *   **Empirical Validation**: Extensive experiments on various tasks and models provide strong support for the effectiveness of the proposed method.\n    *   **Scalability**: Demonstrated effectiveness on large language models like Llama-2-7B.\n\n*   **Cons**:\n    *   **Hyperparameter Sensitivity**: The performance of Flat-LoRA depends on the choice of the perturbation strength (σ), and finding the optimal value might require tuning.\n    *   **Limited Theoretical Depth**: While the paper provides a clear empirical demonstration, a more in-depth theoretical analysis connecting the proposed approach directly to improved generalization in the context of LoRA could be beneficial.\n    *   **Potential Overhead (Minor)**: While minimal compared to SAM, Flat-LoRA does introduce a small additional memory overhead for storing filter norms and potentially a slight increase in computation time for generating and applying perturbations.\n\n**4. Classification of Concerns**:\n\n*   The **hyperparameter sensitivity to perturbation strength (σ)** is a **minor concern**. While tuning might be required, the ablation study provides some guidance on reasonable ranges.\n*   The **limited theoretical depth connecting the approach directly to LoRA generalization** is a **minor concern**. The strong empirical results mitigate this, but further theoretical investigation could strengthen the paper.\n*   The **potential minor overhead in memory and computation** is also a **minor concern**, as the experiments show it to be quite small compared to the benefits achieved.\n\n**5. Overall Assessment**:\n\nThe paper presents a **novel and effective approach, Flat-LoRA, for improving the generalization of Low-Rank Adaptation by optimizing for flatness in the full parameter space**. The method is well-motivated, theoretically grounded in Bayesian expected loss, and supported by extensive and convincing experimental results across a diverse range of tasks and models. The **key strengths** of the paper lie in its **significant performance improvements over standard LoRA, its computational and memory efficiency compared to sharpness-aware methods, and its ease of integration with existing LoRA techniques**. While there are minor concerns regarding hyperparameter sensitivity and theoretical depth, they do not significantly detract from the overall contribution.\n\n*   **Points of agreement:** I agree with the authors' premise that the flatness of the loss landscape in the full parameter space is crucial for generalization, even when using parameter-efficient methods like LoRA. The experimental results support their claim that Flat-LoRA can lead to significant performance improvements across various tasks and models. I also agree that the memory efficiency of their proposed perturbation strategy is a significant advantage, especially for large-scale models.\n*   **Anything learned from the target:** I learned a valuable insight into the potential limitations of solely focusing on the optimization landscape within the low-dimensional space of LoRA and the importance of considering its relationship with the full parameter space. The proposed method for efficiently approximating the effect of perturbations in the full parameter space through random seeds and filter norms is also a noteworthy technique.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. We address your concerns point by point as follows:\n\n--- \n **Q1:** The performance of Flat-LoRA depends on the choice of the perturbation strength ($\\sigma$), and finding the optimal value might require tuning.\n\n **A1:** \n We fully understand your concern. In practice, we suggest $\\sigma=0.05/0.10$. Our paper includes different tasks and networks (e.g., ViT, T5, LLama, SDXL) and shows that $\\sigma=0.05/0.10$ leads to consistent generalization improvements.\n\nTo further address this issue, especially in relation to different sizes of neural networks, we propose an improved perturbation generalization scheme that employs a scaling factor to make $\\sigma$ independent of the network width (Proposition 3.2). To validate this approach, we conducted experiments on the larger ViT-L/14 model. The results demonstrate that the optimal $\\sigma$ can be transferred from ViT-B/32 to ViT-L/14, and in all scenarios, the performance surpasses that of LoRA. We will include these discussions in the revision.\n \n CIFAR-100| LoRA | $\\sigma=0.01$ | $\\sigma=0.05$ | $\\sigma=0.10$ | $\\sigma=0.15$ | $\\sigma=0.20$ |\n---|---|---|---|---|---|---\nViT-B/32 | 87.74±0.13 | 88.14±0.22 | 88.37±0.41 | 88.65±0.35 | 88.64±0.23 | 88.06±0.31 \nViT-L/14 | 92.13±0.17 |  92.33±0.07 |  92.63±0.11 |  93.11±0.13 |  92.98±0.21 | 92.46±0.03\n \n ---\n \n **Q2:** While the paper provides a clear empirical demonstration, a more in-depth theoretical analysis connecting the proposed approach directly to improved generalization in the context of LoRA could be beneficial.\n\n **A2:** Thanks for your suggestion. Understanding the training dynamics and generalization of LoRA remains an open question. The discussions on this topic are active and often rely on assumptions like the NTK regime [1] or rank-1 perturbation [2]. Flat-LoRA additionally introduces random perturbations and raises more challenges to deal with randomness. Overall, exploring the generalization properties of Flat-LoRA is indeed interesting but quite hard. We would like to leave this for future work.\n \n[1] LoRA Training in the NTK Regime has No Spurious Local Minima, ICML'24\n\n[2] Gradient dynamics for low-rank fine-tuning beyond kernels, arxiv\n\n ---\n **Q3:** While minimal compared to SAM, Flat-LoRA does introduce a small additional memory overhead for storing filter norms and potentially a slight increase in computation time for generating and applying perturbations.\n\n **A3:** Flat-LoRA indeed introduces extra memory for storing random seeds and extra time for generating random perturbations. However, this additional overhead is minimal compared to the total training cost—for example, it adds only 0.5% extra memory and 2.4% extra training time on LLama fine-tuning, which is negligible.", "claims_and_evidence": "Evaluation of the claims made in the Flat-LoRA paper:\n\n* **Flat-LoRA improves generalization:** The paper claims Flat-LoRA improves both in-domain and out-of-domain generalization. This is supported by experimental results across various tasks, including natural language understanding, image classification, dialogue generation, mathematical reasoning, coding abilities, and text-to-image generation. Tables 1, 2, 3, and Figure 3 show improved performance compared to standard LoRA.\n*   **Flat-LoRA maintains computational and memory efficiency:** The paper claims Flat-LoRA integrates seamlessly with existing methods while maintaining computational and memory efficiency. Unlike SAM, it avoids additional gradient steps and remains memory-efficient by storing only the random seed and filter norms. Table 7 shows the memory and time usage, indicating minimal overhead compared to LoRA.\n*   **Filter structure and Input dimension:** The approach considers filter structure and input dimension. The variance introduced during the forward pass by random weight perturbation is independent of the input dimension.\n*   **Flat Minima and Generalization:** Flat minima in the loss landscape improve generalization and robustness to distribution shifts. This is well-known through various literature papers. \n*   **Storing random seed for memory efficiency:** Storing only the seed for the random generator and filter norms allows for the re-construction of εW when needed and requires minimal memory overhead.\n*   **Mixed precision training:** Flat-LoRA facilitates memory-efficient integration of perturbation injection during precision casting in mixed-precision training.\n\n**Potential Issues to Investigate Further:**\n*   **Low-rank adaptation may exhibit sharper loss landscapes:** The paper states that low-rank adaptation may lead to sharper loss landscapes in the full parameter space, which Flat-LoRA mitigates. There is no theory to support it. \n*   **Hyperparameter Sensitivity:** The paper mentions setting the random perturbation strength σ to specific values (e.g., 0.05 for T5-base, 0.15 for CLIP ViT-B/32). An ablation study on the variance magnitude was performed and the results are shown in Table C3. It may be worth checking the sensitivity of Flat-LoRA to this parameter across different tasks and model sizes.\n*   **Scope of Improvement:** While Flat-LoRA shows consistent improvements, the magnitude of these improvements varies across different tasks and datasets. It is important to examine scenarios where Flat-LoRA's benefits are less pronounced.\n*   **Integration Complexities:** The guidelines mentions that potential overheads or limitations should be addressed. The flat loss objective can be seamlessly integrated with previous approaches to yield consistent improvements. A clear guideline or step in that area can be helpful.", "methods_and_evaluation_criteria": "*   **Methods**:\n    *   The paper introduces **Flat-LoRA**, which optimizes low-rank adaptation within a flat region of the full parameter space. This addresses the problem of standard LoRA, which may find solutions that are sharp in the full parameter space, potentially harming generalization.\n    *   Flat-LoRA uses a **Bayesian expectation loss objective** with random weight perturbations to smooth the loss landscape, promoting convergence to flatter minima. This is a computationally efficient alternative to sharpness-aware minimization (SAM).\n    *   The method includes a **refined random perturbation generation strategy**, considering weight magnitude and model width scaling, to improve generalization performance.\n    *   The paper also introduces a method for **storing random seeds** to ensure memory efficiency.\n\n\n*   **Evaluation Criteria**:\n    *   The paper compares Flat-LoRA with other LoRA variants, including PiSSA, LoRA-GA, DoRA, AdaLoRA and LoRA+. Flat-LoRA can be seamlessly integrated with previous approaches.\n    *   The paper evaluates Flat-LoRA on diverse tasks: **natural language understanding, image classification, dialogue generation, mathematical reasoning, coding abilities, and text-to-image generation**.\n    *   The models were evaluated based on performance metrics suitable for each task, such as **accuracy** for natural language understanding and image classification, **first-turn score with GPT-4** for chat tasks, **accuracy** for math tasks, and **PASS@1 metric** for code tasks.\n    *   The study includes **out-of-domain generalization** experiments using corrupted datasets and instruction-following benchmarks.\n    *   **Ablation studies** are conducted to assess the impact of different LoRA ranks and perturbation variance.\n    *   The **memory and time costs** of Flat-LoRA are compared to those of LoRA.\n    *   The loss landscape is visualized to demonstrate that Flat-LoRA achieves a flatter loss landscape compared to LoRA.\n\n*   **Appropriateness**:\n    *   The **methods** seem appropriate for the problem. Flat-LoRA directly addresses the limitations of LoRA by seeking flatter minima in the full parameter space, which is expected to improve generalization. The use of Bayesian expectation loss and refined perturbation strategies aims to achieve this efficiently.\n    *   The **evaluation criteria** are also well-suited. The tasks cover a wide range of applications relevant to large language models and computer vision models. The use of both in-domain and out-of-domain datasets, along with ablation studies, provides a comprehensive assessment of Flat-LoRA's performance and robustness.\n    *   The evaluation includes **comparison with SAM**, showing that Flat-LoRA achieves comparable or superior performance to SAM while requiring less memory and training time.", "theoretical_claims": "Yes, I checked proposition 3.2 and its derivations in Equations 8 and 9. But I have not thoroughly verified Equation 9. Also, Not sure how it behaves with the dimensionality of W.", "experimental_designs_or_analyses": "*   **Hyperparameter Tuning and Selection:** The paper mentions specific values for the random perturbation strength $\\sigma$ for different models and datasets. While an ablation study on $\\sigma$ is presented in Appendix C, it's focused only on CIFAR-10/100 with CLIP ViT-B/32. **The paper lacks a thorough justification for the chosen values of $\\sigma$ for other tasks and models.** It's possible that the reported improvements are sensitive to this hyperparameter, and the selected values might not be optimal across all settings. A more systematic approach to hyperparameter tuning, or at least a wider exploration of $\\sigma$ values across different experiments, would strengthen the validity of the results.\n\n*   **Comparison with Baselines:** While Flat-LoRA is compared to standard LoRA and other LoRA variants, the depth of the comparison could be enhanced.\n    *   For instance, in the comparison with other LoRA variants (Table 5), the paper shows consistent improvements. However, **it doesn't delve into whether these improvements are statistically significant.** Authors have Reported confidence intervals but no explanation was provided around them in Section 4.6. \n    *   The comparison with SAM (Table 6) highlights the memory and time efficiency of Flat-LoRA. However, the choice of perturbation radius $\\rho$ for SAM is based on a limited exploration ($\\rho \\in \\{0.005, 0.01, 0.05, 0.1, 0.2, 0.3, 0.5\\}$). It's possible that a different value of $\\rho$ could lead to better performance for SAM, potentially altering the conclusions of the comparison.\n    *   The paper mentions stronger baselines achieved for Llama-2-7B compared to previous work. While this is a positive aspect, it also raises the question of **whether the relative improvements of Flat-LoRA would hold against even stronger, more recent baselines** that might have emerged since the submission of this work.\n\n*   **Out-of-Domain Generalization Analysis:** The out-of-domain experiments on corrupted CIFAR-100-C and instruction following are valuable. However, the analysis could be more nuanced.\n    *   For CIFAR-100-C (Figure 4), the performance gains increase with corruption severity. While this suggests Flat-LoRA's robustness, **a statistical analysis of these gains would be beneficial to confirm their significance at different corruption levels.**\n    *   For instruction following (Table 4), the improvements on DROP and Human-Eval are highlighted as more pronounced. **A discussion on why Flat-LoRA shows a greater advantage on these specific tasks compared to MMLU and BBH could provide more insight into the method's strengths.**\n\n*   **Loss Landscape Visualization:** The loss landscape visualizations (Figure 6) are qualitative. While they visually suggest a flatter landscape for Flat-LoRA, **it's important to acknowledge that these are projections along random directions in the high-dimensional parameter space.** The flatness observed in these 2D projections might not fully capture the characteristics of the loss landscape in all relevant directions. A more quantitative measure of flatness, if feasible, could complement these visualizations.\n\n*   **Scope of Applicability:** While the paper demonstrates Flat-LoRA's effectiveness across various tasks, **the underlying reasons for its varying degrees of improvement across different modalities and datasets are not thoroughly investigated.** Understanding the conditions under which Flat-LoRA provides the most significant benefits would be valuable for future research and application.\n\nIn conclusion, while the experimental design is broad and covers multiple aspects, a more critical perspective reveals potential limitations in the thoroughness of hyperparameter tuning, the statistical rigor of baseline comparisons, the depth of out-of-domain analysis, and the quantitative nature of the loss landscape evaluation. Addressing these points could further strengthen the claims made in the paper.", "relation_to_broader_scientific_literature": "The key contributions of the Flat-LoRA paper are significantly related to the broader scientific literature in the field of parameter-efficient fine-tuning and the understanding of loss landscapes in deep learning. Specifically, Flat-LoRA builds upon and extends prior work in Low-Rank Adaptation (LoRA) and methods for finding flat minima. Here's a breakdown of how Flat-LoRA connects to the literature:\n\n*   **Building upon Low-Rank Adaptation (LoRA):** Flat-LoRA directly addresses a potential limitation identified within the LoRA framework [Hu et al., 2022]. While LoRA efficiently reduces the number of trainable parameters by optimizing low-rank matrices, the authors of Flat-LoRA observed that **a flat minimum in the LoRA optimization space might still correspond to a sharp region in the full parameter space**. This idea is illustrated in Figure 1 and further supported by Figure 6 and the discussion in Section 3.2. Flat-LoRA aims to improve upon standard LoRA by explicitly considering the flatness of the loss landscape in the full parameter space, which is a novel perspective compared to many existing LoRA enhancements.\n\n*   **Addressing Limitations of Sharpness-Aware Minimization (SAM) in the Context of LoRA:** The paper discusses Sharpness-Aware Minimization (SAM) [Foret et al., 2020], a well-established technique for improving generalization by seeking flat minima. The authors acknowledge the potential of integrating SAM with LoRA (LoRA-SAM) [Li et al., 2024a]. However, they highlight several limitations: **SAM applied to LoRA parameters only optimizes sharpness in a restricted space**, **SAM requires an additional gradient step, doubling training cost**, and **computing sharpness in the full parameter space with SAM is memory-intensive**. Flat-LoRA offers an alternative approach to achieving flat minima that aims to overcome these computational and memory overheads.\n\n*   **Leveraging Bayesian Expectation Loss and Random Weight Perturbation (RWP):** Flat-LoRA's core idea of using a Bayesian expectation loss objective [Duchi et al., 2012; Bisla et al., 2022] to smooth the loss landscape and pursue flat minima is rooted in prior work. This line of research suggests that minimizing the expected loss under random weight perturbations can lead to better generalization. Flat-LoRA builds upon this by designing a **refined random perturbation generation strategy** that considers the filter structure and input dimension, differentiating it from simpler RWP methods like Gaussian Model Perturbation (GMP) [Wang & Mao, 2021] or basic random noise injection [Wu et al., 2022; Li et al., 2024b]. The memory efficiency of Flat-LoRA, achieved by storing only the random seed and filter norms, directly addresses a key concern when applying perturbation-based methods to parameter-efficient fine-tuning.\n\n*   **Connection to the Broader Understanding of Flat Minima and Generalization:** The paper's motivation stems from the widely held belief that **flat minima in the loss landscape are linked to improved generalization and robustness**. Flat-LoRA contributes to this understanding by proposing and demonstrating a method to find flatter minima in the full parameter space specifically within the context of efficient fine-tuning using LoRA. The experimental results, particularly the improved out-of-domain generalization on corrupted datasets and instruction following, provide empirical support for this connection in the context of Flat-LoRA.\n\n*   **Orthogonal to Other LoRA Enhancement Techniques:** The authors explicitly state that Flat-LoRA's approach of optimizing the sharpness of the loss landscape in the full parameter space is **orthogonal to other proposed methods for enhancing LoRA performance**, such as adaptive rank allocation (AdaLoRA [Zhang et al., 2023a, 2023b]), decomposition of weight updates (DoRA [Liu et al., 2024]), improved initialization (PiSSA [Meng et al., 2024]; LoRA-GA [Wang et al., 2024]), and learning rate adjustments (LoRA+ [Hayou et al., 2024]). The experiments in Section 4.6 demonstrate that Flat-LoRA can be seamlessly integrated with some of these methods (e.g., Flat-PiSSA, Flat-DoRA), leading to further performance improvements, highlighting its potential as a complementary technique.\n\nIn summary, Flat-LoRA contributes to the scientific literature by:\n\n*   **Identifying a limitation of standard LoRA** regarding the sharpness of the loss landscape in the full parameter space.\n*   **Proposing an efficient alternative to SAM** for finding flat minima in the context of LoRA, leveraging Bayesian expectation loss and a refined random perturbation strategy.\n*   **Demonstrating improved generalization** (both in-domain and out-of-domain) across a wide range of tasks and model sizes.\n*   **Offering a memory and computationally efficient method** that can be readily integrated with existing LoRA training pipelines and is orthogonal to other LoRA enhancement techniques.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "Highlights when comparison with SAM, following claims will help other areas of research as well:\n\n*   **LoRA-SAM optimizes sharpness in a restricted space (the LoRA parameter space), which may not effectively improve generalization**.\n    *   The paper explicitly states that LoRA constrains optimization to a much lower-dimensional space. Figure 1 illustrates this by showing that a flat minima in the LoRA space (blue curve) can still exhibit sharp directions in the full parameter space (red curve).\n    *   Section 3.2 discusses applying SAM to LoRA parameters (Eqn. 2). It argues that focusing solely on this restricted space might have limitations because during inference, the LoRA adaptation is merged into the pre-trained weights. A solution good in the LoRA space might be in a sharp region of the full parameter space, potentially harming generalization.\n    *   Equation (4) approximates the weight perturbation applied to the full parameters when SAM is applied to LoRA, showing it is roughly proportional to $(\\nabla_W L)A^TA$. The paper argues this implies that SAM on LoRA only optimizes sharpness along the column space spanned by $A$, which is a small subspace of the full parameter space.\n    *   **Empirical evidence is provided in Table 5 and Table 6**, which show that applying SAM constraints solely to LoRA parameters (referred to as LoRA-SAM or LoRA+SAM A,B) does not consistently lead to significant improvements in generalization on GLUE datasets. In Table 6, LoRA+SAM applied to A and B even performs worse than standard LoRA on both CoLA and MRPC.\n\n*   **SAM requires an additional gradient step, doubling the training cost and rendering it impractical for large models**.\n    *   The introduction to Section 2.1, which discusses flat minima and generalization, mentions that SAM doubles the training time compared to regular training, limiting its applicability to large-scale training.\n    *   Section 3.2, when proposing Flat-LoRA, reiterates that directly applying SAM to optimize the sharpness of the full weight space doubles the training cost, which is less desirable for large models.\n    *   Table 6 explicitly compares the training time, showing that LoRA+SAM (whether applied to A,B or W) incurs 2x the training time of standard LoRA and Flat-LoRA.\n\n*   **Computing sharpness in the full parameter space necessitates calculating gradients and storing perturbations for all weights, which contradicts the principles of parameter-efficient fine-tuning**.\n    *   Section 3.2 explains that computing sharpness in the full parameter space requires calculating gradients and storing perturbations for all weights, contradicting the principles of parameter-efficient fine-tuning.\n    *   When comparing with SAM applied to the full parameter space (SAM on W or LoRA+SAM W), Table 6 indicates that it requires $O(m \\times n)$ additional memory to store adversarial weight perturbations, making it impractical for parameter-efficient training.\n    *   In contrast, Flat-LoRA is presented as a method that addresses these issues by employing Bayesian expectation loss with efficient random weight perturbations that can be stored as random seeds, requiring only $O(m)$ additional memory.", "other_comments_or_suggestions": "**Presentation in Tables**: In some tables (e.g., Table 1, 2, 3), the ± standard deviation values are provided. Ensuring consistent precision in reporting these values across all tables could improve the overall presentation.\n\n**Detailed Evaluation**:\n\n*   **Novelty, Relevance, and Significance**:\n    *   **Novelty**: The idea of explicitly optimizing for flatness in the full parameter space within the LoRA framework, using a Bayesian expectation loss with a memory-efficient random perturbation strategy, is slightly **novel** but not unique. \n    *   **Relevance**: Parameter-efficient fine-tuning (PEFT) methods like LoRA are highly relevant due to the prohibitive costs of fine-tuning large-scale pre-trained models. Improving the generalization capabilities of these methods is crucial for their broader applicability. The problem of sharp minima in the context of low-rank adaptation is a pertinent issue, making Flat-LoRA's approach highly relevant to the PEFT research community.\n    *   **Significance**: The experimental results demonstrating **consistent improvements in both in-domain and out-of-domain generalization across a wide range of tasks and models** highlight the **significance** of Flat-LoRA. The fact that Flat-LoRA achieves these gains with minimal additional computational and memory overhead compared to standard LoRA makes it a practically significant contribution. The ability to integrate Flat-LoRA with other LoRA variants and yield further improvements also underscores its significance.\n\n*   **Soundness**:\n    *   The paper provides a clear formulation of the Flat-LoRA objective based on Bayesian expected loss.\n    *   The motivation for considering the full parameter space's loss landscape is well-illustrated and argued.\n    *   The proposed random weight perturbation generation strategy is theoretically justified, with Proposition 3.2 demonstrating its input-dimension independence.\n    *   The memory efficiency argument, based on storing seeds and filter norms, is sound.\n    *   The **extensive experimental validation across diverse tasks (natural language understanding, image classification, dialogue generation, mathematical reasoning, coding abilities, and text-to-image generation) and model sizes (T5-base, CLIP ViT-B/32, Llama-2-7B, SDXL) provides strong empirical evidence** for the effectiveness of Flat-LoRA.\n    *   Ablation studies on LoRA rank and perturbation variance further support the design choices of Flat-LoRA.\n    *   The comparison with SAM and other LoRA variants helps to contextualize Flat-LoRA's performance and efficiency.\n    *   The visualization of the loss landscape in the full parameter space offers qualitative evidence of Flat-LoRA's ability to find flatter minima.\n    *   **One potential area for further strengthening soundness could be a more theoretical analysis of why optimizing the Bayesian expected loss in the full parameter space with the proposed perturbation strategy leads to better generalization in LoRA fine-tuning.** While Lemma 3.1 provides some insight into the smoothing effect, a more direct connection to the generalization benefits in the context of LoRA could be valuable.", "questions_for_authors": "1. Hyperparameter Sensitivity of $\\sigma$: The paper includes an ablation study on the perturbation strength $\\sigma$ in Appendix C, showing optimal ranges for CIFAR-10 and CIFAR-100. However, could the authors provide more general guidance or intuition on how to select an appropriate value for $\\sigma$ when applying Flat-LoRA to new tasks or models where such ablation studies might be computationally prohibitive? For example, are there any heuristics based on learning rate, batch size, model size, or dataset characteristics that could inform the choice of $\\sigma$? If the authors can offer some practical guidelines, it would significantly increase the usability of Flat-LoRA and positively impact my evaluation by addressing a key practical concern. Conversely, if the optimal $\\sigma$ is highly task-specific without clear indicators for its selection, it would remain a potential limitation.\n\n2. Underlying Reasons for \"Flat-LoRA (all)\" Performance: Appendix A explores extending the random weight perturbation to all layers. The results show improvements, but they are less pronounced than when applying Flat-LoRA to linear layers only. Could the authors offer more insight into why perturbing all layers (including layernorm, biases, embeddings, etc.) does not yield the same level of performance gain as perturbing only the linear layers in the LoRA modules? Understanding the reasons behind this could provide valuable insights into where the flatness of the loss landscape is most critical for generalization in the context of LoRA and could guide future research directions. A clear explanation would enhance the understanding of Flat-LoRA's mechanism and potentially improve my evaluation of the paper's depth.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "None"}}, {"review_id": "pVdgSRX4Qf", "reviewer": "Reviewer_cWcV", "summary": "The authors present an approach to improving generalization in PEFT. They argue that generalization is strongly correlated with flat minima and that existing empirical approaches are not directly applicable to PEFT. To address this, they propose Flat-LoRA, an extension of SAM that facilitates the discovery of flat minima in the entire parameter space while training within a subspace. By leveraging existing techniques, such as the relaxation of inner optimization problems using random directions and on-the-fly noise generation, the authors demonstrate that Flat-LoRA consistently improves over baseline methods, albeit with a small difference.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your detailed and constructive comments. We address your concerns point by point as follows:\n\n ---\n **Q1:** Table 4 does not provide variance.\n\n **A1:** Following your suggestion, we rerun the experiments and report the variance as follows:\n \n| Method  | MMLU  | DROP |  BBH|  Human-Eval|\n| ---     | ---    | --- | --- | --- |\n| Full FT | 52.36±0.45 | 38.23±0.47 | 35.38±0.35 | 15.44±0.35|\n| LoRA    | 51.22±0.38 | 37.26±0.63 | 34.77±0.22 | 13.01±0.93 |\n| Flat-LoRA | 51.88±0.55 | 38.18±0.71 | 35.22±0.26 | 15.24±0.61 |\n\n \n ---\n **Q2:** Tables 1, 2, and 5 require more analysis to prove that the difference is statistically significant, as most of the improvement of Flat-LoRA is lower than the standard deviation of the metrics.\n\n **A2:** Following your suggestion, we conduct statistical analysis using t-test as follows:\n\n (Table 1) Hypothesis: Flat-LoRA is better than LoRA\n \n T5-base | MNLI | SST-2 | CoLA | QNLI | MRPC\n ---|---| --- | --- | --- | ---\n p (r=8) | 0.186 | 0.038 | 0.043 | 0.186 | 0.027\n p (r=16) | 0.623 | 0.117 | 0.023 | 0.002 | 0.046\n \n (Table 2) Hypothesis: Flat-LoRA is better than LoRA\n ViT-B/32 | CIFAR-10 | CIFAR-100 | Cars | SVHN | DTD \n ---|---| --- | --- | --- | ---\n p (r=8) |0.006 | 0.008 | 0.137 | 0.125 | 0.063 |\n p (r=16) | 0.001 | 0.001 | 0.047 | 0.112 | 0.046 | \n \n (Table 5) Hypothesis: Flat-X is better than X, where X is base method, can be LoRA, PiSSA, etc.\n T5-base | LoRA | PiSSA | LoRA-GA | DoRA | AdaLoRA | LoRA+\n ---|---|---|---|---|---|---\n CoLA| 0.043|0.36|0.488|0.088|0.202|0.037\n MRPC| 0.027 | 0.126| 0.301|0.199|0.183|0.313\n \n \nWe observe that, in most cases, the improvements are significant (i.e., p < 0.05). However, we acknowledge that in some cases, such as in Table 5, the improvements are relatively modest. This could be attributed to the strong baseline performance of the LoRA variants, which leaves limited room for further improvement. Moreover, we find that on large-scale training, the improvement is more significant as below, showing the scalability of flat-LoRA.\n\n(Table 3) Hypothesis: Flat-LoRA is better than LoRA\n  Llama2-7b | MT-Bench | GSM8k | Human-Eval\n ---|---|---|---\n p (r=8) | 0.007 | 0.003 | 0.007\n \n ---\n **Q3:** Some of the relevant baselines are mentioned but are not used in comparison (Balancedness-Aware Regularization [3], LoRA-Pro [4], GaLore [1], CorDA [2]).\n\n **A3:** Thanks for providing the very relevant works. We compare them below and will add the comparision and citations in the revision.\n \n| T5-base   | oBAR [3]       | nBAR [3]       | LoRA-Pro [4] | GaLore [1]     | CorDA [2]      | Flat-LoRA\n|---|---|---|---|---|--- |---\n| MRPC | 88.58±0.35 | 88.63±0.42 | 89.23 $\\pm$ 0.33 | 88.90±1.12 | 89.76±0.52 | 89.59±0.37\n| CoLA | 83.07±0.87 | 82.78±0.68 | 83.17 $\\pm$ 0.28 | 83.14±0.57 | 83.38±0.47 |83.61±0.38\n\n[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, ICML'24\n\n[2] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning, NeurIPS'24\n\n[3] Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems, NeurIPS'24\n\n[4] LoRA-Pro: Are Low-Rank Adapters Properly Optimized? ICLR'25\n\n ---\n **Q4:** Improvement in T2I task is shown only qualitatively and requires additional analysis using common datasets (i.e., measure CLIP-T/CLIP-I on DreamBench).\n \n**A4:** Thanks for your valuable suggestions. Following your suggestion, we finetune an SDXL model under the same setting as Figure 3, using the public dataset provided by DreamBooth, including 30 tasks, where each task is evaluated with 25 prompts and 4 different seeds. The CLIP-I and CLIP-T scores are reported below, which show consistent results with Figure 3. The model with Flat-LoRA exhibits both higher subject fidelity (CLIP-I) and prompt fidelity (CLIP-T). More comparisons of sample images can be found in [Figure 3](https://anonymous.4open.science/r/Flat-LoRA/rebuttal.pdf).\n\n|                     | Real Image | LoRA  | Flat-LoRA |\n| ------------------- | ---------- | ----- | --------- |\n| CLIP-T ($\\uparrow$) | -          | 0.299 | 0.311     |\n| CLIP-I ($\\uparrow$) | 0.881      | 0.819 | 0.825     |\n\n\n---\n**Q5:** Lines 169-171 duplicate lines 167-169.\n\n**A5:** Thanks for your careful reading. We will fixed it in the revision.", "claims_and_evidence": "The proposed method optimizes using the same framework as SAM and does not provide a mathematical analysis of the algorithm's convergence. However, experimental results demonstrate a clear advantage of the method. Experiments with OOD data show that Flat-LoRA improves generalization, supporting the paper’s claim.", "methods_and_evaluation_criteria": "The paper presents results on common datasets (GLUE, InstructEval, MT-Bench) for NLP tasks as well as small-scale qualitative T2I experiments to provide sufficient evidence for Flat-LoRA’s effectiveness.", "theoretical_claims": "The mathematical derivations appear correct to the best of my knowledge.", "experimental_designs_or_analyses": "To prove that the proposed method improves generalization, the authors analyze it in multiple experimental setups, following the widely accepted frameworks and using the same metrics and datasets. As the improvement over most of the benchmarks is minute, it can be beneficial to provide statistical analysis to justify the difference, e.g., in Tables 1 and 2. Improvement in T2I task is shown only qualitatively and requires additional analysis using common datasets (i.e., measure CLIP-T/CLIP-I on DreamBench).", "relation_to_broader_scientific_literature": "The paper references relevant prior work and discusses concurrent approaches. The proposed method complements these existing approaches and can be effectively combined with them.", "essential_references_not_discussed": "Some recent PEFT methods are missing, including GaLore [1], CorDA [2].\n\n[1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection\n[2] CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning", "other_strengths_and_weaknesses": "Strengths:\n\n- Paper is well-written and easy to read\n\nWeaknesses:\n\n- Table 4 does not provide variance\n- Tables 1, 2, and 5 require more analysis to prove that the difference is statistically significant, as most of the improvement of Flat-LoRA is lower than the standard deviation of the metrics.\n- Some of the relevant baselines are mentioned but are not used in comparison (Balancedness-Aware Regularization [3], LoRA-Pro [4], GaLore [1], CorDA [2])\n\n[3] Implicit Regularization of Sharpness-Aware Minimization for Scale-Invariant Problems\n\n[4] LoRA-Pro: Are Low-Rank Adapters Properly Optimized?", "other_comments_or_suggestions": "- Lines 169-171 duplicate lines 167-169.", "questions_for_authors": "See weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces Flat-LoRA, which is basically LoRA done using an approximate version of sharpness-aware minimization (SAM), thereby achieving flatter minima and therefore better generalization. 

The idea is intuitive, and the authors perform a diverse set of experiments --- both on LLMs and text-to-image models, on well-known benchmarks --- to back it up. However, the technique is a relatively straightforward combination of two well known optimization tricks, and the gains over regular LoRA on some of the benchmark experiments were rather miniscule; therefore, enthusiasm was a bit tempered. I recommend an accept since the paper is well written and makes a solid (if not terribly surprising) contribution to the field.
Decision: Accept (poster)