Bi-LoRA: Efficient Sharpness-Aware Minimization for Fine-Tuning Large-Scale Models
Authors: Yuhang Liu
Tao Li
Zhehao Huang
Zuopeng Yang
Xiaolin Huang
Venue: ICLR 2026 Poster (2026)
Keywords: low-rank adaptation
efficient training
generalization
Primary area: optimization
TL;DR: We propose Bi-LoRA that introduces an auxiliary LoRA module to model SAM’s adversarial weight perturbations, enhancing generalization while avoiding doubled computational costs.

Abstract: Low-Rank Adaptation (LoRA) enables parameter-efficient fine-tuning of large pre-trained models. Yet LoRA can face generalization challenges.
One promising way to improve the generalization is Sharpness-Aware Minimization (SAM), which has proven effective for small-scale training scenarios. In this paper, we propose **Bi**-directional **Lo**w-**R**ank **A**daptation (Bi-LoRA), which introduces an auxiliary adversarial LoRA module. This design explicitly decouples sharpness optimization, handled by the auxiliary module, from task adaptation, performed by the primary module. Such a separation yields two key benefits. First, it transforms the sequential computation of primary LoRA update and adversarial perturbation into a parallel form, which roughly halves the time and conquers the main obstacle of applying SAM in LoRA. Second, it provides perturbations from the auxiliary module that do not collapse into the restricted optimization subspace of the primary module, enabling broader sharpness exploration and flatter minima. Bi-LoRA simultaneously achieves both efficiency and effectiveness within a single framework, as verified by extensive experiments across diverse architectures and tasks.

Published as a conference paper at ICLR 2026
BI-LORA: EFFICIENT STARPNESS-AWARE MINIMIZA- TION FOR FINE- TUNING LARGE-SCALE MODELS
Yuhang Liu' ; Li Zhehao Huang' Zuopeng X
Xiaolin Huang
Institute of Image Processing and Pattern Recognition, School of Automation and Intelligent Sensing, Shanghai Jiao Tong University MoBB Key Laboratory of System Control and Information Processing (Shanghai) (yuhangling iu, 1i. tao, kinght_h, yzpeng, aoluannnnnggggn justu..e etu.euu.cc edu.cn
ABSTRACT
Low-Rank Adaptation (LoRA) enables parameter-effiient fine-tuning of large pre-trained models. Yet LoRA can face generalization challenges. One promis- ing way to improve the generalization is Sharpness-Awan Minimization (SAM), which has proven effective for small-scale training scenarios. In this paper, we propose Bi-directional Low-Rank Adaptation (Bi-LoRA), which introduces an auxiliary adversarial LoRA module. This design explicitly decouples sharpness optimization, handled by the auxiliary module, from task adaptation, performed by the primary module. Such a separation yields two key benefits. First, transforms SAM's sequential computation of adversarial perturbation and gradient descent into parallel form, which roughly halves the time and conquers the main obstacle of applying SAM in LoRA. Second, it provides perturbations from the auxiliary module that do not collapse into the restricted optimization subspace of the primary module, enabling broader sharpness exploration and flatter minima. Bi-LoRA multaneously achieves both efficiency and effectiveness within a single framework, as validated by extensive experiments across diverse architectures and tasks. Code is available at https: //ggttttthbbtb www.eo/////eeeeeeeeeeeee ements /Bi-LORA.
INTRODUCTION
The paradigm of pretraining followed by fine-tuning has become the de facto standard in ma- chine learning, demonstrating state-of-the-art performance across various tasks (Devlin et al., 2018; Kolesnikov et al., 2020; Dosovitskiy et al., 2021; Radford et al., 2021). However, as model sizes continue to grow, full fine-tuning (Full FT) becomes memory-prohibitive in resource-constraaied settings. The most successful approach for reducing memory cost in fine-tuning large-scale model is Low-Rank Adaptation (LoRA) (Hu et al., 2022), which introduces trainable, task-speeific low-rank matrices to model weight updates. LoRA greatly lowers memory requirements by significantly reducing the trainable parameters and has become one of the most popular solutions for fine-tuning and deploying large models due to its simplicity and performance comparable to Full FT.
Despite LoRA's memory efficiency, it still faces generalization challenges when fine-tuned with limited data. In such scenarios, the risk of overfitting becomes particularly acute and can significantly compromise model performance (Li et al., 2025; Deng et al., 2025; Lin et al., 2024; Li et al., 2024b).
Motivated by the connection between flatness of the loss landscape and generalization, one promising direction for improving generalization is to seek flat minima (Hochreiter Schmidhuber, 1994; 1997). Sharpness- enhances generalization by formulating optimization as a min-max problem, effectively minimizing the worst-case loss within local neighborhood. It has achieved state-of-the-art performance in small-scale training scenarios (Chen et al., 2022; Zhuang et al., 2022). However, SAM requires calculating adversarial perturbation of model parameters each training step, incurring additional memory overhead of a copy of the model weights and doubling the training time. When applied to large-scale models, such extra memory cost and computation becomes particularly pronounced.
Minimization (SAM) (Foret et al., 2021) is widely used technique that
"Equal Contribution.
I "ooresponning Author.

---

Published as a conference paper at ICLR 2026
Fine- MuniiTnnng
Inference
1
Trainable
B1
B=
Pretrained Weight Wo Egmmon
Pretrained Weight Wi pmmn
Frozen
Q
Gradient Descent
Gradient Ascent Normalization
A
Ag
A
HFC
Figure 1: Overview of our proposed Bi-LoRA. During fine-tuning (Left), Bi-LoRA utilizes two opposing LoRA modules: the primary module (B) optimized by conventional gradient descent for task-specific adaptation, while the auxiliary module (B2,2) performs gradient ascent for sharpness optimization. These two modules are decoupled and simultaneously optimized. During inference (Right), only (B) is retained and merged with the pretrained weights.
Thus, a natural approach to finding flat minima when fine-tuning large models, without sacrificing memory efficiency, apply SAM to the LoRA parameters, referred to LoRA-SAM (Li et al., 2024a). However, this straightforward integration still suffers from the doubled training cost of SAM. To address this essential problem, we introduce an auxiliary adversarial LoRA module decouple the sharpness optimization from the task adaptation. This design, Bi-directional Low-Rank Adaptation (Bi-LoRA) shown in Figure 1, enables updating both modules in one forward and backward pass, making SAM practical for fine-tuning large-scale models.
Decoupling the two LoRA modules also enables broader exploration of perturbations. During inference, the LoRA parameters merged into the pretrained weights, should care about the sharpness of the full parameter space. contrast, LoRA-SAM limits adversarial perturbations to a restricted subspace (see Proposition 1). Worse, LoRA-SAM collapses quickly into it (see Figure 2c), thereby optimizing sharpness only there. Empirically, Bi-LoRA alleviates this issue: its decoupled auxiliary perturbation space converges more slowly than the primary optimization space, enabling exploring perturbation beyond the restricted subspace to capture sharpness and promote flatter minima. Bi-LoRA maintains LoRA's efficiency while improving generalization, making a strong alternative for fine-tuning large-scale models.
Our contributions can be summarized as follows:
We propose Bi-LoRA, which introduces an auxiliary LoRA module to model SAM's adversarial weight perturbation, decoupling it from LoRA optimization. This design enables simultaneous optimization of SAM's two steps with minor additional memory costs.
We point out that directly applying SAM LoRA parameters can only optimize the sharpness within the restricted subspace, thereby limiting its potential to improve generalization. Bi-LoRA broadens the sharpness exploration: the auxiliary perturbation module converges more slowly than the primary one, enabling exploration beyond the restricted subspace.
Extensive experiments across wide range of fine-tuning tasks, including natural language under- standing, mathematics, code generation, chat, instruction following, and Text-to- generation, demonstrate that Bi-LoRA achieves superior generalization performance.
ISSUES OF LORA-SAM
In this section, we formalize LoRA-SAM and analyze its subspace collapse issue during training. Related work on LoRA and SAM is summarized in Appendix E.
2.1 PRELIMINARIES
Given a pre-trained weight matrix Wo E mx LoRA utilizes low-rank matrices and A to model the weight change AW during the fine-tuning
W Wo + AW W. + BA,
(1)
2

---

Published as a conference paper at ICLR 2026
Cosine Siilarit
6.33
6.00
2200
2200
Steps
(a) Norm of diffcrent components
(b) Norm ratio
(c) Cosine similarity to final
Training statistics for LoRA-SAM during fine-tuning. (a) and (b): Frobenius norms of different terms and the ratio of the Frobenius norms of (Be + Ann) to that of (eneA) in Eqn. (5). We monitor fixed LoRA module and more similar experiments could be observed that the norm of the third term (εвеλ) is several orders of magnitude (> 10") smaller than that of the first B are initialized to zero by default (Hu et al., 2022). (c): Cosine similarity between the final LoRA parameters and those during fine-tuning. The LoRA parameters converge rapidly during fine-tuning. The experiments are conducted on CoLA with the T5-base model.
Figure 2:
be found in Appendix T. It
terms, making negligible. Note that || is initially zero since
where B E R and A E x with & min n). Here we omit the scaling factor a/r for the sake of simplicity the equation, as can be easily incorporated into B and A.
During gradient back-propagation, the loss gradient w.r.t. B and A is computed using the chain rule:
ac (VW
Oc AA
B (VwL),
dB
where is the loss objective to be minimized.
2.2 LORA-SAM
For optimizing the sharpness while keeping memory efficiency, natural idea is to apply SAM over the LoRA parameters (LoRA-SAM) which formulates the following optimization target:
min max B.A [(e.,c))]|sp
<Wo + (B (A + eA)),
where cg E Rmxr CA E Rex= are the adversarial weight perturbations over low-rank matrices, (2, eA denotes the norm of weight perturbations (a typical setting is the f2-norm), and p is the neighborhood radius.
To efficienlly solve the inner maximization problem in Eqn. (3), Foret et al. (2021) employ first-order Taylor expansion for approximation, and the resulting perturbations are calculated as follows:
Oc 6B = p 0B / Fintal
OC AA
dc
2
cA =
Foouts
Fioral
0B
+
aA
where || denotes the Frobenius norm. Then we can rewrite the perturbation given by Eqn. (3) as
ew Be + A EBEA-
(5)
Since both ça and ep are several orders of magnitude smaller than the original matrices A and B, the cross term н is negligible, evidenced by Figures 2a and 2b, which report the magnitudes of these terms. Therefore, Eqn. (5) can be approximately simplified as:
ew [BB (VwL) + (Vw C)TT)
(6)
where e (V AT + ||BT (Vw C) Now one can observe that the perturbation of LoRA-SAM is confined to restricted subspace, as demonstrated below:
Proposition (Perturbation Space LoRA-SAM), The effective weight perturbation in LoRA-SAM can be decomposed into two terms: BB (Vw L) and (Vw L)A A. The column space of the first term is given by Co)(B), while the row space of the second term is given by Row(A).
The space for adversarial weight perturbation in LoRA-SAM is primarily dominated by Col(B) and Row (A), from which it follows that LoRA-SAR SAM only cares about the sharpness within the subspace defined by B and A, failing to capture the sharpness in broader space.
3

---

Published as a conference paper at ICLR 2026
Moreover, B and A converge rapidly during training, shown in Figure 2c. This leads to the fast convergence of the column and row spaces BB) and Row(A), further restricting the subspace for sharpness optimization and hindering the effectiveness of LoRA-SAM, potentially leading to suboptimal performance. Appendix U further corroborates this subspace collapse phenomenon.
In Figure 4, we compare the loss landscape flatness of different methods. We observe that LoRA- SAM achieves the flattest loss landscape within the LoRA parameter space as in Figure 4a, reflecting its original caaippesss-mnimizatiiin objective. In contrast, Figure 4b shows that while LORA-SAM yields a flatter landscape than LoRA in the full space, the loss still rises sharply with perturbation magnitude. This indicates the limitation of applying perturbation in a restricted subspace, highlighting the importance of considering sharpness over a broader space beyond LoRA subspace.
3 BI-LORA: BI-DIRECTIONAL LOW-RANK ADAPTATION
In this section, we introduce Bi-LoRA, a novel LoRA variant, which contains an auxiliary module beyond the primary module in regular LoRA. The aim of the auxiliary module is to decouple the sharpness optimization from task adaptation, which makes the training efficient and enhances the generalization improvement. Specifically, the proposed Bi-LoRA takes the following formulation,
W Wo + AiA1 + B2A2,
(7)
where the first LoRA module (A1, B1) serves as the primary module responsible for task-specific adaptation, similar to standard LoRA, while the second module (A2, B2) acts as the auxiliary LoRA module for modeling adversarial perturbation in SAM. With these two modules, the Bi-LoRA's optimization objective is given below,
min max. L <WW + BiA + B2A2),
(8)
BL Aa
where p is the neighborhood radius that controls the magnitude of perturbations in the original SAM. After training, we discard the auxiliary modules (A2, B2). as they serve solely to optimize sharpness during training and guide the primary module (A1, B1) towards a flat region. Consequently, the adapted weights are reduced to the primary module, i.e., W = Wo+ + B₁ A₁, preserving the original LoRA structure and ensuring no additional computational overhead during inference.
In LoRA-SAM, the adversarial perturbation is calculated as Eqn. (4), where the backpropagation on en) and on (A, B) must be carried out sequentially. Now in Bi-LoRA, the task adaptation and the perturbation are decoupled, as shown in Eqn. (8). Then one can simultancously perform task adaptation and sharpness-aware minimization in one gradient step. Specifically, during each iteration, the primary LoRA module (A1, B1) is updated through standard gradient descent, while the auxiliary LoRA module (A2, B2) is updated through gradient ascent for sharpness optimization.
The bi-directional gradient update approach embodies the core concept of our Bi-LoRA. Concretely, one update step for Bi-LoRR is formalized as follows:
B B$- mn (Vw L) AT
A AA - BI (VwL),
(9)
B
Byt + 122 ( W L) T
Aptl A + BA (VwL),
where V = ac
oW W= 4/ A is the gradient of the loss L w.r.t. the merged weight W, k denotes the iteration index, and 7/, 772 are learning rates. We use the same learning rate for both LoRA modules in our experiments, though other choices are possible (see Appendix Q).
The independent backpropagation for the two LoRA modules (A1, B1) and (A2, B2) makes the updates parallelizable. This eliminates the doubled computational cost typically incurred by LoRA- SAM. The following proposition ensures that the adversarial direction induced by (A2, B2) can still increase the inner objective, in non -eggativvv alignment with the SAM's perturbation direction. The Proof and further discussions are provided in Appendix F.I.
Proposition (Aligmment of Bi-LoRA's ascent direction with previous full gradient). Let G. = C(W, + B1, + é) denote the full gradient step t, with = B2, After one BoRL update,
(Gi, ē+t1 ēt) 2 0,
ie. Bi-LoRA increases the inner objective along the previous SAM's perturbation direction.
4

---

Published as a conference paper at ICLR 2026
2.0
LoRA SAM
LoRA-SAM
Training LosS
0.0675
Training Loss
0.0650
0.5
0.0600
0.0
0.0
0.1
0.3
0.0
0.1
0.2 Perturbation Magnitude
0.3
0.4
0.5
Perturbattion Magnitude
(a) LoRA parameters
(b) Full parameters
Figure 4: Loss landscape visualization along random "filter-normaizzed" directions following Li et al. (2018), focusing on (a) LoRA parameters and (h) full parameters. It can be observed that while LoRA-SAM attains the greatest flatness within the LoRA subspace, which aligns with its Sharpness-minmmizatee objective, it is not the flattest in the full parameter space. This distinction critical for inference, as LoRA parameters are ultimately merged into the pretrained weights. In contrast, Bi-LoRA delivers a substantially greater improvement on flatness in the full parameter space. All experiments were averaged over five independent runs with T5-base fine-tuned on CoLA.
Decoupling sharpness optimization from task adaptation not only makes the backpropagation parallelizable but also partially alleviates the in- consistency of LoRA-SAM, where perturbations applied to restricted subspace (A, B) while inference is determined by the full parameters (see Proposition 1). Bi-LoRA eliminates the de- pendence adversarial weight perturbations on the primary LoRA module. In Eqn. 8, we see the column space for perturbations in Bi-LoRA spanned by Col(B2), which independent of the LoRA optimization space, i.e., (1)
1.0
0.8
Cosine Similarity
0.6
Main LoRA (opcimization) final weight
Ausillar LoRA (partuebstion) to fird weight
0.4
Auxiliary LoKA each step
0.2
0.0
500
1000
1500
2000
2500)
Training Step
Although Bi-LoRA's perturbation space Col(B2) (auxiliary) remains subspace, it is strictly decoupled from the optimization space Col(B,) (primary). Moreover, we find that the auxiliary modules converge much more slowly than the primary ones, preserving flexibility for marpnesss-ware updates (see Figure and Appendix L). Therefore, Bi-LoRA enables more effective capture of sharpness in the full parameter space. And Figure confirms that Bi-LoRA achieves a significantly flatter loss landscape in the full parameter space, leading improved generalization.
Figure 3: Cosine similarity between the main LoRA (task adaptation, red), the auxiliary LoRA (perturbation, blue), and their trajectories (green) during training T5-base on CoLA. The auxiliary converges only in the last 20% of steps, substan- tially slower and independent the main.
Numerous studies have discussed SAM's convergence and generalization (Andriushchenko & Flam- marion, 2022; Dai et al., 2023; Khanh et al., 2024). For Bi-LoRA, the following proposition estab- lishes that the Bi-LoRA objective (Eqn. 8) is essentially equivalent to the vanilla LoRA optimization with this regularization term.
Proposition 3 (Bi-LoRA is Regularized LoRA), From the gradient-norm perspective, SAM can be expressed as
C(W) w C(W)22.
the inner maximization of Bi-LoRA is solved to convergence and attains its optimum, its objective reduces to a low-rank counterpart:
L(W, B1A ( Cl) Bi.A:
min
where || (l(r) denotes the Ky Fan r-norm (the sum of the top-r singular values). Thus, ideally, Bi-LoRA can be viewed as LoRA equipped with an explicit low-rank gradient-norm regularizer.
Proofs and further details are deferred to the Appendix F2. Proposition coincides with the regularization view of SAM (Yue et al., 2023). Bi-LoRA can be analyzed in the way, differing
5

---

Published as a conference paper at ICLR 2026
Algorithm Bi-LoRA
1: Input: Initial weight Wo, learning rates 1/, radius p. number of LoRA layers N
2: Output: Adapted weight W for inference
3: Initialize LoRA modules By. Al. Bg. A2:
4: k <0;
5: while not converged do
6: Sample mini-batch data B;
7: Apply gradient descent to primary and ascent auxiliary LoRA modules via Eqn. (9);
8: Clip the auxiliary modules via Eqn. (10);
9: k 1:
10: end while
Remove the auxiliary LoRA modules A2;
12: return 8 = Wo + Af
only in that its regularizer is low-rank norm. In practice, however, perform single inner step to preserve high efficiency, which is crucial for fine-tuning models. This choice introduces a gap between the practical Bi-LoRA and analyses that require optimality of the inner optimization. Given our application-orian orienntd focus, we do not delve deeply into detailed theoretical discussions.
constraint to keep the perturbation sufficiently small to avoid disrupting normal model training. Thus, we apply global clipping to the auxiliary module (A2, B2) after each update, by constraining their total Frobenius norm, which maintains the weight perturbation within a controlled magnitude. Specifically, suppose that there are N LoRA layers. The i-th auxiliary module is scaled as follows:
Norm Constraints. In Eqn. (8), impose
B A
cnocom B2
if
Cuerm p.
(10)
p/corm
Ale
Notice the normalization is applied only if the total Frobenius
N auxiliary LoRA modules
Gnоrm exceeds the neighborhood radius p. where Crorm = =V Σ | 1 12.. This ensures the
perturbation remains constrained within the preset p-norm ball. The overall procedure for Bi-LoRA summarized in Algorithm
EXPERIMENTS
In this section, evaluate Bi-LoRA across diverse models and benchmark tasks. We test Bi-LoRA's capacities on: (1) Llama 2/3.1 models (Touvron et al., 2023; Dubey et al., 2024) for mathematical reasoning, coding, dialogue, and instruction following; (2) Qwen 2.5-14B (Qwen et al., 2025), larger backbone for instruction- folluetion-followinngg and (3) SDXL for text-to-image generation. We further demonstrate that Bi-LoRA be integrated with existing LoRA variants to provide consistent improvement, and conduct ablation studies to examine its hyperparameter sensitivity. More results on natural language understanding tasks with T5-base are provided in Appendix I.
4.1 BASELINES
We mainly compare with the following baseline methods:
Full FT fine-tunes all model parameters.
LoRA applies low-rank adaptation to all linear modules.
LoRA-SAM applies sharpness-awan aware minimization over LoRA parameters.
Random perturbation baselines: Random Output Perturbation (ROP) perturbs logits y with
Gaussian noise c, i.e., ÿ = y + c, <~ (O, p²T); Random Weight Perturbation f perturb either full weights or only LoRA adapters with Gaussian noise.
SAM variants, including LoRA-BBARRnBAR (Li et al., 2024a), Flat-LoRA (Li et al., 2025), LoRA-ES. DRRA-ESAM (Du et al., 2022a). LoRA-LooKSAM (Liu et al., 2022b), S2 SAM (Ji et al., 2024), and WSAM (Yue et al.. 2023).
LoRA variants, including LoRA-GA (Wang et al., 2024), PISSA (Meng et al., 2024), DoRA (Liu et al., 2024), HiRA (Huang et al., 2025), and DeLoRA (Bini al., 2025).

---

and evaluate on Published as a conference paper at ICLR 2026
Table Results of fine-tuning Llama 2-7B and Llama 3.1-8B on different tasks. "Cost" indicates the gradient steps per training iteration, e.g., one step (Cost x 1) for Full FT, LoRA, and Bi-LoRA.
Llama 2-7B
Llama 3.1-88
Method
Cost
GSMSK
HumanEval
MT-Bench
MMLU
DROP
HEval
BBH
FT
59.74.20000
33.122
6.16g0.00
20.31
51.52 45
41.45 68
44.78
LoRA
x1
58.21+0.344
24.75+0.22
5.92+0.10
63,38+0.39
49.82+0.0 54
43.15+0.98
42,82+0027
LoRA-SAM
X2
39.16g0.52
26.59
5.97
63.46.0.39
50,94
44.36
43,4940.40
ROP
x1
10
25.41
6.05 1..00
63.63 10.20
49.96
42:27
43.47.10229
RWP full
x1
go.30
26.26
6.01 10.12
63.40
50.11
44.31 1.0.
43.35
AWP_LeRA
1o.8
24.92.0.1
1..11
63.50
50.1610
42.68 108.100000
44.10
LoRA-oBAR
x1
59.2640.58
26.30 ss
5.97
63.62.0.12
49.92 48
43.49
43.44
LoRA-1BAR
X1
59.72.1.22
26.50 aa
6,1010.00 os
63.4510.10
49.80 oa
45.23 90
43.39
Flat-LRRA
59.44
26.67 as
5,98 os
63.67 63.67 #0.33
50.44.0 17
44.31 78
40.19
LORA-ESAM
x1xx
1..33
24.84
5,80 13
61.79 11.65
49.31 so
42.48
43.40
LoRA- LoRA-LaOKSAM
X1x
58.5510.2s
25.28
5.9410.00
63.4510.29
50.34.0.38
42.48.0.54
43.28.10:20
Bi-LoRA
60.32+0.30
27.20+0.42
6.26+0.00
40.67
51.53+0.55
46.12+0.89
43.45.0.51
To more rigorously evaluate the effectiveness in improving generalization, we adopt a stronger training protocol with larger learning rate than prior works (Wang et al., 2025; 2024). The same protocol is applied across all methods for fairness. Unless otherwise specified, all results are averaged over three independent runs with standard errors. See Appendix R for detailed hyperparameter settings and more training protocol.
4.2 RESULTS ON LLAMA MODELS
Setting. We evaluate the performance of Bi-LoRA on Llama 2-7B/3. across four tasks: mathe- matical reasoning, code generation, dialogue generation, and instruction following, following Wang et al. (2024); Li et al. (2025); Ren et al. (2024). Llama 2-7B is fine-tuned on the first three tasks, while Llama 3.B is used for instruction following. Each task focuses on a specific capability and uses well-established datasets and metrics for training and evaluation, as detailed below:
Mathematical reasoning. Our model fine-tuned on 100k subset of MetaMathQA and evaluated on GSMSK. Performance is measured by accuracy.
Code generation. We fine-tune our model on 100k subset of the HumanEval benchmark, using the PASS@ @1 metric.
Dialogue generation. We train our model on the WizardLM dataset and evaluate on MT-Bench.
The response quality is assessed with GPT-4, and report the first-turn score 10-point scale. Instruction following. Fine-tuned on Cleaned Alpaca (Taori et al., 2023) and evaluated on INSTRUCTEVAL (Chia al., 2023), reporting exact match for MMLU, DROP, and BBH, and PASS@1 for HumanEval.
In experiments, following Du et al. (2022a), (Liu et al., 2022b), LORA-ESAM perturbs 50% of the parameters on the top-50% sharpness-sensitiee data, while LoRA-LooKSAM applies SAM perturbation every five steps, and thus their costs are denoted as "x I.x".
Results. We begin with Llama 2-7B for math, code, and chat tasks. The results, presented in Table 1 (left), demonstrate Bi-LoRA's superior performance. Compared LoRA, Bi-LoRA achieves improvements of 2.11% accuracy on GSMSK, 2.45% on HumanEval, and 0.34 on MT-Bench. Importantly, these gains are even more pronounced than those achieved with the smaller T5-base model (Appendix 1), and Bi-LoRA operates at similar speed as LoRA, underscoring its scalability. Furthermore, Bi-LoRA narrows the gap between LoRA fine-tuning and full fine-tuning, and notably outperforms full fine-tuning on GSM8K and MT-Bench tasks, which are not attainable by competing methods. In contrast, we observe that LoRA-SAM does not consistently improve upon LoRA. Random- perturratrrtttiioonnnn baselines provide only marginal or task-specific gains relative to LoRA-SAM. showing that trivial random noise perturbation offers limited generalization improvement. Unlike efficient-SA. SAM variants that either incur extra cost (LORA-ESAM, Look-Loo compromise performance on certain tasks (e.g., Flat-LoRA on DROP), Bi-LoRA achieves consistent gains on all benchmarks at single-step computation.
Next, we turn to the instruction following task with Llama 3.1-8B. As shown in Table 1 (right), Bi-LoRA outperforms almost all baselines, surpassing LoRA by 0.29% on MMLU, 1.71% DROP,
7

---

Published as a conference paper at ICLR 2026
Table Results of fine-tuning Qwen 2.B-14B on instruction-following tasks. "Cost" indicates the gradient steps per training iteration, e.g., one step (Cost x for LoRA and Bi-LoRA
Method
Cost
MMLU
DROP
HEval
BBH
Avg.
Vanilla
x1
79. 79.28
50.55
10.93+1.13
57.47 03
64.81
LoRA-SAM
x2
40.19
54.54
70.53
58.25
65.63
ROP
x1
79.22+0.28
52.74
71.34
57.8440.49
65,28
RWP full
x1
78.92
54.75+1.01
71.13+1.47
57.76+0.07
65.64
LoRA
79.07 x1
51.69
71.34
57.45
64.89
LoRA-OBAR
x1
79.37+0.22
55.70+0.61
69.92.40.0
58.53
65.88
LoRA-nBAR
x1
79.34
55.96.10.45
69.9210.20
58.67 1.0.4
65.97
Flat-LoRA
x1
79.51 10.20
54.90 11.06
71.54 10.20
58. 1..45..55
66.02
LoRA-ESAM
x1xx
79.55+0.03
54.83
65.0440.81
58.43 58.43
64.46
LoRA-LOoKSAM
x1.x
79.63+0.23
56.35
67,88
58.41
65.57
Bi-LoRA x1
79.67
56.49
71.34
58.9340.05
66.61
2.97% on HumanEval, and 0.63% on BBH. While LoRA-SAM and the two types of baselines show improvements over LoRA, Bi-LoRA achieves HumanEval datasets, while requiring only half the training time. Overall, Bi-LoRA offers better accuracy and efficiency trade-off than both random perturbation and efficient SAM baselines.
substantial gains, particularly DROP and
4.3 RESULTS ON QWEN MODEL
To demonstrate Bi-LoRA's performance across model size and architecture, we next experiment on Qwen 2.5, a distinct and larger backhone compared to LlaMA 2/3.1. We focus on instruction- following tasks. Table shows that Bi-LoRA achieves the highest average score, improving over LoRA-SAM by 0.98%. Compared to the strongest alternative (Flat-LoRA) across both random perturbation and efficient SAM variants baselines, Bi-LoRA delivers additional 0.59% average gain. This indicates its applicability across architectures and scales.
4.4 RESULTS ON DIFFUSION MODELS
Setting. We apply Bi-LoRA subject-driven generalization task and finetune SDXL (Podell et al., 2023) via Dreambooth (Ruiz et al., 2023) on 3D Icons dataset, which contains 23 square-icon images.
Table 3: CLIP 12T and T2T similarity (%) for LoRA and Bi-LoRA the 3D Icons dataset.
Method CLIP 12T (1)
CLIP T2T (1)
LoRA
32.43
42.27 12.08
Bi-LoRA
33.1410.41
46.79 12.37
Results. Table 3 reports the average CLIP image-text (12T) and text-text (T2T) similarity scores six prompt instances of the form "a ToK icon of a <Instance>, in the style of TOK", each evaluated across 200 runs. Com- pared with LoRA, Bi-LoRA improves the av- erage 12T and T2T similarities by 0.71 and 4.52, respectively. demonstrating its stronger personalization capability. As shown Fig- ure 5, in the second row, the image generated by Bi-LoRA artfully merges the fluffy rabbit with the icon, whereas LoRA either fails to in- tegrate the rabbit into the icon (fourth) or even does not generate icon (third). Furthermore, Bi-LoRA better preserves the attributes the
Training Tinget
LoRA
Bi-LoRA
Prompt: ToK lcon o fluff rohbir, tn the syle Tok. Figure 5: Images generated by SDXL fine-tuned with LoRA and Bi-LoRA on the 3D icon datasets, where each column uses the seed for fair comparisons. rabbit, such the eyes in the first column. More qualitative examples shown in Appendix K.
8

---

Published as a conference paper at ICLR 2026
Table 5: Ablations of each component in Bi-LoRA using Llama 3.1-8B on instruction following tasks. We compare vanilla LoRA, Dual-LoRA (two descent branches with global clipping), and Bi-LoRA under different clipping schemes (no clipping, per-layer clipping, and global clipping).
Method
Clipping
Adversorial
MMLU
DROP
HEval
BBH
Avg.
LoRA
X global
X
63.38
49.82.20554
43.15
20.8
49.79
Dual-LoRA
x
63 a..12
49.97-40.17
42.71 42.71 40.47
43.25-00116
49.86
LiRL (no clip)
x
61.33-0.42
43.31
40.24 as
41.96-20.48
47.71
Bi-LaRA laer-l clip)
per-layer global
63.4 63.49:1..26
50.26.20300
43.29:10
43.06
50.03
Bi-LoRA (global clip)
63.67 +0.15
51.53+0.33
46.12+t0.89
43.45-00.15
51.19
4.5 INTEGRATION WITH OTHER LORA VARIANTS
In this section, we evaluate Bi-LoRA's effec- tiveness when combined with existing advanced LoRA variants. Specifically, we consider three LoRA variants, including LoRA-GA (Wang et al.,
Table 4: Results of fine-tuning T5-base on MRPC and CoLAA using various LoRA variants, both standalone and combined with Bi-LoRA.
2024), PiSSA (Meng et al., 2024) and DoRA (Liu et al., 2024), and fine-tune the T5-base model on the MRPC and CoLA datasets, with detailed train- ing settings provided in Appendix R.I. As shown in Table 4, integrating Bi-LoRA improves the av- erage score across MRPC and CoLA by 1.45%, achieving gains of up to 1.80% on MRPC and 1.83% on CoLA. These results confirm that Bi-LoRA can be seamlessly integrated with previous approaches and deliver consistent improvements.
Method
MRPC
CoLA
Avg.
LoRA-GA
88.81
58.8721.00
73.84
PISSA
88.15 10:21
58 58.6619.47
73:41
DoRA
88.81 20.39
59.8920.72
74.35
LoRA-GA + Bi-LoRA
89.62+0.24
60.70+0.27
75.16
PISSA + Bi-LoRA
89.95 $0.50
59.77 59.77 tu.s
74.86
DoRA Bi-LORA
89.54_1.13
60.77
75.16
4.6 ABLATIONS AND HYPERPARAMETE SENSITIVITY
Setting, In this section, we ablate Bi-LoRA to quantify each component's contribution, and investigate the hyperparameter sensitivity from three factors: (1) auxiliary learning rate 1/2, (2) ranks of primary (r1) and auxiliary r2 LoRA modules, and (3) joint sensitivity to neighborhood radius p and auxiliary rank r2. We fine-tune Llama 3.1 3.B and evaluate its performance on the instruction following tasks.
Component- wise ablations of Bi-LoRA. We examine whether the improvements Bi-LoRA arise from the additional capacity introduced by the second LoRA branch or from the adversarial ascent step. To this end, we ablate the clipping schemes and the usage of the auxiliary LoRA module.
Table 5 shows that Dual-LoRA, which shares the same two-LoRA architecture as Bi-LoRA but updates both LoRA modules with gradient descent, only slightly outperforms vanilla LoRA (+0.07 Avg.). Intuitively, the parallel descent branch in Dual-LoRA behaves similarly to merging with another LoRA update whose norm is constrained by the global radius p. which prevents it from fully leveraging the increased rank and representational capacity. This indicates that the performance gains of Bi-LoRA primarily stem from the adversarial ascent step rather than from merely adding an extra LoRA branch. We further compare global clipping with two alternatives: no-clipping and per-layer clipping. Removing clipping leads to noticeable degradation (-3.48 Avg.), as the perturbations become excessively large. And per-layer clipping yields only marginal improvements over LoRA (+0.24 Avg.). These observations support global clipping as the more stable and effective choice. A more detailed explanation of why we adopt global clipping is provided Appendix G.
Sensitivity to auxiliary learning rate 72- Table 6 shows that Bi-LoRA is insensitive to auxiliary learning rate T2: the average score remains within a narrow band of [50.55, 51.19) as 1/2 varies from le-4 to le-3. This indicates that Bi-l Bi-LoRA is robust to 7/2 in a relatively wide range. Interestingly, the best average performance is obtained when 7/2 = 3e-4, which coincides with the optimal primary learning rate 7)1 given by Ren et al. (2024). This suggests that using shared lcarning rate for the primary and auxiliary branches yields more coordinated optimization, consistent with our observation in Table A7. Hence, in all main experiments, we simply //2 !h, reducing hyperparameter tuning while retaining strong performance.
Sensitivity to ranks r1 and 72- Figure 6a shows that the average point rises monotonically up the auxiliary rank r2 = 8, which we adopt as the default for fine-tun fine-tuning scenario. Additionally, further increasing ra does not yield significant improvements even downgrades the performance.

---

Published as a conference paper at ICLR 2026
Table Sensitivity of Bi-LoRA the auxiliary learning rate 72 with Llama 3.1-8B on instruction following tasks, with the same primary learning rate f) in Section 4.2.
MMLU
DROP
HEval
BBH
Avg
lc-4
63.89
10.25
51.05 10.13
45. 10.35
10.14
50.73
3c-4
63.67 #0.15
51.53
46.12
43.45
51.19
5e-4
63.47
51.23
45.9340.73
43.05::0.12
50.92
le-3
63.40zo.17
51.32
40.51
42.96.0022
50.55
0.01
49.80
50.10
50.72
50.34
51.0
50.6
Average score
0.05
50.8
Average score
50.51
50.76
51.19
51.00
-0.4
50.49
50.54
50.78
50.75
50.6
0.1
50.31
50.82
50.85
50.89
50.4
50.08
50.83
-50.0
0.2
50.13
50.27
50.66
50.75
50.2
50.0
49.8
2
8
16
4
16
(a) Effect of and r2.
(b) Effect of and rz.
Figure 6: Sensitivity of Bi-LoRA to primary (r1) and auxiliary (r2) LoRA ranks and perturbation
radius p with Llama 3.I-BB on instruction-folloiin tasks.
Sensitivity to neighborhood radius and auxiliary rank r2. Figure 6b further examines the joint effect of p and r2. The heatmap also corroborates that enlarging 72 from 2 to 8 beneficial. Overall, these trends suggest that and govern complementary aspects of neighborhood size and auxiliary capacity and can be tuned largely independently for given architecture and dataset, line with previous work of SAM.
4.7 TRAINING TIME AND MEMORY COST
Table 7: Peak memory and time per optimization step (relative to LoRA in parentheses).
Model & Dataset
Method
Memory (GB)
Time (s)
LoRA
23.69
8.93 (100%)
Liama 3.1-8B Cleaned Alpaca
LARA-SAM
24.00
Bi-LORA
24.32
9.71 (109%)
We present detailed comparisons of memory and time per optimization step for LoRA, LoRA-SAM. and Bi-LoRA on Llama 3-B (Cleaned Alpaca). Table shows that Bi-LoRA significantly reduces the LoRA-SAM's training overhead, decreasing it from over 210% to about 1 110% relative to vanilla LoRA. This gain comes from jointly performing optimization and perturbation, eliminating SAM's extra gradient step. For memory, Bi-LoRA incurs only a minimal overhead (< 0.7GB), as it adds lightweight auxiliary module. Additional comparisons on training time are provided in Appendix
CONCLUSION
In this paper, we propose Bi-LoRA, a novel dual-LoRA variant that leverages an auxiliary LoRA module to enhance the generalization performance of low-rank adaptation. Bi-LoRA decouples optimization from weight perturbation for better optimizing the sharpness the loss landscape, allowing both to be updated in single backward pass without requiring additional gradient steps, as in SAM. Extensive experiments across diverse tasks and architectures demonstrate Bi-LoRA's efficiency and effectiveness in improving generalization.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
The research leading to these results has received funding from National Key Research Development Project (2023YFFI 104202) and National Natural Science Foundation of China (62376155). The authors would like to thank the chairs and reviewers for their thoughtful comments on this paper. Yuhang Liu also gratefully acknowledges the financial support provided by Prof. Xin Yang at Shanghai Jiao Tong University.
REFERENCES
Maksym Andriushchenko and Nicolas Flammarion. Towards understanding sharpness-aware mini- mization. In International Conference Machine Learning (ICMI.), 2022.
Massimo Bini, Leander Girrbach, and Zeynep Akata. Decoupling angles and strength low-rank adaptation. In International Conference Learning Representations (ICLR), 2025.
Xiangning Chen, Cho-Jui Hsieh, and Boqing Gong. When vision transformers outperform resnets without pre-training or strong data augmentations. In International Conference Learning Representations (ICIR), 2022.
Yew Ken Chia, Pengfei Hong, Lidong Bing, and Soujanya Poria. Instructeval: Towards holistic evaluation of instruction-tuned large language models. arXiv preprint (tt.:2306000777,, 2023.
Yandi Dai, Sai Praneeth Karimireddy, Sebastian Stich, and Martin Jaggi. The crucial role of normalization in sharpness-awe aware minimization. In Neural Information Processing Systems (NeurIPS), 2023.
Jiaxin Deng, Qingcheng Zhu, Junbiao Pang. Linlin Yang, Zhongqian Fu, and Baochang Zhang. Eflat-lora: Efficiently seeking flat minima for better generalization in fine-tuning large language models and beyond. arXiv preprint (rXXiv2508.00022) 2025.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. In Advances in Neural Information Processing System (NeurIPS), 2023.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:/810.0480) 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao. Wei Shen, Limao Xiong, Yuhao Zhou, Xiao Wang, Zhiheng Xi, Xiaoran Fan, Shiliang Pu, Jiang Zhu, Rui Zheng, Tao Gui, Qi Zhang, and Xuanjing Huang. LoRAMoE: Alleviating world knowledge forgetting in large language models via MtE-style plugin. In Proceedings of the Association for Computational Linguistics (Volume 1: Long Papers), 2024.
Jiawei Du, Hanshu Yan, Jiashi Feng, Joey Tianyi Zhou, Liangli Zhen, Rick Siow Mong Goh, and Vincent Tan. Efficient sharpness-awert minimization for improved training of neural networks. In International Conference on Learning Representations (ICLR), 2022a.
Jiawei Du, Daquan Zhou, Jiashi Feng, Vincent Y. F. Tan, and Jocy Tianyi Zhou. Sharpness-aware training for free. Advances in Neural Information Processing Systems (NeurIPS), 2022b.
Abhimanyu Dubey, Abhinay Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21788 2024.
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness minimization for efficiently improving generalization. In International Conference on Learning Representations (ICLR), 2021.
11

---

Published as a conference paper at ICLR 2026
Sepp Hochreiter and Jürgen Schmidhuber. Simplifying neural nets by discovering flat minima. In Advances in Neural Information Processing Systems (NeurIPS), 1994.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural computation, 1997.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhus Yuanzhi Li, Shean Wang. Lu Wang.
and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-efficient hadamard high-rank adaptation for large language models. In International Conference on Learning Representations (ICLR), 2025.
Jie Ji, Gen Li, Jingjing Fu, Fatemeh Afghah, Linke Guo, Xiaoyong Yuan, and Xiaolong Ma. A single-step, sharpness-aware minimization is all you need to achieve efficient and accurate sparse training. In Advances Neural Information Processing Systems (NeurIPS), 2024.
Weisen Jiang, Hansi Yang. Yu Zhang, and James Kwok. An adaptive policy employ sharpness aware minimization. In The Eleventh International Conference on Learning Representations (ICLR), 2023.
Pham Duy Khanh, Hoang-Chau Luong, Boris S. Mordukhovich, and Dat Ba Tran. Fundamental convergence analysis of sharpness-aware minimization. In Neural Information Processing Systems (NeurlPS), 2024.
Minyoung Kim, Da Li, Shell X Hu, and Timothy Hospedales. Fisher sam: Information geometry and sharpness aware minimisation. In International Conference on Machine Learning (ICML), 2022.
Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung. Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision (ECCV), 2020.
Soroush Abbasi Koohpayegani, Navaneet K. L., Parsa Nooralinejad. Soheil Kolouri, and Hamed Pirsiavash. NOLA: compressing lora using linear combination of random basis. In International Conference on Learning Representations (ICLR), 2024.
Jungmin Kwon, Jeongseop Kim, Hyunseo Park, and Kwon Choi. ASAM: adaptive sharpness-aware minimization for scale-invariant learning of deep neural networks. In International Conference Machine Learning (ICML), 2021.
Bingcong Li and Georgios Giannakis. Enhancing sharpness-awand optimization through variance suppression. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Bingcong Li, Liang Zhang, and Niao He. Implicit regularization of sharpness-awar minimization for scale-invariant problems. Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. Advances in Neural Information Processing Systems (NeurIPS), 2018
Siwei Li, Yifan Yang, Yifei Shen, Fangyun Wei, Zongqing Lu, Lili Qiu, and Yuqing Yang. LoRASC: Expressive and generalizable low-rank adaptation for large models via slow cascaded learning. In Findings of the Association for Computational Linguistics: EMNLP. 2024b.
Tao Li, Lei Tan, Zhehao Huang, Qinghua Tao, Yipeng Liu, and Xiaolin Huang. Low dimensional trajectory hypothesis is true: Duns can be trained in tiny subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2022.
Tao Li, Pan Zhou, Zhengbao He, Xinwen Cheng, and Xiaolin Huang. Friendly sharpness-aware minimization. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2024c.
Tao Li, Zhengbao He, Yujun Li, Yasheng Wang, Lifeng Shang, and Xiaolin Huang. Flat-lora: Low-rank adaptation over a flat loss landscape. In International Conference on Machine Learning (ICML), 2025.
12

---

Published as a conference paper at ICLR 2026
Yang Lin, Xinyu Ma, Xu Chu, Yujie Jin, Zhibang Yang, Yasha Wang, and Hong Mei. Lora dropout as a sparsity regularizer for overfitting control. arXiv preprint arXiv:2404.099610, 2024.
Shih-Yaan Yang Liu, Chien-Yr Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Swang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In International Conference on Machine Learning (ICML), 2024.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable Sharpness-awarr minimization. In Proceedings of the IEFE Conference Computer Vision and Pattern Recognition (CVPR), 2022a.
Yong Liu, Siqi Mai, Xiangning Chen, Cho-Jui Hsieh, and Yang You. Towards efficient and scalable sharpness-aware minimization. In Conference on Computer Vision and Paltern Recognition (CVPR), 2022b.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PISSA: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arxiv preprint arXiv:2307.01952. 2023.
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412.15115. 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research (JMLR), 2020.
Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Melora: Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In Proceedings of the Association for Computational Linguistics (Volume I: Long Papers), 2024.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani. Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
Rohan Taori, Ishaan Gulrajani. Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-followee llama model, 2023.
Chunlin Tian, Zhan Shi, Zhijiang Guo, Li Li, and Cheng zhong Xu. HydraloRA: An asymmetric LoRA architecture for efficient fine-tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092888 2023.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Superglue: A stickier benchmark for general-purper language under- standing systems. In Advances in Neural Information Processing Systems (NeurIPS), volume 32, 2019a.
13

---

Published as a conference paper at ICLR 2026
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: multi-task benchmark and analysis platform for natural language understanding. In International Conference Learning Representations (ICLR), 2019b.
Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In Advances in Neural Information Processing Systems (NuerIPS), 2024.
Zhengbo Wang. Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters properly optimized? In International Conference on Learning Representations (ICLR), 2025.
Yichen Wu, Hongming Piao, Long-Kai Kai Huang, Renzhen Wang, Wanhua Li, Hanspeter Pfister, Deyu Meng, Kede Ma, and Ying Wei. SD-IoRA: Scalable decoupled low-rank adaptation for class incremental learning. International Conference on Learning Representations (ICLR), 2025.
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu. Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsich, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for loRA optimization. In International Conference on Learning Representations (ICLR), 2025.
Yun Yue, Jiadi Jiang, Zhiling Ye, Ning Gao, Yongchao Liu, and Ke Zhang. Sharpness-aware minimization revisited: Weighted sharpness as a regularization term. In ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
Jingfan Zhang, Yi Zhao, Dan Chen, Xing Tian, Huanran Zheng, and Wei Zhu. Milora: Efficient mix- ture of low-rank adaptation for large language models fine-tuning, In Findings of the Association for Computational Linguistics: EMNIP. 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for murameter-efficien fine-tuning, In International Conference on Learning Representations (ICI.R), 2023.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory- LLM training by gradient low-rank projection. In International Conference on Machine Learning (ICML), 2024.
Juntang Zhuang, Boqing Gong, Liangzhe Yuan, Yin Cui, Hartwig Adam, Nicha Dvornek, Sekhar Tatikonda, James Duncan, and Ting Liu. Surrogate gap minimization improves sharpness-aware training. In International Conference on Learning Representations (ICLR), 2022.
14

---

Published as a conference paper at ICLR 2026
APPENDIX
A ETHICS STATEMENT
This paper proposes Bi-LoRA, a parameter-effiiien fine-tuning method that introduces adversarial perturbations to improve generalization of large language models, which can reduce memory and energy costs, and thus broaden access to LLM research.
For dataset use, we use only publicly available datasets under their licenses, with human subjects or personal data involved.
Regarding safety and privacy, recognize potential risks. By reducing hardware barriers, Bi- LoRA could also enable malicious actors to efficiently fine-tune models for harmful applications or exacerbate dataset biases if such biases are present. Furthermore, the auxiliary adversarial adapter, while improving robustness, introduces additional parameter pathway that could hypothetically be exploited for backdoor insertion or undesired behaviors if misused. To mitigate these risks, we will anonymously release only training code and experimental scripts in the supplementary materials that follow community standards for responsible research, and we rely exclusively on open benchmarks with no sensitive personal data. We encourage future users of Bi-LoRA to adopt similar safeguards, including dataset audits for bias, robust model evaluation, and adherence to ethical guidelines on downstream applications.
B REPRODUCIBILITY STATEMENT
We provide detailed experimental settings and hyperparameters in Appendix ensure reproducibil- ity. The Bi-LoRA implementation and example experiment scripts will be released anonymously in the supplementary materials. All models, benchmarks, and evaluation metrics used this paper either open-sourced or explicitly identified when closed-source resources are involved.
C LLM USAGE STATEMENT
use large language models (LLMs) as general-purpost tools at the following stages of this work.
Writing, We employ both closed-source and open-source LLMs solely for text polishing, including grammar correction, clarity improvement, and minor stylistic refinement. Importantly, they are not used to generate research ideas, technical content, and substantive arguments.
Benchmarking. For evaluation on the MT-Bench benchmark, we use the GPT-4 API to obtain GPT-4 scores, following the standard evaluation protocol for MT-Bench.
No part of the research design, methodology, or core scientific contributions relied on LLMs. Their role is restricted to auxiliary assistance in language refinement and standardized benchmark scoring.
D DETAILS ON SHARPNESS-ANE AWARE MINIMIZATION (SAM)
Let L(W) denote the empirical loss of model parameters W. Standard ERM can converge to sharp minima where small parameter perturbations significantly increase loss. SAM's goal is target the training toward flat regions by minimizing the worst-case loss inside a:
min
max L(W + W [ell
(A1)
where p 0 is the neighborhood radius, || is kind of norm, and is the perturbation. Intuitively, Eqn, (A1) prefers parameters whose entire neighborhood has uniformly low loss (flat minima), which has been empirically linked improved generalization. A practical single-step approximation solves the inner maximization by first-order ascent and then updates W using the gradient evaluated at the perturbed point.
Denote by L(W: B3) the mini-batch loss on B and by V its gradient. A typical SAM update computes normalized gradient ascent perturbation and then descent step the same mini-batch
15

---

Published as a conference paper at ICLR 2026
Algorithm A1 Sharpness-Aware Minimization (SAM)
1: Input: Initial weights Wo; learning rate n7, radius p. optimizer Opt(-); max steps K
2: Output: Final weights W
3: Initialize model weights W <- Wo
4: for = 0, 1..., K do
5: Sample mini-batch B,
6: g - WwW(W; B.) g
[gradient at clean weights] (first-order worst-case direction)
7:
E+ P.
1g
8:
Wadv W +
9: Jadv + Vw aWavv BB)
( (use the same batch to produce perturbed gradient) [e.g.. SGD/AdamW step)
10: W + Opt( W, Gadv; 1)
11:
end for
12: return W
Be but at different parameter values:
V WwL(W,; B.)
Et = p
(A2)
V W C(W;; B,)
W411 W 1f Vw + ce; B,),
(A3)
i.e., two gradient evaluations per iteration. Eqn. (A2) determines the perturbation direction at W (clean weight), while Eqn. (A3) computes the update at the perturbed parameter W, + e, (perturbed weight).
E RELATED WORK
Low-Rank Adaptation (LoRA). LoRA (Hu et al., 2022) is a widely adopted parameter-efficien fine- tuning (PEFT) method. It models weight updates via low-rank matrices without incurring additional inference costs. Many studies have been proposed to improve the performance of LoRA (Kooh- payegani et al., 2024; Dou et al., 2024; Tian et al., 2024; Wang et al., 2025; Yen et al., 2025; Wu al., 2025). On the resource perspective, AdaLoRA (Zhang et al., 2023) dynamicalll adjusts the rank allocation, while MELORA (Ren et al., 2024) trains multiple mini-LoRA modules in parallel to cut down on trainable parameters. From the optimization perspective, LoRA-GA (Wang et al., 2024), PISSA (Meng et al., 2024) and MiLoRA (Zhang et al., 2024) accelerate convergence and boost performance through enhanced initialization, and DoRA (Liu et al., 2024) decomposes the adaptation into magnitude and direction for better optimization. In this paper, we enhance LoRA optimization by optimizing the loss landscape sharpness of the full parameter space through an auxiliary LoRA module. Our approach is orthogonal to prior works.
Sharpness-Awar Aware Minimization (SAM). SAM formulates the optimization objective as min- max problem to seek flat minima, encouraging model parameters to reside in regions with consistently low loss values, yielding state-of-the-arr generalization and robustness. Despite its effectiveness, SAM doubles the computational cost because its inner maximization is approximated via additional gradient ascent step, thereby limiting its application to large-scale models. Several SAM variants have been developed enhance its generalization performance (Kim al., 2022: Li & Giannakis, 2023; Li et al., 2024c). ASAM (Kwon& et al., 2021) l) introduces adaptive sharpness to address SAM's weale-depnndency issue, while GSAM (Zhuang al., 2022) jointly minimizes surrogate and perturbed losses to locate flatter region. Mcanwhile, other works aim to improve SAM's training efficiency (Du al., 2022h; Jiang et al., 2023; Ji al., 2024). LookSAM (Liu et al., 2022a) applies adversarial perturbations only periodically. Recent studies have also focused on applying SAM to LoRA fine- tuning, aiming to enhance generalization while maintaining training efficiency. For example, BAR (Li al., 2024a) makes SAM's implicit balancedness regularization explicit for scale-invariant tasks like LoRA, achieving SAM-level generalization gains with significantly less computation. Flat-LoRA (Li et al., 2025) replaces the costly inner maximization with random weight perturbation to tackle the coupling issue in LoRA-SAM without sacrificing efficiency. this paper, we introduce a novel framework that integrates SAM with LoRA through dual LoRA modules. Our method employs
16

---

Published as a conference paper at ICLR 2026
auxiliary LoRA modules generate adversarial perturbations decoupled from optimization, incurring minimal additional overhead.
F PROOFS
F.1 PROOF AND DISCUSSION OF PROPOSITION ALIGNMENT OF BI-LORA'S ASCENT DIRECTION WITH PREVIOUS FULL GRADIENT
Proof. Let
G. = VwWWW + BI, +
(A4)
denote the full gradient step with & B2, the auxiliary perturbation.
One update step for Bi-LoRA is formalized as
B B1,411 = B1, G,
A1,411 = A1. 1/1 Bi, G2, Gr,
(A5)
B2 B2, + 72
A A2,t 12,
where 7/1, 722 are step sizes. Expanding the auxiliary perturbation Eqn. (A5) gives
E+1 B2, A2,411 = + (GA42, + B2, B2, + O(
(A6)
Thus, the alignment between the previous gradient Eqn. Eqn. (A4) and the update of the auxiliary perturbation is
(G,, - Et) = -12 (12
1 + 1B O 20,
(A7)
since both terms the right-hand side nonnegative (squared Frobenius norms).
Therefore, Eqn. by Eqn. (A7), the auxiliary update (B2, A2) always increases the inner objective in the direction of the full gradient Eqn. Eqn. (A4). This proves that the Bi-LoRA perturbation aligns with previous SAM's perturbation direction.
As reported in several efficient SAM variants, reusing the previous ascent direction can still yield strong generalization (e.g., S" SAM (Ji et al., 2024) in the sparse training scenario). In general, aligning with the previous, not the current direction, has benefit on efficiency. The reason that aligning with the previous gradient works may be attributed to the www-dimension/low-ree gradients (Li et al., 2022; Zhao et al., 2024), especially in later stage of training, or in fine-tuning stage. Due to the www-dimensioo/llwwratt property, the consecutive gradient directions tend to exhibit similarity. And the intuition behind Bi-LoRA is that adversarial perturbations across different steps are "continuous": the optimal adversarial perturbation iteration likely be close to the optimal perturbation at iteration é + 1, given that the weight differences between the continuous steps are small under LoRA optimization. This means we can obtain the adversarial perturbation at step + with "slight" adjustment based on the perturbation at t. As a result, this method can effectively optimize the sharpness while maintaining the same training efficiency as regular LoRA training.
properties of the
F.2 PROOF AND DISCUSSIONS OF PROPOSITION 3: EOUTVALENCE BETWEEN SAM AND BI-LORA
Proof. SAM solves the min-max objective
min max
L(W +), |el
(A8)
where p > O controls the perturbation radius.
first-order Taylor expansion gives
L(W + R (VV
(A9)
17

---

Published as a conference paper at ICLR 2026
The inner maximization is which is attained at Substituting Eqn. (A11) into Eqn. (A8) yields the regularized objective
max cll2ssp
(VwL(W),c)
(A10)
VwC(W)
- Vw L(W))2
(A11)
+ V (())22
(A12)
For Bi-LoRA, the weight matrix is decomposed W Wo A1 A2.2. The training objective
is
min B
max Ag Aalle
L(Wo + B1A1 B2A2),
(A13)
where the perturbation is explicitly parameterized as
== B2A2, rank(E) Sr, ||ā| s...
(A14)
At 0 Wo B1 expanding gives
[( + [(0) (VoL,
(A15)
Thus the inner maximization becomes
max
(VoL,
(A16)
rank(F) Selees
Let the SVD of V&L be
VoL= = =VV
22.2...
(A17)
The optimum of Eqn. (A16) is achieved
PU,
(A18)
where U,, V, are the top-r singular vectors of VoL, and rank((*) = with ||²*||-
Substituting Eqn. (A18) into Eqn. (A16) gives
(V 2) Σ Vo(Vo = PlVool)(
(A19)
where
|VoBll)) (VoC
(A20)
the Ky Fan r-norm.
Hence, Bi-LoRA reduces to the regularized objective
min L(W, + A1) + V All())
(A21)
A.B.
When r covers all singular values, the Ky Fan norm becomes the nuclear norm, and Bi-LoRA recovers the SAM objective. Conversely, SAM specializes Bi-LoRA when only the sharpest r singular directions are penalized.
And Proposition represents the ideal objective of Bi-LoRA and serves illustrate our motivation in- tuitively. In practice, however, we adopt simultaneous optimization approach formalized in Eqn. (9), which does not introduce additional gradient steps and thus maintains efficiency. Experimental results demonstrate that our approximate approach already significantly improve performance compared to LoRA.
Furthermore, Figure shows that, the auxiliary LoRA module continues broad exploration, while the primary module converges rapidly to a nearly fixed point; once the primary stabilizes, the auxiliary then begins converge quickly. This behavior suggests that at each parameter point, there exists (though inefficient) convergent multi-step scheme for the auxiliary module, which is consistent with the intuition of Proposition 3.
18

---

Published as a conference paper at ICLR 2026
Table Al: Results of fine-tuning Llama 3.1-8B on instruction-followiiin tasks with more LoRA- variants and SAM-based methods.
Method
MMLU
DROP
HEval
BBH
Avg
LoRA
63.38
09.82
43.15
42.82.
49.79
PISSA
13.5940.14
50.20
43.86+0.12
43.25
50.22
DoRA
63.5840.22
50.53
44. 44.10
42.9840.79
50.30
DeLoRA
63,4940.32
51 51.40=0.2s
45.53
42.72.10.29
50.78
HiRA
63.62
51.17 20.26
45.12
42.89
50.70
WSAM
63,42.1011
50.63=0.14
42.88
43.1640.52
50.02
-2-AMM
63.73
50.80
43.09
42.44
50.02
Bi-LoRA
63.67
51.53
46.12
43.45 10.15
51.19
G REASONS OF USING GLOBAL CLIPPING
We clip using the total Frobenius norm over all auxiliary LoRA modules in Bi-LoRA for the following reasons.
Faithfulness to SAM. The StM-stylee inner maximization is defined with single p-ball over the full parameter vector, i.c., max Sp [(w + <). In Bi-LoRA, the perturbation is implemented entirely through the auxiliary LoRA modules, so constraining the global Frobenius norm of all auxiliary modules (Eqn. (8) is the faithful analogue of SAM's original formulation.
Avoiding scale explosion and instability from per-layer normalization. If each layer is normal- ized independently, every layer receives a perturbation with norm so the overall perturbation magnitude roughly scales with the number of adapters. This not only leads training instability, but also makes the same incomparable across architectures with different depths or adapter place- ments. SAM's original global constraint is designed precisely to avoid such architecture- scaling effects.
Empirical evidence. Table 5, global clipping consistently outperforms both per-layer clipping and no clipping in terms of final performance and training stability under the same p.
In summary, global clipping over all auxiliary LoRA modules yields scale-aware neighborhood that is consistent with SAM's objective, comparable across architectures, and empirically more stable, which motivates our choice of this clipping scheme.
H MORE BASELINES WITH LLAMA 3.B-8B ON INSTRUCTION FOLLOWING TASKS
Table Al further compares Bi-LoRA with four LoRA variants (PiSSA, DoRA, HiRA, DeLoRA) and two SAM-based methods (WSAM, S2-SAM) on Llama 3.1 instruction following tasks. Bi-LoRA attains the highest average performance of 51.19, outperforming the strongest LoRA variant by about 0.4% and vanilla LoRA by roughly 1.4% points.
I RESULTS ON NATURAL LANGUAGE UNDERSTANDING
Setting, We fine-tune the T5-base model (Raffel al., 2020) on multiple datasets from GLUB (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a) benchmarks. including MNLI. SST2, CoLA, QNLI, MRPC, BoolQ, CB, COPA, RTE, and WIC. following Wang et al. (2024); Li et al. (2025). Performance is evaluated on the development using accuracy as the primary metric, except for CoLA, where the Matthews correlation coefficient is used.
Results. We first focus the GLUE datasets. From the results in Table A2, observe that Bi-LoRA outperforms both LoRA and LoRA-SAMA achieving an average improvement of 0.47% and 0.32%, respectively. It is worth noting that Bi-LoRA achieves these gains with the same training speed as
19

---

Published as a conference paper at ICLR 2026
00.3
59.93
59.59
59.95
59.76
59.25
80.0
59.41
59.54
59.79
60.77
60.36
59.42
58.0
58.5
58.0
16
58.49
59.04
59.30
59.45
58.98
59.74
57.0
(LoRA)
32
r2
Figure Al: Performance CoLA under varying ranks of primary (r1) and auxiliary (12) LoRA modules.
LoRA, whereas LoRA-SAM doubles the training time. Moreover, the gains are more pronounced on smaller datasets, with Bi-LoRA outerforming LoRA by 1.36% on CoLA and 0.82% on MRPC. In contrast, LoRA-SAM shows no clear advantage over vanilla LoRA, indicating its limited ability to enhance generalization due constrained sharpness optimization.
Next, we evaluate on the SuperGLUE datasets, which feature more challenging language understand- ing tasks. To reduce variance from the limited evaluation sizes CB (56). COPA (100), and RTE (277) samples, we increase the number of runs from 3 to 20 for CB and COPA, and to 5 for RTE. Table A3 shows Bi-LoRA demonstrates more significant advantages over LoRA and LoRA-SAM by 0.69% and 0.60% average, while LoRA-SAM yields minimal gains and even hurts some datasets (e.g., CB). These results confirm the effectiveness of Bi-LoRA in enhancing generalization.
Table A2: Results on fine-tuning the T5-base model on subset of GLUE datasets. "Cost" indicates the gradient steps per training iteration, e.g., one step (Cost x 1) for Full FT, LoRA, and Bi-LoRA.
Dataset Size
Cost
MNIJ (393k)
SST2 (67k)
CoLAA (8.6k)
QNLI (105k)
MRPC (3.7k)
Avg.
Full FT
x1
85.5720.09
94.27+0.24
56.60
93.18
87.30
83.38
LORA
x1
86.25
94,23 +0.30
59.41 +0.62
93.25 +0.06
88. 33.5640..2e
84.34
LoRA-SAM
x2
86.25
94.46+0.17
59.80
93.21 10.16
88.7340.62
84.49
Bi-LoRA
x1
86.33 86.3310.08
94.34
60.77
60.77.0.38
93.25 13.25100.0
89.38 10.26
84.81
Table A3: Results on fine-tuning T5-base with a subset SuperGLUE datasets. "Cost" indicates the gradient steps per training iteration, e.g., one step (Cost x x) for Full FT, LoRA, and Bi-LoRA.
Dataset Size
Cost
BoolQ (9.4k)
CB (0.25k)
CОPA (0.4k)
RTE (2.5k)
WIC (5.4k)
Avg.
Full FT
x1
72.19
92.2640.24
64.67
84.48
68,3440.19
76.39
LoRA
x1
72.20-0.21
92.86-20.27
63. 63.8040.22
83. 23.1020.20
68, 00.3000.32
76.
LoRA-SAM
x2
72.47 20.42
92.32.1.225
64.20 .2
83.4720.47
68.55
76.20
Bi-LoRA
x1
72.25
92.86-10.32
64.60
83.61
70.69
76.80
J HYPERPARAMETER SENSITIVITY
J.1 EFFECT OF THE AUXILIARY RANK ON CoLA
To further examine the sensitivity to the auxiliary rank r2, fine-tune T5-base on the CoLA dataset with r1 E (4, 8, 16} and 72 {2, 4, 8, 16, 32, 64}.
Figure Al shows the performance of Bi-LoRA with varying the primary (r1) ) and auxiliary (r2) LoRA ranks. We observe that integrating the auxiliary module consistently enhances performance. Notably, T2 generally delivers good results, which replicates the pattern reported in Section 4.6. confirming that the sweet-spot generalizes from NLU tasks to instruction tuning. Therefore, we suggest using r2 task-agnostic default.
20

---

Published as a conference paper at ICLR 2026
J.2 EFFECT OF THE NEIGHBORHOOD RADIUS
From Table A4, Bi-LoRA performs consistently well across p = {0.01, 0.05, 0.1), showing its effectiveness to moderate perturbations during training. The ability to achieve strong performance across a range indicates that Bi-LoRA can effectively adapt to varying levels of perturbation, demonstrating its practicality for diverse applications.
Table A4: Results on CoLA, SST2 and GSM8K under different values of the neighborhood radius p
P
CoLA
SST2
GSM8K
0,01
60.15
60.15
59.36
0,05
60.77+0.39
60.15+0.36
59.74
0.1
60.18 :0.18
60. 15 10.36
60.65
0.2
58.9540.81
60.15
58.83
K QUALITATIVE RESULTS ON SDXL
As discussed in Section 4.4, Bi-LoRA improves both image-text (I2T) and text-text (T2T) CLIP similarity on the 3D Icons dataset. visualize these gains, Figure A2 presents generated images from SDXL models fine-tuned by LoRA and Bi-LoRA under identical promptss.
Observations. Across instances, Bi-LoRA tends to preserve the intended icon identity and style more faithfully. These qualitative differences align with the measured CLIP improvements, i.c., larger average CLIP I2T/T2T scores shown in Table also exhibit clearer textual correspondence in Figure A2.
L CONVERGENCE ANALYSIS OF MAIN AND AUXILIARY LORA MODULES
We measure the cosine similarity between the weights of the main and auxiliary LoRA modules and their inal weights on CoLA and SST2 with T5-base. The results indicate that the auxiliary module converges substantially slower than the main one, preserving flexibility for sharpness-aware updates.
21

---

Published as a conference paper at ICLR 2026
(a) LoRA Fluffy Rabbit
(b) Bi-LoRA Fluffy Rabbit
(c) LoRA Curly Cloud
(d) Bi-LoRA Curly Cloud
(c) LoRA Roaring Lion
(l) Bi-LoRA - Roaring Lion
LoRA
Dripping Water Drop
(h) Bi-L.oRA Dripping Water Drop
(i) LoRA Hopping Frog
(i) Bi-LoRA Hopping Frog
(k) LoRA Smiling Sun
Bi-LoRA Smiling Sun
Figure A2: Images generated with SDXL fine-tuned using LoRA (left) and Bi-LoRA (right) on the 3D icon dataset. Each row corresponds different prompt the form ToK icon of <instance>, in the style of ToK". Images in the same row are generated with the same random seed for fair comparison.
22

---

Published as a conference paper at ICLR 2026
Cosine Similarity
Cosine Similarity
0.6
Main LoRA (eptimintioo) weight
0.6
Main LoRA (optimiznion) final weight
Auciliary LuRA (penurlution) final
Auciliary LoRA (poturivtion) to final weight
0.4
Main and Auniiary LokA at moch sup
0.4
Main Amiilary LoRA stop
0.2
0.2
0.0
0.0
2500
1500
Training Stop
Training Step
(a) CoLA: 772 711
CoLA: 172
107
1.0
1.0
Cosine Similarity
0.8
Cosine Similarity
0.8
0.6
Main LoRA (opimistiion) final weight
0,6
Main LefA (oprimizatiun to final weight
Autiliary (penurbation) final weight
Ausiliary (pettrration)) weight
0.4
Main and Auallary LeRA at ioch stp
0.4
Main and Ausilary LoRA at each stop
0.2
1000
2000
3000
4000
5000
6000
1000
2000
3000
4000
5000
6000
Training Step
Training Step
(c) SST2: 12 = 71 =
(d) SST2: 172 = 10% = 3
Figure A3: Cosine similarity between the main and auxiliary LoRA weights and each final weights during training on CoLA and SST2 using T5-base, under different auxiliary learning rates (122). The auxiliary module (perturbation, blue lines) converges in the final 20% of steps, significantly slower than the main (optimization, red lines) module. A larger 722 = 1077 further slows down convergence. Moreover, the auxiliary module's trajectory remains independent of the main module throughout training (green lines).
M BI-LORA IN QUANTIZED SETTINGS
We apply Bi-LoRA in quantized setting, where the transition to lower precision cause the model to jump from flat region into a sharper region with higher local loss, reducing generalization on unseen data. Following QLoRA (Dettmers al., 2023), quantize the pretrained model NF4 and adopt the setup described Section 5.3.
As shown in Table A5, Bi-QLoRA outperforms QLoRA and QLoRA-SAM by 2.58% and 2.36% on GSM8K, respectively. This gain is attributed to Bi-LoRA's ability to maintain flatness under quantization by decoupling optimization and perturbation during training.
Table A5: Performance on GSM8K under quantized NF4 settings. Results are averaged over runs.
Method
GSM8K (%)
QLoRA
57.77
QLoRA-SAM
57.89 10.29
Bi-QLoRA
59.26
N EFFECT ON ADVERSARIAL ROBUSTNESS
Figure 4 already shows that Bi-LoRA is more robust than vanilla LoRA against random perturbations applied both in the LoRA subspace and in the full parameter space. We now investigate robustness to adversarial perturbations. Concretely, fine-tune T5-base CoLAA and report Matthew's
23

---

Published as a conference paper at ICLR 2026
Table A6: Results on CoLA with adversarial parameter perturbation. Perturbation strength a from to 5.0.
Perturb space
Method
a
a 0.5
1.0
a 2.5
a 4.0
5.0
LoRA
60.62
59.42
57.47
42.07
24.22
19.19
LoRA-SAM
60.95
59.14
58.44
52.12
41.33
30.61
Bi-LoRA
61.12
59.36
59.14
48.47
33.24
24.20
Perturb foll space
Method
a 0.3
1.0
2.5
a 4.U
5.0
LaRA
60.62
$2.69
44.42
19.08
11.39
6.55
BI-LORA
61.12
56.29
49.00
32.91
24.83
18.06
Correlation Coefficient. Following (Kim et al., 2022), we apply the worst-case parameter perturbation
V
0
(A22)
evaluating attacks restricted to the LoRA subspace (0 (B, A)) and (ii) the full parameter space (# = W', with perturbation strength a from 0.5 to 5.0. Results are shown in Table A6.
In the LoRA subspace, Bi-LoRA consistently suffers smaller performance degradation than vanilla LoRA. LoRA-SAM shows the strongest resilience, which aligns with its explicit optimization of sharpness in this subspace. the full parameter space, degradation more substantial across all methods, yet Bi-LoRA still mitigates the impact more effectively than LoRA-SAM.
These findings are consistent with our random perturbation results, demonstrating that Bi-LoRA provides robustness to adversarial parameter perturbations, particularly when the attack targets the full parameter space.
O TRAINING AND TEST METRIC CURVES
We fine-tune T5-base on MRPC 10 epochs, tracking both training and eval losses and metrics. The results in A4 shows nearly identical training losses across methods, yet Bi-LoRA attains lower validation loss and higher accuracy. Importantly, the smaller train-val generalization gap demonstrates Bi-LoRA's improved generalization.
P COMPUTE RESOURCES
We utilize two types of GPUs: the NVIDIA RTX 4090 24GB and the A100 80GB. For NLU and Text- to- immam generation tasks, computations are performed on a single RTX 4090. All other experiments conducted on single A100.
Q LEARNING RATES FOR TUNING THE Two MODULES IN BI-LORA
In experiments, both modules in Bi-LoRA the same learning rate of 5 10 Initially, we explored an unequal learning rate strategy. However, results in Table A7 show that tuning 72 provides only limited improvement in performance. Considering the trade-off between gains and the additional effort for hyperparameter tuning, we recommend using the same learning rate for both modules.
R EMPIRICAL DETAILS
This section provides detailed empirical setups used for the previous experiments.
24

---

Published as a conference paper at ICLR 2026
LoRA
LORA LoRA-SAM
0.22
LoRA-SAM Bi-LoRA
Bi-LORA
Thining Loss
0.20
Evalsation
0.18
0.16
2
0.14
0
200
400
600
800
1000
1200
200
400
600
800
1000
Step
Step
(a) Training Loss
(b) Evaluation Loss
100
90
Training Accuracy (%)
88
86
Evaluation
84
LoRA
LoRA
LoRA-SAM
LORA-SAM
Bi-LORA
8
10
Epoch
Epoch
(c) Training Accuracy
(d) Evaluation Accuracy
10
Generalization Gop (%)
LoRA
LoRA-SAM
Bi-LORA
2
Epoch
(e) Generalization Gap
Figure A4: Training and evaluation loss, accuracy, and generalization gap of T5-base fine-tuned the MRPC dataset.
Table A7: CoLA accuracy under different learning rate of the auxiliary module 7/2 settings (with ? 5x for the main LoRA module).
722
1 10
x 10 (main)
5 10
CoLA
60.2340.35
60.77+0.55
60.44
25

---

Published as a conference paper at ICLR 2026
We tune the neighborhood radius p over (0.005, 0.01, 0.05, 0.1, 0.2, 0.5) for LoRA-SAM and Bi-LoRA and search learning rates among {2e-5, 5e-5, le-4, 2e-4) for Full FT and [5e-5, le-4, 2c-4, 5e-4, 1e-3) for LoRA and its variants.
R.1 EXPERIMENTS ON THE GLUE AND SUPERGLUE DATASETS
The hyperparameter configurations for all GLUE and SuperGLUE experiments (Sections I, 4.5 and 4.6) are detailed in Tables A8 and A9. As discussed in Section I, we set the neighborhood radius P 0.01 and 0.05 for LoRA-SAM and Bi-LoRA, respectively, in the GLUE experiments. For SuperGLUE, we use p values of 0.05 and 0.1 for LoRA-SAM and Bi-LoRA, respectively. For full fine-tuning, we the learning rate le-4 while keeping all other hyperparameters unchanged. We use the same learning rate of 5e-4 for both the primary and auxiliary LoRA modules in Bi-LoRA.
Table A8: Hyperparameter configurations for tine-tuning T5-base using LoRA- -asdd methods on the GLUE datasets.
Hyperparameter
MNLI
SST-2
COLA
QNLI
MRPC
Learning Rate
5c-4
Batch Size
32
Epochs
3
10
Max Sequence Length
256
LoRA Rank
LoRA Alpha
16
LR Scheduler
Cosine
Target Modules
All
Warmup Ratio
0.03
Evaluation Metric
Accuracy
Accuracy
Matthews Corr.
Accuracy
Accuracy
Table A9: Hyperparameter settings for fine-tuning T5-base using LoRA-based methods on the SuperGLUU datasets.
Hyperparameter
BoolQ
CB
COPA
RTE
WIC
Learning Rate
5e-4
Batch Size
Epochs
10
50
50
10
10
Max Sequence Length
256
LoRA Rank
8
LoRA Alpha
16
LR Scheduler
Cosine
Target Modules
All
Warmup Ratio
0.03
Evaluation Metric
Accuracy
R.2 EXPERIMENTS ON LLAMA MODELS
The hyperparameter settings for Llama 2-7B and Llama 3.1-8B are listed in Tables A10 and A11.
The neighborhood radius p is set to 0.01 and 0.05 for LoRA-SAM and Bi-LoRA. For full fine-tuning, set the micro-batch size to 4, 2, 2, and 8 for the respective four tasks, and the learning rate is set to 5e-5. We use the same learning rate of 5e-4 for both the primary and auxiliary LoRA modules in Bi-LoRA.
26

---

Published as a conference paper at ICLR 2026
Table A10: Hyperparameter configurations for fine-tuning Llama 2-7B Math, Code, and Chat tasks using LoRA-based methods. "Math", "Code" and "Chat" correspond to mathematical reasoning, code generation, and dialogue generation tasks, respectively, described in Section 4.2.
Hyperparameter
Math
Code
Chat
Learning rate
5c-4
Batch Size
32
Micro-Batch size
4
Epochs
Max Sequence Length
1024
LoRA Rank
LoRA Dropout
0.0
LoRA Alpha
Target Modules
All
LR Scheduler
Cosine
Warmup Ratio
0.03
Table A11: Hyperparameter configurations for fine-tuning Llama 3.1-8B on Cleaned Alpaca dataset.
Hyperparameter
Cleaned Alpaca
Learning rate
3e-4
Batch Size
128
Micro-Batch size
4
Epochs
Max Sequence Length
256
LoRA Rank
8
LoRA Dropout
0.0
LoRA Alpha
16
Target Modules
All
LR Scheduler
Cosine
Warmup Ratio
0.03
27

---

Published as a conference paper at ICLR 2026
R.3 EXPERIMENTS ON QWEN MODEL
We first search the learning rate {le 5, 5e 5, 1e 4.3e 4.5e 4) for LoRA, Bi-LoRA and LoRA-SAM, and perturbation radius E 0.05, 0.01}. For all random noise baselines we tune the variance p² {0.1, 0.01, 0.005, 0.001}.
We set the perturbation radius to = 0.1 for Bi-LoRA and = 0.05 for LoRA-SAM and other SAM variants, use variance p² 0.001 for all random-noise baselines. Other hyperparameters summarized in Table A12.
Table A12: Hyperparameter configurations for finettuning Qwen 2.5- 14B on Cleaned Alpaca dataset.
Hyperparameter
Cleaned Alpaca
Learning rate
5e-5
Batch Size
128
Micro-Batch size
Epochs
3
Max Sequence Length
256
LoRA Rank
8
LoRA Dropout
LoRA Alpha
16
Target Modules
All
LR Scheduler
Cosine
Warmup Ratio
0.03
R.4 EXPERIMENTS ON DIFFUSION MODELS
The finetuning dataset, 3D Icons', contains 23 training images, all featuring square icon.
The hyperparameter settings for Diffusion Models are listed in Table A13. We finetune the model for 500 steps with a constant learning rate of 2e-4, and set the for Bi-LoRA to 0.05. The LoRA rank set to 4, with auxiliary rank of 4 for Bi-LoRA. We use the same learning rate of 2e-4 for both the primary and auxiliary LoRA modules in Bi-LoRA. The instance prompt used for DreamBooth training is "a TOK icon, in the style of TOK", and the validation prompt used for generating the specific icons is formatted "a ToK icon of a <Instance>, in the style of ToK". We follow the scripts implemented by Hugging Face?.
Table A13: Hyperparameter settings for fine-tuning SDXL. using Dreambooth the 3D) Icons dataset
Hyperparameter
Value
Learning Rate
2e-4
Batch Size
Micro-Batch Size
Train Steps
500
Resolution
1024
Validation Epochs
25
LoRA Rank r
4
LoRA Dropout
0.0
LoRA Alpha a
8
Target Modules
WK. Wo, Wv, Wo Constant
LR Scheduler
Warmup Steps
/tttgps://uggiinggscee. co wt//ataaets///aaacooooooat htt.aasttttliitt//eeeesssms icon
Pntppa http:///t/tttto/////eea/oo www.c//.../tocooccoccooocooocoot www.:/ggggnntteeeeooeeoooooo trvsser /bbl /baa in examp les www.eemoothh//AAAeee..
28

---

Published as a conference paper at ICLR 2026
R.5 EFFICIENT SAM VARIANTS FOR LORA FINE-TUNING
To ensure fair comparison, we adopt the results of Flat-LoRA from Li et al. (2025), and follow the default setup of et al. (2024a) for LoRA-oBARMBARR with a = 0.25. And following Du et al. (2022a), (Liu et al., 2022b), LoRA-ESAM perturbs 50% of the parameters the top-50% sharpness-sensitive data, while LoRA-LooKSAN applies SAM perturbation every five steps. All other hyperparameters aligned with those used in prior experiments.
S TIME COMPARISONS ACROSS TASKS
In this section, we compare the per-step optimization time for LoRA, LoRA-SAM, and Bi-LoRA across range of benchmarks. 'The results are organized based on different datasets and tasks, including experiments on the GLUE and SuperGLUE datasets, as well as Llama experiments on domain-specific datasets. All experiments were conducted on single NVIDIA RTX 4090 GPU 24GB, using the same hyperparameters as in the previous experiments.
Table A A14; Optimization time (s) per step for T5-base on GLUE benchmarks. The average time across tasks also reported.
Avg.
Dataset
MNLI
SST2
CoLA
QNLI
MRPC
Time
Full FT
0.758
0.538
0.549
0.756
0.681
0.654
LoRA
0,458
0.310
0.273
0.508
0,442
0.398
LoRA-SAM
1.036
0.573
0.546
0.950
1.039
0.829
Bi-LoRA
0.476
0.325
0.305
0.513
0.508
0.425
Table A15: Optimization time (s) per step for T5-base SuperGLUE benchmarks. The average time across tasks is also reported.
Avg.
Dataset
BoolQ
CB
COPA
RTE
WIC
Time
Full FT
0.622
0.415
0.451
0.689
0.544
0.544
LoRA
0.276
0.276
0.253
0.263
0.286
0.271
LoRA-SAM
0.519
0.534
0.568
0.688
0.484
0.559
Bi-LoRA
0.304
0.285
0.278
0.276
0.262
0.281
Table A16: Optimization time (s) per step for training Llama2 on math (MetaMathQA),, code (Code- Feedback) and chat (WizardLMM) tasks.
Method
MetaMathQA
Code-Feedback
WizardLM
LoRA
2.81
5.10
5.13
LoRA-SAM
5.52
10.10
10.53
ROP
2.88
5.41
5.24
RWP full
3.21
6.02
5.88
RWP LoRA
2.97
5.56
5.38
LoRA-OBAR
2.90
5.41
5.21
LoRA-nBAR
2.89
5.41
5.24
FlAA-
3.42
5.92
5.78
LoRA-ESAM
4.20
8.13
7.35
LoRA-LooKSAM
3.47
6.49
6.29
Bi-LoRA
3.12
5.68
5.43
29

---

Published as a conference paper at ICLR 2026
Table A17: Optimization time (s) per step for training Llama 3-8B on the instruction foolooinn dataset Cleaned Apaca.
Method
Cleaned Alpaca
LoRA
8.93
LoRA-SAM
19.23
ROP
9.43
RWP full
12.35
RWP LoRA
10.10
LoRA-oBAR
9.35
LoRA-nBAR
9.35
Flat-LoRA
9.90
LoRA-ESAM
13,44
LoRA-LookSAM
11.49
Bi-LoRA
9.71
Table A18: Optimization time (s) per step for training Qwen 2.5-14B on the cntrruttionfollovviin dataset Cleaned Alpaca.
Method
Cleaned Alpaca
LoRA
15.47
LoRA-SAM
32.37
ROP
15.87
RWP full
21.28
RWP LoRA
16.67
LoRA-oBAR
15.87
LoRA-nBAR
15.87
Flat-LoRA
17.06
LORA-ESAM
21.79
LoRA-LooKSAM
19.07
Bi-LoRA
16.63
30

---

Published as a conference paper at ICLR 2026
x10
x10*
10
[ I- [cacall
Norm
Ratio
2
1000
1500
2000
2500
500
1000
1500
2000
Steps
Steps
(a) Norm of different components in the key of the first self-atiention layer in the fifth encoder block
(b) Norm ratio of the key of the first self-attention layer in the fifth encoder block
x10
x10
[0c.l
AArole [Gasale
10
B8 + Acali Faafel
Norm
Ratio
0
500
1000
1500
2000
2500
Steps
2000
Steps
(c) Norm of different components in the output projection second self-attention layer in the eighth encoder block
(d) Norm ratio of output projection in sec- ond self-attention layer in the cighth encoder block
-10
[
x10*
10
IAsel
Bc, Acali |Extalr
IcAcolr
Norm
Ratio
2500
Steps
500
1000
1500
2000
2500
Steps
(c) Norm of different components in the value of the first self-attention layer in the ninth encoder block
(f) Norm ratio of the value of the first self-attention layer in the ninth encoder block
Figure A5: Analyses of training statistics for LoRA-SAM (a) and (c) and (c): Frobenius norms of different terms. And (b), (d) and (f): ratio of the Frobenius norms of the first two terms (B + <B to that of the third term (C)EA) in Eqn. (5) during fine-tuning. Models are fine-tuned on CoLAA with T5 for 10 epochs.
T NORM AND RATIO ANALYSIS ACROSS LAYERS
In addition to Figures 2a and 2b, we extend the analysis of LoRA-SAM's training dynamics to additional self-attention layers across different encoder blocks to verify the consistency of our observations.
Figures A5a, A5c and A5e show the Frobenius norms of different terms in the key, output projec- tion and value of self-attention layers the 5th, 8th, and 9th encoder blocks, respectively. And Figures A5b, A5d and A5f present corresponding norm ratios. As observed, the norm of the third higher-order term remains several orders of magnitude smaller than the first two terms, confirming its negligible impact.
31

---

Published as a conference paper at ICLR 2026
U RESTRICTING PERTURBATION IN THE OPTIMIZATION SUBSPACE LIMITS LORA-SAM
We evaluate the performance of LoRA-SAM Llama 3.1-8B for the instruction following bench- mark, sweeping the rank over E {4,8, 16, 32, 64). Table A19 shows that while the average score rises monotonically with rank, the gain per doubling quickly reduces. The upward trend confirms that restricting perturbation in the optimization subspace does limit LoRA-SAM, and that increasing the rank partially mitigates it. Ideally, if LoRA-SAM becomes full parameter SAM, and the limitation would disappear. Our results show that even at r = 64, LoRA-SAM still lags behind our Bi-LoRA with primary rank = 8, auxiliary rank = 8, whose wider exploratory subspace is achieved with decoupling the (auxiliary) perturbation and (primary) optimization subspace. Moreover, LoRASAM's SAM-style perturbation doubles computation, whereas Bi-LoRA keeps the cost close to vanilla LoRA.
ma n) for each weight matrix W E R
Table A19: Instruction-tuning results fine-tuning Llama 3.1 3.1-8B using LoRA-SAM with varying rank.
Rank
MMLU
DROP
HEval
BBH
65,50
50.93
40.85
43,14
50.11
66.05
51.92
40.85
42.91
50.43
64.45
50.93
44.51
42.43
50.58
64
65.81
50.72
43.90
42.37
50.70
V THEORETICAL COMPARISON WITH OTHER EFFICIENT SAM VARIANTS DESIGNED FOR LORA FINE-TUNING
We provide a brief theoretical comparison showing the difference between Bi-LoRA and two efficient SAM variants, LoRA-o/nBAR and Flat-LoRA.
Starting from the common SAM objective miny |a L(W + +)., all three methods approxi- mate this minimax objective under a low-rank constraint, yet they differ in how the approximation is carried out. LoRAo/BBAR utitlizes SAM's implicit regularization, it replace the inner maximiza- tion with an explicit balanceness penalty, leading to minn,, L (Wo + BA) + p B|| AAT] (take oBAR and [| 2 ||A||] for example) with a looser O (p²) approximation. Flat-LoRA and Bi-LoRA both target LoRA-SAM's restricted-subspacee issue, i.e., the perturbation z BB (VwL)+ + (V A A is confined to Col(B) and Row(A). Yet they pursue flatness dif- ferently. Flat-LoRA replaces the inner maximisation with random weight perturbation (RWP) expectation, optimising min 1E~, (W, + B₁ A1 + +e,, thus encouraging expected flat- ness rather than worst-case sharpness. Bi-LoRA introduces a gradient-driven auxiliary adapter ( B2, A2 ) so that ming, A, | AgllEp C (Wo + B1 + B2 allowing a more targeted explo- ration of the sharp regions through learnable perturbations. From the perspective of gradient- norm regularization, Bi-LoRA essentially replaces SAM's f-rnrrm constraint with a rank nuclear norm constraint, thereby inheriting similar theoretical guarantees on generalization and convergence.
32


Reviews:
[{"review_id": "JhySxAQobA", "reviewer": "Reviewer_b4TB", "summary": "The paper introduces a new method for LoRA training. In particular, the paper focuses on incorporating Sharpness-Aware Minimization (SAM) into LoRA training to improve the generalization performance of LoRA. The paper notes that direct naive application of SAM to LoRA limits the performance and incurs overhead. As such, the paper proposes to introduce independent perturbation via an auxiliary LoRA module, decoupling the sharpness optimization and task adaptation. As a result, the proposed method, Bi-LoRA, significantly reduces the training cost and prevents the perturbations from collapsing into the restricted optimization subspace. The experiemntal results demonstrate the strong performance of Bi-LoRA.", "questions": "- Could authors provide an intuitive explanation as to how the Bi-LoRA is able to give appropriate aligned perturbations, despited the fact that optimization and perturbation are decoupled?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The proposed idea of decoupling the perturbation from the optimization has clear motivations.\n- The proposed idea is simple and effective.\n- The proposed method does not bring significant overhead and yet brings performance improvement.", "weaknesses": "- The paper does not compare against other LoRA variants, such as PiSSA and DoRA on the entire benchmark.\n- The paper notes that Bi-LoRA shares similar perspective with WSAM (Yue et al., 2023) that views SAM as regularization. However, the paper does not experimentally compare against WSAM.\n- The paper does not provide theoretical justification as to how Bi-LoRA achieves better generalization compared to LoRA and LoRA-SAM.\n- Bi-LoRA introduces additional hyperparameter that seems to need to be tuned for each dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lA6Mmet0SH", "reviewer": "Reviewer_bc19", "summary": "The paper introduces Bi-LoRA, a dual-adapter fine-tuning scheme in which a primary LoRA module is optimized by gradient descent for task adaptation while an auxiliary LoRA module is optimized by gradient ascent to approximate SAM-style perturbations in a single forward/backward pass. The decoupled design aims to mitigate the perturbation-subspace restriction observed when applying SAM directly to LoRA (LoRA-SAM) and to promote flatter minima in the full parameter space with minimal extra memory and near-LoRA time cost. The authors provide analytical support (e.g., alignment of the auxiliary ascent with the full gradient and a connection to a low-rank gradient-norm/Ky Fan regularizer under ideal inner maximization) and report consistent empirical improvements over LoRA and efficient SAM variants across LLaMA-2/3.1, Qwen-2.5-14B, T5-base (GLUE/SuperGLUE), and SDXL DreamBooth. At inference, the auxiliary adapter is discarded and only the primary LoRA is merged into the base model.", "questions": "1) Can you report a dual-LoRA (both descent) control with the same rank and clipping as Bi-LoRA, to isolate the effect of adversarial ascent?\n2) How sensitive are results to ρ, (r1, r2), and the ascent/descent step sizes, and is there a regime where multiple inner ascent steps improve performance at acceptable compute cost?\n3) Did you evaluate per-layer clipping (vs. global) and can you provide compute-parity details (hardware, precision, gradient checkpointing) to ensure fair efficiency comparisons across baselines?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) The method is simple and practical: a bi-directional LoRA scheme that achieves SAM-like sharpness-aware training in a single forward/backward pass with near-LoRA compute and modest memory overhead.\n2) The paper offers clear analytical insights, including a derivation of LoRA-SAM’s perturbation subspace limitation and supporting theory for the auxiliary ascent direction (e.g., gradient alignment and a Ky Fan–style regularization view).\n3) Empirical results are broad and consistent across LLMs and diffusion models, showing gains over LoRA and efficient SAM baselines, compatibility with other LoRA variants, and robustness under quantization.", "weaknesses": "1) The causal source of gains is under-isolated: without a “dual-LoRA (both descent)” control, it remains uncertain how much improvement comes from adversarial ascent versus added optimization degrees of freedom during training.\n2) Theoretical grounding for the practical single-step inner ascent is limited; results rely on an idealized inner maximization, leaving a gap between the theory and the implemented algorithm.\n3) Ablations and baselines are incomplete: the choice of global vs. per-layer clipping, sensitivity to ρ/rank/step sizes, and comparisons to broader single-pass SAM approximations / LoRA variants are not fully explored.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bDpjqXzKsL", "reviewer": "Reviewer_6yuC", "summary": "This paper proposes Bi-LoRA, introducing a *bi-directional* (dual-module) structure for integrating Sharpness-Aware Minimization (SAM) into Low-Rank Adaptation (LoRA). The method adds an auxiliary LoRA module that performs gradient ascent to model adversarial perturbations, while the primary LoRA module performs gradient descent for task adaptation.\n\nThis design allows both updates to be computed simultaneously in one forward/backward pass, avoiding the doubled training cost of standard SAM, while also decoupling sharpness optimization from task adaptation.\n\nEmpirical results on a broad range of tasks (LLMs like LLaMA-2/3.1, Qwen-14B, and SDXL diffusion models) show that Bi-LoRA improves generalization performance over LoRA and LoRA-SAM with only minor additional memory and time cost.", "questions": "1. How does the choice of the neighborhood radius (ρ) and the auxiliary LoRA rank (r_2)? Is there a scaling rule or heuristic for choosing these jointly?\n2. Why you apply the normalization only when the total Frobenius norm over $N$ auxiliary LoRA modules instead of do normalization for each auxiliary LoRA modules?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. **Clear Motivation.** The paper clearly identifies the inefficiency and subspace limitation of LoRA-SAM and logically motivates the need for decoupling SAM’s sharpness optimization from LoRA’s adaptation.\n2. **Elegant Design.** The proposed bi-directional scheme (gradient descent for main LoRA, ascent for auxiliary LoRA) is simple yet conceptually neat. It transforms SAM’s two sequential steps into a single parallel step without additional training time.\n3. **Theoretical Analysis.** The paper provides analytical support (Propositions 1–3), clarifying LoRA-SAM’s subspace restriction and proving that Bi-LoRA’s ascent direction aligns with SAM’s perturbation direction.\n4. **Strong Empirical Validation.** Comprehensive experiments across architectures (transformers, diffusion models) and tasks (math reasoning, code, chat, instruction following, and image generation) show consistent improvement. The results are credible and diverse.\n5. **Efficiency Retained.** The method achieves near-LoRA efficiency (≈1.09× training time, +0.7 GB memory) while roughly halving SAM’s cost, making it practical for large-scale LLM fine-tuning.\n6. **Compatibility with Existing Methods.** The paper demonstrates that Bi-LoRA can be seamlessly integrated with LoRA variants such as LoRA-GA, PiSSA, and DoRA, yielding further gains.", "weaknesses": "1. **Overclaim**.  Bi-LoRA has a higher loss in Figure 4(a) than the LoRA-SAM, thus, it is a overclaim that Bi-LoRA achieves a significantly flatter loss landscape in LoRA parameter space in Line 252-253. \n2. **Ablation Clarity**. The effect of each component (e.g., gradient clipping in Eqn. (10)) is not fully isolated. It’s unclear how much each contributes to final performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8dkfY6phzN", "reviewer": "Reviewer_K3nD", "summary": "To address the generalization issue of LoRA, this paper proposes a method called Bi-LoRA. Specifically, besides the standard LoRA module trained via gradient descent, the method introduces an auxiliary LoRA module trained via gradient ascent. By training these two modules in parallel, the approach achieves an effect analogous to sharpness-aware minimization (SAM). Extensive experiments demonstrate that the proposed method attains superior or comparable performance across various settings.", "questions": "1. In Figure 3, the convergence curve of the auxiliary module appears to converge rapidly after the primary module has already converged. Providing a reasonable explanation for this phenomenon would help deepen the understanding of the role and effect of the auxiliary module.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The proposed method is simple and easy to implement, while effectively reducing computational overhead compared to traditional SAM.\n\n2. The experiments are comprehensive, covering both image generation and language tasks.\n\n3. The paper is clearly written and easy to follow.", "weaknesses": "1. The motivation of the paper is confusing. It is unclear why the computation of perturbations needs to consider a broader space. Moreover, the proposed method does not seem to alleviate this issue, as the perturbation space it explores is still constrained.\n2. Although the proposed method is theoretically shown in Proposition 3 to be equivalent to SAM under the corresponding assumptions, this equivalence does not hold in practice due to the conflict between its assumptions and the observed experimental behavior.\n3. Similarly, the argument presented in Proposition 2 is not convincing, as the update direction of the auxiliary module at iteration t is applied to the optimization of iteration t+1, making its alignment with the SAM perturbation direction at iteration t conceptually questionable.\n4. The improvement brought by the proposed method is quite limited; however, this is understandable.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)