Stable-LoRA: Stabilizing Feature Learning of Low-Rank Adaptation
Authors: Yize Wu
KE GAO
Ling Li
Yanjun Wu
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
parameter-efficient fine tuning
low-rank adaptation
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper proposes Stable-LoRA, a weight-shrinkage optimization strategy that enhances stability of LoRA feature learning.

Abstract: Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as $W=W_0+sBA$, where $W_0$ is the original frozen weight, $s$ is a scaling factor and $A$,$B$ are trainable low-rank matrices. Despite its robust empirical effectiveness, the theoretical foundations of LoRA remain insufficiently understood, particularly with respect to feature learning stability. In this paper, we first establish that, LoRA can, in principle, naturally achieve and sustain stable feature learning (i.e., be self-stabilized) under appropriate hyper-parameters and initializations of $A$ and $B$. However, we also uncover a fundamental limitation that the necessary non-zero initialization of $A$ compromises self-stability, leading to suboptimal performances. To address this challenge, we propose Stable-LoRA, a weight-shrinkage optimization strategy that dynamically enhances stability of LoRA feature learning. By progressively shrinking $A$ during the earliest training steps, Stable-LoRA is both theoretically and empirically validated to effectively eliminate instability of LoRA feature learning while preserving the benefits of the non-zero start. Experiments show that Stable-LoRA consistently outperforms other baselines across diverse models and tasks, with no additional memory usage and only negligible computation overheads. The code is available at https://github.com/Yize-Wu/Stable-LoRA.

Published as a conference paper at ICLR 2026
STABLE-LORA: STABILIZING FEATURE LEARNING OF LOW-RANK ADAPTATION
Yize Wu Ke Gao', Ling Li Yanjun Wuln
'Intelligent Software Research Center, Institute of Software, CAS, Beijing, China University of Chinese Academy of Sciences, Beijing. China (wuyize2021, gaoke, 1il liling, yan feitunnnnnnssaa.ineeee weeesiscss...... an
ABSTRACT
Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient method for fine-tuning Large Langauge Models. It updates the weight matrix as W Wn + sBA, where Wo the original frozen weight, s is a scaling factor and A.B are trainable low-rank matrices. Despite its robust empirical effectiveness, the theoretical foundations of LoRA remain insufficiently understood, particu- larly with respect feature learning stability. In this paper, first establish that, LoRA can, in principle, naturally achieve and sustain stable feature learning (i.e., be self-stabilized) under appropriate hyper-parameters and initializations of A and B. However, we also uncover a fundamental limitation that the necessary non-zero initialization of A compromises self-stability, leading suboptimal performances. To address this challenge. we propose Stable-LoRA, a weight-shrinkan optimiza- tion strategy that dynamically enhances stability of LoRA feature learning. By progressively shrinking A during the earliest training steps, Stable-LoRA is both theoretically and empirically validated to effectively eliminate instability of LoRA feature learning while preserving the benefits of the non-zero start. Experiments show that Stable-LoRA consistently outperforms other baselines across diverse models and tasks, with no additional memory usage and only negligible com- putation overheads. The code is available at https://gttuu..com/ Sooom/iiz--ii TizeeWW/Sttable LoRA.
INTRODUCTION
Low-Rank Adaptation (LoRA) (Hu et al. 2022) is an effective and widely adopted parameter- efficient method for fine-tuning Large Language Models (LLMs). Unlike full fine-tuning, which updates all model parameters, LoRA freezes the original weight matrix Wo and introduces two low- rank trainable matrices, A and B, with the weight matrix updated by the multiplication of A and B. Formally,
W Wo + sBA, Wo E Rm A x B Rm
where is scaling factor. By choosing << min(m, n), the number of trainable parameters is reduced from ma to (m + n)r, substantially lowering computation and memory overhead while retaining strong learning capacity.
The effectiveness of LoRA A has been demonstrated through massive experiments across various mod- els and tasks, and recent studies (Hayou et al., 2024b Zhang & Pilanci 2024 Kalajdzievski 2023) have also begun to theoretically explore the fine-tuning dynamics of LoRA. However, no prior work has established theoretical explanation for such robust effectiveness. In this paper, we first provide a theoretical analysis showing that, with appropriate syper-parameters and initializations of A and B, LoRA can naturally achieves stable feature learning with respect to model width n (informally, the learned features scale as ((n")). Furthermore, once this stability is achieved, it will be sus- tained throughout the entire training process. Such self-stabilizing property provides a theoretical foundation for the observed effectiveness and robustness of LoRA.
According to the analysis, the ideal initialization for ensuring self-stabilizatiio is to set both and B to zero However, this leads to practical issues of saddle-point halting (Zhang & Pilanci
Corresponning author.

---

Published as a conference paper at ICLR 2026
stable f- false
for each step f:
if not stable and Aule
Hr
Weights
+
B
A
Ag (1 A)
W' R
else: stable truc
A411 A: 79.s
B+4 <- B, 79's
Figure I: Illustration of Stable-LoRA. The weight-shrinkage operation is emphasized as patch to the gradient-descent procedure.
2024), information loss and gradient vanishing/explosion (He et al., 2013). The mostly-adopted and theoretically proven-effective (Hayou et al. (2024a) solution is to initialize only B to zero and non-zero. Nevertheless, we both theoretically and empirically demonstrate that such non-zero initialization Ao compromises stable feature learning and hence causes suboptimal performances, which motivates the design novel LoRA optimization strategies.
To address this problem, propose Stable-LoRA, weight-shrinkage strategy for LoRA opti- mization that dynamically enhances the stability of feature learning. We conclude from theoretical perspectives that the irttalization-induue instability is long-tern problem whereas others short-termed. Therefore, Stable-LoRA adopts the non-zero Ag for benefits and progressively shrinks A training proceeds. Specifically, shrinkage ratio À (0 < X 1) is applied to A in the earliest steps of training, updating A according to
A (1 A)A, -ngaa
(as shown in Figure [ This exponential decay diminishes the instability introduced by Ao while still preserving its advantages for early training. Shrinkage stops once the stability condition is sat- isfied--secifficllly when the average norm of A becomes no larger than that of B (see Section 4 We theoretically proved that sufficient shrinkage of A guarantees the prevention of potential insta- bility, thereby ensuring stable feature learning throughout training.
We evaluated Stable-LoRA across different model architectures and tasks, where uniformly out- performs AdamW and other baselines. Importantly, Stable-LoRA incurs no additional memory usage and introduces only negligible computational overhead proopertiss that are particularly im- portant in the resource-constrained scenarios where LoRA is most commonly applied.
PRELIMINARY
2.1
FEATURE LEARNING OF LORA
Consider training weight matrix W with input Z, such that the output Y WZ. In LoRA, the original weight is frozen as Wo and two low-rank trainable matrices A and B are introduced, so that the updated weight becomes W Wo sBA, where is scaling factor. Given learning rate y), the parameter updates at training step t are:
A A, ng'a B.+1 B₁ -ng's
where gA and JB are the optimizer-processed gradients. The change in output after these updates given by:
AY; s(A, -ng -n94)Z -
(1)
-sngg sn Bn...''2 snj 9'99'2
We particularly interested in the properties of AY, it serves the "learned feature" at step Specifically, AY, directly influences the inputs to downstream layers and ultimately the model's output, representing the contribution of LoRA updates the model.
2

---

Published as a conference paper at ICLR 2026
2.2 STABLE FEATURE LEARNING
As neural networks continue to scale, understanding their training dynamics with respect to param- eter growth becomes increasingly important (Hayou et al.] 2024b] Zhang Pilanci 2024 Hayou er al. 2019). Much of this analysis has focused on the regime of model width, since in most archi- tectures the parameter count dominated by width, while depth (i.e., the number of layers) plays comparatively smaller role. In this regime, a desirable property is that learned features remain "stable" as width increases-ther neither explode nor vanish numerically. Such stability crucial for ensuring meaningful representations full performance potential upon trained tasks.
be learned, thereby allowing the model to achieve its
In the context of LoRA, stable feature learning requires that the output update AY, does not scale positively or negatively with model width n, otherwise it would explode or vanish as n increases. Formally, this requirement
be expressed AY, ((n") = (11).
Definition 1. (LoRA stable feature learning) LoRA feature learning stable, Ay @(1) for all training steps t.
Note that Definition T is slightly different from similar concepts in prior works (Hayou et al. 2024b
Zhang & Pilanci 2024). It does not require intermediate representations (e.g. A,2) to individually scale as (1), but only constrains the final output update. This relaxation is motivated by practical considerations: for an actual (finite) n, the scale of intermediate representations can compensate for each other through multiplicative interactions and yield an overall stable output. For example, it is acceptable for components of AY, scale as (n)) and V O(n -)n, as long as AY, UV = (1) is ensured.
2.3 F-FNCCTION
For convenience of notation, introduce ry-function to characterize the scaling behavior of vari- ables with respect to the model width n. It is defined as follows: for real-valued scalar variable v, we have V = (n?le)) For a k-dimensional tensor variable = (vo. Uk-1), 1). we define y| max v, 0 s k)), which captures the dominant scaling behavior among its components.
By definition, y-function obeys the following properties under clement-wise operations:
Multiplication: For two real-valued variables and v', >/v x v'] > > +
Addition: For two real-valued variables v and v', 7(v + v²] = max((lu),,71'))
With this notation, the condition for stable feature learning can be succinctly expressed as >|YY]
2.4 OPTIMIZED GRADIENT
Modern optimizers such as Adam and AdamW (Kingma & Ba, 2014) are generally preferred over Stochastic Gradient Descent (SGD) in fine-tuning scenarios. These optimizers typically normal- ize gradients through momentum mechanisms (e.g., exponential moving averages in Adam), which effectively prevents the entries of gradients from becoming excessively small or large. In the fol- lowing analysis, we assume that the normalized gradients have all entries to be O(1), condition theoretically justified by the internal dynamics of such optimizers and commonly observed in prac- tice (Hayou et al. 2024b). In the context of LoRA, this assumption applies individually optimized gradients of each low-rank matrices, i.e., 9As 9B (()).
3 LORA IS SELF-STABILIZED
In this section, we present theoretical analysis of LoRA fine-tuning dynamics, showing that LoRA self--tabillied with potential appropriate choices of paraeetersseters and initializations Ag and Bn-
Recall from Definition T and Equation II that stable feature learning requires all 3 components of AY to be ((1). Formally, using the multiplication property of y-ffnccion and the results that
3

---

Published as a conference paper at ICLR 2026
yl94] = [996=] (as established in Section 2.4 we have the following contraints:
> + (6) - o()) > + = (dg ((1))
+ + =0 (1))
(2)
Readers may notice that it is sufficient for AY, = (1) if just one component is O(1) and the others
are o(1. As [611] 2 7/6a] and 7[02] 2 7|6s] always hold (later explained in Section 3.1 it suffices to only justify why 81 and 02 are restricted to be (
Assume that 61 o(1). To maintain AY, @(1), we must have 62 (1), implying that the output update is dominated by 82. This situation corresponds to fixing the matrix B and only training A (gs matrices. The same argument applies if 62 = o(1). Therefore, for effective and balanced learning,
in Equation (T)). which clearly suboptimal compared training with both
both 61 and 82 must scale as (1))
Now we focus on the value of 7/44,)), [B.] and >[94 Z]. which ultimately correlate to the choices of s and 17.
3.1 VALUE OF 21A AND (B:).
We begin by stating an assumption on the value of v19' Z] and some clarification of its soundness. Assumption 1. With optimized gradient g'a E TR"x and input E IR"** we have >l9', Z| = 1.
Consider an extremely simplified optimizer, which normalizes each entry of the gradient to its sign, i.e.,
aLL
9 = sign(
0A
where 0Le denotes the raw (non-optimized) gradient of A at step t. By the chain rule, we have dA
aL, =B] dY, Z,
0A
where dY, is the gradient of the output Y. Define
S = =B=
so that the gradient becomes
θL. 5t x 2 = (St
0A
Therefore we have
OL, 0A
9'A sign(
sign(S* x 2) sign(S") sign(2)
Hence,
027 ("i sign( [)) (sign (Z)" 7) s)))
Since Z always holds and 5 (1)) if is a stable gradient, we conclude that g Z (n)), the same as > 1.
As more sophisticated optimizers generally preserve the sign of gradient (Yang et al. 2013). this assumption is well justified and serves as a foundation for our subsequent analysis.
Equation
2 then refined as:
[[s] + + 0
7[s] + + +1-0 0
(3)
7[s]+ + 2

---

Published as a conference paper at ICLR 2026
Next, analysis the value of >{A,2] 2] and y/Bi] by induction. Recall from Section addition property of y-function, we have
2.1 and the
>|A4Z] - max(m{A,- 7 [ max (>) [B-- 7 (1)) = [B,] 2 [17
+ + = M|, 2 > +
(4)
which immediately implies that > 2 >|0>] and || 2 ? [63] (used as a conclusion above). is quite intuitive that d3 is less significant than 51 and 62, as 03 is quadratic in the typically small learning rate 77; indeed, this term often even neglected some prior analysis (c.g.. Yen et al.
2025
3.2
IMPACT Ao AND Bo-
Based on the induction relations of Equation pairs:
4 we have the following two equivalent condition
||42 >/n] {Aa S vinl +1 [B.]] = 7vi ?Bal S 7[1]
(5)
which indicates that the y-values of the components are closely related to the initializations Ag and Bo.
Importantly, to satisfy Equation Equation (5) must both hold or neither: if only one of them an equation, then we will definitely have AA,4 * [BBB] + 1, which leads to an undesirable situation that 7(01] # 7/62). Hence, there are only two acceptable cases for Ao and By:
. Case 1. 2 < >11] + and [(Bo] - 7 [v]
. Case 2. 2 > >b1 + 1 and yBoo] > 7/1l
Among them, Case 2 is undesirable because the initial values dominate the training results, over- riding contributions from learned updates. In contrast, Case ensures that gradient-based updates govern the learning process. More importantly, if Case I is satisfied, the left-hand side conditions of Equation (5) are also satisfied, which turn ensures that all constraints in Equation are met simultaneously. This leads to unified expression of ōs:
[ = x52) = [63] >[8) + 2y(x) +
(6)
Therefore, with appropriate initializations satisfying Case 1, tuning & and v such that 7[0] 2n +
1 O results in >|YY;] and stable feature learning will be naturally (without any extra opera- tions) achieved and sustained throughout training, validating its empirical robust effectiveness.
Theorem 3.1. (Self-stability of LoRA) LoRA can naturally achieve and sustain stable feature learn- ing, if the hyper-parameters and n are tuned such that 7 + 2>/v]] + O, and the initializations Ao and By satisfy 7 [Ao S ((n] - and 7 [Bo] S >v1]-
4 STABLE-LORA
Case suggests that an ideal initialization strategy is to set both Ao and Bo to zero, ensuring that [A] B.] -o,, which guarantees the satisfaction of Case 1 with arbitrary n) However, while stable feature learning is a necessary condition for effective training, it is not sufficient on its own. With setting Bg 0 feasible, initializing Ao introduces two empirical issues: (1) the combination A 0 and B = 0 is saddle point with zero gradient, leading to halting of training; (2) the initial input to A 0, resulting in complete information loss for learning and possible gradient vanishing/cxplosion. The common solution is to set Bo 0 and sample the entries of A from a distribution with o² = n -1 which addresses both issues and has been theoretically HHyoou al. 2024a) and empirically (He et al. 2013) shown to be beneficial.
From the perspective of feature learning, this initialization yields [[Bb] --0 < { for arbitrary n. According to Theorem 3.1 we must also have >|A,Z] S >/n]
ensure stability. However, due to the non zoro entries in An, this condition imposes a lower bound on ry: it cannot be arbitrarily small, but must be sufficiently large absorb the magnitude of An In practical scenario where
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Stable-LoRA
Input:
Learning rate 7,, shrink rate A, weight decay rate w, initializations Ao. Bo Shrink if the stable condition not satisfied sluble +- false
for training step t do
if not stable and | > |BIE then
A, = (1 A)
else
slable +- true
end if
Update parameters with optimized gradients and weight decay
A.+1 A, Vig's mw Bi+1 = Be 19988 mwB.
end for
learning rate is usually small, this condition on is typically violated (as supported by empirical results in Section [.T]. Moreover, this issue cannot be resolved solely by altering initialization or tuning hyper-parameters: adjusting Ao cannot reduce AgZ uniformly across the batch, since Z varies while Ao is fixed; tuning is also ineffective since is not involved in the condition. Consequently, these limitations motivate the development of novel optimization strategies.
We discover that the problem of instable feature learning is fundamentally different from other issues: it is long-term problem whereas others are short-termed. While the saddle-point and issues arise only at the beginning of training, they naturally resolve as training proceeds with parameters moving away from the saddle point and meaningful signals be- ing propagated to B. In contrast, feature- feature-learning instability occurs at the start if y|Ao > >/7l +1 and persists throughout training due to the induction in Equation 4 This leads to a key idea: in- stead of modifying Ao from the beginning (which would exacerbate the other two problems), we can gradually reduce its negative impact as the training proceeds. The solution would be optimal Ao can serve its etrly-stage positive role while its adverse effects diminish to desirable level over time.
Based on this, we propose Stable-LoRA, weight-shrinkage optimization strategy applied to matrix in earliest steps of training, to mitigate instability of LoRA feature learning. An overview of Stable-LoRA is shown in Figure [ and the detailed procedure is provided in Algorithm Specifi- cally, at an early step I, before parameter updates, A first shrinks as
A (1 A)A. - n9
where À (0 À < 1) is the shrinkage ratio, after which A continues to be updated with gi Shrinkage is applied at every step until a stable condition is met: achieves a comparable average norm scale B, i.e. ||
|| (with in denominators canceled). The design of stable condition is motivated by the terms d1 sng' A, Z and 82 = sn Z, which reach similar scales when A, and B, do. Since 7(22] = is always ensured, satisfying this condition also guarantees 7 = 0. The practical effectiveness of this stable condition has been demonstrated in Section 5.3
Stable-LoRA can robustly prevent instable feature learning for any learning rate ry: after N steps of shrinking, we have
Av (1 A)AN-1 - 199 N-1
(1 A -11 -1jnn -
(1 ^) Ao 11 (7(11 N)
(1 - -1 ())
= (1 A Ao 17 N n

---

Published as a conference paper at ICLR 2026
0.25
0.8
0.20
r103
0.15
Avb Nom (10
0.15
0.10
0.10
0.06
0.001
0.005A
0.007 A
0.005A
0.201
0.005
0.001 8
0.005
0.0
200
400
600
100
200
400
800
1000
Training steps
Training steps
(a) aproy
(b) Uproj
Figure 2: Averaged norm of A and B on the 0.5B model and HellaSwag. While the scale of B smaller, it grows more rapidly than (gel Iga), indicating practical violation of feature learning stability.
With >((1 1141 S 7/1] 0 for all & E Zt, we have AZZ S 2 = Therefore,
=ax( A| yA4 7/1 7 --1 2], +
= -max(Ny/1 >| A Z) +
With N and/or À sufficiently large, Ny(1 - \a + 7 [Ao2] will finally drop below 7/v] + 1, and hence stable feature learning is achieved from step N onward and persists throughout the rest of training.
Stable-LoRA orthogonal to existing optimization strategies such as gradient optimization (like AdamW) and weight decay, as formally described in Algorithm More importantly, Stable- LoRA incurs negligible overheads during training: it requires no additional memory usage, since the shrinkage operation can be done in-place (and should be, for acceleration) with the pre-shrinkage value no longer used after that. This property particularly crucial since LoRA is commonly used in cemmory-onssrainge scenarios. Computation overhead arises from computing the Frobenius norms ||-||F and performing the shrinkage, which is negligible (as shown in Table because (1) the oper- ations are relatively light-weight compared to others, and (2) they applied only during the initial steps.
5 EXPERIMENTS
We evaluated Stable-LoRA and other baselines under these experimental settings:
Datasets. The tasks involve two fine-tuning scenarios: multi-choice question answering (QA) and thain-of-thought (CoT) reasoning. The QA datasets include HellaSwag (Zellers et al. 2019). 1. So- ciallQa (Sap et al. 2019), OpenbookQA (Mihaylov et al.) 2018), ARC-Easy and ARC-Challenge (Clark et al. 201 8). For CoT reasoning, we focus mathematical tasks where models are trained on MetaMathQA Yu et al. (203) and evaluated on GSM8K (Cobbe et al. 2021). The exact match accuracy is used evaluation metric for all tasks.
The experiments are conducted on the 0.5B and 1.5B models from Qwen-2 YYng et al, 2024 and IB and 3B models from LLaMA-3.2 (Dubey et al. 2024).
Models.
Baselines. Besides AdamW, we compared our proposed method against several other baselines, including stable feature learning methods of LoRA+ (Hayou et al. 2024b) and Riemann Precondi- tioned Optimization (Zhang & Pilanci 2024), and a transformation. invariant optimizer LoRA-RITE
Yen et al. 2025).
LoRA+ claims achieve stable feature learning by setting learning rate of B

---

Published as a conference paper at ICLR 2026
Model
Method
HellaS.
SIQA
ObQA
ARC-E
ARC-C
Avg.
A
65.65
67.04
63.00
66.92
47.10
61.94
LoRA+
64.75
67.91
64.40
67.80
47.87
62.55
0.5B
Riemann
60.79
66.94
65.00
67.30
47.35
61.48
LORA-RITE
62.73
66.89
66.00
67.13
46.50
61.85
Stable-LoRA
66.73
68.27
67.00
69.15
48.89
64.01
AdamW
83.76
71.70
73.20
76.81
54.01
71.90
LoRA+
83.39
71.19
73.20
75.88
53.16
71.36
IB
Riemann
77.41
70.62
71.20
73.99
52.13
69.07
LoRA-RITE
82.38
71.60
71.00
76.73
53.50
71.04
Stable-LoRA
84.41
72.26
73.80
77.36
54.78
72.52
AdamW
88.28
77.33
83.60
86.20
70.14
81.11
LoRA+ 87.94
77.38
83.20
85.94
70.65
81.02
1.5B
Riemann
86.46
76.87
82.40
85.82
69.71
80.25
LoRA-RITE
87.97
77.23
83.00
86.36
70.82
81.08
Stable-LoRA
88.52
77.64
84.00
86.99
72.61
81.95
AdamW
93.39
79.89
83.20
88.38
72.78
83.53
LoRA+
93.54
79.94
83.00
88.26
72.35
83.42
3B
Riemann
92.61
79.89
82.60
88.01
71.42
82.91
LORA-RITE
93.21
80.25
83.40
88.26
71.50
83.32
Stable-LoRA
93.59
80.25
84.00
88.68
73.63
84.03
Table 1: Task accuracies of models on datasets of question-anssen answering tasks.
larger than A. Riemann Preconditioned Optimization adopts matrix preconditioning 94 and gø- LoRA-RITE achieves invariant transformation equilibration of LoRA using unmagnified gradients.
Configurations. Unless otherwise stated, use the following training configurations. We train aproj and Uproy of the attention block with rank r 8. We conducted careful tuning of hyper-parametert by searching 7) from 5e-5 to 8e-4 and from 2.0 64.0. Each value of Ag sampled from [-1/n, 1/n] following (He et al.) 2015) and Bu = 0. For LoRA+, we set vn 4rja following its recommendation for decoder-only models. We search the shrinkage tio over À E [0.0005, 0.001, 0.002, 0.005] and report the best result (more detailed results about each value are Table 8). The algorithm of AdamW is adopted as the gradient optimizer for Stable-LoRA. Each reported accuracy is the best of 3 random runs. More detailed configurations are specified in corresponding sections Table
5.1 DYNAMIC ANALYSIS
Figure provides dynamic analysis the training process by visualizing the change Frobenius norms ||e of matrices A and B. The results indicate that the problem of o[A[A T|nl + does occur practice:
During LoRA training. ||B||r grows rapidly from small values, meaning that gn is large while the value of B is small. Meanwhile, || remains at a steady but larger value, meaning that 94 small and A large. Recall that 8 A,Z and 62 sn Z. the larger gø and A makes 8₃ dominate over 82 (1661] > Furthermore, ||A|| Fr never drops below its initial value, indicating that the negative influence of initialization persists throughout training.
(g(])) (82]). This consistent with our theoretical analysis that {[A > >(n + 1.
Stable-LoRA effectively mitigates this negative effect and hence promotes stable feature learning. Moreover, the value of || in early training stages remains unaffected when Stable-LoRA de- clines |||||r_ confirming that our method preserves the benefit of non-zero Ag.
8

---

Published as a conference paper at ICLR 2026
1B
3B
Method
1000
2000
5000
1000
2000
5000
A Stable-LoRA
23.88
27.14
31.08
51.48
53.45
58.83
24.56
27.75
31.84
52.16
54.74
59.44
Table 2: Task accuracies of models on (math) CoT reasoning tasks for different training steps.
Target
Model
qF
glvo
glvogud
AdamW W
Stable-LoRA
AdamW W
Stable-LoRA
AdamW
Stable-LoRA
0.5B
61.54
62.46
63.03
63.48
64.35
64.75
IB
71.90
72.80
73.76
73.96
75.04
75.44
1.5B
80.27
80.75
80,94
81.15
81.71
81.82
3B
83.79
84.13
84.54
84.93
85.31
85.60
Table 3: Task accuracies of models on the combined dataset of question-answering tasks.
5.2
MAIN RESULTS
5.2. RESULTS OF MULTI-CHOICE QUESTION ANSWERING
presen[s the results the QA datasets. As demonstrated, Stable-LoRA consistently outper- forms other methods across models and datasets, achieving up to a 4% increase in accuracy. While other baselines may boost performances specific tasks models, the improvements are incon- sistent. In contrast, Stable-LoRA offers not only improved accuracies but also greater robustness across tasks and models.
5.2.2 RESULTS OF CHAIN-OF-THOUGHE REASONING
Chain-of-Thhught (CoT) is widely used approach in tasks that require multi-step reasoning. We trained models to learn to reason in CoT format spontaneously without explicit prompting (i.e., giving the question directly without instructing the model "think step by step"). We used math- reasoning datasets as representative reasoning tasks. The results in Table 2 show that Stable-LoRA again outperforms AdamW, maintaining performance advantages CoT tasks.
5.2.3 ABLATIONS
To evaluate the generalization capability of our method, we perform ablation studies along two di- mensions: dataset formulation and target modules. For the dataset formulation study, construct a unified QA dataset by combining all datasets (see details Section C.I). We also vary the target matrix components of the model to which LoRA is applied. As shown in Table Stable-LoRA sistently improves task accuracy across different LoRA configuration settings. Additional ablation results provided in Section [33
5.3 MEMORY AND COMPUTATIONAL. COSTS.
Stable-LoRA introduces no additional memory usage compared to LoRA, the shrinkage operation conducted in-place. Table 5 compares the training time of Stable-LoRA with baseline methods. The results show that StableLLRRA incurs only marginal (0.6%) increase in training time, indi- cating that the scalar-matris multiplication involved in shrinkage is far less costly than gradient computation and parameter updates.

---

Published as a conference paper at ICLR 2026
Method
0.5B
1B
1.5B
Stable-LoRA
62.46
72.80
80.75
84.13
Stable-LoRA w/o the stable condition
62.52
72.72
80.61
84.10
Table 4: Task accuracies on the combined QA dataset from Stable-LoRA with/without the stable condition.
Method
AdamW
LoRA+
Riemann
LoRA-RITE
Stable-LoRA
Time(s)
217.4
217.4
235.5 +8.3%
317.3 +46.0%
218.8
+%
+0.0%
+0.6%
Table Comparison of training time of different methods on 0.5B model and HellaSwag.
5.4 JUSTIFICATION FOR THE STABLE CONDITION.
To justify the stable condition, we conducted experiments where the stable condition is removed and A shrinks whenever |||||/// > Bl|p//m comprees task accuracies with and without stop- ping at the stable condition, and the results show that further shrinkage beyond the stable condition does not lead to noticeable impact on performance, which aligns with our previous analysis.
RELATED WORKS
6.1 STABLE FEATURE LEARNING.
There are existing works established upon initialization schemes for stable feature learning, from the perspective of width and depth. In scenario of width, (Glorot & Bengio, 2010) proposed Xavier initialization to stabilize the variance of activations, and (He et 2015)) improved it for non-linear activation functions (like leaky ReLu). (Yang & Hu 2021) introduced μP parameterization for ensuring feature learning in the safinite-width scenario. Related literature about the depth limit includes (Hayou 2022 Schoenholz al.] (2016) etc.. Stable-LoRA is specifically targeted at width scenario, so it is theoretically discussed upon the width-related initialization method of (He et al. 2015) and shows empirically strong results.
6.2 STABLE FEATURE LEARNING FOR LORA.
The concept of LoRA stable feature learning originates from LoRA+ (Hayou et al. 2024b). which suggests choosing a larger learning rate for than A (1/ > 74)) (Zhang & Pilanci 2024) proposed matrix-preconditioned optimizer to achieve stabilization. Beyond width-related stability, (Kala-) jdzievski 2023) studies the stability with respect to rank r and recommends using scaling factor a/ T rather than a/r. Our definition of stable feature learning is slightly different from the mentioned work, where we do not demand intermediate states to be ((n²)) due to practical considerations.
7 CONCLUSION
This paper addresses the challenge of stabilizing feature learning in Low-Rank Adaptation (LoRA). We first establish that, under appropriate hyper-parameters and initializations of A and B, LoRA can principle be self-stabilized during the training process regardless of model width, which provides theoretical foundation for the robustness and effectiveness of LoRA. However, we further reveal that the non-zero Ay compromises this staf-stability which leads to performance degradation. shrghk-shrinkage strategy that mitigates instability caused by An while preserving its benefits. Stable-LoRA shows superiority over various tasks and models, with additional memory usage and only marginal computation overhead.
Stable-LoRA is hence proposed
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENT
This work is partially supported by the NSF of China (under Grant 92364202), and Major Program of ISCAS (Grant No. ISCAS-2D-202402).
REFERENCES
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 1803::080005553 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110. 2021.
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llm.. Advances in neural information processing systems, 36: 10088-10115,, 2023.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad AI-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint arXiv:2407.21780 2024.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256922 JMLR Workshop and Conference Proceedings, 2010.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: low-rank adapters are secretly gradient com- pressors. In Proceedings of the 41st International Conference on Machine Learning, pp. 17554- 17571, 2024.
Soufiane Hayou. On the infinite-depth limit of finite-width neural networks. arXiv preprint arXiv:2210.000 10.00888 2022.
Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International conference on machine learning, pp. 2672-2680. PMLR, 2019.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. Advances in Neural Information Processing Systems, 37:1 1111111015517044 2024a.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In International Conference on Machine Learning, pp. 17783-17806. PMIR, 2024b.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2022.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LIm-adapters: An adapter family for parameter-efficient fine-tuning of large language models. In Proceedings of the 2023 Conference on Empirical Methods Natural Lan- guage Processing, pp. 5254-5276, 2023.
Qiushi Huang. Tom Ko. Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Con- ference on Learning Representations, 2025.
Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. arxiv preprint 232.:23312.03732.. 2023.
11

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Shih- Yang Liu, Chien-Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang. Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning. 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2381-2391, 2018.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social iqa: Common- sense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463 4473, 2019.
Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein, Deep information propagation. arXiv preprint arXiv:1611.01223 2016.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jin- gren Zhou, Jinze Bai, Jinzheng He, Junyang Lin, Kai Dang. Keming Lu, Keqin Chen, Kexin Yang. Mei Li, Mingfeng Xue, Na Ni, Pei Zhang, Peng Wang, Ru Peng, Rui Men, Ruize Gao, Runji Lin, Shijie Wang, Shuai Bai, Sinan Tan, Tianhang Zhu, Tianhao Li, Tianyu Liu, Wen- bin Ge, Xiaodong Deng, Xiaohuan Zhou, Xingzhang Ren, Xinyu Zhang, Xipin Wei, Xuancheng Ren, Xuejing Liu, Yang Fan, Yang Yao, Yichang Zhang, Yu Wan, Yunfei Chu, Yugiong Liu, Zeyu Cui, Zhenru Zhang, Zhifang Guo, and Zhihao Fan. Qwen2 technical report, 2024. URL wwt. https://arxiv: otgo abs 2207
Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-widhh neural networks. In International Conference on Machine Learning, pp. 11727-11731 PMLR, 2021.
Liu Yang, Steve Hanneke, and Jaime Carbonell. A theory of transfer learning with applications to active learning. Machine learning, 90(2):161-189, 2013.
Jui-Nan Yen, Si Si, Zhao Meng. Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhen- guo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2023.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can ma- chine really finish your sentence? In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791 -8000, 2019.
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation mod- cls. In Proceedings of the 41st International Conference Machine Learning, pp. 2024.
12

---

Published as a conference paper at ICLR 2026
A RELATED WORKS OF LORA VARIANTS.
Besides LoRA, there are some similar strategies of low-rank adapters proposed. DoRA (Liu et al, 2024) seperately updates the normalized matrices and its norm, leading to more similar behav- ior to Full Fine-Tuning. VeRA (Kopiczko et al. 2024) creates a random and frozen matrix and learns vector scalings of the columns, achieve extreme memory saving. HiRA (Huang al.
2025 element-wise product of W and BA to enlarge the rank of updates. QLoRA (Dettmers et al,
2023 reduces computation costs by quantizing pretrained weights down to smaller bits, and only set higher bits for trainable parameters, to save memory usage. Flora (Hao et al., (024) achieves overall high-rank updates by resampling low-rank projection matrices for each step and accumulating them periodically.
B DETAILS OF ALGORITHM
Algorithm [ shows the detailed procedure of Stablc-LoRA, where the orthogonality of our method gradient optimizers and weight decay is demonstrated.
Note that Stable-LoRA is conceptually different from weight decay from several perspectives:
Theoretical motivation and formulation. Weight decay is based on a Bayesian prior assuming that model weights follow Gaussian distribution centered at zero. It introduces an additional regularization term in the loss function, so the rate w is multiplied by y and then applied to the parameters.
A= A mw = (1 - -nw)
In contrast, Stable-LoRA directly shrinks the weights using À which is independent of the learning rate:
= XA = A)A
With y << in almost all cases, Stable-LoRA results in significantly faster decay com- pared to weight decay.
Scope of application. Weight decay is applied uniformly to all trainable parameters, and Stable-LoRA targets only at the matrix A, with the explicit purpose of reducing the influ- ence of An-
Application schedule. Weight decay is applied throughout the entire training process, while Stable-LoRA is only applied before the stable condition achieved.
C DETAILS OF EXPERIMENTS
C.1 DATASETS
Table the detailed information of datasets. All datasets include test sets but no validation sets; therefore apply weight decay to mitigate overfitting. For the QA datasets, we use the templated versions provided by (Hu et al. 2023).
The combined QA dataset includes 8K samples each from HellaSwag and SIQA, as well all available samples from ObQA, ARC-E, and ARC-C, resulting in approximately 24K samples in total.
C.2 HYPER-PARE PARAMETERS
The hyper- syper-parametert of experiments (except for specified ones) are listed in Table 7 To find the optimal combination, 77 and s are thoroughly searched across wide ranges. The weight decay ratio is set to 0.01 to prevent overfitting, and the dropout ratio is 0.0 setting a non-zero value of it cannot boost performance in our experiments.
We the training steps to 1,000 for each individual QA dataset and 3,000 for the combined QA dataset, ensuring that the effective number of training epochs per QA task remains consistent be- tween the separate and combined settings.
13

---

Published as a conference paper at ICLR 2026
Dataset
#Train
#Test
HellaSwag
39905
10042
SIQA
33410
1954
ObQA
4957
500
ARC-E
2251
2376
ARC-C
1119
1172
MetaMathQA
40000
1319
Table 6: Detailed information about datasets.
parameer
Value
Learning rate "
[5e-5, (1-8811)--4 2.0 to 64.0 (interval 1.0)
Scale factor 8
Target modules
Oprojs Pproj
Rank r
Weight decay ratio
0.01
Dropout ratio
0.0
Batch size
16
Learning rate scheduler
linear
Warm-up steps
100
AdamW 31,82
0.9, 0.999
Table 7: papper-prameters..
D ADDITIONAL RESULTS
D.1 RESULTS OF DIFFERENT Às
We show results of different Às on QA datasets in Table The results show that Stable-LoRA can robustly outperform AdamW with a variety of A, demonstrating the principle effectiveness of the method. À should be considered as newly-introduced hyper-parameter, which could and should be tuned for optimal performances. The results of X 0.005 is often suboptimal, suggesting that an excessively larger À could result in loss of information from previous training steps and hence downgraded performances.
D.2 RESULTS ON LARGER SCALE
Table 9 shows results Llama-3. 1-B with targets of gkvogud on the math-reasoning datasets, which verifies performance superiority of Stable-LoRA on the larger-scaled model.
D.3 RESULTS OF MORE ABLATIONS
Table 10 shows the results of ablation studies different models, target modules and datasets, where Stable-LoRA shows uniform superiority of performances across the involved tasks.
D.4 COMPARISON WITH LoRA VARIANTS
Table 11 reports the performance of Stable-LoRA and DoRA on the QA benchmarks. The results show that Stable-LoRA empirically outperforms DoRA across these tasks.
We note that Stable-LoRA cannot be directly applied to substantially different variants such as DoRA, since the variants fundamentally modifies the original LoRA parameterization (e.g.DoRA BA resulting in distinct training dynamics.
changes m [[
14

---

Published as a conference paper at ICLR 2026
Target
Method
0.5B
1B
1.5B
3B
A
61.54
71.90
80.27
83.79
Stable-LoRAx=o..eo
62.36
72.80
80.75
84.13
qu
Stable-LoRAA
0.000
62.13
72.37
80.55
84.08
Stable-LoRAA
62.05
71.92
80.44
83.71
StableLoRAxx=0000
62.46
72.29
80.50
83.85
AdamW
63.03
73.76
80.94
84.54
Stable-LoRAy 0.0005
63.21
73.96
81.09
84.80
glco
Stable-LoRA 000.001
63.15
73.54
81.12
84.85
Stable-LoRA x
63.48
73.84
81.15
84.93
Stable-LoRAx
63,09
73.78
80.89
84.53
AdamW
64.35
75.04
81.71
85.31
Stable-LoRAA 000000
64.37
75.42
81.71
85.60
gkvogud
Stable-LoRA A=
64.53
75.44
81.82
85.32
Stable-LoRA 000002
64.75
75.20
81.74
85.56
Stable-LoRAx-o.00 = 00.005
64.33
75.18
81.71
85.23
Table 8: Task accuracies of different Às the combined QA datascts.
Step
AdamW
StableLoRAAxa..00:
StableLLoRAj=o.00
Stable-LoRAx=..02
1000
68.61
69.75
70.05
69.60
2000
70.96
71.42
71.49
71.87
3000
71.72
74.37
73.39
72.02
Table Task accuracies of Llama-3.1-8B with targets of gkvogud on the math reasoning dataset.
E
LIMITATIONS
The effectiveness of Stable-LoRA has been validated two LLM fine-tuning scenarios. However, its performance in other settings-such as additional datasets cross-modal tasks (e.g., vision)- remains to be explored. We employ AdamW as the gradient optimizer, and compatibility with alternative optimizers has not yet been systematically evaluated. For initialization, we adopt the commonly used scheme Ag ~ [-1nn, 1/n] and B, = 0, rather than alternative strategies.
15

---

Published as a conference paper at ICLR 2026
Model
Target
Method
HellaS.
SIQA
ObQA
ARC-E
ARC-C
Avg.
AdamW
68.00
68.42
67.20
68.73
48.98
64.27
glvo
Stable-LoRA
68.98
68.83
67.80
69.57
49.83
65.00
0.5B
AdamW
68.76
69.14
67.60
69.49
48.04
64.61
gkvogud
Stable-LoRA
69.08
69.50
68.80
70.79
49.74
65.58
AdamW
85.09
73.44
74.60
77.2
55.20
73.11
glvo
Stable-LoRA
86.21
73.90
74.60
78.16
56.51
73.88
IB
AdamW
85.93
74.36
76.80
77.44
55.03
73.91
gkvogud
Stable-LoRA
86.72
74.72
77.40
78.58
57.17
74.92
AdamW
89.13
77.69
84.20
86.91
70.56
81.70
glvo
Stable-LoRA
89.32
78.05
85.00
87.63
72.35
82.47
1.5B
AdamW
89.72
78.66
84.80
87.12
71.25
82.31
gkvogud
Stable-LoRA
89.83
78.66
85.80
87.79
72.35
82.89
AdamW
94.02
80.45
83.60
88.43
72.44
83.79
gkvo
Stable-LoRA
94.23
80.76
85.00
88.89
74.06
84.59
3B
AdamW
94.41
80.81
84.80
88.51
72.61
84.23
qkvogud
Stable-LoRA
94.50
81.01
85.60
88.72
74.23
84.81
Table 10: Task accuracies of models and different targets on QA datasets. The top and bottom value are from Adam V and Stable-LoR. respectively.
Model
Method
HellaS.
SIQA
ObQA
ARC-E
ARC-C
Avg.
DoRA
82.47
71.49
73.60
76.98
53.84
71.68
IB
Stable-LoRA
84.41
72.26
73.80
77.36
54.78
72.52
DoRA
93.38
80.04
82.80
88.47 88.68
72.27
83.39
3B
Stable-LoRA
93.59
80.25
84.00
73.63
84.03
Table 11: Task accuracies of DoRA and Stable-LoRA 1B and 3B models, qu and QA datasets.
16


Reviews:
[{"review_id": "v3njdn2OIe", "reviewer": "Reviewer_33Pn", "summary": "LoRA is the method that is the most commonly use to finetune LLMs. However it does have some training challenges. This study looks at those problems and proposes a clear analysis, especially around the initialization of LoRA--and propose a novel solution. Empirical results are provided to support the claim.\n\n** Strength **\n- very important problem in the world of LLM and post-training especially\n- the experiments are quite sound to validate the study\n- the interpretation of LoRA is interesting albeit hard to grasp for readers not familiar with the field\n\n** Weakness **\n- enormous typo (e.g. one right in the title, adaption, or in the abstract, Langauge) which makes the entire script feel quite odd\n- could the author discuss a possible link with https://arxiv.org/pdf/2410.09692? who also checked at initialization and training dynamics?\n- lacking some geometric/visual intuition that would help reader better grasp the results\n- needs better scoping of limitation and future work", "questions": "Please see summary", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Please see summary", "weaknesses": "Please see summary", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "G59yKgcar8", "reviewer": "Reviewer_1UrF", "summary": "The paper investigates whether LoRA can achieve and sustain stable feature learning. By introducing a $\\gamma$-function as the main analytical tool, the authors prove that stable feature learning is attainable under specific hyperparameter settings and when $A = B = 0$. However, in practices, $A$ and $B$ cannot be set to 0 at the same time. To mitigate this issue, the paper proposes a progressive shrinkage strategy that gradually reduces $A$ during the early training stages. Extensive experiments suggest that the proposed Stable-LoRA method consistently outperforms baseline models.", "questions": "1. Line 47: Should the asymptotic complexity be $O(n)$ instead of $O(n^0)$,  if this is \"with respect to model width $n$\" as stated in line 46.\n2. Line 131: The definition of $r[v]$ by $v=\\Theta(n^{\\gamma[v]})$ is kind of informal. Written in this way, it implicitly invites a proof of existence.\n3. Line 132: $\\gamma[\\overrightarrow{v}]:=\\max(v_i, 0 < i < k)$. Should it be $0\\le i$?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper introduces the $\\gamma$-function as a novel analytical construct for understanding LoRA’s stability behavior.\n* Theoretical analysis yields an interpretable condition—$A=B=0$—under which stable feature learning can be achieved.\n* The proposed progressive-shrink mechanism is a practical solution to approximate the ideal condition $A=B=0$ and address the constratins that in practices, we cannot set both $A$ and $B$ to 0.", "weaknesses": "* Definition 1 lacks rigorous justification. While it aligns with prior empirical observations, it represents only one possible stability condition. The framework built upon it may have limited generalizability, which can only be supported by broader empirical studies.\n* The definition of the $\\gamma$-function appears mathematically infeasible. Although it seems inspired by logarithmic properties (e.g., \n$\\log(x) + \\log(y) = \\log(x \\times y)$ and $\\log(x + y)$ is dominated by $\\max(\\log(x), \\log(y)$). The equation $\\gamma[v + v'] = \\max(\\gamma[v], \\gamma[v'])$ may not always hold, even with a hidden constant in the $\\Theta(\\cdot)$. Thus, the $\\gamma$-function may be valid for qualitative reasoning, but not for formal proof.\n* Empirical results (e.g., Table 1) show only moderate improvements without reporting standard deviations or confidence intervals. It is therefore difficult to determine whether the observed gains are statistically significant or within expected variance.\n* The writing quality could be improved. Several sections are hard to parse and would benefit from clearer exposition and more precise mathematical notation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ew5Wo6HWvg", "reviewer": "Reviewer_S7AS", "summary": "This paper introduces Stable-LoRA, a novel optimization strategy that enhances the training stability of LoRA. The authors first theoretically demonstrate that LoRA possesses self-stabilizing properties under appropriate hyperparameters and initialization, but commonly used non-zero initialization often undermines this stability. To address this, Stable-LoRA dynamically shrinks matrix A during the early training phase. Experimental results demonstrate the effectiveness of Stable-LoRA and with negligible additional computational or memory overhead.", "questions": "See details in the weaknesses part.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This paper is solid in its theoretical contribution. The motivation and concepts are well illustrated, making the work easy to follow. The algorithm design is simple yet elegant, and the stability stopping criterion is theoretically justified.\n\n* Empirical results demonstrate both the effectiveness and stability of the proposed method. Moreover, it is computationally efficient, introducing only a minor additional runtime overhead. The approach is also compatible with existing LoRA setups without requiring any architectural modifications.", "weaknesses": "* The experimental settings and details are somewhat limited and unclear. First, how are the experiments on the QA datasets conducted? Is the model fine-tuned on a mixed training dataset and then evaluated on several benchmarks, or is it fine-tuned on one QA dataset and tested accordingly? If it is the latter case, I would suggest conducting additional experiments on general language understanding and dialogue datasets such as WizardLM to better assess the model’s generalization ability. Moreover, the experimental settings for the reasoning tasks are also limited.\n\n* How many steps does Stable-LoRA require to reach its stable mode? I think the shrinkage of the LoRA matrix $A$ shares a similar intuition with the weight decay mechanism in AdamW. How about comparing Stable-LoRA with AdamW using a relatively large weight decay parameter when the model shows instability?\n\n* Finally, Stable-LoRA should also be applicable to other LoRA variants, such as AdaLoRA. Have the authors tried this? If the method generalizes well to other setups, it would be quite interesting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iga8GuzJHa", "reviewer": "Reviewer_iYHY", "summary": "This paper provides a theoretical analysis of LoRA’s fine-tuning dynamics and argues that LoRA can achieve self-stabilization under proper initialization and hyperparameter choices. The authors further propose *Stable-LoRA*, a simple weight-shrinkage strategy applied to the A matrix during early training to mitigate instability. Experimental results show consistent improvements across several reasoning and QA benchmarks, demonstrating the method’s effectiveness and efficiency.", "questions": "Please see the weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper offers a clear theoretical perspective on the stability of LoRA fine-tuning, supported by a consistent use of the γ-function framework to characterize scaling behavior. The derivations are mathematically well-structured and connect intuitively to optimization dynamics. The proposed Stable-LoRA method is simple yet effective, introducing negligible computational overhead while improving training stability across tasks. The work also contributes conceptually by bridging theoretical analysis and practical optimization design within the PEFT paradigm.", "weaknesses": "1. About Assumption 1: \n\n   If the activation $Z$ is normalized under a fan-in scaling scheme  (i.e., each input element scaled by $1/\\sqrt{n}$, which is common in linear layers or attention projections),  then each component of $Z$ becomes $\\Theta(n^{-1/2})$.  \n\n   Consequently, the product $g_A^t Z$ involves a summation over $n$ such terms,  resulting in an overall magnitude of $\\Theta(\\sqrt{n})$.  This implies that $\\gamma[g_A^t Z] = \\tfrac{1}{2}$ rather than $1$.  Could the authors clarify whether Assumption 1 still holds under common fan-in scaling conventions?  In particular, does the assumed normalization (e.g., layer normalization, residual connections, or other schemes)  preserve the activations at the $\\Theta(1)$ scale?\n\n2. In Section 3.2, the authors assume that $\\gamma[A_0 Z] \\le \\gamma[\\eta] + 1 \\Rightarrow \\gamma[A_t Z] = \\gamma[\\eta] + 1.$ However, “≤” only provides an upper bound and does not guarantee equality during training. This step effectively assumes convergence of the recursive relation $\\gamma[A_t Z] = \\max(\\gamma[A_{t-1}Z], \\gamma[\\eta] + 1),$ without proving that the sequence will reach equality. Could the authors provide a justification (analytical or empirical) showing that the recursion indeed converges to $\\gamma[A_t Z] = \\gamma[\\eta] + 1$ rather than remaining strictly below that bound?\n\n3. The paper claims that when only one condition in Eq. (5) holds, $\\gamma[A_t Z] = \\gamma[B_t] + 1 \\Rightarrow \\gamma[\\delta_1] = \\gamma[\\delta_2],$ which is said to be “undesirable.” However, δ₁ and δ₂ represent the contributions of matrices A and B to the output update ΔYₜ. Having the same γ-scale simply implies that both pathways contribute symmetrically, which is not necessarily detrimental to learning stability. Could the authors clarify why $\\gamma[\\delta_1] = \\gamma[\\delta_2]$ must be considered suboptimal? Is there theoretical or empirical evidence that symmetric scaling between A and B leads to instability?\n\n### Cons\n\n1. The baselines considered in Table 1 are too limited. Recent PEFT methods such as DoRA[1] and AdaLoRA  also address optimization stability and efficiency. Including these methods would provide a fairer and more comprehensive comparison.\n\n2.  The backbones used are relatively small (≤ 3B parameters). To substantiate the claim of general stability, experiments on larger or more recent architectures (e.g., Llama-3.2-8B, Mistral-7B, or Mixtral-8×7B) are necessary. This would also help assess scalability and compatibility with modern large-model training dynamics.\n\n   [1] Liu, X., Li, Y., Zhou, T., Wang, K., & Qiu, X. (2024). *DoRA: Weight-Decomposed Low-Rank Adaptation*. arXiv preprint arXiv:2402.09353.\n\n   [2] Zhang, R., Xu, H., Cui, Y., Liu, T., & Zhang, Y. (2023). *AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning*. arXiv preprint arXiv:2303.10512.\n\n## Typos\n\n1. Lines 101: There is a dot(.) at the begining of 101 lines. Maybe its should be placed on the end of Eq. (1). \n\n2. Lines 202: The same issue as in line 101 — punctuation should not appear at the beginning of a new line.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)