The Primacy of Magnitude in Low-Rank Adaptation
Authors: Zicheng Zhang
Haoran Li
Yifeng Zhang
Guoqiang Gong
Jiaxing Wang
Junxing Hu
pengzhang liu
Qixia Jiang
Venue: NeurIPS 2025 spotlight ()
Keywords: LoRA; Low-Rank Adaptation; Deep Learning; Large Pre-trained Models
Primary area: deep_learning

Abstract: Low-Rank Adaptation (LoRA) offers a parameter-efficient paradigm for tuning large models. While recent spectral initialization methods improve convergence and performance over the naive “Noise \& Zeros” scheme, their extra computational and storage overhead undermines efficiency. In this paper, we establish update magnitude as the fundamental driver of LoRA performance and propose LoRAM, a magnitude-driven “Basis \& Basis” initialization scheme that matches spectral methods without their inefficiencies. Our key contributions are threefold: (i) Magnitude of weight updates determines convergence.
We prove low-rank structures intrinsically bound update magnitudes, unifying hyperparameter tuning in learning rate, scaling factor, and initialization as mechanisms to optimize magnitude regulation. 
(ii) Spectral initialization succeeds via magnitude amplification.
 We demystify that the presumed knowledge-driven benefit of spectral component essentially arises from the boost in the weight update magnitude.
(iii) A novel and compact initialization strategy, LoRAM, scales deterministic orthogonal bases using pretrained weight magnitudes to simulate spectral gains. Extensive experiments show that LoRAM serves as a strong baseline, retaining the full efficiency of LoRA while matching or outperforming spectral initialization across benchmarks.

com The Primacy of Magnitude in Low-Rank Adaptation
Zicheng Zhang' Haoran Li Junxing Hu!
Yifeng Zhang' Pengzhang Liu"
Guoqiang Gong'
Jiaxing Wang'
Qixia Jiang'
University of Chinese Academy of Sciences
Abstract
Low-Rank Adaptation (LoRA) offers a parameter-efficient paradigm for tuning large models. While recent spectral initialization methods improve convergence and performance over the naive "Noise & Zeros" scheme, their extra computational and storage overhead undermines efficiency. In this paper, we establish update magnitude as the fundamental driver of LoRA performance and propose LoRAM, magnitude-driven "Basis & Basis" initialization scheme that matches spectral methods without their inefficiencies Our key contributions are threefold: (i) Mag- nitude of weight updates determines convergence. We prove low-rank structures intrinsically bound update magnitudes, unifying hyperparameter tuning in learn- ing rate, scaling factor, and initialization as mechanisms to optimize magnitude regulation. (ii) Spectral initialization succeeds via magnitude amplification. We demystify that the presumed knowledge-drives benefit of the spectral component essentially arises from the boost in the weight update magnitude. (iii) A novel and compact initialization strategy; LoRAM, scales deterministic orthogonal bases using pretrained weight magnitudes to simulate spectral gains. Extensive experi- ments show that LoRAM serves as a strong baseline, retaining the full efficiency LoRA while matching outperforming spectral initialization across benchmarks.
Introduction
The rise of large pretrained models 2 5 has driven urgent needs for parameter-efffcient Among these, Low-Rank Adaptation (LoRA) 7 stands out for its efficiency, flexibility, and stability. By freezing pretrained weights and injecting trainable low-rank matrices, LoRA enables the update of less than 1% of the parameters, significantly reducing memory and compute costs. Its aong-and--layy nature achieves easy integration into diverse models, facilitating model sharing and federated learning [12 [3.. Additionally, LoRA helps prevent catastrophic forgetting [14, making well-suited for continual learning. These advantages have led
fine-tuning (PEFT) methods 66
7
9
10 II]
to its wide adoption in multilingual NIP [15 16 17 18] and multimodal applications 19 20
21
22
Despite achieved efficiency, the low-rank reparameterization constrains practical performance and convergence [14]. Besides the well-known "representation bottleneck" [23
10
24
25|
26 27
28 29 LoRA highly sensitive hyperparameters due to the non-convex and non-smooth loss landscape 30 Effective training relies on careful tuning of rank [31
32
33 scaling factor [34],
learning rate 35 initialization strategies 37 30, and preconditioning
30
38
Recent works
increasingly leverage information from pretrained weights [40
41
42
task specifc data 43 44
39
pioneers the use of Singular Value Decomposition (SVD) for LoRA initialization, employing spectral components of pretrained
improve the "Noise Zeros" baseline. Among these, PiSSA 40
"Corresponding author: zhangrichengo@@d.com
Codee available here
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Magnitode
Pretrained Matrix
Residual Matrix
Residual Matrix
gel U.VS
Residual W B (
gith =
W E Ig xmm
Bundom
W
g(0) A WSy"
4
(a) Full Fine-tuning
(b) LoRA
(c) Spectral initialization
(d) LoRAM
Figure We propose LoRAM, a magnitude-driven initialization method that enhances both the convergence and performance of LoRA while maintaining its efficiency. Unlike spectral initializa- tion, which precomputes and stores singular components (U, V,S) [40, LoRAM uses deterministic orthogonal bases and derives scaling from pretrained weight statistics. This elegant simplification is grounded our analysis of LoRA through novel lens of magnitude dynamies, where we show that the benefits of spectral values in scaling weight update magnitude can be effectively approximated.
weights to significantly enhance convergence and performance. Subsequent works 44 extend to diverse matrix decompositions, fostering wave of knowledge-driver initialization schemes.
39
While spectral initialization 40
43 showcases considerable promise in convergence and performance, two fundamental challenges persist. First, they introduce complexity by requiring additional matrix decomposition and storage overhead, undermining usage efficiency in resource- constrained settings [45 25 46 and hindering seamless integration with deep learning libraries [6 47)
Second, their success remains poorly understood. The common, justification (40
41
43 48 that
spectral components preserve features better than random alternatives lacks theoretical grounding, Although recent works parameter gradients, the non-convex nature of LoRA renders training dynamics unpredictable.
44 suggest that some specific initialization may approximate full-
In this paper, demystify the knowledge-drive drivve intuition behind spectral initialization and reveal that its effectiveness primarily stems from the magnitude scaling of weight updates. We design a minimal "Basis& & Basis" initialization strategy, which demonstrates comparable performance without the overhead of SVD operations. Specifically, our key contributions can be summarized as:
We identify weight update magnitude as a fundamental principle for analyzing and improving LoRA's training dynamics. This principle unifies previously independent factors, such as learning rate, scaling, and initialization, revealing their shared ability to control weight update strength and achieve comparable amplification effects when properly configured.
We demonstrate that initialization scheme eritically shapes LoRA's weight magnitude dynamics. Theoretically, we prove that LoRA naturally produces smaller updates than full fine-tuning, which limits its convergence and expressiveness. Moreover, show that spectral initialization amplifies updates, providing a principled explanation for its effectiveness beyond knowledge-drive intuition.
Guided by the magnitude principle, we propose Magnitude-driven Initialization (LoRAM) to make LoRA initialization efficient again. LoRAM employs a logarithmic magnitude factor to retain the benefits of spectral scaling, while directly scaling deterministic orthogonal bases to eliminate the need for decomposition and storage. Extensive experiments on language and vision- languaee tasks establish LoRAM as strong and practical baseline, surpassing prior initialization schemes.
2 Magnitude Principle for Characterizing LoRA Dynamics
2.1 Preliminaries and Notations
Given a pretrained weight matrix W 2 LoRA 7 reparameterizes the forward pass as Wx + WLARAT Wa + a(BA B(O) ((0))
where B E Raxr, E Re are trainable low-rank matrices with r < min(n, m), and a scales the update magnitude. The initialization term [(0)_A()) can be absorbed into W for the convenience, as illustrated in Figure (b). Given a loss function L, LoRA updates are computed as:
V VAL T By
aL
T aB L) V a ØL x T AT a L)T.
(2)
by
Magnitude metric. To elucidate how LoRA affects the training process, we analyze the dynamics of the parameters A, B, and the resulting weight update WLORA. Specifically, we define the weight

---

magnitude as 2/ bi[LLRA = 1 LORA P which serves as central metric in our study. Assuming
independent and zero-mean entries 49]. the expected weight magnitude is given by [|r]BA]]
r (B(u(B) [J] B [[]]] In the asymptotic regime where m and are large, w|WLoRA] is approximated with the variance of WLARA, and v|BA] R r [ v(A]. Our analysis motivated by the fact that
LoRA introduces no change to the pretrained weights initially, i.e., =|W O. while its effect emerges gradually through training. Therefore, we use the term "magnitude" instead of "variance" to highlight the cumulative growth of WLORA- In Appendix B we also present theoretical insight showing that parameter magnitude is key determinant of LoRA's expressiveness.
2.2 Effect of Hyperparameters on Update Magnitude
Let AW
LoRA denote the weight update at step t for LoRA framework, and W:(1) = =ii Aw
LoRA
represent cumulative adaptation. Given learning rate 1/, we expand A Awi(t) rules, leading to the following formulations for the update magnitude"
LoRtA using gradient update
AW LORA
an (B ( + ( A + VVV ALD))
(3)
and v/AW (1) LORAl 10 (M(B() M (V + ~IV [V
These equations indicate the complex interplay of multiple hyperparameters, distinct from the full- parameter updates given by AW(O = Vn T,( We investigate the interplay among the learning rate 1), scaling factor a, and initialization magnitude, revealing a quantifiable equivalence relationship. Proposition 1 (Parameter Scaling Equivalence). For LoRA layers defined in Eq. ① consider decomposing the scaling factor a = a' ga where d', A 0'g E RR+ Under the commonly used optimization frameworks with negligible numerical errors, the following parametrization schemes exhibit dynamical equivalence throughout training: For all iterations t 2 0, AW:(t) AW!l))
LORA LORA
and W(Ook
LoRA. where (t)), ( and (()) represent the re-parameterized versions.
LORA-
Original
SGD
Adam
Representation
BBr
a' BÁz
a' BAr
Initialization
(0)
Ama, Rd) Bad 1 a.
B00) B. A a. B(0) Bgu
Learning Rates
DA 0, 0 ni QANA, ogne n1 OANA, 1/8 agio
Remarks. This equivalence underscores how hyperparameters collectively regulate update magnitude, effectively reducing the search space for optimal contigurations. A striking implication is that, increasing Ni in LoRA+ [35) is identical to scaling in RsLoRA [34] under the "Noise & Zeros" initialization, highlighting the critical role of initialization magnitude in shaping LoRA's training dynamics, which is rarely discussed in prior works. For the non-zero initialization, it is advisable to first adjust the initialization magnitudes and learning rates for moderate improvements, as modifying inherently combines the effects of both and may result in drastic and unpredictable changes.
To demonstrate the joint effect of hyperparameters experiment using a 5-layer MLP with "Noise & Zeros" initialized LoRA layers, setting the interme- diate dimension 400 and the LoRA rank 25. The network trained on synthetic data under various hyperparameter settings, consequently using SGD and Adam optimizers with r = 5 x 10-5, As shown in Figure aa), all settings with a 16 result in identical loss trajectories and parameter evolution, confirming the theoretical predictions. In contrast, weight updates deviate significantly when using a 1 with 7) scaled by 4, indicating that equivalence holds only under specific rules.
LoRA dynamics, conduct controlled
2.3 Magnitude Limitation Rooted in Low-Rank Structure
Guided by the established equivalence framework, we fix a = to climinate the interference of scaling factors, which is the most commonly-used configuration in practical implementation. In the following, we prove initialization magnitudes and other factors critically influence the weight update. Proposition 2 (Parameter Magnitude Dynamics), Consider LoRA parameters updated with the same learning rate n,. Assume: A AV I21), B(d) NN I),1), L() NN of, 1), and
See Appendix for derivation and proof
3

---

Training dynamies under different hyperparamders
Magnitude dynamies under different initialization
1.0
SGD region
region
500
1.0
Loss
0.5
0.8
40
Norm Diffarence
0.0
0.5-
30
0.4-
20
vGA
0.00
3000
5000
2000
5000
Figure 2: (a) Validation of Proposition 1 Each curve represents model with unique hyperparameters. The norm difference (right axis) aggregates Frobenius norm discrepancies between the baseline model (black) and others across layers. Purple and other curves share identical learning rates but diverge due to differing initialization magnitudes. Equivalent optimization trajectories emerge from diverse hyperparameter combinations under both SGD and Adam optimizers. (b) Validation of Proposition 2 The black curve represents random orthogonal initialization. Parameter magnitudes predominantly governed by initialization scaling, resulting in smaller weight changes compared to conventional linear layers. This necessitates the magnitude scaling in enhancing LoRA performance.
(K V VAL E((((, VBL = Under these conditions, the parameter magnitudes
Vt [E JEYBBon evolve linear dynamical system. Its exponential solution admits the linearized approximation under small-n regime:
NB
[o3 tr V
V: =
T
oo 1T
(5)
where γ4 = of 7B mr/ This further yields the evolution weight update magnitude:
u(W/()) LoRal 2 kinl ())2t²) where 7 v70o2, kn = rm(mo' no
(6)
Remarks. This analysis uncovers essential properties of LoRA initialization and optimization trajectory. First, since "A and 7B are very small values (< 1), the magnitudes of parameters A(1) and (t)) remains nearly unchanged throughout training, potentially constraining the representation capacity of the learned model. Moreover, unlike full-parameter tuning, which evolves at a linear rate y, the low-rank structure introduces a proportional factor ky, significantly slowing updates. For instance, the naive "Noise & Zeros" initialization yields kı r. while the dimension in large models like LLaMA B1 is in the thousands or more. Despite the quadratic term accelerates growth, small gradients may temper this effect in the later training stages.
Figure 2 ()) visualizes LoRA magnitude dynamics during training. We use the same network as in Figure a) but apply nonzero initialization with σ = OB 20 We explore a regular MLP with the same magnitude denoted as "Linear", and group of networks with larger initialization magnitudes. Notably, while the loss decreases significantly, the magnitudes of A and B remain nearly unchanged throughout training. The magnitude evolution W reveals that the basic LoRA with theoretically k +े grows substantially slower than the regular MLP. Increasing the initialization magnitude effectively accelerates the growth of W, aligning with our theoretical analysis.
Integrating analyses in this section, we derive magnitude principle for LoRA development:
A valid improvement to LoRA convergence will enhance weight update magnitude v/ [WLoRA)
As shown in Proposition 2 magnitude principle could unify and explain improvement factors in existing works, including the learning rate, scaling factor, gradients, rank and initialization schemes.
3 Demystifying Spectral Gains with Magnitude Principle
The sheer scale of modern neural networks complicates the determination of optimal LoRA initializa- tion across layers. Drawing inspiration from recent spectral initialization methods 40 43
39
which have demonstrated improved convergence and task performance, we reinterpret their effec- tiveness through the lens of the magnitude principle and introduce a magnitude-driven initialization

---

(a) DeBERTa-y3-baal
LLaMA-2-7B
(c) FLUX.1-12B
032
452
plr]
302
992
155
1,00
1.02
Q
6.75
9:30
250
600
15.0
1000 2000
2003
0003
2000
Illustration of spectral gain factor Q[r] defined in Eq. (0) and spectral concentration factor p[r] defined in Eq. (8) across DeBERTa-v3-base [50], LLaMA-2-7B [3] and PLUX.I--22 [51]. Values are computed from uniformly sampled layers. The white dotted line represents the linear growth rate of naive LoRA weight magnitudes, while spectral initialization exhibits faster growth.
Figure 3:
Due to its concave nature, approximate the spectral gain factor using logarithmic function.
method called LoRAM. Notably, these methods requires extra SVD computations and storage, lead- ing increased resource overhead and implementation complexity. In contrast, LoRAM mitigates these drawbacks and achieves even better performance. In the following, we take the seminal work PiSSA [40] as a representative baseline and ablate other methods in experiments (see Section 4.3)
3.1 Magnitude Gain in Spectral Initialization
The PiSSA method [40] initializes LoRA using the spectral decomposition of pretrained weight matrix W USV which has a rank of R[W]. The spectral initialization is defined as:
A AsvD VSSV.Tr 4,m
B(O)_ BSVD U..rVS,,
(7)
where S, E R contains the top-r singular values, and U E R V E Rmxm are the left and right singular vector matrices. While prior works (40 41 43
48 attribute PiSSA's success to
its ability to preserve principal components, we show that its key advantage lies in singular value weighting. By redistributing dominant variance into the initialization, PiSSA facilitates adaptive magnitude updates across layers, accelerating convergence.
Consider the statisties of the top-r singular values, we define the spectral concentration factor
p|r]
(8)
Σ
This factor captures the concentration of energy in the top-r singular values. We then reformulate
v(AsvD) E, [s] W W mR[
((BYVD) E, [s]
mplr|e/
<R[W]
Essentially, p(r) acts scaling factor that redistributes variance from the pretrained weight matrix, influencing the magnitude of updates during training. Since p[r] monotonically decreases with r, its impact is more pronounced for smaller r, making particularly relevant for LoRA applications.
Spectral Gain Factor. Taking the above magnitudes into the dynamics in Eq. ( further derives:
h Qm + m))[W]
Q(r)
& 1.
(10)
Given that w[W]- Ommin( : })), this results in gain factor of at least Q(r). which we term as the "spectral gain factor". As shown in Figure 3 the spectral gain factor Q/r] exhibits bounded variation in [0, 1] with characteristic concavity. which can be formally derived via Jensen's 's inequality, Although SVD components are not completely independent preventing the theoretical monotonic increase in Q(r). this concavity also suggests that the gain effect more pronounced when is small, reinforcing the effectiveness of spectral initialization for LoRA in the aurameter-efficient manner.
3.2 Efficient Magnitude-driven Initialization with LoRAM
We propose LoRAM to achieve similar magnitude update rate in Eq. (10) like PiSSA while eliminating LoRAM initializes the parameter matrices as:
spectral computation. As depicted Algorithm I
1
(0) ALORAM = TT B(O) BLORAM = B. B
(1))
o +
5

---

Algorithm LoRAM Initialization Procedure
Pretrained weight W Ruxm target rank Output: Initialized parameters A B(),, W
Input:
nn 2m f- gatsssiis(,, r), get basis(m,
Generate basis matrices, e.g. Eq
12
8 logminta m)) (r)-2([] 1/4
> Compute magnitude gain factor Initialize parameters
,n 9 mm
O 10)
W + 3 2n 3- T W- B 4n T
Here d, and P... denote the first r columns of an and m-dimensional orthogonal basis matrices, respectively. Given that v/$,,| = a as PiSSA with k = Q[r] (m + [W] We can also derive v viB(0), A = Q[r]u[W], implying LoRAM inherently ensures numerical stability and moderate corrections to the pretrained weight.
and v/(mm] = m LoRAM achieves the similar magnitude
Logarithmic Gain Factor. Due to the concave nature of Q[r). we approximate its analytical form
using an asymptotic expansion: Q[r]= logmin(n,, (r). As illustrated in Figure 3 this logarithmic function effectively captures the monotonic increase nature, providing a predictable improvement than LoRA and PiSSA particularly when using small rank r.
To eliminate the need for storing initialization buffers, we adopt an analytic approach instead of random initialization. Specifically, we employ the Discrete Sine Transform (DST) basis due to its simplistic mathematical definition:
Deterministic Basis Matrix.
Pm [
2 sin m++1
[++1)++1)
0 S i.j < m.
(12)
This formulation constructs orthogonal matrices of arbitrary dimensions, ensuring reprodueibility across different devices while providing provable statistical properties: E[ 0 and 1/ [( 1
One may wonder if randomness is required for initialization, we find it unnecessary in LoRA. In fact, DST even slightly outperforms random strategies in our ablation experiments (see Section 4.3
Efficiency and Compatibility. Since / and avoid complex matrix operations, LoRAM retains the efficiency and storage footprint of naive LoRA. As it only modifies initialization, LoRAM remains plug-and-play. integrating seamlessly into any pipeline that supports standard LoRA. This is especially valuable for modern large models built on fixed and highly optimized frameworks, In contrast, other initialization methods require costly preprocessing, such as matrix generation 4 which complicates adoption in standard workflows.
or decomposition
40 42
4 Experiments
We conduct comprehensive experiments to evaluate LoRAM efficiently implemented via the PEFT we assess performance on language tasks and extend the evaluation to rision-language tasks, demonstrating LoRAM's 's generalization across diverse models and modalities. All experiments are run on servers with NVIDIA H800 GPUs.
library 6 Following conventional settings 40 41 35
While extensive research on LoRA has explored aspects like structural modifications and rank control, these directions largely orthogonal to our focus hyperparameter analysis within the naive LoRA framework. In line with this, we compare LoRAM with the naive LoRA (ICLR 2022) (7). as well as several representative hyperparameter tuning strategies. We first consider weight-driven initialization (marked "S"), including PiSSA (NeurIPS 2024) [40], which uses the top-r singular vectors and values of pre-trained weights; MiLoRA (NAACL 2025) (4I). which utilizes the last r singular vectors and values; and OLoRA 42 which applies orthogonal initialization via QR decomposition. All these methods adopt a fixed scaling factor
Baselines.
1. We then include RsLoRA 34 (marked "f"), which enhances performance by setting = vr, and LoRA+ (ICML 2024) 35 (marked "i"),, which increases the learning rate with the recommended 7// = 41/A. We also evaluate data-driven initialization in the ablation study, including LoRA-GA (NeurIPS 2024) 52 and CorDA (NeurIPS 2024) [ which require extra pipeline to leverage training data information. Most of these baselines have been integrated and validated in the PEFT library.

---

Table 1: Comparison of LoRAM versus hyperparameter tuning baselines NLG tasks. Experiments conducted with LLaMA2-7B model using two ranks, reporting mean std results (%) over three runs. Bold and underlined values represent the best and second-best performances, respectively.
Rank Param
Method
GSM8K
MATH
HumanEval
MBPP
Commonsense
N/A
6738M
Full FT
60.34 = 1.32
11.74 1: 0.63
32.30 + 1.26
39.27 :: 1.01
79.20 1.20
LoRA
31.51 0.31
4.16 0.27
15.98 +
28.65 +: 0.47
66.56 1.21
RsLoRA 1
39.04 = 0.53
4.94 + 0.40
18.85 = 0.66
28,10 = 0.64
73.24 = 0.84
16
40M
LoRA+1
31.69 0.64
3.98 0.38
18.54 0.52
28.00 + 0.81
72.19 1.43
MiLoRA
29.70 0.42
4.18 U.21
14.69 = 0.66
27.23 + 0.53
67.90 1.20
OLoRA
35.83 0.58
4.80 + 0.53
16.58 = 0.38
27.44 + 0.76
73.48 1.09
PISSA
37.68 0.45
5.16
18.37 +
28.62 +
73.72
LoRAM
40.32 = 0.43
5.30 0.37
18.92 = 0.55
28.83 + 0.63
75.19 1.10
LoRA
40.27 0.70
4.72
0.43
20.11 = 0.32
28.84 + 0.37
73.64 1.13
RsLoRA
50.38 0.37
7.32 0.28
21.32 + 0.70
30.73 + 0.43
77.01 = 1.17
128
320M
LoRA++
40.41 = 0.67
5,28 0.53
20.71 = 0.88
29.13 + 0.78
78.19 1.33
MiLoRA
39.81 0.89
5.18 0.58
20.39 -
29.95 +
74.29 1.09
OLORA
50.10 = 0.64
7.01 + U.56
20.72 = 0.67
30,21 + 0.89
78.61 = 0.97
PISSA
51.48 0.77
7.04 0.54
21.62 0.48
31.07 0.68
77.28 0.98
LoRAM
51.12 0.73
7.25 0.68
22.03 + 0.56
31.53 + 0.72
77.81 0.96
Table 2: Comparison of LoRAM versus hyperparameter tuning baselines on GLUE benchmark.
Experiments conducted with the DeBERTa-v3-bas model using rank reporting results three runs. Bold and underlined values represent the best and second-best performances, respectively.
Method
# Param
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Full
184M
88.31
93.57
89.46
67.26
92.80
91.52
83.75
86.87
LoRA
1.33M
90.23
95.87
84.06
63.56
93.88
90.55
50.18
87.20
RsLoRAt
1.33M
90.33
95.64
86.38
64.85
93.97
90.26
60.31
88.37
LoRA+1
1.33M
90.37
95.32
87.54
64.79
94.32
90.93
65.37
89.20
MiLoRA
1.33M
90.28
95.75
87.00
62.58
93.97
90.83
54.87
87.74
OLORA
1.33M
90.19
94.83
88.72
65.59
93,36
90.74
74.09
88.52
PiSSA
1.33M
90.38
95.64
89.21
65.06
93.84
91.35
74.36
88,90
L
1.33M
90.34
95.29
89.95
65.53
94.08
91.70
74.72
89.93
4.1 Evaluating the Performance on Natural Language Tasks
Nature Language Generation (NLG). As shown Table we conduct supervised fine-tuning of LLaMA 2-7B 3 on math, coding, and commonsense reasoning tasks. Our setup strictly follows
PiSSA 40), using the Adam optimizer with batch size of 128, a learning rate of 10-5 warmup ratio of 0.03, and no weight decay. All experiments are performed
subsets containing 100K data points for one epoch to minimize training overhead. For math tasks, the model tuned on MetaMathQA 5 and evaluated on GSM8K (54) and MATH [53] validation sets. For coding tasks, we use CodeFeedback (55) as training dataset, with evaluations on HumanEval [56] and MBPP (57]. For commonsense tasks, model is tuned Commonsensel I 70K (58). and report averaged accuracy on eight sub-datasets. The results Table demonstrate that LoRAM consistently outperforms LoRA variants across diverse tasks and rank settings, without requiring matrix decomposition.
Nature Language Understanding (NLU). We evaluate the NLU performance by fine-tuning the DeBERTaa-v--bass model [50] with a rank of 8 on eight tasks in the GLUE benchmark (59]. We utilize scripts from the Transformers Library [47] to ensure a fair comparison. All methods are trained with a learning rate of 1 x 10 for 3 training epochs, except for MRPC, which uses 5 epochs due to its smaller size. We report overall matched and mismatched accuracy on MNLI, Matthew's 's correlation CoLA, Pearson correlation on STS-B, and accuracy the other datasets. As shown in LoRAM achieves competitive performance against PiSSA across most tasks.
Table 2
4.2 Evaluating the Performance Vision-Language Tasks
Text-to-image synthesis. We adapt the advanced FLUUX1-112 (11 to address the image customiza- tion task, implementing LoRA, PiSSA, and LoRAM under identical configurations: learning rate of 1 x 10-1, batch size of and 1,000 iterations. We set the rank to 8, optimizing 9.3 million

---

Table 3: Comparing LoRAM with other LoRA variants on LL for multimodel tasks. Bold and underlined values indicate the top and second-best performances, respectively.
Method
MMEc
MME
MMMU
A12D
ChartQA
OCRBench
TextVQA
ScienceQA
Full FT
280
1541
0.355
0.583
0.251
0.361
0.597
0.722
LoRA
278
1402
0.331
0.557
0.231
0,333
0.536
0.684
RsLoRA
274
1385
0.334
0.573
0.227
0.328
0.539
0.694
LoRA+1
283
1389
0.341
0.565
0.229
0.331
0.545
0.690
MiLoRAS
285
1354
0.340
0,564
0.220
0.335
0,536
0.681
OLORA
288
1404
0.345
0.565
0.228
0.330
0.540
0.677
PISSAE
311
1411
0.344
0.564
0.232
0.338
0.547
0.686
LORAM
308
1406
0.350
0.571
0.238
0.336
0.551
0.700
Doy backpack
Teapot
Vage
Boot
Reference
LoRA
PISSA
LORAM
Figure 4: Comparison of LoRA, PiSSA, and LoRAM on image customization task. Experiments conducted with the state-of-the-art FLUX. 1-12B model using rank 8.
parameters while maintaining computational efficiency single GPU. The training data and prompt template adhere to DreamBooth' 's protocol [22]. Qualitative results in Figure demonstrate that LoRAM exhibits marginally superior performance in detail fidelity compared to PiSSA.
Image-to-text generation. Following the pipeline of LLaVA 60), we employ CLT---VIT-L/I as the visual encoder, ficuna-13B [62] as the text decoder, and new visual resampler 63 as the connector. the pre-training stage, we fine-tune only the perceiver resampler using the CC-595K dataset [60] for one epoch. During the subsequent instruction-tume
stage, we fine-tune both Vicuna and the resampler using a 656K mixture dataset (60). The learning rate is set to 2 10-5, and the batch size is 128. We follow the official implementation and use a rank of 64. As shown in Table 3 LoRAM achieves favorable performance across multiple multimodal benchmarks.
Training curves. We provide representative training loss curves of diverse initialization methods in Fi It can be noticed that LoRAM is able have faster convergence rate the early stages compared to other LoRA variants and incur smaller losses in the end.
4.3 Ablating the Magnitude Principle in LoRA improvements
We conduct ablation experiments on LLaMA-2-7B 3 under the NLG setting, focusing on the effect of magnitude gain factor, the choice of basis matrix, and the validation the magnitude principle.
Magnitude Gain Factor. As shown in Table 4 we first evaluate different values of Q(r) and observe that increasing its value slightly improves performance. We further introduce a "tracking mode" that adjusts 3 in Algorithm based on the initialization magnitudes from reference methods. Specifically, Ba
we set = V
where Bet and And are initialization matrix using weight-driven 40 41
4 44.. Under this mode, LoRAM essentially matches the performance of prior methods, confirming that magnitudes govern its performance. We also observe that PiSSA,
or data-driven approaches

---

Tuning DeBERTa QQP
(b) Tuning LLaMAZ MetaMath
Instruction tuning LLaVA
Pelrt
04
24
0.2
02
3,000
30c
Figure 5: Illustration of training loss curves. LoRAM achieves comparative convergence dynamics to PiSSA across diverse models and benchmarks. See tables and texts for the evaluation results.
Table 4:
Results of ablation study. "Orth." denotes the random orthogonal initialization. The left and right sides of the slash indicate the results of the method and the tracking mode, respectively, The average value is calculated over all the ranks and tasks to compare the overall trend of change.
Qirl
Basis
Wrighn
Drta-driven
RRLLRA
Rank
Task
lug
log
log
Orth.
Gaussian
MiLoRA
PISSA
CurDA
LoRA-GA
LoRA-GA
LoRAM
MATH
5.08
5.30
5.18
4.74
4.62
4.18/4.05
5.16/5.10
4.60/4.14
3.73/3.76
7.94
7.22
16
GSM8k
40.1
40.3
40.7
36,3
35.8
29.7/ 29.6
37.6/36,7
36.2/30.7
45.7/300 30.9
51.5
52.1
MBPP
28.8
28.8
28.3
28.6
27.5
27.2/27.8
28.6/
25.7
28.3/27.8
33.9
31.5
HumanEval
17.1
18.9
17.1
17.7
17.7
14.6/ 14.7
18.3/17.
15.2/
19.5/17.7
22.0
18.3
MATH
7.52
7.25
7.51
7,40
7.62
5.18/5.04
7.04
6.26/5.04
9.18/7.32
9.08
11.1
GSM8K
50.7
51.1
50.4
50.2
49.8
39.8/39.2
49,4
40,3
54.4/50.8
53.6
59.4
128
MBPP
31.5
31.5
32.3
31.3
32.8
29.9/30..
39.0/29.6
29.6/29.9
30.2
31.7
38.1
HumanEval
20.7
22.0
22.6
23.2
22.6
20.3 19.5
21.6/ 20.3
20.1 18.9
20.4/20.7
26.8
31.7
Average value
25.1
25.5
25.6
24.9
24.8
21.1
/21.2
25.0/ 24.3
22.7/21.6 27.4/336
29.5
31.1
which selects the top-r singular values, outperforms MiLoRA, which selects the last singular values. This indicates that leveraging the large dominant singular values enhances performance.
Basis Matrix. We find that the choice of basis matrix generally has limited impact. For instance, replacing the DST basis with a random orthogonal or Gaussian matrix just results in only minor performance degradation. In tracking mode, substituting the SVD-derived basis with DST does not significantly impact performance. notable exception LoRA-GA [44), which approximates the -ull-parameter gradient at initialization. Nonetheless, we emphasize that tracking mode fails not due to incorrectness of magnitude principle, but because the LoRA-GA matrix form maximizes gradient magnitudes of Eq. 2 making it irreplaceable by alternatives and validating magnitude principle
Upper Bound of Magnitude Scaling. While increasing update magnitude generally improves performance, the benefit is not unlimited. For example, applying RsLoRA to LoRA-GA yields a clear gain rank 8, but the improvement diminishes and may even reverse rank 128. This suggests that magnitude scaling should be applied conservatively at higher ranks, since larger ranks inherently amplify updates, as demonstrated in our Proposition 2 Given that data-driven methods involve costly gradient and SVD computations, we recommend LoRAM with RsLoRA as more efficient and scalable alternative for accelerating LoRA convergence and performance.
5 Conclusion
In this paper, we explore the magnitude principle of Low-Rank Adaptation (LoRA) and introduce a novel magnitude-driven initialization strategy, LoRAM, that bridges the gap between efficiency and performance. Our work demystifies the prevailing awareness surrounding spectral initialization methods, demonstrating that their success primarily stems from the amplification of weight update magnitudes. By focusing on magnitude regulation as the key driver of convergence, we provide unified perspective that connects seemingly disparate hyperparameter adjustments, such as learning rate, scaling factor, and initialization schemes, under single framework.
Limitations and Future Work. Despite the advancements introduced this work, several challenges remain open for future research. First, LoRAM mimics spectral initialization magnitudes rather than seeking optimal ones; exploring alternative strategies could yield further gains. Additionally, different layers may benefit from tailored magnitude settings, motivating joint optimization with learning rate and rank. Finally, our work does not explicitly address optimization dynamics and convergence properties. These directions remain valuable for advancing parameter-efficienl fine-tuning.
"See Proposition in Appendix. We prove LoRA-GA initialization maximizes LoRA gradient magnitude.

---

References
[1] OpenAl Team. Language models are few-shot learners. In NewrIPS, 2020.
[2] OpenAT Team. Gpt-4 technical report. arXiv preprint arXtv:333.008774 2023.
[3] Meta Team, Llama 2: Open foundation and fine-tuned chat models, arXiv preprint arXiv:2307.09280
2023.
[4] Qwen Team. Qwen technical report. arXiv preprint arXiv:2309.16609 2023.
[5] DeepSeek Team. Deepseck-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[6] S Mangrulkar, S Gugger, L Debut, Yo Belkada, S Paul, and B Bossan. Peft: state-of-the-ar parameter- efficient fine-tuning methods. https: // github. com/ huggengface, peft 2023.
[7] Edward J. Hu, Yelong Shen, Phillip Wallis, 7eyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022.
[8] Brian Lester, Rami Al Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In EMNLP, 2021.
[9] Junxian Hc, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirk-arikt, and Graham Ncubig. Towards a unified view of parameter-effiient transfer learning. In ICLR, 2022.
[10] Ali Edalati, Marzieh S. Tahaei, Ivan Kohyzev, V. Nia, James J. Clark, and Mchdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. In NeurIPS Workshop, 2023.
[11] Dan Zhang, Tao Feng. Lilong Xue, Yuandong Wang, Yuxiao Dong, and Jie Tang. Parameter-efficient fine-tuning for foundation models. arXiv preprint arXiv:2501. 13787, 2025.
[12] Sara Babakniya, Ahmed Roushdy Elkordy, Yahya Ezzeldin, Qingfeng Liu, Kee-Bong Song, Mostafa El-Khamy, and Salman Avestimehr. Slora: Federated parameter efficient fine-tuning of language models. In NeurlPS Workshop, 2023.
[13] Yuxuan Yan, Shumpu Tang, Zhiguo Shi, and Qianqian Yang. FeDeRA: Efficient fine-tuning of language models in federated learning leveraging weight decomposition. arXiv preprint arXiv:2404. 18848, 2024.
[14] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. TMLR, 2024.
[15] Haokun Liu, Derek Tam, Mohammed Muqoeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. In NeurIPS, 2022.
[16] Ning Ding. Yujia Qin. Guang Yang. Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-effliient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 2023.
[17] Xianghui Sun, Yunjie Ji, Baochang Ma. and Xiangang Li. A comparative study between full parameter and lora-based fine-tuning on chinese instruction data for instruction following large language model. arXiv preprint ar 23v:2304.08888 2023.
[18] Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Amav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, and Devvret Rishi. Lora land: 310 fine-tuned Ilms that rival gpt-4, a technical report. arXiv preprint arXiv:2405.007
2024.
[19] Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Yc, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality, arXiv preprint arXiv:2304. 144.12304,,,178, 2023.
[20] Yuwei Guo, Ceyuan Yang, Anyi Rao, Zhengyang Liang, Yaohui Wang, Yu Qiao, Mancesh Agrawala, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. In ICLR, 2024.
[21] Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets, arXiv preprint 2r1 2023.
10

---

[22] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
[23] Nam Hyeon-War Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for
communiiation--fffccennt federated learning. In ICLR, 2022.
[24] Bojia Zi. Xianbiao Qi, Lingzhi Wang. Jianan Wang, Kam-Fai Wong, and Lei Zhang, Delta-lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2330.02441 2022.
[25] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. In JCML, 2022.
[26] Ting Jiang. Shaohan Huang. Shengyue Luo, Zihan Zhang, Haizhen Huang. Furu Wei, Weiwei Deng. Feng Sun. Qi Zhang. Deging Wang, et al. Mora: High-rank updating for parameter-efficien fine-tuning, arXiv preprint arXiv:2405.12133 12130, 2024.
[27] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. Relora: High-rank training through low-rank updates. In ICLR, 2024.
[28] Wenhan Xia, Chengwei Qin, and Elad Hazan, Chain of lora: Efficient fine-tuning of language models via residual learning. In ICML, 2024.
[29] Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang. Peiyi Wang. Qingxiu Dong, Liang Chen, and Zhifang Sui. Periodiclora: Breaking the low-rank bottleneck in lora optimization. arXiv preprint 21..xi22422..64444 2024.
[30] Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. In ICLR, 2025.
[31] Yuchen Zeng and Kangwook Lec. The expressive power of low-rank adaptation. In ICLR, 2024.
[32] Qingru Zhang, Minshuo Chen, Alexander W. Bukharin, Pengeheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficien fine- tine-tuning. In ICLR, 2023.
[33] Ning Ding, Xingtai Lv, Qiaosen Wang, Yulin Chen, Bowen Zhou, Zhiyuan Liu, and Maosong Sun. Sparse low-rank adaptation of pre-trained language models. In EMNLP, 2023.
[34] Damjan Kalajdzievski. rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:23/2.03733 2023.
[35] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. In ICMI,, 2024.
[36] Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez de Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry in low-rank adapters foundation models. In ICML, 2024.
[37] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics. In NewrlPS, 2024.
[38] Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. In ICMI., 2024.
[39] Yuanhe Zhang, Fanghui Liu, and Yudong Chen. One-step full gradient suffices for low-rank fine-tuning, provably and efficiently. arXiv preprint arXiv:2502.01235. 2025.
[40] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In NewrlPS, 2024.
[41] Hanqing Wang, Zeguan Xiao, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficiet lim finetuning. In NAACI, 2025.
[42] Kerim Büyükakyü.. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.01775. 2024.
[43] Yibo Yang. Xiaojie Li. Zhongzhu Zhou, Shuaiwen Leon Song. Jianlong Wu, Liqiang Nie, and Bernard Ghanem. Corda: Context-oriented decomposition adaptation of large language models. In NeurlPS, 2024.
[44] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In NewrIPS, 2024.

---

[45] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki Markus Asano. Vera: Vector-based random matrix adaptation. In ICLR, 2024.
[46] Tim Detimers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Imms. In NeurlPS, 2023.
[47] HuggingFace Team. Transformers: State-of-the-art natural language processing. In EMNLP, 2020.
[48] Yuren Mao, Yuhang Ge, Yijiang Fan, Wenyi Xu, Yu Mi, Zhonghao Hu, and Yunjun Gao. A survey on lora of large language models. Frontiers of Computer Science, 2025.
[49] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.
[50] Pengeheng Hc, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deherta using electra-style pre-training with gradient-disentanlled embedding sharing. In ICLR, 2023.
[51] Black Forest Labs. Flux. https: // githbb. com/ black forest- labs/ fluz 2024.
[52] Zhengbo Wang. Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? In ICLR, 2025.
[53] Longhui Yu. Weisen Jiang. Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In ICLR, 2024.
[54] Karl Cobbe, Vineet Kosaraju. Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jemry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 2021.
[55] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Openeodeinterpreter: Integrating code generation with exccution and refinement. arXiv preprint arXiv:2402. 14658, 2024.
[56] Qinkai Zheng, Xiao Xia, Xu Zou, Yuxiao Dong, Shan Wang, Yufei Xue, Zihan Wang, Lei Shen, Andi Wang. Yang Li, et al. Codegeex: A pre-trained model for code generation with multilingual evaluations on humaneval-x. In KDD, 2023.
[57] Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry. Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 108.0000000 07732, 2021.
[58] Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ec- Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Lee. LLM-tadpters: An adapter family for parameter-efficient fine-tuning of large language models. In EMNLP, 2023.
[59] Alex Wang, Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: multi-task benchmark and analysis platform for natural language understanding. In EMNLP Workshop, 2018.
[60] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024.
[61] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021.
[62] Lianmin Zheng, Wei-Lin Chiang. Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-hench and chathot arena. In NewrlPS, 2024.
[63] Zonghao Guo, Ruyi Xu, Yuan Yao, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. In ECCV, 2024.
[64] Tero Karras, Samuli Laine, Miik Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In CVPR, 2020.
[65] Robin Rombach, Andreas Blattmann. Dominik Lorenz, Patrick Esser, and Björ Ommer. High-resolution image synthesis with latent difffusion models. In CVPR, 2022.
12

---

[66] Zicheng Zhang, Yinglu Liu, Congying Han, Tiande Guo, Ting Yao. and Tao Mei. Generalized one-shot domain adaptation of generative adversarial networks. In NeurIPS, 2022.
[67] Zicheng Zhang, Bonan Li, Xuecheng Nie, Congying Han, Tiande Guo, and Luoqi Liu. Towards consistent video editing with text-to-image diffusion models. In NeurIPS, 2023.
[68] Bonan Li, Zicheng Zhang, Songhua Liu, Weihao Yu, Xinchao Wang, et al. Top-down compression: Revisit efficient vision token projection for visual instruction tuning. arXiv preprint arXiv:2505. 1194,, 2025.
[69] Ai Han, Junxing Hu, Pu Wei, Zhiqian Zhang. Yuhang Guo, Jiawei Lu, and Zicheng Zhang. Joyagents- rl: Joint evolution dynamics for versatile multi-llm agents with reinforcement learning. arXiv preprint arXiv:2506. 19846, 2025.
[70] Menglin Yang. Jialin Chen, Yifei Zhang, Jiahong Liu, Jiasheng Zhang. Qiyao Ma, Harshit Verma, Qiannu Zhang. Min Zhou, Irwin King, Low-rank adaptation for foundation models: A comprehensive review. arXiv preprint (tt::550100066. 2024.
[71] Pengjie Ren, Chengshun Shi, Shiguang Wu, Menggi Zhang, Zhaochun Ren, Maarten de Rijke, Zhumin Chen, and Jiahuan Pei. Mini-ensemble low-rank adapters for parameter-efficient fine-tuning. In ACL, 2024.
[72] Yinqiao Li, Lingi Song. and Hanxu Hou. Loran: Improved low-rank adaptation by a non-linear transforma- tion. In EMNLP Findings, 2024.
[73] Yibo Zhong, Haoxiang Jiang, Lincan Li, Ryumei Nakada, Tianci Liu, Linjun Zhang, Huaxiu Yao, and Haoyu Wang. Neat: Nonlinear efficieet adaptation of pre-trained models. arXiv preprint arXiv:2410.01870. 2024.
[74] Yeming Wen and Swarat Chaudhuri. Batched low-rank adaptation of foundation models. In ICLR, 2024.
[75] Shih-Yang Liu, Chien-Yr Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decoppoeed low-rank adaptation. In ICML. 2024.
[76] Sebastian Raschka. Practical tips for finetuning Ilms using lora. https: // magazine.
sebastianraschk.. com/ p/ pwsptital--tss--aaaaeeep-pp--see for ftw.r/fffenssnng--. 1mms 2023.
[77] Yang Lin, Xinyu Ma, Xu Chu, Yujic Jin, Zhibang Yang, Yasha Wang, and Hong Mci. Lora dropout as a sparsity regularizer for overfitting control. arXiv preprint arXiv:2404.09611 2024.
[78] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-effiiiet low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.00 033003 2023.
[79] Klaudia Balazy, Mohammadreza Banaci, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint 27564 2024.
[80] Yixiao Li, Yifan Yu, Chen Liang, Pengeheng He, Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. Loftq: Lora-fine-tnning-wwar quantization for large language models. In ICL,R 2024.
[81] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In AISTATS, 2010.
[82] Fabian Paischer, Lukas Hauzenberger, Thomas Schmied, Benedikt Alkin, Mare Peter Deisenroth, and Sepp Hochreiter. One initialization to rule them all: Fine-tuning via explained variance adaptation. arXiv preprint arXi::2410007170, 2024.
[83] Uijeong Jang. Jason D. Lee, and Ernest K. Ryu. Lora training in the NTK regime has no spurious local minima. In ICMI. 2024.
[84] Sadhika Malladi, Alexander Wettig, Dingli Yu, Danqi Chen, and Sanjecv Arora. A kernel-based view of language model fine-tuning. In ICML, 2023.
13

---

A Related Work
Advances in Low-Rank Adaptation. The growing scale 2
64
65 of large pre-trained
models across various domains urgently demands efficient fine-tuning methods [88
T9
20
21 22
66 67
LoRA 7 has garnered significant attention leveraging low-rank structures to
represent weight updates, with subsequent studies expanding its foundations and applications [70 48 Existing improvements LoRA primarily focus on three directions:
The low-rank constraint creates bottlenecks when learning complex features. Recent approaches
enhance expressiveness through iterative stacking of LoRA modules [27
28
24
29]
high-order
matrix operations 23 [0O and customized architectural designs [7] 26 72
73
74
32
75
The non-convex landscapes pose challenges for numerical optimization and hyperparameter
tuning [30]. Researchers have explored to improve scaling factor [3 76 learning rate [35].
dropout rate 777), optimizers [30
38 39
44], and initialization strategies 40
37
39
Recent works reduce computational and memory overhead through freezing parameters designing more compact adapters
77 36
45
79 25 and parameter quantization 40 46
80
Knowledge-Driven Now-Rank Initialization. Proper initialization critically influences neural net-
work training outcomes [1 49
The standard LoRA (7) initializes its low-rank matrices with random noise and zeros, referred to as "Noise & 7ero" scheme, demonstrated to hinder convergence. Recent
advances address this limitation by leveraging knowledge from pre-trained weights 40. 41 42 [2]
or task-specific data 139
43 44 Most prominent approaches involve Singular Value Decomposi-
tion (SVD), which allows flexible control over matrix rank. As a pioneer, PISSA [40] initializes
LoRA weights using principal singular components of pre-trained matrices, aligning adaptation directions with the most significant parameter variations to accelerate convergence. Subsequent propose driaen strategies that incorporate domain knowledge into adapter construction. For instance, LoRA-GA [44] aligns low-rank gradient directions with full fine-tuning counterparts during initialization. While these methods boost performance and retain the core LoRA structure, their theoretical effects on optimization dynamics remain unclear. More importantly, they less pluggable than LoRA, requiring extra computational pipelines and storage for SVD buffers.
works [43
44 39 82
Optimization Dynamics LoRA. LoRA exhibits inherently nonlinear and non-convex optimization dynamics that complicate theoretical analysis [30]. Most existing theoretical studies are limited to simplified and idealized scenarios, such as the lazy-training regime [83 S4 and infinite-width
limit 335 37 A common goal across existing studies involves ensuring feature learning stability to prevent unstable or collapsed training dynamics. For instance, (34 proves that improper scaling factors induce gradient collapse in high-rank adapters, proposing modified scaling mechanisms to stabilize forward and backward propagation. Recent work [35] 36 37 reveals critical asymmetries in LoRA: The two low-rank matrices exhibit divergent distinct impacts on optimization trajectories. This asymmetry motivates (35] to employ distinct learning rates for each matrix. For SVD-based initializa- tion, (39] investigates principal components derived from single-step full fine-tuning gradients [44), providing convergence guarantees yet under restrictive assumptions, Moreover, such analyses still fail to explain the efficacy of alternative approaches like weight-driven initialization (40]
The intricate nature of LoRA and its numerous advancements motivates us to seek a streamlined principle explaining its empirical success and guiding practical applications. We identify update magnitude amplification as a key mechanism, unifying seemingly disparate elements, such as scaling factors 34 [4], initialization strategies 440 44), and learning rates [35], into cohesive perspective.
B Lower Bound on Representation Error
Proposition (Lower Bound on Representation Error). Consider the function class of LoRA- parameterized linear models:
H ((W + aBA)x B RuXr, ARRRN,,AEE Brxm ,I[A S M1, sM)).
Let R(f) = E(z.g)~D((([(( [[(f(z), v) denote the expected regression loss under data distribution D. with squared error loss {(f(x),)) = lly fxx)]l Define f" arg min, R(f) as the globally optimal predictor and fix = arg min R(f)) as the optimal predictor within H. With loss of generality, we consider the optimal predictor is linear, i.e., (x) W* x. Assume that the input covariance matrix is positive definite, i.e., ED Ep [xxx ] 0. and the magnitudes is limited,
14

---

satisfying or orvnnMMM < I WollE. Then, the representation error is strictly positive and
lower bounded:
R(F+) R(f*) 2 Amin (Ep) (IW" Wollr - -ar > 0,
where Amin (-) denotes the smallest eigenvalue of the matrix.
Proof.
Due to the linear learnability, we have that y(a) (x) = W*r. x. For any W, we have that
R(W) EnnD [[[[ - W' "=r=]"
EnnD (W- NW-WW -W*) ((W - ==)))²
(13)
Ir ((W - -W - -W
Amin (ED) wl
The last inequality comes from Amin (Ep) > O.
Due to v/A] & M1, v/B]= & M2, we have that
|| S rrmMi ||]] rrM2. Further, for any A, B H. we have that
|W + aBA - W
|Wo - WWI|F ||o:BAllr
V IW M
IWo - W' - -alAA|FF BlIF |BI|
(14)
IWo W*|lF arv mmm,,
0.
Therefore, we have that
R(f*) R(f")
min MA]SMM
R(Wo + aBA)
2 Amin (Ex)
min
W + aBA - W
2 Amin (Ed)
(l - M2 0.
The first inequality comes from Eq.
13 and the last inequality comes from Eq. 14
C Proof of Main Theorems
C.1 Proof of Parameter Scaling Equivalence
Proposition (Parameter Scaling Equivalence). For LoRA layers defined in Eq. ① consider decomposing the scaling factor a = a' A where a', a.A. O E R*. Under the commonly used optimizution frameworky with negligible numerical errors, the following purametrization schemes exhibit dynamical equivalence throughout training: For all iterations t 2 0, AW(i) LORA = (W)(t) LoRA
and ((tt)
W-lt) LoRA where A(t), Bit) and ((() represent the re-parameterized versions.
LoRA
Original
SGD
Adam
Representation
aBAr
a' BÁx
a' BÁz
Initialization
(0)
Ama. ((0)
Bay
1 Amm, 5000 ar Bas
( a Amu, (00) az Bun
Learning Rates
nA
0,
ni a'MA. OBBi
nA asna,
Proof. Given LoRA's weight decomposition WI.RA aBA where > 0, the gradient updates follow:
Aw LoRA ((B(++1) ((+++))+++))))))) ((0+)))(00A99
(15)
with learning rates 7)4, MB for parameters A and B, respectively. We define reparameterized parame- ters À = a A, A, B B, and a' a'= = The LoRA projection becomes:
WI,RA = aBA = a' BA.
(16)
This transformation preserves the functional form while redistributing scaling factors.
15

---

Proof for SGD
Under initial conditions:
( A
(17)
B(0) an Binit-
(18)
Gradients for reparameterized parameters:
VAL= V V = L'V
VWL,
(19)
(20)
With learning rates "i a'ITAA, 11B 7 and 1 - 0:
1(++) = A( - aAAaV V 2L(9)
(21)
= =a (an T
(22)
-a (A (B)(((() (
(23)
= aa (4") - NAV.NAA AL ((+1
(24)
(25)
Similarly for B:
B(++1) aB [B9 7B8 VBL BBB())
(26)
These equations propagate through all > 0 via mathematical induction. Compare weight increments:
AW²(t)
LoRA = a' (1( q(++1 B(9 A
(27)
('aa ((+]) - A())
(28)
(29)
Aw((t) LORA'
(30)
Proof for Adam Adam maintains exponential moving averages (m, v,) for gradients. Under
parameter scaling P = kP, gradients transform V kV pL. The momentum terms inherit scaling factors:
m = 3imp + (1 PL
(31)
Up = B20p + (1- (2)
(32)
The pratical gradients for Adam:
VtP mp/ V
(33)
=B8mp ...]/ Bzvp
(34)
When setting 0) for alleviating numerical errors, have VtP ViP. Setting np knr
cancels scaling factors, preserving update magnitudes. With learning rates Π = a 7g = Q
and
1(++1) = 7 aATA V I,(0
(35)
aa1A(1) CATA (())
(36)
= (1141+11)
(37)
Similarly for B. These equalities propagate through all > via mathematical induction.
16

---

C.2 Proof of Proposition 2
For the convenience, we split the proof of Proposition 2 into two parts.
Proposition 3 (Parameter Magnitude Dynamics).
Consider LoRA parameters A and B(t) updated A (0,00²²), B o Vw NN, of D), ([ V 0. Under these conditions, the parameter norm T [
with learning rate 1/- Assume:
and B(A V VAL [9)]
vector [E|| (+) [[:.], Elll
evolve as the following linear dynamical system (left) with
closed-form solution (right):
+
14 -
e1+1 =
1+
7B
7A 0
et
El =
7B
2
X
X1
7
LA
X1+
2
eu-
(38)
2
2
where TA mm 2. 7B nnf of and x== =1 VATB- Similarly, the parameter magninudes
vr EI evolve linear dynamical system:
V: I
71
(39)
(YA
Proof. Following the gradient descent update rule, the parameter dynamics are:
A A: A
(40)
B B, BT.
where the gradient relationships VAL= = (V L) and VBL = (VWL)A are derived from the LoRA architecture. Expand the Frobenius norm for A4+1:
| 11:- = || Ar - (A, V V. + 1/2 V L.
(41)
Taking expectations (using independence (A,, V [xL,) 0):
E||A++112 E|A.1 2 + 1/ 2 B V A 1.
(42)
Similarly, for Bi+1:
E| B+ 13 B| + B RLL
(43)
Expand the norm of V = B L):
| = Tr((V II
B).
(44)
Taking expectations (using gradient entry independence):
E V A Tr (B (VV = moj B
(45)
where W LV
mo In follows from the i..d assumption gradient entries.
For = L)A
E V B Tr (AE (VwL))AT) no7.11
(46)
Substitute gradient norms results:
E A || - Ar 13--
-/² B₁
(47)
E B 11
E
B, ? no? E Arll?
Define parameters and state vector:
E A
YA mo
BB
noo
=
E Be
(48)
The recursive system becomes:
1
YA
0
7A
)
et+1
7B
1
e,
=
+
BB
)e.
(49)
Similar results for i, can be obtained using the same proof techniques.
17

---

Proposition 4 (Linearized Dynamics Approximation), For sufficiently small learning rate n, the closed-form solution admits the first-order approximation:
))
[[A01
[o + tYB
Vi V
(1+t)
TAA
B 0
Vu.
where V.
vBB(0]
B + toYr
(50)
This yields the update variance expansion under small-n regime:
wlW/o LORAl se ki 1
(51)
with y - =1 k1 r(mr]A + 1²) and k2 rmmny/ [/
Proof. For a sufficiently small learning rate n, the product YA7B = mnn of is very small. Hence, the cigenvalues
t = VTABB
(52)
can be approximated via first-order Taylor expansion. Thus, for small 4, the closed-form solution for can be approximated as
7B
I+t
Vo.
(53)
0
By definition, we set
v v
To3 + tBB
(54)
L
[
o + Yoo
which agrees with the first-order expansion of the linear system.
D Analyzing LoRA-GA from Magnitude Principle
Proposition 5 (LoRA-GA initialization maximizes low-rank gradient magnitude). Given a gradient matrix W L Raxm with SVD decomposition Ww = USV the optimal rank-r matrices A Brxm and B E &aar that maximize the Frobenius norm of the gradients a L and L
P in Eq. 2 under the constraints || and |B|| = r. are given by:
A' = 1,m
B²=U..r
(55)
where Verrr and U. contain the first right and left singular vectors V respectively.
Proof. Let G = Vw Rmxm and write its SVD G U. with singular values o 2 02
2 0. From the LoRA gradient expressions we have
VAI. = BTG,
VBL= GAT.
Note that VAL does not depend on A and V BL does not depend on B, hence the two maximization problems decouple.
(i) Maximizing
VB 17P w.F.l under || = r. Let A' == AV. Then ||A' = |||| = r and IGA T 11: = ( 112 = (15 11 = (r (A
Form the Lagrangian L(A', = (A)) yields the stationarity condition A'S² = XA' Thus the rows of A'' lie in the eigenspaces of S To maximize the trace, one places the Frobenius norm onto the coordinates corresponding to the largest diagonal entries of $2 i.c. the first singular values. Taking A"* = [,, [] attains the maximum
( TA - r). Taking derivative w.r.t. A'
GA
max a
and A = T V (up to orthonormal transformations within the chosen r-subspace).
(ii) Maximizing V I w.t.t B under ||B|] = Analogously let B' = UT B. Then
|B
G tr (TB' S²
18

---

and the same Lagrange argument gives that the optimal choice concentrates norm on the first coordinates, producing
G
o
max
|
==1
and one may take B* US
Hence the stated solutions A V. For This completes the proof.
and B U.er are optimal, with the above maximal values.
E Details on Key Formulas
E.1 Weight Update Magnitude
v v(Aw(()) LORA ra?n?
+ (VVV
Proof.
Given the LoRA parameter update rule:
AW LoRA am
VBL (()) + 011))
(56)
where we retain first-order terms 17 under small learning rate assumption.
Assumptions:
(A1) Independence: B TL A VgL 11
(A2) Zero-mean initialization: B[B,,] = TE| [A =0 0
(A3) Spatial homogeneity: var [By] r|B], varl =
For entry (m, n) in AWr(t)
LORA"
var [(W)) mm)] R 222r2var var
Bmnk (V (LL))n + (V (VnL)mi ARn)
(57)
Σ L(VBL)mAenn ))
122 var Σ Bmu (V (VaL)an
+ var
(58)
By assumptions (AI)-A33):
var
Σ Bmx ( VAL)ann =
Σ var BariBmaarr (aaas/aar((AA (VAL)) V
(59)
rrliBlu
(60)
Similarly:
var Σ (VBL)mbAen rr(VBL]u(A)
(61)
Combining these terms:
1 (l LORA R a²12 (
(62)
= ra?nn
+ +VV
(63)
where vBB), u{A] inherit their magnitudes from initialization scheme, and v v(VI. terms reflect task-specific loss landscape characteristics.
D
19

---

E.2
Spectral Concentration Factor
Let the singular values of the pretrained weight matrix W be (s)) R[W] top-r singular values as
Define the average of the
E2 [s]
8/
(64)
1=1
and the average of the squares of all singular values as
R W
ERiwl[s*] R[W]
Σ 5
(65)
Because the square function convex, Jensen's 's inequality implies
E, [s] < E,
(66)
with equality if and only if all 8; (for i r) are equal. Moreover, since lower singular values generally contribute less to the overall energy, as increases the value of E, [s] decreases relative to ER[w]s²", making pfr) monotonically decreasing function of r.
E.3 Variance of AsvD and BSVD
Recall that the PiSSA initialization is given by
AsvD = S.T.V.
(67)
where S, is diagonal matrix containing the top-r singular values W. Assuming that the columns of V (and similarly, U) form an orthonormal basis, the variance of V.,r (taken elementwise) is approximately v[Vyr] z 1 (or 1 for U), since for an orthogonal matrix the energy is uniformly distributed. Thus, the variance of ASVD can be expressed as:
v(AsvD)= Tr(y STV/TVV STV VS.
E 81 = [s[ss)
(68)
mr To connect this with the variance of W, note that
mrr
R(
[W]
Σ
3
(69)
mn
Thus, we can relate [s] r/[] via the spectral concentration factor pir]. Incorporating a factor of to account for the dimensions of AgvD, we obtain:
n p|r] v[W]
v( v(AsvD)= V R[]
(70)
similar argument, with the roles m and n interchanged, leads to the expression for v v(BsvD).
E.4 Spectral Gain Factor
The dynamics of the LoRA weight update variance are captured by an expression of the form:
"[ kiyt + 2
(71)
where k is the linear evolution rate. Substituting the variance expressions derived for AgvD and BSvD into the dynamical system (see Eq. (6). we obtain:
h plr] (m + n)
v[W].
(72)
R[
Defining
Qirl A p|r]r
(73)
R[W]'
this expression becomes:
Qr](( n)p[W]
(74)
with the constraint < Q(1) < 1. When v[W]- (min(1 1/n)), the factor Q(r) effectively quantifies the amplification of the weight update magnitude due to the spectral initialization.
20

---

Proof. Recall that
p|r| _ ( 8;)
(75)
R[W] E [
and the spectral gain factor is defined as
Q11l plr] R[W]
(76)
Let m R[W]. By Jensen's 's inequality (or by the Cauchy-Schwarz Schwarz inequality), we have
(77)
Therefore,
plr] v
(78)
m
2i==
since
Substituting this bound into the definition of Q(r)
yields
Qr) = p|r]
m r =1 1.
(79)
This completes the proof that Q(r) < 1.
L
Further Clarifications
In this section, we provide additional details regarding the experimental setup for theoretical validations and justify the core assumptions underlying our propositions.
F1 Clarification on Figure 2 Experimental Setup and Optimizer Dynamics
The experiments illustrated in Figure 2 were conducted in controlled environment to isolate and validate our theoretical claims regarding hyperparameter equivalence and magnitude dynamics. The setup uses 5-layer MLP with an intermediate dimension of 400, where LoRA modules are trained to mapping for randomly generated synthetic data. This simple setting effectively tests the fundamental fitting capabilities of LoRA.
Figure 2 (a) is specifically designed empirically validate Proposition I reproduced with clearer separation in Figure We deliberately use two different optimizers, SGD for the first 2,500 steps and Adam thereafter, to demonstrate that when the hyperparameter product a' held constant, LoRA exhibits equivalent training trajectories regardless of the optimizer. This equivalence is confirmed by the overlapping loss curves. Furthermore, the norm difference || // AW((t) LaRAl l between the baseline and other equivalent settings remains near zero, confirming that the learned weights themselves evolve identically.
F.2 Clarification Figure 3 Direct Relationship to PiSSA
The goal of Figure 3 is to visualize the source of magnitude gain in spectral initialization methods, for which
PISSA 40)
representative example. The link is direct:
The solid colored curves for the spectral concentration factor (p(r)) and spectral gain factor (Q(r) are calculated directly from the SVD of the pretrained weights. This is precisely the mechanism that PiSSA employs for initialization. Therefore, these curves illustrate the inherent magnitude dynamics of -initiiiial model.
The plots reveal two key insights from analysis of PiSSA. First, the plot of p(r] shows that spectral energy is highly concentrated in the top singular values, explaining why PiSSA is particularly effective low ranks. Second, the plot of Qir quantifies the significant magnitude gain that PiSSA (colored lines) achieves over the naive "Noise & Zeros" baseline (white dotted line). This is precisely the gain that LoRAM is designed mimic efficiently without performing SVD.
21

---

1.0
Norm Difference
1.5
Norm Difference
5O
Loss
0.4
0.4-
0.2
1000
2000
3000
4000
5000
2000
3000
0000
5000
SGD dynamics under diffeerent lyperparamelers
(b) Adam dymamics under different hyperparameters
Figure 6: A detailed view of the validation of Proposition separating the SGD and Adam opti- mization phases. Each curve represents model with unique hyperparameter combination. The norm difference (right axis) aggregates the Frobenius norm discrepancies between the baseline model (black curve) and others across all layers. The results show that diverse hyperparameter can produce identical optimization trajectories, confirming our theoretical equivalence framework.
G Further Analysis on the Interplay of Initialization, Learning Rate, and Performance
Our primary experiments were conducted under unified hyperparameter configuration to ensure fair and controlled comparison. However, different initialization strategies may achieve optimal performance under varying hyperparameters, particularly the learning rate. To provide more nuanced understanding, we conducted supplementary analysis investigating the performance of LoRAM, PiSSA [4), and MiLoRA
41 under different learning rates.
Specifically, we compare the results from our original experiments using moderate learning rate 10 10-5) with new results obtained using a higher learning rate (2 x 10 10-4), which aligns with the setting used in the original MiLoRA paper. The detailed results for ranks 16 and 128 are presented in Table 5 and Table O respectively.
Table 5: Performance of PiSSA, LORAM, and MiLoRA with different learning rates at rank 16.
Learning rate
2 x 10-5
2 x 10-4
Method
PiSSA
LoRAM
MiLoRA
PiSSA
LoRAM
MiLoRA
GSM8K
37.68
40.32
29.70
52.46
53.29
46.62
MATH
5.16
5.30
4.18
8.80
8.92
6.18
HumanEval
18.37
18.92
14.69
24.47
25.62
17.14
MBPP
28.62
28.83
27.23
31.22
31.74
28.65
Commonsense
73.72
75.19
67.90
77.02
79.11
76.67
Table 6: Performance of PiSSA. LoRAM, and MiLoRA with different learning rates rank 128.
Learning rate
2
10-5
2 10-4
Method
PiSSA
LoRAM
MiLoRA
PiSSA
LoRAM
MiLoRA
GSM8K
51.48
51.12
39.81
58.37
59.28
54.66
MATH
7.04
7.25
5.18
11.46
10.76
9.20
HumanEval
21.62
22.03
20.39
26.81
29.95
28.02
MBPP
31.07
31.53
29.95
36.54
37.80
33.35
Commonsense 77.28
77.81
74.29
67.09
74.23
79.01
The results reveal several key patterns:
At lower rank 16), all methods generally benefit from the higher learning rate, showing improved performance across tasks. This suggests that in low-rank settings, larger learning rate can enhance convergence speed.
a higher rank (r 128), a clear divergence emerges. While most methods still improve, PiSSA exhibits a notable performance degradation the Commonsense dataset. Its training loss
22

---

curve in this high-rank, high-LR configuration plateaus early, suggesting that the amplified updates overshoot the effective descent direction.
These findings align perfectly with our magnitude principle. PiSSA's use of principal singular components leads to stronger initial magnitude amplification, which necessitates smaller optimal learning rate to maintain stability. Conversely, MiLoRA, which initializes with smaller minor singular components, can benefit from larger learning rate. This analysis reinforces our claim from Section 4.3) "Magnitude scaling should applied conservatively at higher ranks, since larger ranks inherently amplify updates."
This supplementary analysis underscores that the optimal learning rate is intrinsically linked the magnitude scaling introduced by the initialization method.
H Efficiency Analysis: Computational and Memory Overhead
A primary motivation for LoRAM is to achieve the performance gains of spectral initialization methods while preserving the computational efficiency and minimal memory footprint of the original LoRA framework. In this section, provide detailed comparison of the theoretical and practical overhead associated with LoRAM, LoRA, and PiSSA.
H.1 Theoretical Complexity
The theoretical time and space complexities for the initialization phase of each method are summarized in Table 7 LoRA's overhead is minimal, stemming from random matrix initialization. LoRAM maintains this same linear complexity, as its deterministie DST basis is generated efficiently through vector multiplication. In contrast, PiSSA's complexity dominated by the SVD of the pretrained weight matrices, a computationally intensive operation. Furthermore, PiSSA requires storing both the original and decomposed components, effectively doubling the space requirement compared LoRA and LoRAM.
Table 7: Theoretical initialization time and space complexities of LoRA, LoRAM, and PiSSA.
Method
LoRA
LoRAM
PiSSA
Time Space
(mr + ar) (r(mm + n))
Omr + nr) O(r(m + n))
Omin((m² mn²)) 0(2r(m +n))
H.2 Practical Performance
We empirically validated these complexities by measuring the practical initialization time and memory usage while fine-tuning LLaMA-7B on an 8-GPU server. The results, shown in Table S highlight two distinct workflows for PiSSA:
1. Pre-processing: This approach first computes and saves the residual model after subtracting the low-rank approximation. While the initialization itself is faster, this process incurs a substantial one-time storage cost, requiring over 12.5 GB to store the residual weights in addition to the LoRA adapter weights.
2. Direct Workflow: This method computes SVD on-the-fly, which avoids the large storage overhead. However, it significantly slower (often taking over 10 minutes in our setup) due to known bottlenecks related to CPU-based SVD computations before GPU transfer.
As shown in the table, LoRAM's practical performance is nearly identical to that of the standard LoRA implementation, demonstrating its exceptional efficiency successfully eliminates the significant time and space overhead introduced by SVD-based methods like PiSSA, confirming its utility lightweight yet powerful initialization strategy.
I Scope and Limitations of the Magnitude Principle
While our work establishes the magnitude of weight updates as a fundamental driver of LoRA's performance, it is crucial to clearly define the scope and limitations of this principle. Our central
23

---

Table 8: Practical initialization time and space cost of LoRA, LoRAM, and PiSSA on LLaMA-7B.
Metric
LoRA
LoRAM
PiSSA (pre-process)
PiSSA (direct)
Time -16)
1.36s
0.95s
51.32s
10min
Time (r = =128)
4.23s
2.19s
57.79s
> 10min
Time 512)
8.50s
5.19s
110.31s
10min
Space (r = -16)
152MB
152MB
12.5GB + 152MB
305MB
Space (r 128)
1.2GB
1.2GB
12.5GB 1.2GB
2.4GB
Space (r 512)
4.8GB
4.8GB
12.5GB 4.8GB
9.6GB
claim is that magnitude plays primary, not universal, role in the success of various hyperparameter- tuning strategies. The principle provides a coherent and predictive lens for unifying these strategies, rather than suggesting that magnitude is the sole determinant of all outcomes.
The applicability and effects of magnitude scaling are subject to several important considerations:
Interplay with Other Hyperparameters: As indicated by analysis in Proposition the magnitude principle is not limited to initialization magnitudes (o2, [ alone. is intrinsically linked other key factors, including the LoRA rank (r), the learning rate (1)) and the task task-dependent gradient variance (o2). The final performance a result of the complex interplay among all these components.
Interaction with Rank Size: A key finding, highlighted in our ablation studies and consistent with our theoretical analysis, is that the benefits of magnitude scaling are not monotonic. We empirically observe that the performance gains from increased magnitude scaling tend to diminish and can even reverse at higher ranks. This is because larger ranks inherently amplify updates, as predicted by Proposition
2 suggesting that magnitude should be scaled more conservatively in high-rank settings.
On Optimality: Our work aims to identify and validate update magnitude as core mechanism influencing LoRA's training dynamics, thereby demystifying the success of methods like PiSSA. We do not claim to have identified an optimal magnitude scaling strategy. Determining the optimal magnitude for different models, tasks, and layers remains a challenging and important direction for future research.
In summary, the magnitude principle serves as powerful analytical tool for understanding and designing LoRA-basee methods, but its application should be contextualized within the broader optimization landscape.
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction have clearly reflected the theoretical and empirical contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: We have declared the limitations in the article, including some theoretical assumptions, and have cited relevant literature to show that these assumptions are commonly used in the field. We have also included additional technical limitations at the end of the main paper.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model seeesspcciiiiaiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer: [Yes]
Justification: We have provided full proofs for each proposition in the Appendix, and for the important formulas in the paper, we also give the corresponding derivation in the Appendix.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All benchmarks, as well as the experimental setup, are from or based on work that has been open-sourced. We have given relevant references in the paper and have given specific hyperparameter settings. Moreover, our proposed LoRAM does not require tuning parameters, as illustrated in Algorithm 1.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
26

---

reproducible, they 5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Since we have conducted extensive experiments on several datasets, providing detailed scripts is intricate. The code will be made available upon acceptance of the article.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co
public//uides//odeee http:///oooeuummmsiiooooooooo
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ //nips.ccppblll... wwcppublic/ggudeeeoo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All benchmarks, well the experimental setup, from based work that has been open-sourced.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have provided the variance of the test results, and also the range of fluctuations in the training curves.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
27

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Our experiments are based on server with 8 H800 GPUs. The timing of
experiments is diverse depending on the datasets, while there will be no differences between
LoRAM and other approaches on the same dataset, since they do not modify the training process.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
We have carefully checked and followed the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper presents general fine-tuning technique that not specific to actual application scenarios.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
28

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: The paper poses such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We have properly cited the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
29

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
30

---

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non ntn-standard components.
Please refer to our LLM policy hattap///aaurrss.cc///oofereeees for what should or should not be described.
31


Reviews:
[{"review_id": "sJtU3BjNRj", "reviewer": "Reviewer_PbY3", "summary": "LoRAM introduces a magnitude-aware initialization scheme for LoRA that replaces SVD with deterministic orthogonal bases scaled by pretrained weight statistics. While preserving the standard LoRA training pipeline, this approach achieves comparable or superior performance with significantly reduced computational overhead. By reinterpreting spectral methods through the lens of update magnitude, the paper provides a unifying perspective on various hyperparameter strategies and highlights magnitude scaling as a key lever for efficient adaptation.", "questions": "Please refer to weakness", "limitations": "Please refer to weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "(+) Provides a unified theoretical framework linking LoRA hyperparameters through update magnitude. This perspective allows for a coherent interpretation of prior LoRA variants such as RsLoRA and LoRA+, which represents a clear theoretical contribution.\n\n(+) The ablation studies are appropriately designed and help isolate the effects of key components in the proposed method.\n\n(-) The authors reinterpret SVD-based initialization—commonly understood as preserving principal directions—as primarily contributing to performance through magnitude amplification. This is an insightful and valid perspective. However, it is important not to dismiss the spatial or spectral interpretations of SVD entirely. The original LoRA paper analyzes the similarity between $W$ and $\\Delta W$ in weight space, and later methods such as MiLoRA and CorDA explicitly leverage the structure of principal and minor components to mitigate forgetting and enhance transferability. Notably, MiLoRA achieves strong performance even when modifying only small singular components, making its underperformance in this paper somewhat surprising and potentially in need of further clarification.\n\n(-) The overall narrative may slightly overstate the universality of the magnitude principle. In practice, the benefit of increasing update magnitude likely depends on factors such as rank size, model architecture, and task characteristics. A more nuanced discussion of when magnitude scaling is helpful—and when it may result in diminishing returns or instability—would strengthen the overall framing.\n\n(-) As shown in Figure 5(a), the improvement in convergence speed may not be as pronounced as implied, and appears moderate relative to other methods.\n\n(-) While LoRAM achieves consistent performance gains in Tables 1 and 2, the improvements over strong baselines are relatively modest. The results are solid and reliable, but in some cases the margin may be too small to be considered practically significant.", "was_revised": true, "final_justification": "While the theoretical motivation and related issues have been adequately addressed in the rebuttal, I remain unconvinced that the underperformance of MiLoRA and other baselines has been fully explained. Although the authors have discussed possible causes—such as differences in optimal learning rates—it is still unclear whether these baselines were appropriately tuned for each task. The appendix does not provide sufficient detail to verify this. As a result, this concern remains unresolved.", "rebuttal": "**General Response**\n\nWe sincerely thank Reviewer PbY3 for the thoughtful and constructive feedback. We greatly appreciate your recognition of the strengths of our work, including the **insightful theoretical framework, the appropriate experimental design and the solid performance**. \n\nRegarding the concerns raised, we categorize them into two types of issues:\n1. **Differences in Interpretation**: We feel that concerns on the dismissal of spectral interpretation (Q1) and the overstatement of the magnitude principle (Q3), may arise from differences in interpretation of our work. We hope to further clarify these points to mitigate concerns.\n2. **Method Performance**: Concerns regarding findings such as the underperformance of MiLoRA (Q2) and the modest improvements (Q4) are valid points that we believe are addressable through further explanation. \n\nWe apologize for any confusion caused by points we intended to convey, but which may not have been sufficiently clear. We will clarify and address these issues in the revision.\n\n---\n**Q1. Concern on Dismissal of Spectral Interpretations**\n\nWe agree that spectral interpretation is significant. In fact, we do not dismiss the spectral view, but rather provide a deeper and mechanistic understanding of its effectiveness through the lens of the magnitude principle.\n\n1. **Relationship to prior spectral analyses**. The original LoRA paper [1] (Sec 7.3) observed that the adapted weight $\\Delta W$ hugely amplify features absent in $W$. **Our work extends this by explicitly modeling the training dynamics behind this amplification**, further revealing how LoRA's inherent low-rank structure can weaken this expected effect, and crucially, how subsequent hyperparameter tuning further amplify update magnitudes to enhance performance. \n   * **Spectral values**. Our findings suggest that in most spectral initialization schemes, the major performance gains are primarily driven by the magnitude amplification resulting from scaling the singular values.\n   * **Spectral bases**. As shown in lines 257-259, we also acknowledge that the spectral matrices from full gradients (in LoRA-GA) have significant impacts. We prove its effectiveness stems from maximizing gradient magnitudes. This directly aligns with our magnitude principle. We will highlight this finding in Sec 2.3.\n2. **Relationship to Minor-component methods**. Methods like MiLoRA and CorDA tune minor components to preserve key information and mitigate forgetting, while there still lacks a theoretical framework to explain their effectiveness. **Our study, within the scope of enhancing convergence and performance, showcases that leveraging minor components typically yield less magnitude amplification.** This suggests their benefits in avoiding overfitting and providing a more stable solution might stem from this inherent lower magnitude scaling.\n\n---\n**Q2. About the underperformance of MiLoRA**\n\nIn summary, PiSSA generally outperforms MiLoRA at typical fine-tuning learning rates (e.g., 2e-5), though this advantage diminishes or reverses with larger learning rates.\n\n**Intuitive explanation:** MiLoRA directly uses the smallest singular values and bases, which, being close to zero, can induce a vanishing problem.\n\n**Practical guidance:** Our theory predicts that PiSSA's stronger magnitude amplification from principal singular values necessitates a smaller optimal learning rate compared to MiLoRA. We demonstrate this as follows:\n\n1.  **Consistent Findings with PiSSA**. Our experiments, conducted at a moderate learning rate (2e-5), align with the PiSSA paper's setups and findings that \"principal components consistently lead to reduced training loss and enhanced accuracy\" (Figure 8, PiSSA paper [2]).\n2.  **Experiments with Large Learning Rates**. To better understand MiLoRA's comparative performance, we conducted experiments at a higher learning rate (2e-4), matching that used in the MiLoRA paper. The tables below reveal several interesting patterns:\n    1.  **At a lower rank (r = 16)**, all methods benefit from the increased learning rate (2e-5 to 2e-4), showing improved performance across tasks, indicating higher learning rates can enhance convergence in low-rank settings.\n    2.  **At a higher rank (r = 128)**, a divergence emerges. While most datasets still show improvement, PiSSA exhibits a notable performance drop on the Commonsense dataset, falling below its rank-16 performance and MiLoRA's.\n    3.  **Training curves**. PiSSA's loss plateaus early and fails to decrease at high-rank, high-LR configurations, suggesting amplified updates overshoot the effective descent step, preventing loss reduction.\n\nThese findings are **fully consistent with our claims in Section 4.3 (lines 262–263)**: \"Magnitude scaling should be applied conservatively at higher ranks, since larger ranks inherently amplify updates.\"\n\n\n\n\nTable 1: Results of PiSSA, LoRAM and MiLoRA with different learning rates ($r = 16$).\n\n|               | PiSSA | LoRAM | MiLoRA | PiSSA | LoRAM | MiLoRA |\n|---------------|-------|-------|--------|-------|-------|--------|\n| Learning rate | 2e-5  | 2e-5  | 2e-5   | 2e-4  | 2e-4  | 2e-4   |\n| GSM8K         | 37.68 | 40.32 | 29.70  | 52.46 | 53.29 | 46.62  |\n| MATH          | 5.16  | 5.30  | 4.18   | 8.80  | 8.92  | 6.18   |\n| HumanEval     | 18.37 | 18.92 | 14.69  | 24.47 | 25.62 | 17.14  |\n| MBPP          | 28.62 | 28.83 | 27.23  | 31.22 | 31.74 | 28.65  |\n| Commonsense   | 73.72 | 75.19 | 67.90  | 77.02 | 79.11 | 76.67  |\n\nTable 2: Results of PiSSA, LoRAM and MiLoRA with different learning rates ($r = 128$) .\n\n|               | PiSSA | LoRAM | MiLoRA | PiSSA | LoRAM | MiLoRA |\n|---------------|-------|-------|--------|-------|-------|--------|\n| Learning rate | 2e-5  | 2e-5  | 2e-5   | 2e-4  | 2e-4  | 2e-4   |\n| GSM8K         | 51.48 | 51.12 | 39.81  | 58.37 | 59.28 | 54.66  |\n| MATH          | 7.04  | 7.25  | 5.18   | 11.46 | 10.76 | 9.20   |\n| HumanEval     | 21.62 | 22.03 | 20.39  | 26.81 | 29.95 | 28.02  |\n| MBPP          | 31.07 | 31.53 | 29.95  | 36.54 | 37.80 | 33.35  |\n| Commonsense   | 77.28 | 77.81 | 74.29  | 67.09 | 74.23 | 79.01  |\n\n---\n**Q3. Concern on the Overstatement of the Magnitude Principle**\n\nWe fully agree that the applicability and limitations of the magnitude principle deserve careful discussion and should not be overstated. Below, we clarify our intent and how we have already addressed some of these concerns:\n\n1. **Primacy, Not Universality**.\n   1. As stated in our title, our central claim is that **magnitude plays a primary—not universal—role** in the success of various LoRA hyperparameter-tuning strategies. We do **not** argue that magnitude alone determines all outcomes, but rather that it provides a coherent and predictive lens across settings. \n   2. As indicated in lines 134–135, the magnitude principle is **not limited to initialization magnitudes** $\\sigma_A^2$ and $\\sigma_B^2$,  but also includes rank $r$, learning rate $\\eta$, and gradient variance $\\sigma_L^2$ (task-dependent). \n   3. Our LoRAM focuses on **the impact of initialization magnitude in PiSSA**, serving as a concrete evidence of the magnitude principle.\n\n2. **Self-Consistency and Validation**.  We believe our statements are consistent and validated by extensive experiments and ablation studies, as recognized by other reviewers.\n3. **Discussion of Limitations**.\n   We agree that further discussion will strengthen the paper. We have already addressed the following in the current version:\n   1. **Interaction with Rank Size**: As shown in lines 260–263, we empirically observe that the benefits of magnitude scaling **diminish and even reverse** at high ranks, aligning with our theoretical Proposition 2.\n   2. **Architectural and Task Diversity**: We deliberately evaluated our approach across a variety of architectures and tasks to avoid conclusions based on narrow settings and to support the generality of our findings.\n   3. **On Optimality**: We noted in lines 137–138 and 276–280 that our work does not aim to identify an optimal strategy. Rather, we aim to reveal the key factors that influence performance, leaving this challenging topic to future work.\n\nIn our revision, we will further clarify these points and make the scope and limitations of our claims more prominent.\n\n\n---\n**Q4. Moderate Improvement in Convergence and Performance Over Strong Baselines**\n\nWe appreciate the reviewer’s observation. We would like to clarify the broader context and significance of this result:\n\n1. **Simplicity Grounded in Theory as a Core Contributions**.\n   Our main contribution is not to claim dramatic acceleration in convergence, but to demonstrate that **comparable or better performance can be achieved with significantly lower complexity**. In this light, matching PiSSA’s convergence speed is itself a meaningful contribution as recognized by reviewers.   \n\n2. **Consistent Trend as Theoretical Validation**.\n   The fact that LoRAM tracks PiSSA’s performance is a validation of our central theoretical claim. Since LoRAM mimics the magnitude gain deterministically, the similarity in convergence trajectories supports our hypothesis and underscores the diminishing role of complex SVD-based initialization once magnitude is properly controlled.\n3. **Potential for Further Gains**.\n   As shown in our ablation studies (Sec 4.3), LoRAM can be readily combined with other methods—such as RsLoRA—to achieve further improvements. Despite this, we would like to emphasize that **our work not only aims to push advancements**, but also to **identify and validate the fundamental mechanisms**. \n\nWe believe the ability to **consistently deliver solid performance using a theoretically-grounded, explainable and resource-efficient approach** makes a significant contribution. \n\n---\n[1] Lora: Low-rank adaptation of large language models. In ICLR, 2022.\n\n[2] Pissa: Principal singular values and singular vectors adaptation of large language models. In NeurIPS 2024."}, {"review_id": "oJJpBTFR8N", "reviewer": "Reviewer_eWC2", "summary": "This paper investigates the important problems of training dynamics and initialization in Low-Rank Adapters (LoRA). The authors develop an initialization (LORAM) based on deterministic orthogonal bases with controllable magnitude scaling parameters. This developed theoretically by treating the evolution of LoRA parameter magnitudes as a linear dynamical system and observing the magnitude changes during training. A correspondence between different optimization schemes is developed based on a scaling factor applied to adapters. \n\nThe authors demonstrate that the LORAM initialization achieves competitive performance to leading LoRA schemes across a range of Natural Language Generation, Natural Language Understanding, and Vision-Language tasks with superior performance achieved in several individual benchmarks. They additionally provide a number of additional analyses and ablations for their method, including the use of a 'tracking mode' that backsolves scaling parameters for LORAM based on target initialization schemes.", "questions": "My current view is that this is a decent paper which presents interesting theoretical analysis along with well benchmarked and ablated experiments, but which would benefit substantially from improving clarity (especially for figures). In the current form many of the details are not immediately clear which detracts from the analysis and results. My suggestion would be that simplifying figures and sections by splitting less essential details to the supplementary materials could make this a good paper. \n\nIn addition to the details raised in the Strengths and Weaknesses, there are a few questions which it would be useful to clarify: \n\n* Could the authors clarify Figure 2a and 2b? As described in the Strengths and Weaknesses section these figures are difficult to digest (e.g. the meaning of the SGD region and the Adam region, and the interpretation of the loss and norm differences). My interpretation of b) is that it shows how changing the initial variance of the A and B matrices changes the total magnitude across training, but I am not certain \n* Could the authors clarify Figure 3? It appears to either show how the spectral concentration and gain factor change over training or across the dimension of the layers? In addition the insert in the p[r] row seems to refer to the early Q[r] values. \n* There are a few numbers that are used in the paper (e.g. L126-130, sigma_A = 1/20, k1 = 1/16) where it isn't immediately clear how these were selected and the how it connects with the theoretical analysis \n* In the Abstract the method is described as a `Basis and Basis' initialization scheme, however this isn't clearly defined later\n* The linear dynamical system approach used within Proposition 2 is an interesting and potentially novel way to analyze LoRA. As section 2.3 and the supplementary proof don't directly include citations, could the authors briefly discuss the papers they feel most relevant to the approach used in Proposition 2.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well motivated. Despite the many variants of LoRA there are still many open questions to its training dynamics and optimal initialization. The paper approaches this in an ambitious way: by seeking to develop a correspondence between the training dynamics under different optimization schemes and adapter magnitudes, the authors aim to unify understanding for a number of hyperparameter and design choices involved in LoRA. \n\nThe main strength of this paper is in its theoretical insight. In particular, the linear dynamical system approach for LoRA is interesting. In addition, the use of a deterministic orthogonal initialization (in particular the Discrete Sine Transform) is a very interesting approach which may motivate further exploration into other deterministic bases. \n\nThe experimentation and ablations are thorough and appropriate. Experiments appear to support the central idea that appropriately scaling parameters can boost LoRA convergence, and results show competitive or improved performance on a wide range of benchmarks across modalities (NLG, NLU, Vision-Language). \n\nIn terms of written description, the manuscript is relatively clear - with the related literature presented especially well. \n\nThere are a few important areas where the clarity is reduced: in particular, the figures are very information dense (for example, Figure 2a attempts to combine insights into multiple [a', a_A, a_B] hyperparameter ablation, optimizer [SGD, Adam], losses, and norm differences into a single double-axis figure with overlapping convergence lines [which I believe is meant to indicate they achieve equivalences as per Proposition 1]). This makes the figures very difficult to parse as a reader. Figure font sizes are often much too small to be visible when printed and when zoomed in. Adding in labels for axes (e.g. on Figures 2, 3, 5) would similarly improve the figures. \n\nTables 1, 2, and 3 are presented clearly and show that LORAM is competitive with leading methods, and achieves superior performance for several tests. Presentation for Table 4 is relatively dense (showing ablations for different scaling factors, bases, benchmark initialization methods, backsolved `tracking' initialization, etc) and could benefit from being split into separate tables in the supplementary materials to improve clarity. Bold figures could help make this comparison and the takeaway message clearer. \n\nMy general impression is that this paper has a lot of information that it wants to convey in a short amount of space. This unfortunately makes it relatively difficult to digest as a reader. Spacing these ideas out by focusing on the key message / simplified clear figures in the main paper, and moving less critical ablations to the supplementary materials would greatly improve this paper's potential impact. (E.g. - separating a few of the ablations in Figures 2 and 3, and providing sufficient space to describe this in the supplementary). \n\nSimilar issues appear in some of the mathematical descriptions (very densely written, with some overloading of notation [e.g. it is not clear how B^{(0)} differs from B_{init}], and many details feeling a little \"cramped\" to fit in the full technical details of ablations, e.g. logarithmic gain factor, tracking mode, and the DST basis). Including many technical components in the main paper can mean that there is less space available to develop, motivate, and connect each idea clearly. In contrast, I found the proofs in Supplementary B were relatively clear and well developed. Focusing in on the most important message (things that directly support the LORAM initialization and magnitude analysis), and leaving fuller description in the supplementary would make it much easier to digest (e.g. details of the LORAM initialization are provided relatively late in the method on page 6, and felt difficult to directly connect to the earlier formal linear dynamical system analysis and update magnitude scaling).", "was_revised": true, "final_justification": "While I have remaining concerns on the clarity of the figures / technical details I have raised my review to a borderline accept. The reasons for this are the technical contribution of the paper (which I believe is interesting and relevant), the depth of experimentation, and the author's detailed responses to all reviewers during the review process in which they have sincerely worked to clarify and validate their claims. The authors have committed to improving the clarity of the manuscript and figures for any camera-ready version. I wish them the best on this project and thank them for their engaged and responsive discussion.", "rebuttal": "**General Response**\n\nWe sincerely thank Reviewer eWC2 for the thoughtful and constructive feedback. We are especially grateful for the recognition of our contributions across **theory, methodology, and experiments**, including the linear dynamical analysis, magnitude-based perspective, use of deterministic bases (e.g., DST), and extensive empirical validation.\n\nWhile we appreciate the description of our work as \"decent,\" we understand that the main concerns lie in presentation clarity, rather than the technical soundness:\n\n1. **Visual Density**: We acknowledge that some figures and tables are dense, which resulted from our attempt to present comprehensive findings within the page constraints.\n2. **Information Overload**: With the large number of original contributions, we focused on precision, particularly in the theoretical sections, which may have compromised simplicity.\n3. **Structural Suggestions**: We value the reviewer’s structural suggestions and agree that more space is needed to clearly develop, motivate, and connect each idea. We will revise the organization for greater clarity.\n\nWe regret any confusion and respectfully ask for your understanding. With the opportunity to revise (e.g., in a camera-ready version with an additional page), we are confident we can present the content more clearly.\n\n---\n**Q1. Confusion about Notation $B^{(0)}$ and $B_{\\text{init}}$**\n\nWe appreciate the reviewer’s careful review.  $B^{(0)}$ is used as a **general notation** to represent the initial state of the LoRA matrix $B$ at the training step $0$,  while $B_{\\text{init}}$ refers to the **specific assignment** based on the chosen initialization method, which we specify more precisely as follows:\n\n* **In Proposition 1**, $B_{\\text{init}}$ refers to any matrix of the appropriate shape for generality.\n* $B_{\\text{init}}$ could also take values from three specific schemes:\n\n  1. Zeros: Used in the \"Noise & Zeros\" initialization of conventional LoRA.\n  2. $B_{\\text{SVD}}$ (Eq. 7): Denotes the SVD-based initialization, as used in methods like PiSSA.\n  3. $B_{\\text{LoRAM}}$ (Eq. 11): Represents the initialization using Discrete Sine Transform (DST) bases, as proposed in our LoRAM scheme.\n\n---\n**Q2. Connection between Update Magnitude Scaling (Sec 2.2), Dynamical System Analysis (Sec 2.3), and LoRAM Initialization (Sec 3)**\n\nWe establish the following logical progression:\n\n1. Section 2.2 explains \"Why **weight update magnitude** ($\\nu[\\Delta W _ {\\text{LoRA}}^{(t)}]$) matters,\" showing that various hyperparameters converge to a common effect: they control the update magnitude, which directly influences loss reduction.\n2. Section 2.3 demonstrates \"How **initial variance** ($\\nu[A^{(0)}]$ and $\\nu[B^{(0)}]$) explicitly affect **weight update magnitude** ($\\nu[\\Delta W _ {\\text{LoRA}}^{(t)}]$).\"\n3. Section 3 illustrates \"How **spectral initialization** and **LoRAM initialization** influence **initial variance** ($\\nu[A^{(0)}]$ and $\\nu[B^{(0)}]$), and in turn, affect **weight update magnitude** ($\\nu[\\Delta W _ {\\text{LoRA}}^{(t)}]$).\"\n\n\n---\n**Q3. Clarification of Figure 2**\n\nThese figures illustrate the training dynamics of low-rank adaptation under different initialization settings and optimizers, aiming to provide an intuitive validation and interpretation for the training dynamics and magnitude evolution.\n\n1. **Figure 2a**: This figure shows the training dynamics of LoRA under two optimizers: SGD (for the first 2.5k steps) and Adam (thereafter). This setup validates Proposition 1, which states that when the product $\\alpha' \\alpha_A \\alpha_B$ is held constant, the training trajectories of Adam and SGD should be equivalent. This equivalence is reflected in the overlapping loss curves (black, blue, green, red). To further verify that $\\Delta W _ {\\text{LoRA}}^{(t)} = \\Delta \\tilde{W} _ {\\text{LoRA}}^{(t)}$ across training dynamics, we compute the norm difference $||\\Delta \\tilde{W} _ {\\text{LoRA}}^{(t)} - \\Delta W _ {\\text{LoRA}}^{(t)} ||_F^2$ between the weights learned by the two optimizers and the baseline weights (black curve), accumulated across all layers. This confirms that not only the loss, but also the learned weights, evolve similarly across settings.\n\n2. **Figure 2b**: Your interpretation is correct. This figure shows how varying the initial variance of the $A$ and $B$ matrices affects the update magnitude $\\nu[\\Delta W]$ during training. As shown, larger initialization variance leads to larger parameter updates and faster convergence, which supports our core claims in Proposition 2. Additionally, as noted in Eq.(5), the magnitudes of $A$ and $B$ remain relatively stable throughout training, further emphasizing the importance of their initial values.\n\n---\n**Q4. Clarification of Figure 3**\n\nFigure 3 illustrates how spectral concentration ($\\rho[r]$) and the spectral gain factor ($Q[r]$) vary across different rank configurations (x-axis) and model layers (as shown in the legend).\n\n1. The plot of $\\rho[r]$ shows that $\\rho[r]$ decreases as rank $r$ increases. According to Equation (9), this indicates that top principal components contribute most significantly to initialization magnitude and optimization dynamics.\n\n2. The plot of $Q[r]$ generally suggests that as rank increases, magnitude scaling becomes more significant, with higher-rank configurations leading to larger overall update magnitudes. For the \"Noise and Zeros\" initialization, $k_1 = r(m\\frac{1}{m^2}+0) = \\frac{r}{m}$, resulting in constant growth (represented by the white line). In contrast, PiSSA produces diverse curves (colored lines) due to varying singular values across models and layers. We approximate the spectral gain factor using a logarithmic function (shown for three cases) to moderate the enhancement of magnitude scaling.\n\n3. The inset plot highlights low-rank configurations, which are most commonly used in practice. We will add arrows or labels to make this relationship clearer.\n\n---\n**Q5. Clarification of Numbers (e.g., $\\sigma_A = 1/20$, $k_1 = 1/16$)**\n\nWe apologize for not providing a clearer derivation of these values in the main text.\n\n1. **For $\\sigma_A = 1/20$**: This value is based on the Kaiming initialization principle. As described in lines 100-102, we conducted a controlled experiment using a 5-layer MLP with an intermediate dimension of 400. According to Kaiming initialization, the weight variance is set to $1/\\sqrt{n}$ (where $n$ is the input size) to maintain proper variance. In this case, $\\sigma_A = 1/\\sqrt{400} = 1/20$.\n2. **For $k_1 = 1/16$**: This value is derived based on the experimental setup where the LoRA rank $r = 25$. Taking into account the chosen rank and intermediate dimension, $k_1$ is calculated as: $ k_1 = r(m\\sigma_A^4 + n\\sigma_B^4) = 25 \\left( 400 \\times \\frac{1}{20^4} + 400 \\times 0^4 \\right) = 1/16$\n\n\n\n---\n**Q6. Clarification of \"Basis and Basis\" Initialization Scheme**\n\nAs illustrated in Figure 1 and Algorithm 1 of our paper, our method employs a deterministic orthogonal **basis** (e.g., DST) for each LoRA matrix. This approach contrasts with the conventional \"Noise & Zeros\" initialization, where matrix $A$ is typically initialized with random noise, and matrix $B$ with zeros.\n\n---\n**Q7. Papers relevant to the approach used in Proposition 2**\n\nWe sincerely thank the reviewer for their positive assessment of our approach as **\"interesting and potentially novel.\"** Our thinking was informed by a broader line of research that models the dynamics of neural network training. We were particularly inspired by the general philosophy in work:\n\n1. **Gradient Flow :** Which models parameter training trajectories as solutions to differential equations on a loss landscape [1]. \n2. **Neural ODEs :** Which provide a powerful framework for thinking about deep networks as continuous-time dynamical systems [2].\n\n The modeling of linear dynamical systems is novel and based on the unique update structure of LoRA:\n* The gradient for the matrix $A$: $\\nabla_A L= B^\\top (\\nabla_W L)$ is a function of the current state of $B$.\n* Conversely, the gradient for $B$: $\\nabla_B L = (\\nabla_W L) A^\\top$ is a function of the current state of $A$.\n\nThe optimization creates a feedback loop that can be expressed conceptually as $A_{t+1} = A_t + f(B_t)$ and $B_{t+1} = B_t + g(A_t)$, naturally simplified into the linear dynamical system. The remaining analysis is based on our unique efforts.\n\n---\n[1] Gradient flows: in metric spaces and in the space of probability measures. In Basel: Birkhäuser Basel. 2005\n\n[2] Neural ordinary differential equations. In NeurIPS 2018"}, {"review_id": "e50E7gBXXs", "reviewer": "Reviewer_Eyvp", "summary": "This paper focuses on the parameter-efficient fine-tuning of large language models. The authors investigate the initialization methods of LoRA to improve its convergence and performance. Specifically, they theoretically re-interpret the success of spectral initialization in terms of weight magnitude and unify the hyperparameter for LoRA tuning as the magnitude regulation optimization. Furthermore, the authors propose an initialization method called LoRAM, which serves as a strong baseline for replacing spectral initialization across multiple benchmarks.", "questions": "According to the above weaknesses, I expect the authors to (i) demonstrate the efficiency of the proposed method, (ii) provide the reliance of their assumption.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**:\n1. This paper provides a comprehensive theoretical analysis of the correlation between the LoRA training dynamics and the weight magnitude.\n2. This paper interprets the effectiveness of spectral initialization and unifies the hyperparameters of LoRA in terms of weight magnitude.\n3. The proposed LoRAM is easy to implement and improves LoRA in multiple scenarios.\n4. The proposed LoRAM is evaluated on multiple tasks and achieves comparable performance to spectral initialization methods.\n5. This paper is well-organized and easy to follow.\n\n**Weaknesses**:\n1. The proposed LoRAM is designed to avoid the additional matrix decomposition and storage overhead of spectral initialization methods. Nevertheless, the current experiments only evaluate LoRAM's performance in comparison to spectral methods. Including comparisons of computational costs and memory usage would further demonstrate the effectiveness and efficiency of the proposed approach.\n2. In the experiment of Figure 2, the pre-trained model and dataset should be described.\n3. Since the authors assume that gradients $\\nabla_W L^{(t)}$ follow a normal distribution in Proposition 2, it would be insightful to investigate whether this assumption holds reliably across different models, layers, and downstream tasks.\n4. The LoRAM is designed for improving naive LoRA, which might lack the flexibility to accommodate the current practice of incorporating additional operations into LoRA [1, 2, 3].\n\n[1] Expanding Sparse Tuning for Low Memory Usage, NeurIPS'24\n\n[2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning, NeurIPS'24\n\n[3] Parameter-Efficient Fine-Tuning with Discrete Fourier Transform, ICML'24", "was_revised": true, "final_justification": "The rebuttal of authors addresses my concerns about the LoRAM's memory usage and computation requirements. Thus, I have decided to maintain my positive rating.", "rebuttal": "**General Response**\n\nWe sincerely thank Reviewer Eyvp for the thoughtful and constructive feedback. We appreciate the positive evaluation of our work’s **quality, clarity, significance, and originality**. We are particularly grateful for the recognition of our strengths, including the **theoretical analysis** of LoRA training dynamics, the **unification of LoRA hyperparameters** in terms of weight magnitude, and the **practical contributions** of LoRAM in enhancing LoRA performance across tasks.\n\nWe also value the reviewer’s constructive comments on areas for improvement and address them below.\n\n___\n\n**Q1. Computational Cost and Memory Usage Comparison**\n\nWe appreciate the reviewer's question regarding computational cost and memory usage, a crucial aspect for practical deployment.\n\n1. **Theoretical Complexities** (Table 1): We provide the theoretical computational and memory complexities, where $m$ and $n$ represent the matrix dimensions, and $r$ represents the rank.\n\n   * LoRA: Its complexity primarily stems from random matrix initialization.\n   * LoRAM: Its complexity is driven by the initialization of the DST matrix, computed efficiently via vector multiplication (see provided code).  The magnitude gain factor $\\beta$ can be computed explicitly as  $\\nu[\\Phi _ n\\Phi _ m^\\top]\\approx r\\nu[\\Phi _ n]\\nu[\\Phi _ m^\\top]=\\frac{r}{mn}$ for the DST matrix.\n   * PiSSA: Its complexity is dominated by SVD, which requires significant storage for the initial state and result matrices, leading to a space complexity of $O(2r(m+n))$.\n\nTable 1. Theoretical initialization time and space complexities of LoRA, LoRAM, and PiSSA.\n\n|       | LoRA        | LoRAM       | PiSSA                 |\n|-------|-------------|-------------|-----------------------|\n| Time  | $O(mr+nr)$    | $O(mr+nr)$    | $O(\\min(m^2n, mn^2))$ |\n| Space | $O(r(m+n))$ | $O(r(m+n))$ | $O(2r(m+n))$          |\n\n\n**2. Practical Performance** (Table 2): We tested training LLAMA-7B on an 8-GPU setup. LoRA and LoRAM exhibited similar practical performance. PiSSA has two implementations:\n\n* **Official Default Pre-processing**: Saves the residual model and LoRA results, leading to high space usage. This is the recommended approach.\n* **Direct Transformers and PEFT Workflow**: Computes the SVD matrix on the CPU before transferring it to the GPU (due to discrepancies with fast SVD across different GPU devices). This results in significantly slower performance, consistent with **known bottlenecks mentioned in PiSSA’s official GitHub issues**. In our 8-GPU setup with Deepspeed distribution training, the direct process typically took over half an hour. Since we cannot confirm whether the time was accurately measured, we report only the approximate duration.\n\n\nTable 2. Practical initialization time and space cost of LoRA, LoRAM, and PiSSA.\n\n\n|               | LoRA  | LoRAM | PiSSA (pre-process) | PiSSA (direct) |\n|---------------|-------|-------|---------------------|----------------|\n| Time ($r=16$) | 1.36s | 0.95s | 51.32s              | $>$ 10min      |\n| Time ($r=128$)  | 4.23s | 2.19s | 57.79s              | $>$ 10min      |\n| Time ($r=512$)  | 8.50s | 5.19s | 110.31s             | $>$ 10min      |\n| Space ($r=16$)  | 152MB | 152MB | 12.5GB  + 152MB     | 305MB          |\n| Space ($r=128$) | 1.2GB | 1.2GB | 12.5GB + 1.2GB      | 2.4GB          |\n| Space ($r=512$) | 4.8GB | 4.8GB | 12.5GB + 4.8GB      | 9.6GB          |\n\n\nCode 1. A Pytorch Code for Generating DST matrix\n\n```angular2html\nimport torch\n\ndef fast_dst_matrix(m, n, device='cpu', dtype=torch.float32):\n    \"\"\"\n    generate m x n DST-I matrix.\n    \"\"\"\n    transpose = False\n    if m < n:\n        m, n = n, m\n        transpose = True\n        \n    assert n <= m, \"n must be smaller than m\"\n    \n    k = torch.arange(n, device=device, dtype=dtype).unsqueeze(1)  # shape (n,1)\n    i = torch.arange(m, device=device, dtype=dtype).unsqueeze(0)  # shape (1,m)\n    \n    dst_basis = torch.sin((i + 1) * (k + 1) * torch.pi / (m + 1))\n    \n    scale = torch.sqrt(torch.tensor(2 / (m + 1), device=device, dtype=dtype))\n    \n    dst_matrix = (scale * dst_basis).T\n\n    if transpose:\n        dst_matrix = dst_matrix.T\n    \n    return dst_matrix.to(device=device)\n```\n\n---\n**Q2.Model and Dataset Description in Figure 2**\n\n   \nThank you for pointing this out.\n\nAs mentioned in lines 100-103, we conducted a controlled experiment using a 5-layer toy MLP with an intermediate dimension of 400, where the model weights were initialized randomly. The data were generated from random noise (both random input and random output). Specifically, we fixed the random model weights and trained the LoRA module to fit the data mapping from scratch. This setup is simple but effectively tests LoRA's fitting ability. We will add more details in the revision.\n\n\n---\n**Q3. Assumption of Normal Gradient Distribution in Proposition 2**\n\nThank you for this insightful question. We would like to prove this reliability from three perspectives:\n\n1. **Relaxed to zero-mean and independence**.\n   We state the critical properties required for our derivation are that the gradient entries are zero-mean and independent to simplify the mathematical derivation. While the Gaussian assumption is a standard and convenient way to ensure these properties, it is not strictly necessary.\n2. **Empirical Validation**.\n   We conducted an empirical investigation using LLaMA-2 for three NLG tasks and FLUX.1-12B for image customization. We monitored and visualized weight gradients across various layers. Our findings are:\n   * The gradient entries consistently have zero mean, sampled from various layers and at different training stages.\n   * The gradient distributions are Gaussian-shaped, with a single peak, while the variance varies across layers and   remains stable in the middle of training.\n3. **Prior Works**.\n   This assumption is well-justified and widely adopted in the literature. For example:\n   * \\[1] provides extensive illustrations of normal gradient distributions.\n   * \\[2] makes assumption of zero-mean Gaussian distribution throughout the article, inspiring us to adopt this assumption in our work.\n   * \\[3] presents a great work that derives the mean and variance of forward and backward signals in transformer architectures.\n\n---\n\n**Q4. Flexibility of LoRAM in Incorporating Additional Operations**\n\nWe appreciate the reviewer’s insightful point.\n\nIndeed, as you rightly pointed out, LoRAM is designed to improve the original LoRA method and may lack the flexibility to incorporate the great works you mentioned, which are orthogonal to LoRAM's core idea. \n\nAlthough LoRAM does not directly integrate with these methods, we consider that our theoretical analysis could offer some insights into understanding these approaches, making it a good direction for future research.\n\n---\n\n\\[1] Understanding the difficulty of training deep feedforward neural networks.  In AISTATS, 2010.\n\n\\[2] Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In ICCV, 2015.\n\n\\[3] Transformers get stable: An end-to-end signal propagation theory for language models. In ICML, 2024."}, {"review_id": "gHmBv56U5c", "reviewer": "Reviewer_62N4", "summary": "This paper introduces a unifying principle for understanding and improving Low-Rank Adaptation (LoRA), arguing that the magnitude of weight updates is the primary driver of its performance. The authors present a theoretical framework that connects learning rate, scaling factor, and initialization as different means to the same end: regulating update magnitude. Using this, the authors demystify the success of recent spectral initialization methods, positing that their effectiveness comes from amplifying update magnitudes, not from a more abstract preservation of pretrained knowledge.\n\nBuilding on this insight, the paper proposes LoRAM that initializes the LoRA matrices using deterministic orthogonal bases derived from the Discrete Sine Transform and scales them by a factor carefully calculated to mimic the magnitude gains of spectral approaches. Through extensive experiments on a diverse set of language, coding, and vision-language tasks, the authors demonstrate that LoRAM consistently matches or outperforms more complex initialization schemes while retaining the full efficiency and plug-and-play simplicity of the original LoRA.", "questions": "No further questions", "limitations": "The authors have adequately addressed the limitations of their work in a dedicated section. They correctly note that LoRAM mimics, rather than optimizes, spectral magnitudes and that further gains might be found by jointly optimizing magnitude with other hyperparameters like rank and learning rate.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\nThis paper successfully challenges the findings of previous research by proposing an alternative explanation for the improved performance of spectral decomposition PEFT algorithms.\n\nLoRAM displays comparable or slightly superior performance to spectral PEFT algorithms using a simpler strategy.\n\nThe findings are mathematically supported and insights into the training dynamics are presented in Figures 2 and 3.\n\nLoRAM's performance is evaluated across a range of tasks to comfort as to it efficiency\n\nWeaknesses:\n\n(Minor) The ablation studies do show that while magnitude is the dominant factor, the choice of basis vectors (e.g., DST vs. SVD-derived) and the update direction (as influenced by data-driven methods like LoRA-GA) can still lead to minor performance differences outside of magnitude alone. The paper does acknowledge this.", "was_revised": true, "final_justification": "After reading other reviews which all go in the direction of acceptance, I maintain my score", "rebuttal": "We sincerely thank Reviewer 62N4 for the thoughtful and constructive feedback. We are particularly grateful for the recognition of our work's **quality, clarity, and significance**, especially the acknowledgment that LoRAM **successfully** provides an alternative explanation for the improved performance of spectral decomposition PEFT algorithms. We also appreciate the recognition of the **mathematical support and insights** into training dynamics and the evaluation of LoRAM’s performance across various tasks.\n\nWe concur with the reviewer’s observation that, while spectral value plays a key role, the choice of basis vectors and the update direction influenced by data-driven methods like LoRA-GA can indeed contribute to performance differences. We greatly appreciate the reviewer’s positive outlook on our contributions despite the nuances.\n\nWe remain committed to incorporating feedback and engaging in ongoing improvements through future research."}]

Author rebuttal:

Meta-review: The paper introduces a novel method (LoRAM) and a new theoretical perspective that re-interprets spectral methods through magnitude scaling. The proposed LoRAM initialization achieves competitive or superior performance compared to leading LoRA schemes. The claims are supported by a wide range of experiments conducted across diverse tasks. 

During the rebuttal process, most of the concerns over the results and motivation of the paper are adequately addressed by the authors, and two reviewers increased their final scores. However, some clarity issues on experiment results (e.g. MiLoRA) and figures still remain.

Considering the significant technical contribution and the consensus positive reviews, I would recommend the acceptance of this paper. I strongly suggest that the authors to continue to improve the paper's clarity on experimental results and overall representation quality as suggested by the reviewers for final publication.
Decision: Accept (spotlight)