LoRA-One: One-Step Full Gradient Could Suffice for Fine-Tuning Large Language Models, Provably and Efficiently
Authors: Yuanhe Zhang
Fanghui Liu
Yudong Chen
Venue: ICML 2025 oral (2025)
Keywords: low-rank fine-tuning
linear convergence
subspace alignment
Primary area: theory->learning_theory
TL;DR: Our theory shows that one-step gradient of full-finetuning can be sufficient for low-rank fine-tuning, and devises a theory-grounded algorithm for performance imporvement in real-world tasks.

Abstract: This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA) (Hu et al., 2022) in large language models as a case study. We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one-step full fine-tuning gradient. This result suggests that, by properly initializing the adapters using the one-step full gradient, subspace alignment can be achieved immediately—applicable to both linear and nonlinear models. Building on our theory, we propose a theory-driven algorithm, LoRA-One, where the linear convergence (as well as generalization) is built and incorporating preconditioners theoretically helps mitigate the effects of ill-conditioning. Besides, our theory reveals connections between LoRA-One and other gradient-alignment-based methods, helping to clarify misconceptions in the design of such algorithms. LoRA-One achieves significant empirical improvements over LoRA and its variants across benchmarks in natural language understanding, mathematical reasoning, and code generation. Code is available at: https://github.com/YuanheZ/LoRA-One.

LoRA-One: One-Step Full Gradient Could Suffice for Fine- Tuning Large Language Models, Provably and Efficiently
Yuanhe Zhang' 12 while retain the knowledge from pre-trained models. For scalability, we expect that fine-tuning can be conducted with low computation/memory cost, i.e., parameter-efficient fine-tuning (PEFT) (Houlsby et al., 2019; Han et al., 2024).
Fanghui Liu 23 Yudong Chen
Abstract
This paper explores how theory can guide and enhance practical algorithms, using Low-Rank Adaptation (LoRA) (Hu et al., 2022) in large lan- guage models as case study. We rigorously prove that, under gradient descent, LoRA adapters align with specific singular subspaces of the one- step full fine-tuning gradient. This result sug- gests that, by properly initializing the adapters using the one-step full gradient, subspace align- ment can be achieved immediately-apphle applicable to both linear and nonlinear models. Building on our theory, we propose a theory-driven algo- rithm, LoRA-One, where the linear convergence (as well as generalization) is built and incorporat- ing preconditioners theoretically helps mitigate the effects of ill-conditioning. Besides, our the- ory reveals connections between LoRA-One and other pundiient-lignmee--basss methods, helping to clarify misconceptions in the design of such algorithms. LoRA-One achieves significant empir- ical improvements over LoRA and its variants across benchmarks in natural language under- standing, mathematical reasoning, and code gener- ation. Code is available at: https: // /it.bb... com/ Hoan/Yuanhee///e LLRAA-OrR
One typical PEFT strategy is Low-Rank Adaptation (LoRA) (Hu al., 2022), which learns an approximation of the unknown feature shift A by two low-rank matrices A and B with rank i.e. A s AB under the following initialization (denoted by index O):
[Aoli) ~ (N(, [Bol = 0, a >>0. 0. (LoRA-init)
To improve the performance in the downstream tasks, vari- ous LoRA-basee algorithms have been proposed based on, e.g., refined initialization (Li et al., 2025), learning rates (Hayou et al., 2024), efficiency (Kopiczko et al., 2024), and gradient information (Meng et al., 2024; Wang et al., 2024).
A Although LoRA is conceptually simple, its optimization dynamics are inherently nonlinear and non-convex. There few theoretical understanding of its behavior, e.g., optimiza- tion from lazy-training regime (Jang et al., 2024; Malladi et al., 2023; Liu et al., 2025) to non-lazy training regime (Kim et al., 2025) and generalization guarantees in some simplified settings (Dayi & Chen, 2024). It still remains un- clear how (low-rank) gradient updates in LoRA evolve and which subspaces LoRA will converge to. More importantly, given the application-driven nature of LoRA, rigorous the- orctical understanding should not only explain its behavior but also inform practical algorithm design. The goal of this work is to enhance LoRA's empirical performance through theoretically grounded insights. To this end, we address two key questions at the intersection of theory and practice:
1. Introduction
How to efficiently approximate or learn nonlinear models a central question large-scale machine learning, espe- cially in the era of large language models (LLMs) (Brown et al., 2020; Thoppilan et al., 2022). Fine-tuning (Dodge al., 2020) aims make LLMs perform well on new tasks
Q1: How characterize low-rank dynamics of LoRA and the associated subspace alignment in theory?
Deppatmennt Statistics, University of Warwick, UK. 2 Department of Computer Science, University of Warwick, UK. 2 Centre for Discrete Mathematics and its Applications (DIMAP), University of Warwick, UK. 'Department of Computer Sci- enees, University of Wisconsin Wisconsin-Madiionn USA. Correspondence to: Fanghui Liu <fanghui liu warwick.ac.uk>>.
Q2: How can our theoretical results contribute al- gorithm design for LoRA in practice?
1.1. Contributions
In this work, we theoretically investigate the behavior of gradient descent (GD) update of LoRA parameters (A,, B,) and identify the subspaces they align with. Our theory iden- tifies the optimal initialization strategies in the perspective
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
brgett
0.8
LolA--ne 1.422
Mran prodia
33-
(Randon Initf
One-step Gradient
42
By
93
48-
Chocrd
coor
-13 -04
(a) Alignment pipeline
(b) Alignment validation
(c) Loss landscape
Figure 1: (a) Illustration of the alignment behavior of LoRA with the certain singular subspaces. (b) The mean principle angles within each layer class is estimated for the alignment of fine-tuning T5 base model (Raffel et al., 2020) on MRPC using LoRA. The principal angle measures the distance between the projection matrices of the top-r (LoRA rank) singular subspace of the LoRA matrices and that of the one-step gradient. Note that the principal angles are approximately at initialization and then decrease for alignment. (c) Comparison of trajectories among full fine-tuning (Full FT), LoRA, and LoRA-One OeRA-Oneun under gradient descent. We use a two-layer neural network pretrained on odd-labeled data and then fine-tune on even-labeled MNIST data. Experimental details are presented in Appendix G.I.
of subspace alignment, and we find that it also performs well on some real-world datasets. We term this initialization as spectral imitialization, which leverages the information of one-step full gradient, leading to the theoretical grounded algorithm, LoRA-One. This algorithm incorporated into several architectures achieves promising performance on natural language processing (NLP), reasoning tasks when compared to LoRA and its variants. Our contributions from theory (see Table 1 for summary) to practice are:
ii) Global convergence and generalization guarantees: Under spectral initialization, continuing gradient descent (GD) updates for (A,, B.), we further establish the linear convergence rate of |A, - All for both linear and non- linear models. This linear rate, however, is sensitive to the condition number <(A) of A, leading to unsatisfactory con- vergence performance if A is ill-conditioned. To address this issue, rigorously show that adding preconditioners into the GD update eliminates the dependence on the condi- tion number; see Section 3.2 and Section 4, respectively.
i) Alignment and algorithm design principles: We start by analyzing LoRA for fine-tuning a multi-output linear model. Denoting one-step gradient of full fine-tuning as G we prove that the gradient update aligns A, with the left top-r" singular subspace of G while B, always stays in right top-r" singular subspace w.r.t. Go shown in Fig. I(a), where r" is the rank of A. This alignment phe- nomenon is also empirically verified in real-world fine- tuning tasks, see Fig. 1(b). Based on the alignment re- sults, we compute the singular value decomposition (SVD) of G UG Sg:S VT. The alignment can be directly achieved the certain initialization strategy, termed as spec- tral initialization
Moreover, our theory aims to clarify certain misunderstand- ings in prior algorithm designs. Specifically, identifies the correct subspace for alignment and highlights potential limitations of previous LoRA variants based on gradient alignment-such as LoRA-GA (Wang al., 2024); see the discussion in Section 5.
iii) Performance improvement in numerical and read- world datasets: Guided by our theory, the spectral ini- tialization strategy (Spectral-init) leads to our theoretically grounded algorithm, LoRA-One. As shown in Fig. I(c), our numerical results demonstrate that LoRA-One's trajectory is close to the full fine-tuning (Full FT) and obtain lower loss than LoRA.
Ao = v (Gc
G1/21
G=
[
(Spectral-init)
We conducted experimental comparisons between LoRA- One and standard LoRA-based algorithms across various NLP benchmarks, including natural language understand- ing (NLU), mathematical reasoning, and code generation tasks. For instance, using only (Spectral-init), it takes just one second on some NLU tasks to achieve performance comparable to LoRA which requires tens of seconds. On the HumanEval benchmark, LLaMA 2-7B fine-tuned with LoRA-One achieves a score of 28.66, outperforming stan-
By = v7 1/2 [1ar]
c+
[g
where y is a tuning parameter. By (Spectral-init), we the- oretically ensure that B, AllF is sufficiently small at beginning, see the theoretical results for linear models in Section 3.2 and nonlinear models in Section 4, respectively. demonstrates the sufficiency of using one-step full gradi- ent, which can be numerically verified on several real-world benchmarks, serving as the algorithm principle.
2

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Table 1: Main results in the main text and appendix from subspace alignment to global convergence.
Model
Results
Algorithm
Initialization
Conclusion
Theorem 3.1
GD
(LoRA-init)
Subspace alignment of B,
Theorem 3.2
GD
(LoRA-init)
Subspace alignment of A;
Linear
Theorem 3.3
GD
(Spectral-init)
|| Bo - AlF is small Linear convergence of ||A. A||r
Theorem C.17
GD
(Spectral-init)
Theorem C.21
Precondition GD
(Spectral-init)
Linear convergence rate independent of k(A)
Nonlinear
Theorem 4.2
Precondition GD
(Spectral-init)
Linear convergence rate independent of k(A)
dard LoRA (25.85) by 2.81, while maintaining almost the same time and memory costs.
Matrix Sensing under Gradient Descent: Since LoRA performs fine-tuning using a Burer-Monterio factorization, it admits similarities with matrix sensing problems, includ- ing the symmetric matrix problem with r = r (Li et al., 2018) and r 2 r* (Stöger & Soltanolkotabi, 2021); asym- metric problem with r 2 r" (Soltanolkotabi et al., 2023; Xiong et al., 2024). Regarding initialization, small initial- ization (Ding et al.. 2022) and spectral initialization (Ma et al., 2021) help convergence with theoretical guarantees, which is applied to LoRA under certain specific settings (Xu et al., 2025). Besides, adding preconditioner (Zhang et al., 2021; Tong et al., 2021; Xu et al., 2023; Zhang et al., 2023; Giampouras et al., 2024; Zhu et al., 2024) is beneficial to solve the problem of ill-conditioned ground truth matrix.
Notations For matrix A, let |[Allap denote its operator and |p its Frobenius norm. Let @ denote the Hadamard (i.e., entrywise) matrix product. We use I, to denote the IRnxx-valued identity matrix. The notation UA denotes the left singular matrix of the compact SVD of A and UA.1 denotes the corresponding orthogonal complement. Simi- larly, VA denotes the right singular matrix of A and VA denotes its orthogonal complement. Let U,- (A) denote the left singular subspace spanned by the r" largest sin- gular values of A and (A) denote the left singular subspace orthogonal to U,- (A). Similarly define V,, (A) and V (A) for the right singular subspace. A complete list notations can be found in Table 5 of Appendix A.
Technically, for the alignment part, our theory leverages some techniques from Soltanolkotabi et al. (2023). However, the symmetrization technique used in prior work cannot be applied to decouple the GD dynamics of (A,, B.), posing a challenge in analyzing their individual spectral behaviors. To overcome this limitation, we develop a novel approach that enables a detailed analysis of the distinct spectral dy- namics of A, and B,, which is one technical contribution of this work. In fact, one-step gradient information has been used in deep learning theory, demonstrating that it allows for feature learning under different stepsizes (Ba et al., 2022; Moniri et al., 2024; Cui et al., 2024; Dandi et al., 2025). Besides, for the nonlinear model part, dynamical analysis are normally based on classical gradient- based algorithm (Damian et al., 2022; Lee et al., 2024) for feature learning. Nevertheless, how such model behaves under low-rank up- dates under (Spectral-init)) is still unclear to our knowledge.
1.2. Related Work
Parameter-Efficient Fine-Tuning (PEFT): LoRA (Hu al., 2022) and its variants have received great attention for downstream applications. The variants of LoRA focus imbalance stepsize (Hayou et al., 2024), initialization using SVD of pre-trained weights (Meng et al., 2024), gradient approximation (Wang et al., 2024; 2025) for better perfor- mance, reducing parameters (Kopiczko et al., 2024) effi- ciency, preconditioned algorithm (Zhang& & Pilanci, 2024) for stability.
In theory, the training dynamics and generalization ability of LoRA are rarely discovered. Based on the empirical evi-
dence of kernel behavior of LoRA in Malladi et al. (2023), the global convergence is given by Jang et al. (2024) for LoRA with rank ((VN) under the lazy training (Jacot et al., 2018) as well as Liu et al. (2025) on PL.* condition. Beyond the lazy training regime, Kim et al. (2025) study the loss landscape of LoRA well as its implicit bias. For general- ization, Dayi& & Chen (2024) derive the sample/time com- plexity by exploring the SGD dynamics of rank-1 LoRA, related to single-index model (Arous et al., 2021). In our work, we study the dynamics of LoRA from the perspective of subspace alignment, which has some overlap with matrix sensing as below.
2. Problem Settings
In this section, we introduce the problem setting of fine- tuning pre- pre-trained linear and nonlinear models with the following assumptions for our theory.
2.1. Basic Assumptions
We consider both linear and nonlinear pre-trained models with multiple outputs and thus matrix parameters (instead
3

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
of vectors), which is consistent with LoRA in practice. Assumption 2.1 (Pre-trained model). For the imput æ e R we denote by W E TRd the known pre-trained parameter matrix. We assume that the pre-trained model can be linear or nonlinear with o--) = max } being the (entry-wise) ReLU activation function.
the compact SVD A US with U Rdxr", VT R and S E RR" xr" The left/right singular subspaces spanned by U and V play important role in our analysis.
We write the downstream data in compact form X 3 _N]" E RXxx and the label matrix Y y W.... UN E RN is generated by either linear or non- linear target functions in Assumption 2.3. We introduce the training based on full fine-tuning and LoRA below.
Spe (a) = [ (o WHIT E I
linear [ W:)T] ER nonlinear
Full Fine-tuning: We consider the following empirical risk minimization with a squared loss
Note that our results can handle large dimension d and k. For fine-tuning, we assume there exists an unknown low- rank feature shift A on W that we aim to estimate.
| XW Y
1
linear,
L(W)
F
(2)
The downstream feature matrix W := W + A admits an unknown low-rank feature shift A E Rdxk where Rank (A) = r" < min(d,kk k).
Assumption 2.2.
2N
o [XW) - Y
nonlinear
where the parameter W can be learned by gradient descent (GD) initialized at W i... Wo : W
This assumption is widely used in the literature on LoRA analysis and matrix factorization (Zhang et al., 2021; Stöger & Soltanolkotabi, 2021; Soltanolkotabi et al., 2023; Xiong al., 2024). Next we assume the following data generation process, i.c., label-noiseless and well-behaved data.
LoRA: It updates two low-rank matrices A E RR B E R k for efficiency with the following empirical risk
X(Wi+ +AB)
linear,
Assumption 2.3 (Data generation process for fine-tuning). Given the unknown W the label y is generated by
B), 2N
| AB))
-
nonlinear
(3)
W R [o(( ]. { fid.
i.d.d SG, linear
which can be minimized using GD with stepsize r) > 0
N(O, Ia) nonlinear
where SG' denotes the probability distribution for isotropic centered sub- Gaussian random vectors. We assume that we have N ii.d training data ( 9. J for fine-tuning.
A A. 7/ (A,: B.) B B. 7V (A, A., B.)
(4)
Note that the nonlinear model can be regarded as a special case of multi-index model (Damian et al., 2022; Abbe et al., 2022; Bietti et al., 2023) and Gaussian data is a common assumption in the analysis of single/muuliiindee (Damian et al., 2022; Lee et al., 2024; Oko et al., 2024). We additionally assume that d < N, which coincides with practical settings of LoRA for LLaMA 2-7b (Touvron et al., 2023) on real-world datasets, e.g., MetaMathQA (Yu et al., 2024) and Sede-Feedback (Zheng et al., 2024), where d 128 4096 and is on the order of 105.
Since the true rank r of A is unknown in LoRA, our results will cover two cases: over-ranked (r 2 r²) and exact-ranked (r = r*).² Our results allow for large d, k while r, r ((1), which coincides with common practice.
models
Optimization and Generalization: We are interested in the error A, All under the LoRA training dynamics. Bounds on this error also imply generalization performance, because the generalization error for a new data (æ, T) sat- isfies Eg | o o(W + A,) x ? ||A,B, - A in the nonlinear setting, with equality in the linear setting.
2.2. Full Fine-tuning and LoRA
3. Analysis of LoRA under Linear Model
Our goal is to efficiently recover A by fine-tuning on the downstream data. Let the complete SVD of E Bdck be
In this section, we establish the alignment between LoRA and one gradient of full fine-tuning. This result guides us to design new strategies for speeding up practical LoRA-based algorithms, which achieve this alignment at initialization.
S* O
0
A= VT = U U1
where Ò E dxd and V E look are the left and right singular matrices, and S- E d is a rank-r* diagonal matrix with nonzero singular values {x))"- It admits
We formally define the negative gradient of full fine-tuning
IIn the matrix sensing/completion literature, they often called over- and exaam--aaaatterizze,, respectively.

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
in Eq. (2) for the linear setting after the first step as
0.0
10
-==-wwL(W3) = N (Y X (5)
-2.5
Principal Angle
Q.B.
Risk
-5.0
-7.5
0.6
Note that X is non-singular square matrix (Zeng & Lee, 2024, Lemma 6). Since left multiplication by a non-singular square matrix does not change the rank (Horn & Johnson, 2012, 0.4.6 (b)), we have Rank(G") Rank(A) (A) = r". Then, we denote the singular values of G by () non-increasing order.
-10.0
12.5
100
d=100
d=1000
d=1000
15.0
10
16:1
1c.2
16:3
1c.4
16.2
1e.3
164
Figure 2: Under (LoRA-init), the risk and the alignment to full one-step GD of LoRA with different a² and d, trained via GD on task (3). Left: the log risk under dif- ferent initialization variance a². The risk is defined as 1 | - AlE Right: the best principal angle between the top-r singular subspace of G and A, during train- ing. Smaller is closer. The principal angle is defined as min, || (G*) U,- (At) llop. More experimental details can be found in Appendix G.1.
3.1. Alignment under LoRA Initialization
We first present the results for the alignment of B, by re- calling the notations (-) and (-).
2
Theorem 3.1 (Alignment between G and B.). Under as- sumptions Section 2.1 for the linear setting, consider the LoRA updates (4) with (LoRA-init). We have
V (G*) V.- (B:) = =0, VEN+
Requirement on | can be relaxed under smaller initialization, illustrated by Fig.
One can see that, due to the zero initialization of Bo in (LoRA-init), after the first GD step, it holds that B₁ n G², which has rank < r" and lies in the right top-r" singular subspace of GP. The subsequent GD dynamics of B, is always restricted to this invariant subspace.
The above two theorems characterize the alignment between G and (A,, Bt). Fig. 2 empirically validates Theorem 3.2 in two folds: i) Smaller initialization (a in the x-axis) en- courages better alignment (evaluated by the principal angle), and then better generalization performance of fine-tuning (evaluated by the risk). But in practice smaller initializa- tion would increase the training time for convergence, as a double-edge sword. ii) increasing d leads to longer align- ment time, illustrated by Eq. (6), and worse alignment per- formance, illustrated by the formulation of a. Besides, we also verify this alignment in read-world applications by fine- tuning a T5 base model (Raffel et al., 2020) on MRPC using LoRA, as shown in Fig. I(b). The mean principle angles within each layer class computed in a similar way of Fig. 2. Our empirical results demonstrate that the principal angles decrease from around 1 because of Gaussian initialization to the value around 0.2 ~ 0.4.
Next we build the alignment for At with the notations (+). (-) and R² as the condition number of G Theorem 3.2 (Alignment between G and At. Simplified version of Theorem C.9). For the 2 2r case, under assumptions in Section 2.1 for the linear setting, we con- sider the LoRA updates (4) with [Aolis ~ NO, a²) in (LoRA-init). Then for any constant E (0,1), by taking a a= d d (1()) ) and running gradient de-
scent for t" steps with
In In + Mdp= (Gi))*
Vi
(6)
we achieve the following the alignment on the left singular subspace between GD and At- as below
Proof of Sketch: Here we give proof of sketch of Theo- rem 3.2. The dynamics (4) can be written as
U.. (A..) )lop 20,
(7)
A++1 B
Ia
G|
A
+
nonlinear term,
G
Is
B
with probability at least - C2 exp( (dd) - C2 Cxexp(-r)-
C3 expepNN) C for some constants C1, C2,C2,C0 Cz.
=:2++1
=.H
Z:
where H a time-independent matrix corresponding the linear part of the dynamic Z,1n H'Zz- By Schur de- composition of H (see Lemma C.1), we can obtain the precise spectral dynamics of Z and derive the align- ment between Z, and G see Lemma C.5 for details. We prove that the nonlinear term is well controlled, i.c., Iz 2 llop S || Vt S t", see Lemma C.6. Then
Remark:
The result under the x" S < 2r case more complex and defer this result Theo- rem C.9. The choice of in Theorem 3.2 shows that after 1" = θ be achieved. Our results can cover the standard He- initialization (He et al., 2015) if || 2 ((dtt)).
Ind A,- (CP) in Eq. (6), the alignment can
5

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
nonlinear models, see Lemma D.5 for details. Fig. 3 nu- merically validates that the starting points initialized by (Spectral-init) are consistently closer to the set of global minimizers, whereas those initialized by (LoRA-init) tend to be farther away across different random seeds.
Sentiul Int.
Chonal MERYANNS
LARA Soat
Sonc Start
Seet Eid
Moreover, running gradient descent from points initial- ized by (Spectral-init) requires significantly fewer steps to reach global minimizer, demonstrating the advantages of (Spectral-init). Due page limit, present the global convergence of linear models is deferred to Appendix C.2 and Appendix C.3, respectively. In Appendix C.2, we derive the linear convergence rate A, A Since the con- vergence will be slow if the downstream feature shift A is ill ill-conditioned, i.e., is large. This motivates us add pre- conditioners, then the convergence rate will be independent of accordingly, see Appendix C.3 for details.
12
16
Figure 3: Comparison of the GD trajectories under (Spectral-init) and (LoRA-init) with two different starting points. See more experimental details in Appendix G.I.
the alignment between Z, and G can be successfully transferred to that of Z, see Theorem C.9 for details.
We remark that previous work on matrix sensing (Stöger & Soltanolkotabi, 2021; Soltanolkotabi al., 2023) via symmetrization technique cannot be directly applied to our setting. Such symmetrization technique prevents the alignment results decoupling into two factorized matrices. We extend their technique to decouple the alignment for A, and B, individually via Schur decomposition of H.
4. Analysis of LoRA under Nonlinear Models
Now we focus on the nonlinear setting described in Sec- tion 2, where we consider the exact-rank case r = r" for delivery. We will demonstrate that Ar A|IF still small under the spectral initialization. Besides, the linear convergence rate of |, AllE can still hold.
3.2. Spectral Initialization and Global Convergence
As an example, we demonstrate the equipment of precondi- tion GD on ( (A,, B,) for global convergence
Theorem 3.2 has demonstrated the alignment on the rank- r* singular space of G and (A, Bt). In other words, if we take the SVD of G and choose the certain singular subspace for initialization in (Spectral-init), we can directly achieve the aligmment at this initialization and recover A to some extent, which is the main target of this work.
A.11 A, 7 VAI (A.
B,) (B,B.)
B = B: (A, -1 V B.)
Notice that here we use standard matrix inversion since we can prove that A, and B, stay non-singular across all f 2 0. By denoting W == W + A,B, we have the gradient
By the following standard concentration result for (sub)- Gaussian data: with probability at least - 2C exp( for some constants C > O, we have
V Bi) - Jw, B, (BI - A, Jw.
--1a - op
:::= min
(8)
where we denote
2
3
1 X (XW:) (.W:)oodl @x N
Jw. =
Recall k is the condition number of A and X;- is the --h singular value of A, we have the following result at the spectral initialization.
To deliver the proof, apart from the above-mentioned as- sumptions in Section 2. for the the nonlinear setting, we also need the following assumption.
Theorem 3.3. [One-step gradient can suffice] Under assumptions in Section 2.1 for the linear setting via (Spectral-init), taking in Eq. (8), then with probability at least 20
Assumption 4.1. We assume that i) W 1lap = O (1): |
constant C >> we have
max(A;, Al op) O 1 for m E [4].
Ao Allo S Id/Allop S x:
W lla
The condition i) ensures the balance between dif- ferent neurons within one layer for the downstream teacher model and the task diversity. The condition ii) ensures the signal of downstream feature shift is smaller than the pre- trained ones approximately in the order (kr*)= -1 since the
Remark:
Theorem 3.3 demonstrates that, after one-step full gradi- ent, i.c., using spectral initialization (Spectral-init), Ag Bo able to recover A with small error. This still true for
6

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
signal of adapted weight is generally weaker than the pre- trained weight. Two conditions can be empirically observed in Appendix G.5.
Algorithm LoRA-One for one specific layer
Input: Pre-trained weight W batched data (Dm M== m=11 sampled batch data B, LoRA rank r, LoRA alpha a, loss function L, scaling parameter s
Here we can show that, for the nonlinear model, LoRA training can achieve global linear convergence under (Spectral-init) via preconditioned GD in Eq. (9).
Initialize:
I: Compute V L given B
2: U,S.V + SVD (- WwW(W))
Theorem 4.2 (Simplified version of Theorem D.10). Under assumptions in Section 2.1 for the nonlinear setting and 1.1, with training conducted by Eq. (9) and initialization via (Spectral-init) with setting 7 = 2, we take c = = (v-wva)
3: + S/S and + 1/s
4: An + vT. 17.U1.17) [nar)
5: Bu + VT.S $2/2
and S 1. Then choosing E (c,, f)r small constant
6: Clear Vw L((
20' On > O, with probability at least 1 - 2Cdk exp N for a universal constant C > O, we have
Train:
for 0, ................................ T do
Compute gradients given D++1:
G f V V (e) G V (A,,
|. Al (1 11-27 2) (1-
pd PA. 2.
(10)
Update A++1 B,41 t AdamW (G G 10: end for Return: W Jr Ar
Remark:
We make three remarks here:
i) This theorem is based on || Ao - lp S ptP.- at ini- tialization, see Lemma D.5 for details, which demonstrates that one-step full gradient can be sufficient.
LoRA-One
LoRA-SB LORA-GA
-5-
ti) The convergence rate is independent of condition number of downstream feature shift A, demonstrating the benefits of adding preconditioners.
Log-Rikk
Logeloak
-10
-20
1
Proof of Sketch The complete proof can be found in Ap- pendix D.2. We first compute the expectation of Jw, (see Lemma D.2) and decompose Jw, into (A, +)) where E, is defined in Lemma D.6. The first term is the signal term which can dominate the preconditioned GD dy- namics. The second term E,- == T1 T2 consists of two parts (details see Lemma D.7): the first part 71 is the resid- ual term from Eg [Jw.] which vanishes due to pre-training signal dominance. For the second term 72, it comes from the concentration error of Jw., which can also controlled by large sample size N.
-25
20
40
60
80
100
20
60
80
100
Epochs
Epochs
(a) < y
(b) >> r
Figure 4: The log-risk curve under LoRA- One, LoRA-SB, and LoRA-GA, trained via GD on fine-tuning task (3) under: 1) under-ranked case r < r", 2) over-ranked case > r".
lowing initialization strategy (omit the scaling parameters)
To handle IIA,B, - AllF, we explore its recursion re- lationship in Lemma D.6. The key part is to control [Laa UA, A (1& VB, VH,) (Lemma D.9) and higher order term (Lemma D.8).
Au +-
Üo:] [.1:)] Bo
+ Vas
[x++1:22)]
which aims provide the best 2r approximation of G However, our theory indicates that B, will align to the right-side rank-r" singular subspace of G under random initialization. However, LoRA-GA chooses the (r + 1)-th to 2r-th singular values for Bo, causing the iterates B, to lie outside the desired subspace. As a result, the optimiza- tion may remain trapped in an undesirable subspace and fail to converge to an optimal solution, which can numeri- cally verified by Fig. 4. Moreover, this approach subtracts the gradient for non-zero initialization and thus yields a bi- ased estimate of A, scaling with the model size; see further discussion in Appendix F.
5. Algorithm and Discussions
In this section, we present the LoRA-One algorithm and justify the optimality of our initialization over previous gra- dient alignment based algorithms for fine-tuning.
We present the implementations in Algorithm 1, which is driven by (Spectral-init) (shown in line 3-6). It coincides with the spirit of gradient alignment work, e.g., LoRA-GA (Wang et al., 2024), LoRA-pro (Wang et al., 2025), but the mechanisms for gradient alignment differ significantly, as suggested by our theory. First, LoRA- GA proposes the fol-
Secondly, there is one concurrent work (Ponkshe et al., 2024), LORA-SB, which uses the same singular subspace
7

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
for initialization but only updates r x matrix R from the SVD of G.. Their intuition is to project the fine-tuning updates onto the singular subspace of first gradient step G.. However, the singular subspace of G normally still have distances with the ground truth, which will make their method hard to escape/rotate this subspace with limited degrees of freedom.
sampled batch, which is equivalent (Spectral-init). We remark that low-rank update with small batch for CoLAA and MRPC only costs less than one second but already matches the performance of LoRA which needs tens of seconds. Accordingly, one-step full gradient can suffice for fine- tuning on small-scale datasets, e.g., CoLA, MRPC.
Besides, Table 2 also shows that LoRA-One outperforms other LoRA-based methods on three tasks out of five and achieves the best in average. Significant gains appear on the smaller benchmarks, i.e. CoLA and MRPC. On large datasets such as MNLI and QNLI, LoRA-Pro performs better but is with more cost. This is because, LoRA-pro (Wang et al., 2025) approximates gradient from full fine-tuning at every training step while LoRA-One only conducts at the first step. LoRA-pro adds 10dr² + 6kr + 155r more FLOPs than LoRA-One per pair of matrices for time cost. For memory cost, LoRA-pro on MetaMathQA 100k with rank 8 costs 43.87 GB while our method only costs 21.7 GB for memory management.
Our toy experiments based on (3) in Fig. 4 show that both LoRA-GA and LoRA-SB fail to find global minimizers even in simple linear setting, whereas LoRA-One demonstrates significantly better generalization. More experimental de- tails are presented in Appendix G.I.
6. Experiments
In this section, we conduct experiments to compare LoRA- One with typical LoRA based algorithms across multiple NLP benchmarks. In Section 6.1, we evaluate the ability of one-step gradient on real-world fine-tuning tasks to justify our theory on natural language understanding, i.c. Theo- rem 3.3 and Lemma D.5. In Section 6.2, we evaluate on mathematical reasoning, general knowledge, and code gen- eration tasks, with more data and epochs for further evaluat- ing math reasoning ability in Section 6.3. Furthermore, we compare the time and memory cost across different methods to illustrate our efficiency.
6.2. Natural Language Generation
We fine-tune LLaMA 2-7B (Touvron et al., 2023) on: 1) 100K samples from MetaMathQA (Yu et al., 2024) and evaluate the accuracy based on two types of prompting: (a)
direct prompting, (b) 8-shot Chain-of- -hhhughhh (CoT) (Wei et al., 2022) prompting on GSMSK (Cobbe et al., 2021); 2) Alpaca (Taori et al., 2023) and evaluate on the MMLU (Hendrycks et al., 2021) benchmarks using direct prompt- ing; 3) 100K samples from Mode-Feedback (Zheng et al., 2024) and evaluate the PASS@ I on HumanEval (Chen et al., 2021a). We compare LoRA, LoRA-GA, and LoRA-One with rank 8. The learning rate and batch size are optimized for each method. More experimental details are presented in Appendix G.2.
6.1. One-Step Full Gradient Could Suffice in Natural Language Understanding
We fine-tune T5 base model (Raffel et al., 2020) on a subset from GLUE (Wang et al., 2019)- - MNLI, SST2, CoLA, QNLI, and MRPC. We evaluate the test performance by accuracy (%). We compare LoRA (Hu et al., 2022). LoRA+ (Hayou et al., 2024), P-LoRA (Zhang & Pilanci, 2024), PISSA (Meng et al., 2024), LoRA-GA (Wang et al., 2024), LoRA-Pro (Wang et al., 2025), and LoRA-One with rank 8.
Table 3 shows that LoRA-One consistently outperforms both vanilla LoRA and LoRA-GA across different tasks and prompting methods. LoRA-One achieves 60.44% accu- racy under direct prompting- about 1.18 points higher than LoRA-ane and 55.88% in the few-shot CoT setting, a gain of roughly 2.52 points, indicating it not only strengthens the model's core problem-solving abilities but also its capacity for coherent, multi-step reasoning. On MMLU, LoRA-One also shows superior generalization on the MMLU bench- mark (47.24% vs. 45.73% for LoRA), indicating improved knowledge retention across diverse domains. Finally, LoRA- One excels in code generation, it achieves a PASS@ @ score of 28. 28.66%, nearly 3 points higher than LoRA's 25. 25.85% and also improving upon LoRA-GA, implying better adaptation to structured code synthesis tasks. Moreover, LoRA-One exhibits noticeably lower run-to-run variability compared to
The hyperparameters are optimized for each method. More experimental details are presented in Appendix G.4.
Before experimental comparison, we first access the capac- ity of the one-step full gradient with its low-rank compo- nents these real-world fine-tuning tasks. We approximate the one-step full-batch update G from full fine-tuning by large sampled batch (2048) as G², and best rank-r approx- imation of G for r =8 8 using a smaller sampled batch (8) as Pr (.))). We optimize the learning rate, i.e. 17",, and update the base model by W ''',, and Wt... - PG')) for SST2, CoLA, QNLI, and MRPC, respectively.
The top rows of Table 2 shows that the test performance can be significantly improved over the pre-trained model by the one-step full gradient step with proper selection of stepsize. This improvement is still promising (even better) after taking the best rank-r approximation with smaller
1m- won.llatiio-haaaaae
8

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Table 2: Accuracy comparison on GLUE subset across typical LoRA based algorithms,, as well as evaluation of the pre-training model, one-step gradient update and its low-rank approximation (r = 8, i... (Spectral-init)). Results are reported as accuracy (%) with standard deviations over runs (best bold). The results marked with (*) are sourced from Wang et al. (2024; 2025) under the same setting, and their hyper-parameter selection aligns with our search. The test accuracy on MNLI remains after one-step update thus not reported.
Method
MNLI
SST-2
CoLA
QNLI
MRPC
Avg.
Pre-train
89.79
59.03
49.28
63.48
One-step full gradient
90.94
69.13
70.35
68.38
r = 8 (low rank)
89.91
69.22
76.31
68.38
LoRA
85.30+0.04
94.04
72.84
93.02
68.38_o.m
82.72
LoRA+*
85.8
93.85
77.53
93. 93.14.0..
74.43=1.38
84.95
P-LoRA
85.28
93.88
79.58
93.00+0.07
83.91=1.16
87.13
PISSA*
85.75
94.07
74.27
93.15
76.31 76.31 =0.51
84.71
LoRA-GA
85.70=0.09
94.11
80.57
13.18
85,29=0.21
87.77
LoRA-Pro"
86.03
10.19
81.94
:3.42
86.60
88.44
LoRA-One
85.89
94.53....13
82.04 10.22
93.37 +0.02
87.83
-0.37
88.73
Table 3: Performance comparison across different methods on NLG benchmarks. Results are reported as mean with standard deviations over runs (higher better).
compare LoRA, LoRA+, LoRA-GA, and LoRA-One with rank 8. We evaluate the fine-tuned models on GSM8K with direct prompting. The learning rate and batch size optimized for each method. More experimental details presented in Appendix G.3.
(r=88 GSM8K-D
LoRA
LoRA-GA
LoRA-One
59.26+0.99
56.44+1.15
60.44-0.17
GSM8K-CoT
53.36
46.07+1.01
55.88
MMLU
45.73
45.15
47.24.0..8
62
HumanEval
25.85
26.95+1.30
28.66
Accuracy (%6
60
Table 4: The training time and memory cost of LoRA and LoRA-One from Section 6.2.
56
Training Time (Memory)
Epoch
Epoch 2
Epoch
Epoch 4
(r 8)
LoRA
LoRA-One
Figure: Accuracy comparison over epoch on GSMSK.
MetaMathQA
6h20m (21.6GB)
6h23m (21.7GB)
Results are reported as mean over 2 runs (higher is better).
Alpaca
3h22m (23.4GB)
3h25m (23.4GB)
Code-Feedback
6h24m (22.6GB)
6h26m (22.9GB)
Fig. 5 shows that LoRA-One consistently leads other meth- ods over epochs, suggesting that it scales more effectively with additional training and data. In contrast, LoRA-GA only shows marginal gains over LoRA and LoRA+.
the baselines, indicating better stability under (Spectral-init). Regarding the time and memory cost, LoRA-One takes al- most the same cost as LoRA, as shown in Table across all three datasets. This suggests that PoRA-One delivers its in- tended benefits such enhanced Shaptability--ithheee imposing any meaningful extra time memory cost during fine-tuning.
7. Conclusion
such as improved convergence stability or
This paper theoretically demonstrates how LoRA can be improved from our theoretical analysis in both linear and nonlinear models: the alignment between LoRA's gradi- ent update (A,, B,) and the singular subspace of G and adding preconditioners. Our theory derives the optimal initialization strategy for LoRA, clarifies some potential is- sues behind gradient alignment work, and bridge theory to practice with the promising performance LoRA-
6.3. Math Reasoning on Full Data and Multiple Epochs
Beyond Section 6.2, we further fine-tune LLaMA 2-7B on the complete MetaMathQA (395K) dataset for 4 epochs to access the maximum capacity of math reasoning. Here we

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Impact Statement
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. D. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman,
This paper provides theoretical understanding of low-rank adapters and proposes algorithm design for parameter- efficient fine-tuning, The target of this paper is advance the field of Machine Learning. There might be some poten- tial societal consequences of our work, none which we feel must be specifically highlighted here.
G., et al. Evaluating Large Language Models Trained on Code. arXiv preprint arXiv:2107.03374, 2021a.
Chen, Y., Chi, Y., Fan, J., Ma, C.,, et. Spectral Methods for Data Science: A Statistical Perspective. Foundations and Trends@ in Machine Learning, 14(3):56-806, 2021b.
Acknowledgment
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H.,
Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training Verifiers to Solve Math Word Problems. arXiv preprint arXiv:2110.14106 2021.
Y. Zhang was supported by Warwick Chancellor's Interna- tional Scholarship. F. Liu was supported by Royal Soceity KTP RI 241011 Kan Tong Po Visiting Fellowships. Y. Chen was supported in part by National Science Foundation grants CCF-2233152. We thank Yichen Wang for coding discussions, Zulip² for the project organization tool, and Sulis for GPU computation resources.
Cui, H., Pesce, I.., Dandi, Y., Krzakala, F., Lu, Y., 7 borova, L., and Loureiro, Asymptotics of feature learning in two-layer networks after one gradient-step. In International Conference on Machine Learning, pp. 9662 9695. PMLR, 2024.
References
Damian, A., Lee, J., and Soltanolkotabi, M. Neural Networks can Learn Representations with Gradient De- scent. In Conference on Learning Theory, pp. 5413-5452. PMLR, 2022.
Abbe, E., Adsera, E. B., and Misiakiewicz, T. The merged- staircase property: a necessary and nearly sufficient con- dition for SGD learning of sparse functions two-layer neural networks. In Conference on Learning Theory, pp- 4782-4887, PMLR, 2022.
Dandi, Y., Pesce, L., Cui, H., Krzakala, F. Lu, Y., and Loureiro, B. A Random Matrix Theory Perspec- tive on the Spectrum of Learned Features and Asymp- totic Generalization Capabilities. In The 28th Interna- tional Conference on Artificial Intelligence and Statistics, 2025. URL https: http:///eenneeeeeeeeeeeeeeeseeee 1d=3k7rKKU7RL.
Arous, G. B., Gheissari, R., and Jagannath, A. Online stochastic gradient descent on non-convex losses from high- high-dimensiooal inference. Journal of Machine Learn- ing Research, 22(106):1-511 2021.
/ net.eeiwwwwwwweeeeeeeeee forum?
Ba, J., Erdogdu, M. A., Suzuki, T., Wang, Z., Wu, D., and Yang, G. High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Repre- sentation. In Advances in Neural Information Processing Systems, pp. 37932-37946, 2022.
Dayi, A. K. and Chen, S. Gradient dynamics for low-rank fine-tuning beyond kernels. arXiv preprint arX5v:241..15385, 2024.
Ding, L., Qin, Z., Jiang, L., Zhou, J., and Zhu, Z. A Valida- tion Approach to aver-parameterized Matrix and Image Recovery, arXiv preprint arXiv:2209. 2022.
Bietti, A., Bruna, J., and Pillaud- Pillaud-Vivien, L. On learning gaussian multi-index models with gradient flow. arXiv preprint arXiv:2310. 197993 2023.
Dodge, J., Ilharco, G., Schwartz, R., Farhadi, A., Hajishirzi, H., and Smith, N. Fine- Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping. arXiv preprint arXiv:2002.06300 2020.
Brown, T., Mann, B., Ryder, N., Subbiah, M.. Kaplan, D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. Language Models are Few-Shot Learners. In Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Giampouras, P., Cai, H., and Vidal, R. Guarantees of Preconditioned Subgradient Algorithm for Overparame- terized Asymmetric Low-rank Matrix Recovery, arXiv preprint arXiv:2410. 16826, 2024.
Brutzkus, A. and Globerson, A. Globally Optimal Gradient Descent for ConvNet with Gaussian Inputs. In Inter- national Conference on Machine Learning, pp. 605-614. PMLR, 2017.
Han, Z., Gao, C., Liu, J., Zhang, J., and Zhang, S. Q. Carameter-Efficien Fine-Tuning for Large Models: A Comprehensive Survey. Transactions on Machine Learn- ing Research, 2024. ISSN 2835-8888 8856. URL https: / openreviee net forum? d==1scc8bb22]
Phttps://zu comcom/ 'https: //tas////wwwaeee ck, ac, wtt.kk///eeeec//tttt///////////e sulis /s/
10

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Hayou, S., Ghosh, N., and Yu, B. LoRA+: Efficient Low Rank Adaptation of Large Models. In International Con- ference on Machine Learning, pp. 17783-17806. PMLR, 2024.
Lee, J. D., Oko, K., Suzuki, T., and Wu, D. Neural network learns low-dimensional polynomials with SGD near the information-theerreie limit. Advances in Neural Informa- tion Processing Systems, 37:58716 1-7766,, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Delving Deep into Rectifiers: Surpassing Human-Leevel Performance on ImageNet Classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026- 1034, 2015.
Li, B., Zhang. L., Mokhtari, A., and He, N. On the Crucial Role of Initialization for Matrix Factorization. In The Twelfth International Conference on Learning Represen- lations, 2025. URL https: /ttp///ppp//eeeeee///////e net/ forum?id-YTEweer
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt. J. Mcasuring Massive Multitask Language Understanding. In International Conference on Learning Representations, 2021.
Li, Y., Ma, T., and Zhang, H. Algorithmic Regularization in Over-parameterized Matrix Sensing and Neural Networks with Quadratic Activations. In Conference On Learning Theory, pp. 2-47. PMLR, 2018.
Horn, R. A. and Johnson, C. R. Matrix Analysis. Cambridge university press, 2012.
Liu, X.-H., Du Y., Wang. J., and Yu. Y. On the Optimiza- tion Landscape of Low Rank Adaptation Methods for Large Language Models. In International Conference on Learning Representations, 2025.
Houlsby, N., Giurgiu, A., Jastrzebski, S., Morrone, B., De Laroussilhe, Q. Gesmundo, A., Attariyan, M., and Gelly, S. Purameter-Efticient Transfer Learning for NLP. In International Conference Machine Learning, pp. 2790-2799, 2019.
Loshchilov, I. and Hutter, F. Decoupled Weight Decay Regularization. arXiv preprint arXiv:1711.05101, 2017.
Ma, C.,, Li, Y., and Chi, Y. Beyond Procrustes: Balancing- Free Gradient Descent for Asymmetric Low-Rank Matrix Sensing. IEEE Transactions Signal Processing, 69: 867-877, 2021.
Hu, E. J., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W., al. LoRA: Low-Rank Adaptation of Large Language Models. In International Conference Learning Representations, 2022.
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A Kernel-Based View of Language Model Fine- Tuning. International Conference on Machine Learning, pp. 23610-23641, PMLR, 2023.
Jacot, A., Gabriel, F,, and Hongler, C. Neural Tangent Ker- nel: Convergence and Generalization in Neural Networks. In Advances in neural information processing systems, 2018.
Meng, F., Wang, Z., and Zhang, M. PiSSA: Principal Sin- gular Values and Singular Vectors Adaptation of Large Language Models. In Advances in Neural Information Processing Systems, pp. 121038-121072, 2024.
Jang, U., Lee, D., and Ryu, E. K. LoRA Training in the NTK Regime has No Spurious Local Minima. In Inter- national Conference on Machine Learning, pp. 21306- 21328. PMLR, 2024.
Jia, X., Wang, H., Peng, J., Feng, X., and Meng. D. Pre- conditioning Matters: Fast Global Convergence of Non- convex Matrix Factorization via Scaled Gradient Descent. Advances in Neural Information Processing Systems, 36, 2024.
Mishra, B., Apuroop, K. A., and Sepulchre, R. riemannian geometry for low-rank matrix completion. arXiv preprint u2 2012.
Moniri, B., Lee, D., Hassani, H., and Dobriban, E. A Theory of Non-Linear Feature Learning with One Gradient Step in Two-Layer Neural Networks. In International Con- ference on Machine Learning, pp. 36106-36159. PMLR, 2024.
Kim, J., Kim, J., and Ryu, E. K. LoRA Training Provably Converges Low-Rank Global Minimum It Fails Loudly (But it Probably Won't Fail). In International Conference on Machine Learning, 2025.
Oja, E. A Simplified Neuron Model as a Principal Com- ponent Analyzer. Journal of Mathematical Biology, 15: 267-273, 1982.
Kopiezko, D. J., Blankevoort, T., and Asano, M. VeRA: Vector-based Random Matrix Adaptation. In The Twelfth International Conference on Learning Representations, 2024.
Oko, K., Song, Y., Suzuki, T., and Wu, D. Pretrained Transformer Efficiently Learns Functions In-Context. Advances in Neural Information Processing Systems, 37:77366-77365, 2024.
Target
LeCun, Y. The mnist database of handwritten digits. hup://yann. lecun. com/exab/mnist/, 1998.

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Ponkshe, K., Singhal, R., Gorbunov, E., Tumanov, A.,
Wang, S., Yu, L., and Li, J. LoRA-GA: Low-Rank Adapta- tion with Gradient Approximation. Advances in Neural Information Processing Systems, 37:54905-54931, 2024.
Horvath, S., and Vepakomma, P. Initialization us- ing Update Approximation is a Silver Bullet for Ex- tremely Efficient Low-Rank Fine-Tunme Tuning, arXiv preprint arXiv:2411.19555 2024.
Wang, Z., Liang, J., He, R., Wang. Z., and Tan, T. LoRA- Pro: Are Low-Rank Adapters Properly Optimized? In The Twelfth International Conference Learning Rep- resentations, 2025. URL
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. Exploring the Limits of Transfer Learning with a Unified Text Transformer. Journal of Machine Learning Research, 21 (140):1-67, 2020.
net/ herum? (tw.rrrmmmm=gggTMSS... T=gTTRMU31555
Wedin, P.-A. Perturbation bounds in connection with sin- gular value decomposition. B/T Numerical Mathematics, 12:99-111, 1972.
Soltanolkotabi, M., Stöger, D., and Xie, C. Implicit Balanc- ing and Regularization: Generalization and Convergence Guarantees for Overparameterizze Asymmetric Matrix Sensing. In The Thirty Sixth Annual Conference on Learn- ing Theory, pp. 5140-5144 142.. PMLR, 2023.
Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F,, Chi, E., Le, Q. V., Zhou, D., et al. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. Advances in Neural Information Processing Systems, 35: 24824-24837 2022.
Stöger, D. and Soltanolkotabi, M. Small random initial- ization is akin spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstruction. In Advances in Neural Information Processing Systems, pp. 13831-23344 2021.
Xiong, N., Ding, L., and Du, S. How Over- Parameterization Slows Down Gradient Descent in Ma- trix Sensing: The Curses of Symmetry and Initialization. In The Twelfth International Conference Learning Representationss 2024. URL https: /pppr///////...i...e.. net forum? i dddixddAiGohhhhho
Taori, R., Gulrajani. I. Zhang. T., Dubois. Y., Li, X., Guestrin, C., Liang P., and Hashimoto, T. B. Stanford Alpaca: An Instruction-follovinng LLaMA model. https:/ // httpt/tttbb.////////////tt/.tttt
thub. com/tatsu-laa) stanford_alpaca, 2023.
Xu, X., Shen, Y., Chi, Y., and Ma, C. The Power of Precondi- tioning in Overparameteriiee Low-Rank Matrix Sensing. In International Conference on Machine Learning, pp. 38611-38654 PMLR, 2023.
Thoppilan, R., De Freitas, D., Hall, J., Shazeer, N., Kul- shreshtha, A., Cheng. H.-T., Jin, A., Bos, T., Baker, L., Du, Y., et al. LaMDA: Language Models for Dialog Applications. arXiv preprint arXiv:2201. 08239, 2022.
Xu, Z., Min, H., MacDonald, L. E., Luo, J., Tarmoun, S., Mallada, E., and Vidal, R. Understanding the Learn- ing Dynamics of LoRA: A Gradient Flow Perspective on Low-Rank Adaptation in Matrix Factorization. In International Conference on Artificial Intelligncce and Statistics, 2025.
Tong, T., Ma, C., and Chi, Y. Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient De- scent. Journal of Machine Learning Research, 22(150): 1-63, 2021.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaci, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., et al. LLaMA 2: Open Foundation and Fine- Tuned Chat Models. arXiv preprint arXiv:2307.09288, 2023.
Yu, L., Jiang, W., Shi, H., Jincheng, Y., Liu, Z., Zhang, Y., Kwok, J., Li, Z.,, Weller, A., and Liu, W. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: //openreview.com net/ Fow.mmidd-eeeeee NENhhggD....
Vershynin, R. Introduction to the non-asymptotic analysis of random matrices. arXiv preprint arXiv: 100 1.3027, 2010.
Zeng, Y. and Lee, K. The Expressive Power of Low-Rank Adaptation. In The Twelfth International Conference on Learning Representations, 2024. URL https:// openreview net forum?id-11k22/jm3333
Vershynin, R. High-Dimenssonal Probability: An Intro- duction with Applications in Data Science, volume 47. Cambridge university press, 2018.
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural L Language Understanding. In International Conference on Learning Representations, 2019.
Zhang, F. and Pilanci, M. Riemannian Preconditioned
LoRA for Fine-Tuning Foundation Models. In Inter- national Conference on Machine Learning, pp. 59641- 59669. PMLR, 2024.
12

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Zhang, G., Fattahi, S., and Zhang, R. Y. Preconditioned Gra- dient Descent for Overparameterivzed Nonconvex Burer- Monteiro Factorization with Global Optimality Certifica- tion. Journal of Machine Learning Research, 24(163):
1-55, 2023.
Zhang. J., Fattahi, S., and Zhang, R. Y. Preconditioned Gra- dient Descent for Over-Parameterizer Nonconvex Matrix Factorization. Advances in Neural Information Process- ing Systems, 34:5985-5996, 2021.
Zheng, T., Zhang, G., Shen, T., Liu, X., Lin, B. Y., Fu, J., Chen, W., and Yue, X. Opeereedentrrppeeee:: Integrating Code Generation with Execution and Refinement. In Findings of the Association for Computational Linguistics ACL 2024, pp. 12834-12859, 2024.
Zhu, Z., Wu, Y., Gu, Q., and Cevher, V. Imbalance- Regularized LoRA: A Plug-and-Play Method for Improv- ing Munnng of Foundation Models. In NeurlPS Work- shop on Adaptive Foundation Models, 2024.
13

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Contents
Symbols and Notations
15
Preconditioned LoRA-One
16
Proofs for Linear Model
17
C.1 Proofs for LoRA under Random Initialization
17
C.I.1 SVD and Schur Decomposition
18
C.1.2 Dynamics of Linear Approximation
19
C.1.3 Alignment to Negative Gradient Full Fine-tuning
23
C.2 Gradient Descent under Spectral Initialization
27
C.3 Preconditioned Gradient Descent under Spectral Initialization.
37
D Proofs for Nonlinear Model
40
D.1 Problem Settings and Spectral Initialization
40
D.1.1 Computation of Full Population Gradients
42
D.1.2 Concentration Empirical Gradients
46
D.2 Preconditioned Gradient Descent under Spectral Initialization.
49
Auxiliary Results for Proofs
57
Detailed Comparison with LoRA-GA
58
Experimental Settings and Additional Results
59
G.1 Small-Scale Experiments
59
G.2 Natural Language Generation
60
G.3 Math Reasoning on Full Data and Multiple Epochs
60
G.4 Natural Language Understanding
60
G.5 Empirical Verification of Assumption 4.1
61
14

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
A. Symbols and Notations
this section, we provide a the symbols and notations used in a paper.
Symbol
Dimension(s)
Definition
N(,
σr)
Multivariate normal distribution with mean vector and covariance matrix Sichmann-Laadaaa asymptotic notation
O 2,
|
Fuclidean norm of vector up Operator norm of matrix M Frobenius norm of matrix M Dot product of vectors and Hadamard product of matrix M and N Pre-trained weight matrix
[M lpp
M F
u, v)
Mon N
w
Bdx
A
Bdx
Downstream feature shift matrix
W
Downstream weight matrix W = W A
G
The initial gradient matrix under full fine-tuning Learnable low-rank adapters at step f
A,, B.
Raxr
Rexk
w
Rd
th column of pre-trained weight matrix W
Rd
column of downstream weight matrix W
Rd
th column of adapted weight matrix (W + A. at step
Wi,i
A;
Rd
column downstream feature matrix
[A
Rd
tth column of the product of adapters A.
X
R
Xxd
Downstream matrix
R Nxd
Downstream label matrix downstream data point Inverse of matrix M
Rd
M
M
Pscudo-inverss of matrix M thh singular value of matrix M
d (M)
IR
A
IR
h singular value of downstream feature shift matrix A
N (M)
The condition number matrix M
K
The condition number A: = /jnax/ Xinia
The condition number G ve Amax (G²) /Amin (G²)
Un (M)
The left singular subspace spanned by the largest singular values of the input matrix M The left singular subspace orthogonal to U.. (M)
U (M)
V (M)
The right singular subspace spanned by the m largest singular values of the input matrix M
V (M)
The right singular subspace orthogonal to Vn. (M)
U
The left singular matrix of the compact SVD of A
U
The corresponding orthogonal complement of U
VA
The right singular matrix compact SVD
VA
The corresponding orthogonal complement of VA
a(.)
ReLU activation function
d(-))
The derivative of ReLU activation function
Vw. (W)
The gradient matrix of function f w.r.t. input matrix W
L)A,B)
Loss function under LoRA fine-tuning
L(W)
Loss function under full fine-tuning
N
Number of downstrcam data Input dimension of the data
d
k
Output dimension of the label
77
Learning rates
In theory, random init. scale Ao. In algorithms. standard LoRA alpha.
Table 5: Essential symbols and notations this paper.
15

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
B. Preconditioned LoRA-One
Motivated by our theory of preconditioning (see Theorem C.21 and Theorem 4.2), we propose a preconditioned variant of LoRA-One termed LoRA-One-P. The algorithm is formally presented in Algorithm 2. LoRA-One employ the same initialization, i.c. (Spectral-init), with LoRA-One. For the optimizer, we use a preconditioned AdamW which introduced in (Zhang & Pilanci, 2024) instead of AdamW (Loshchilov & Hutter, 2017) in LoRA-One.
Algorithm 2 LoRA-One-P for a specific layer
Input: Pre-trained weight W batched data (Dm m- sampled batch data B, LoRA rank r, LoRA alpha a, loss function L, scaling parameter preconditioning parameter X
Initialize:
1: Compute 3(W given B
2: U.S. +- SVD ( V ((W3)
3: 5 (/S
4: 7 1/s
5: Ao + v7.Uge (t.os))
S 1/2
6: Bg V7- S $1/2 VT
[iran] [[.arr]
7: Clear L(W
Train:
8: f ...,= do
9: Compute preconditioned gradients given D
G + V B,) - XI,
-
G + (AJA, + ) (Ar, B,)
10: Update A,t1 BB+1 + Adam W (G GB
1I: end for
Return: W + Vr
A, BT
Next, we conduct experiments to justify that LoRA-One-P is more robust to sub-optimal learning rate and can achieve faster convergence under suboptimal choice than LoRA-One. We fine-tune the T5 base model (Raffel et al., 2020) on SST-2 dataset from GLUE (Wang et al., 2019) for one epoch. To ensure fair comparison, we fine-tune on the grid of learning 10-5) and fix other parameters to be the same in
rates {1 x 10-3 2 10-4, 1 x 10-4, ,, 10-5 2 10 Appendix G.4. Additionally, we the preconditioning parameter À 0 in Algorithm 2 to be consistent with Section 4. For each choice of learning rate, run 5 different seeds for both methods and record the test accuracy every 30 steps. Then, we compute the mean and 95%-confidence interval to construct the trajectory of test accuracy during fine-tuning. The results are shown in Fig. 6.
We can observe that, LoRA-One-P demonstrates clear advantages over LoRA-One across all options of tested learning rates, and these benefits manifest in two key aspects: robustness mis-specified learning rates and faster speed of convergence.
Under overly large learning rate (1 x 10-3), LoRA-One hardly makes consistent progress and its wide confidence interval betrays unstable learning. LoRA-One-2 makes stable improvement with much tighter interval. This demonstrates that the preconditioners temper the volatility introduced by too-aggressive rate, yielding both faster and more reliable gains.
When operating in the moderate range (2 x 10"A), both methods eventually attain strong performance and share a similar trending, indicating that LoRA-One achieves strong performance same as LoRA-One-P when the learning rate is well- specified.
At the other extreme, with the learning rates chosen too small (1 1 to 1 x 10-5), the training process of LoRA- One gradually appears to be slow and nearly stalls. In contrast, LoRA-One-P not only converges more rapidly but also maintains stable, high-quality performance even when the learning rate departs substantially from its optimal setting. This "rescue effect" signals that LoRA-One-P can tolerate sub-optimal even under-scaled updates. Its expands "safe zone" for hyperparameter tuning and reduced variance across random seeds which makes it robust and efficient choice for real-world tasks.
16

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
rr= x 10
1r=2x10
1r=1x10 4
0.95
0.95
0.95
0.94
Accuracy
0.92
0.92
0.92
0.91
0.91
0.91
0.90
LoRA-Ore
0.90
0.90
LoRA-One-P
500
1000
1500
2000
500
1080
1500
2000
500
1000
1500
2000
Epoch
Epoch
Epoch
1r=5 x 10 5
1r=2x 10
1r=1x10
0.95
0.95
0.95
0.94
0.94
0.94
Accuracy
0.92
0.92
0.91
0.91
0.91
0.90
0.90
0.90
500
1000
1500
2000
500
1000
1500
2000
500
1000
1500
2000
Epoch
Epoch
Epoch
Figure 6: The trajectory of test accuracy during fine-tuning for LoRA-One and LoRA-One-P under different learning rates.
C. Proofs for Linear Model
In Appendix C.I, we deliver the proofs for alignment Section 3.1. In Appendix C.2, we present the proofs for the main results in Section 3.2 under spectral initialization. Appendix C.3, we give the proofs for precondition GD,
C.1. Proofs for LoRA under Random Initialization
Let X be the fine-tuned data with X E Nxd and the multi-output Y E Nx For simplicity, we define the initial residual
error YA = XW: = X A. Then, denote the negative gradient of Full Fine-tuning after the first step as
G -ww N
x (XW: Y)
X X YA
Rdxk
N
Recall the gradient update for LoRA
A.+1 A;
7 T
X + A,B;) Y
V)BT, BT (X(Wa+ A,B.) -Ā).
N
B41 B,
A X
N
we rewrite compact form
A
A
0
G
A
1
X
A,
B
B
B.
G
0
B.
N
X
0
B
:=H
(1)
La
G
A
17
X
X.A.B
A
GG
It
B.
X
0
B
:=H
17

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
By defining a stack iterate
A
Ao 0
Z: :=
and
Zo: :
E R
(d=)))xr
(12)
B
we can formulate Eq. (11) as a compact form of nonlinear dynamical system
Z++1
H Er+1,
(13)
where H is a time-independent matrix corresponding to the linear part, and E.1 corresponds to the nonlinear part.
C.I.I. SVD AND SCHUR DECOMPOSITION
We recall the complete SVD of E Rd
S
0 ((-r()))))) (dkr"))(k-r))
V
A USV U U1]
where S Diaa (1ī (-)))
(d-ri)xr
Lemma C.I and then extend d + k in Similarly, we recall the complete SVD of GD as G Sc:Sc: V
We derive the Schur decomposition of H under the special case Lemma C.3 via zero padding SVD in Lemma C.2.
Lemma C.1 (Schur Decomposition of H under d = k). Under assumptions in Section 2.1 for the linear setting, given G E Rdxk in Eq. (5) and its complete SVD G::Sc: V.. if = k, then the block matrix H admits the following Schur decomposition
La
G
H
Ia = C
in
(G:)
where is an orthogonal matrix and T is a block upper triangular matrix
1 Üa:
-Üa
La + nSa:
0 Ia n S
C
and
T
V2
Va:
Va
Proof. We prove by verifying the claim. Starting with
ÜG:
-ÜG:
La La+nst nSa
0
Va:
Va
0
Ia - S
To: Sor
nU. Sc: Ua
=;
Va + VVa
Vo: VR Sc:
then we can verify that
T UT Ga
La VV:
n VT H.
=
I
G:
Accordingly, we conclude the result.
Next, we consider the case of d &f k.
Case (d > k:): by zero padding, G and related matrices are given by
G
G b Dxx(d=))
H
G
Ia
and for any 2 have the following related matrices
A
A in
B. B, 0 Orx(d-k1
Z,
Z
H'
H
(B,)
(B.
18

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Case 2 (d < k): Similarly, by zero padding, we define
Gt
Is
G
G
H
O(x-))x
0
11
G
I&
and 14 2 0, we define
A.
A (B.)
A A (B!
A
dkx-)
Z
T
Z
'H'2 Zo.
Then we have the following lemma on the SVD of G Lemma C.2. If d > k, then we have the following SVD of G
G Ua Sax Va:
where
VG
Va
Oxx(d-k) [(d-k)
: and Sc: Sca 0dx(d-1)
(d)
If d < k then have the following SVD Gf G2
G UG SG: Va.
where
UG+
ÜG: (k-xi
(Bx(k=d) [(=-d)
and Sea
O d
Proof. The block construction does not affect the original part of the SVD. It only appends zeros to the singular values and grows the corresponding orthonormal bases as partial identity matrices appropriately.
Now we can apply Lemma C.2 for Lemma C.1 to extend to d + k via the following lemma. The proof is direct and we omit it here.
Lemma C.3 (Schur decomposition of H under & k). Given the defined block matrix H E 2sxx2 with max(d,
have the following decomposition
H =TC
If d > k.
1
Ug:
U
Ia SS
C
T
V2
Va
Va
Ia n SSa
Ifd k,
UG:
Uc:
In + nSa
C
V2
VG:
VG
1 S
1.1.2. DYNAMICS OF LINEAR APPROXIMATION
The target of our proof is to demonstrate that Êt④1 does not effect the dynamics too much such that the dynamics of Z, is close to the following pseudo iterate
A lin (B.in)
Zi H Zo
(14)
The updates of the pseudo iterate follow the trajectory of Oja's Power Method (Oja, 1982). Therefore, we aim to prove that the error between the actual iterate Z, and the pseudo iterate Z, is sufficiently small, which is equivalent to that the actual iterate Z, performs a power iteration during the early steps. First, we obtain the difference between Z, and Z by the following lemma.
19

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Lemma C.4 (Formulation of Ed). Under assumptions in Section 2.1 for the linear setting, given the nonlinear dynamical system (13) and its linear part (14), their difference admits
E, = Ze
Zz Lin
H E., VIENT
(15)
where E, corresponds the nonlinear part in Eq. (11).
Proof of Lemma C.4. We prove it by induction. Recall the formulation of the nonlinear dynamical system Zi+1 such that
H Er-1, we start with the base
Z1 HZ E1 Z E1.
which proves the claim. Next, we assume Eq. (15) holds for / 2 2, then for + 1, we have
Ze+1 = H Ex+1
= H
Z,in
Σ H
Hi-iE;
Ej
- -E
i=
H +++1
E.
E141
(=1
Ziin
H
E.
which proves the claim.
Eg lop is sufficiently small within a certain period, e.g., S T, then we could approximate the early dynamics by
AA41 B B
lim
A Jin
G
A in
Z1+1 =
~
Z
(Bli))
(B.an)
+
[n+
o
(Blin)
via
A B
Ain (Blim
E
Egllop-
op
In this subsection, we will bound E op show that it is actually small up to the initialization. To prove it, we first conduct the dynamical analysis of Z via the structure of H.
Part I: Dynamics of Z
With the algebra fact above, we can derive the precise spectral dynamics of Zfin i.e., ALin and B separately.
Lemma C.5. Under assumptions in Section 2.1 1 for the linear setting, given the pseudo iterate (14) on Z, where two components A and Blin admit the following recursion
A 1o (la+ + (Ia-nisc n)(6:) )ns==)))(d
Ao:
PA
(B.ia)
= (TI++
1 Va:
Uo: Ao
2
Furthermore, if XT X non-singular, P.A. is a full rank matrix and singular values are after the r order. P. is a rank-r" matrix.
20

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. We start with the special case d k and then discuss the case of cl k. For the case of d = k, we have
Z, in = H (CTC)) CT'C Zo
where the last equality follows from the fact that C is an orthogonal matrix. Next, we compute T
La ( nSan)
0
T =
(16)
Odxd
(Ia-nSSi
Then, we can derive the following recursion
2, in -H'2 Zo
La + +S:))
1 UG -
0
C. Zo
v2 Var
Va:
Ia S )²
Oaxd
Üa
La + Sea)
Üma [Ia - [d--1)
C Zo V2
Va
Ia + S
Vo: [I n)
a
[a 1 (Ia+
+ (u-nnSa
Ug *
Ao
Va
(I++n5e)
1
(Ia 1):
UG:
0
a
(Ia + nSa
+ (Ia- n)
Ug: Ao
Va
(Ia++15
(Ia- 1):
|U:: Ao
Next, we extend the results above to d & k. Here we take d > ki
B
1 Va:
[a 1) IIa (Ia-1/c:
U Ao
2
Vo
(Is+ )² - (Is (--n)s))))
)
U UG: Ao 0
which proves the claim. Lastly, we take d < ki
A
- UG
1 -
(Is +la+n5co 1 (Ix (la-nsc)) )' UG3 Ao
1 U
(Ia+n5er
(Ia Va-nsac )² Ü Ao
(k-d))xr
which completes the proof.
Besides, we discuss about some properties of P.a. and P.a. Recall Rank Rank(A) = r", then we have
Ay- Ax- (P.A) - A (Ia + (n n )' + (Ia - ))a))))))))))))))))))))))))))) = A1 S < (d-r").
That means P.A. E Rdxd is a full rank matrix and the singular values are I after the -"tth order. However P... E BRk is a rank-r" matrix.
Part II: Control E
Based on the above results, we are ready to prove that Ex Eellop is small.
21

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Lemma C.6. Under assumptions in Section 2.1 for the linear setting, with LoRA initialization (LoRA-init), given A and G in Eq. (5) and its largest singular value X1 (G consider the following time period
(G9))) Ao 1ap
In
t
3 3 (1 + ()A)(G)))))))))))))))))))))))))
then the following statement holds with probability least 1 2C exp(-NN for universal constant C over random Gaussian data
Eoop - Aollop
(17)
Remark: By choosing proper random initialization variance over Ao, we can ensure 1" > to avoid vacuous upper bound.
Proof. We will prove by induction. Starting from = O, this is trivially true since Zo = Ziin Next, we assume Eq. (17) holds for with 2 1 and prove Exllop I Aollop- To deliver the proof, denote ag : || from Lemma C.5, we know that
1 (1+ | B S
| lin llp S (1 + m (G+) ao.
(18)
Besides, since (A, - Alin) and (B: - Blin) are the sub-matrices of the error term E,, our condition || |Elpl S Aollop we have
A Lin
op S E
op:
(19)
B Blis llap S E lop
implies that
1 Bi-illop S (1 (1+ ||E-----lll
AA-1lo S (1 M ( + 11
2
Besides, according to covariance matrix estimation in the operator norm in Lemma E.1, with probability at least 2C' exexp((NNe) for a universal constant >> 0, we have (taking é 1)
X
X
La
S6=1.
(20)
op
Accordingly, with probability least 1 2C' EEllop can be upper bounded by
EEplopp $n
N X
X
A, - B, B
17
B
A.
X
XA
N
op
<1( + e) A
1 lpp|
B B op
+ 71 + c) A, - 1| B Fl
[using Eq. (20))
< (1 + <)mll A
-
op
B
op
B op + || : (lop)
s (1+ +) <)n (1) (|A
((ppp))
(|
+ |.-ollop)
(I + ||A)in llop E1
[using Eq. (19)
Accordingly, using the upper bound of
||A op and |B in Eq. (18), have
( (1 (G)))))) ap + (Er-1llop)
E op s (1 + e)n (Ai 1-1
a0 +1EEe_1llop)
3 (- (1 + (G)) t-1
00 + 2 E211E1
$ 2(1 + c)n (1+ 31-3
ag + EE 113
< 6n (1 m))i(())
3(-3 ag.
[from our inductive hypothesis]
22

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Then, by Lemma C.4, can conclude that
E lop
H lop
Eillop
op
1=1
s Gnjag
x
1 + 11
4421-3
[using Lemma C.I]
Gnjg x (1 + 11I
Σ (1 M1 (G))
==1
7/1 (G 21
-nag x (1+
[geometric series]
(1 771 (G²)
$ Gnaj x (1+ (1+ 1-1 (1+ m1 (G)) 21+1
27121 (G
3 31+ 31
(21)
(G
(a In
A
Accordingly, when t St
31n(1+-AA (G:jj we have
EEllop S Aollop:
which proves the claim.
C.1.3. ALIGNMENT TO NEGATIVE GRADIENT OF FULL FINE-TUNING
Now apply Lemma C.6 obtain
| A!1m ll.p S Alooolpp.
Recall Lemma C.5, A left singular subspace to the left subspace of the initial negative gradient step G of full fine-tuning. We anticipate that X,- (A,) > x1++1 (A,) for sufficiently large t. Furthermore, if EEllop remains small, then the top-r" left singular subspace of A, can closely align to 's's To prove this alignment, we modify Stöger & Soltanolkotabi (2021, Lemma 8.3) to obtain the following results.
observe that the dynamic of A!tn also follows Oja's Power Method (Oja, 1982), which aligns
Lemma C.7. Under assumptions Section 2.1 for the linear setting, recall P.A. je ((I+
(Ia- - (i-n)so)) ) Ue: as Rexd symmetric matrix Lemma C.5, we assume that
Ar+41 (Pt)((PA)))lll)) + EE < X,- (P. (Paii (U. (PA))
that can be satisfied under certain conditions (discussed later). Then the following three inequalities hold:
Ap (P.. A + EE) 2 Ay (P.A) Amii (UT. (P.A) )o) - || Eellop
(22)
( Ao + E,)- S A )| ||Aollop + E llop
(23)
dr--1 (P.)) )| Aullop + || EEllop
J JT.._IPP (PA) (PA Ao + E.) (P.())) (U, (P,A) An) Ay- [|
(24)
|
where U denotes the left singular subspace spanned by the k largest singular values of the input matrix M and UR.1 (M) denotes the left singular subspace orthogonal to Us (M).
This lemma can help us derive the principle angle of the left singular subspace between A and A, Note that the assumption comes from the necessary condition of Wedin's 's sin & theorem (Wedin, 1972). In the next lemma, we aim to derive the time threshold which can fulfill this assumption.
23

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficienlly
Lemma C.8. Under assumptions in Section 2.1 for the linear setting, given Aollop for any (0, 1), taking
In
811Aoolop
Ammmn (U., (P())A))
1. In +1A,- (G9))
then Eq. (24) holds with probability least
20 exp(=NN) a universal constant C over random Gaussian data, i.e.
| T (P. Ao + Es)llop S0.
Remark: To ensure that the 0-alignment phase still falls into the early phase in Lemma C.6 for |E S ||Aollop we need to choose proper initialization for Au- We will detail this in Theorem 3.2 later.
Proof. First, Ap- (P.A) in Lemma C.5 can be lower bounded by
X, (P.A)
Ar (Ia nSar + (Ia r)aa)))
Ap (Is + )S=))))))))))))
(25)
(1 nd, (G*))²
Recall Lemma C.5, we have ( and Lemma C.6 with Eellop S 1lAoolopp we define the following threshold and upper bound it
Ay- +1
(PA)] A
Aollop + Eellop
7:= X, (P.A) Amin (U,. (P.A)
A llop
[using Lemma C.5, C.6]
1 (1 + mp, (G:))* Amin(U,) (PA.)
2
4 4||Aulle llop Xmnin (U,,. (P.A An)
exx (- In (1 + ndp (G*)) -t)
(26)
Set 0 E (0, 1), let (q.(2 8. then have that
| [ Ao + Es)loop 50.
The time to achieve this angle can be upper bounded by
Al1Aoll op Amin (U,. (P.A) Aa)
0
exp (- In (1 n,,, (G*)) .t)
S
2
which implies that
In
AllAollor 0Ammn (PA))A))
5
In nd,- (Gh))
Finally we conclude the proof.
Theorem C.9. [Full version of Theorem 3.2] Under assumptions in Section 2.1 for the linear setting, recall G defined in Eg. (5) with its condition number x², we consider random Gaussian initialization Ag E Rddr with [Aulas ~ NN0, in (LoRA-initt, for any E (0, 1), let g o(1) be chosen such that
BE
A (G+) 27d
ifr" S 2r",
24r
2
(G5) 27d
ifr 2 2r'
24v7
24

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Then if we run gradient descent for 1x steps with
ifr* Sr 2r Iy (G=1)
E S
In 20y/d
ifr 2 2r",
In( (G+)
we have the following alignment on the left singular subspace between G and A,-
(G))) (A.-)) 00
1 - Ci CC-Ce-----
ifr* Sr < 2r*, ifr 2 2r",
with probability at least
1 cxcx(ppp Cs exp(-r) - Cexp(-NN
for some positive constants C, C1 C2 C3 Cs,Cs. Here U,- (At.) denotes the left singular subspace spanned by the r* largest singular values of Ar and Ur,,1 (M) denotes the left singular subspace orthogonal to U, (M). that we can select any pair of stepsizes (n) m) that satisfies the conditions to > 1, 17 2 17. and Ç(77, 17) = (11)
Proof. For ease of description, we denote Ag == oT E R where T is standard random Gaussian matrix with zero-mean and unit variance. Here we aim to choose proper a to ensure that 0-alignment phase in Lemma C.8 still falls into the early phase in Lemma C.6, i.e.
81 Aollo
(G))
In
In
QAmin (U) (PA))A))
3 A 1
= 6- (
In (1 + 1/ (G²))
3 (1 17/11 (G²)
8|1Aollop
In (1 + n/,,-
A (G
In
In
0 0Amin' (U,L. (P())Ao)
31n (1 17,1 (G))
3| Ap 10
In 1+ (at))
8|| 8llooo
(1
nb1 (
0mmin (U,. (P.A)
A. 1ip
(1102 (ct)
81| lop
3 A| 11
(lm(31n)((Ct)
Amin (U,, (P.A) (PA))A)) AA))
A (G2)
8|T
3||AAul2
1n(1+p/.. G
by setting 1 = 3 3ln(++q) (Gt)
Amiu (U,.. (P.(P)))T)
A1 (Ga)
8||T||o opp
3||T| 2 op (G*)
a²
Amiu (U, (P.)))
In the next, will discuss how to pick up a. According Lemma E.3, we need consider the following two the relationship between 7* and r.
Case 1. r* < r < 2r": by Lemma E.2 and Lemma E.3, with probability at least 1 - C1 exp(-d) - (C2E) Cxexp(-r) for some positive constants C1 C2 C3, we have
T op <1, rdmin (U. $1. 3vd
(27)
Here we pick
00
Ai (Ga)
24
27d
25

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
then recall Lemma C.8 the alignment, we take a here
1
(-VwI(w1))
U. (A,-)
81 op
3 op
a
Amin (U,- (P()T)
(GH)
8| op Amin (U,- (P(A)T)
3
00 24rVd d
(Ai(G*)
(G
27d
8||T lop Amin Amnn(U,- (UA-(P(ATT)
|T
Og 24rVd d
3xch
9d
| (T
|
0g since i 2 1/3rc and
24rv E (0,1)
3r /r (UT. (PA)T)
9d
Then using Eq. (27), with probability at least 1 - C1 - C's exp(-r) for some positive constants C1 C2, C3, we have
(-r.i U,- (Ar:) 20.
And compute the upper bound of t"
In MAmin (U,, (PA
8||A|las
In 24ryd
In In + 77y- (G*))
In (1 + 7 (G¹))
Case 2. 2 2r by Lemma E.2 and Lemma E.3, with probability least C3 exp((pp) C3 exp(-r) for some positive
constants C4, Cs, we have
op < 1,
Amiu (PA.)) 11
3vd
Here we pick
0
A1 (Gt)
24vd
27d
Similarly, we can obtain
|07 (-VwT(whh) U, (Ai)l
5 3vddunn (U,,. (PA)TT
T 0
|
|
op
2
$ 0.
9d
And we can compute the upper bound t" as
In 24v 8' < In (1 + TTpp (G9)
Theorem C.10. Under assumptions in Section 2.1 for the linear setting, using the LoRA initialization for Bo = O, then for any time-step N4,, we have
VIT (-Vw V, (Bi)l =0.
26

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. We prove by induction. Recall the complete SVD of A in Eq. (1) as
0, ((d-r') A = Ü [U U1]
S
(d-r)))
0 (d-r*) x (d-r)))
V1
Fort = recall G
XA in Eq. (5), we have
N
B G.V.
A
XAV, -Oyx(d-r))-
N
Assume B, V Oxx(d-r)) holds for any E N and / then
B VI B.V !
XA
A, -xx(ddrr))
N
N
which completes the claim.
C.2. Gradient Descent under Spectral Initialization
For notational simplicity, we denote Σ X X the following content. Recall the negative gradient of Full Fine-tuning the first step in Eq. (5), write it here again
G V-
Y
ΣA Ug. V
(28)
In this section, according to Lemma L, the following statement
|| Lal| = < min
1 c
2
for some small constant e,
(29)
2k
k3
2
holds with probability at least - 2C excpp--2 for universal constant C > 0. We propose the following initialization
scheme (Spectral-init)
Ao = UG:
11/2 [1r]]
Bo
5//2 &
V
[.1mr]
[1x]]
[.1b]]
First, we have the following lemma.
Lemma C.11. Under assumptions in Section 2.1 for the linear setting, with spectral initialization (Spectral-init), recall
K = (44 (A), then with probability at least with probability - 2C exp(-eeN),peeeeeeeeeeeeeeee N),Noorr for a universal constant C > 0,
we have
|, Allp <cllAllop S x;-
(30)
2
and
Ap. (Ao) 2 x
Vx 2
2 Apt (Bo)
(31)
Proof. Due to rank (G*) = r" and 2 r², then Bo = G Accordingly, by Eq. (29), with probability at least
1 - 2eexppccee we have
Ag - A -
Ag - G*
+||G* -Alla
| A llop
E La
A
[using Eq. (28))
<
Ia
|
lop
el|
A
lop
1
A
[using Eq. (29)
2x
X
27

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Then, using the above result and Weyl's inequality, we have the upper bound A,- (AgBo) X1 (Ao) A,- (Bo) and the lower bound
Ay- (AoBo) Ay- (G*) A,- (A) -| Alppp d, (A) |AgBoo All
Now we are ready to give the lower bound of Ap. (Bo). Because of Ag Bu = G under spectral initialization, we have
d1 (Ao) & (CH)) & - Ia op
A1(A) & Von(A)) with high probability at least 2C 2C'xx(--22
where we use G ΣA and the concentration results on Σ. Then combining the above two inequalities, 2,- (Bo) lower bounded by
Apt (Bo) 2 A, (o)
12
\ (Ao)
A (Ao)
2
by taking 6 <
1 2x"
The lower bound of A,- (Ao) can be obtained similarly.
The following lemma indicates B,'s GD dynamics stay in the low-dimensional target subspace under the spectral initializa- tion.
Lemma C.12. Under assumptions Section 2.1 for the linear setting, with spectral initialization (Spectral-init), during the iteration, for any N+, we always have B, V (ax(d-rr)- where V1 comes from the complete SVD of in Eq. (1).
Proof. We prove it by induction. First, recall the SVD of A in Eq. (1), we have
G.V εAV -ax(ddr1))
and
2/22
B
$
V1
G:
[1r]]
[.1r]]
1/2
U
V
[1>]]
[1
--2
S
U
ΣAV
[1mr]
Odx(d-r))
Next, We prove by induction. Starting from & 1. using the above two equations, have
B Bo V
A X
X( + AoBo) Y V
N
B, V
A X X B.
N
Odx(d-r')-
Assume B, V1 -dxxdd-r)) holds for any 2, 3, ..., then at + 1, we have
B+s1 V1= = =B 1 A X XABBV + GMA
= -dx(d-r')-
Accordingly we finish the proof.
Under spectral initialization, we have already demonstrated that A, is close to A. In the following content, we aim to track how ||A, All,p behaves (in a local sense), which is a critical ingredient to study both the loss and risk LoRA training. In this regime, there is no significant difference on setting different step-size 171 and 72. For ease of description, we set 771 = 1/2 : n).
28

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Here can characterize the operator norm of (A,Be A) as
|| A - oloop
A,B. - 4) IV vi]
[by unitary invariance of operator norm]
op
|A,BBV - US
[by Lemma C.12]
(UU +ULUT) )(
(A.B.V -US op
I (o(U UT -
+ +|IULUIA
op
op
s U A,B,V - S + | V
(32)
signal space
complementary
where the first term denotes the loss in the signal space |
ABV
S llop and the second term denotes the complementary
space decay | ABV lop complete SVD of A in Eq. (1) as
Next, we need new parametrization to track the dynamics of these two terms. Recall the
A ÜS (U U1]
S ((d-r)xx-
O (d-rr)
V
0 (d-rt))x
V1
For notational simplicity, we denote
A = U
A = U, B. = BY, B.V = B.
and thus
R. = (A,B: - A)V, R = A - SS, R = A BY
Accordingly, Eq. (32) can be reformulated as RRello, <| + || Ilap By Lemma C.12, we have B (k-rr)
Vt E N Next, we can track RG and R. via the following two lemmas.
Lemma C.13. Under assumptions in Section 2.1 for the linear setting, with spectral initialization (Spectral-init), we have the following reparametrized iterates
A A. R (BX)² - - ( ( Ia] R. (B A A -nR (BY) T -NU ( Ia) R. (BY)
(33)
(34)
BV, BY 17 (A4) T R -1 (A R²
- (A()) UT (S- La) R2 17 (A
U. ( (5-14) -la) R.
(35)
Proof. Recall the gradient update for A141, we have
A.11 A. E (A - A) (B.)
A. 17 B, (B.) -11 (E T.) (A,B, (B.)
Recall R. = (A,B. A)V and A V we have
UT A.t1 = UT A, T (A,B, A) (VV + VVVI (B.)
nTUT (5 (S-1)) (A.B, A) (VV VV. (B.)
=UT A (A.B. - AV) - -TU ($- La)
[by Lemma C.12]
=UT A. A ABV - (B.V) πUT ($- Ri (B.V)
29

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Accordingly, the recursion for A is reformulated
- = AV -n ( - nn (i (I-Ia) Is) R. (B.)²
Similarly, can obtain
A - (BY) NUI ( ($- Id) Rr (BN)
Regarding the recursion for B,+1, we can derive in a similar way
B4+1 V B (A.) (A, - VVV
B,V (AD (UU +ULUI) (A,B, - A)V
(A.) (UU +UUU)) (2- Ia)
which implies
BY = =B. -n (A R2 - 11 (A7+) R - 1 (AH) UT I ($- Id) R. - 1/ (A
T
U. { Ia) R..
In the next, we are able to characterize the upper bound of || R llop
Lemma C.14. Denote M, := max
R llop
R,
]pp] under assumptions in Section 2.1 for the linear setting, with
spectral initialization (Spectral-init), then we choose with probability at least 2C exp(-e? for a universal constant C' > 0, we have
R 1lo 5 (1 (AZ. (4 + 22. (BK))
Mr
+ 2 B
M. + n/ |
BB
M 2n A
llap
|B llap M²
op
op
op
+ |
A U1
Me + n²
|B||
1l
A || M²
lpp
op
BB + 2
A U1
Mr + 2
| 41l
1?
M₁
op
2n |
|B
Mr + 2 || l1.p
B 11 op M?
op
+ 2m/// |A/
11.p
A
Mr + 2n |
||
|B
llop M²
op
op
+ + 2
A
|
||
1lop M₁
and
R
11 op
1 1- (2 (AU-) X²- (B\)) Mr
(36)
+ 21j
BB
2
Me + 1/ || 1l.p || |B op
M₁ + 21f² ||4411 1lop
BB
1l M2
op
+ +
M, 1/ |
|1
A mp M2
+ +2n2c
B
AU
M 2 ||
A
A Mr
+ +
|
|B
M, + 2n |
|B
M?
lopp
op
+ 2me
|
M₁ + 2
-
||
BB V
M²
op
op
+ 2e2
AA
|B
llop
M3
30

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. Here we first track the dynamics of R₁ We have
R A By S
Ri Ti (B By n (S (S-1a) R. (BY) BY
- -nAY (AT) R + n Ri (BX) (A!) Ri + u (Si 13-1, Ia) Re (B.) (AP)* Ri
- -n (A!+)
Ri + Ri (BK)² T (A7*) T R - 1²²² (5- Ia) R. (BX) (A!*) R
- - (A UT (5 1Ba1a) Rd
R (BY) (AL) UT (5-1a) Ia) Rr
+ + (S (S- Ia) Re (BY)* ( UJ ($- Ia) R
- -n (AP U. ( (2- - Ia) R
+ R (BY) (A!- T
) U_[ (f- - Is) Rr
+ U (i- Is) R. (BY) (A! ) UI I Ia) Re
Then, we take operator norm over the above equation. Hence, with probability at least 1 - 20 @CCeppepe2 for a universal constant C' > 0, we have
|R op
<
11 (A2- (AP) + X2. (BK)) |1a
llop
n
+
B
1
| llop
172
| op
BB
!!
R
1
+
+
||
Il.p
BV
||
Rilp
|e
op
op
+1
A
1 op
| op
+ n)²
||
|
||.p
A
U1
R !lop
|
op
opp
+ BB
||
A
U1
|R
!l
|
|| op
+ mE
|| w
2
op
RR
olop
op
+ +
A
BB
| + r7222
||A4
1lop
|
||
|2 1
op
op
+ TE ||
V A
|R
+ T
A l!,,
|BY
||
||
|
ol
||
lop
op
op
+
A
|B op
Next, we take maximum over R; llop and || [2, on the right hand side above. Recall M. = max {1
R
1l
31

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
using the fact that Rellop S 2M,, we have:
R Ri op - 1 1 (1, (A!) + X2. (BK)) Me
2 B M, + n
| 1lop |
M2 2n/² ||
|
!lop
||
llap Mi
op
+ A |lap
Mr + n² |B
1l.p
A
!!
M2
lopp
op
+ + |B
A -
M2 + 2ne |
!| M₁
op
op
+ 2 | ||
|
Mr + 41/222 || 1lap
|B op M²
op
op
+ 2 || | ll.p
-
Mr + 2272 A:-
|| |
llap M
op
2
|
||B||
M₁
lopp
op
Next, we track the dynamics of R,
We have
R4 = A B.
= R. - -RR (B.) B. - nUT (5 Io- Re (BY) B.
n (
R + 1/ R (B\)² T (A!)² R, + U ( Ra (BM) (A!) Ri
- - (4!
T
R +12 (BM) T (AQ*) T
R, + U (s- - In) Ra (BM) T (AQ+) R
- - (A!) UT (S- La) R
+ R (BM) T (A!)²
T UT (5-1a) R₂
T + UU (S (S-Ia) R. (BY) T "A!!) L ($- Ia) Rr
mAA (AF ) UI S 13-1. -Ia) R
T
+ 1 R (B\) T (A/=)
T U. (I-1a) -aa) R²
1/ U (8 1.) R. (BX)
T ( UI (S (S- Ia) R.
Then, take operator norm over the above equation. With probability at least 1 - 2C 22exp(-e2N) for universal constant C'> O, we have
S (1-1
(nain
(A!=) + X2. (BX)))
R
+ 17
|
2
[ op + n ||A |l.p
|BY 1lop
llop
|R|||
+
||
11.p
|| 1lap
| llop
R
loop
lop
+ +w| ||
U A
V1
||22||
llop
+ n)²
||
A
V1
|
2
op
lop
op
op
op
+ + | BY
A
R opp
RRlllog op
+ 7c
[A
U
A
Relloog
op
op
op
op
+ + 1
BV op
|| |
+ 17/22" ||
U
1lap
|BY op
R
12
op
+ mE
A
1
2
[Rlloop
+ +
A 1.,p
L op
|R
R
llop
0p
op
+ +
A U1
||B.
R& 1
op
op
32

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Next, we take maximum over R llop and || llop on the right hand side above. Recall M, = max using the fact that R llop
[i
R
llop)
S 2M,,, we have:
R 1lop 1 1 11- 17
P (4 X². (B\)) M.
+ 2mp
BY 1
Mr + r/ | llop [| op
+ 21/²e
||
llop
B.y
llop My
op
+ + |lop
A -
Me + r/² | llop
A 11
M
op
+ 22/2c
B
A U
M₁ + +nn
,
A
Mr
op
2n |A4|
|B
M, + e2
|||
|B
1lop
M
+
2ne
A
op
M₁ + 21Je
1 !,p
| M
op
e
2
|
|
llop
M3.
Finally we conclude the proof.
Before we move to the main proof, we need to establish a strict upper bound on A, and B..
Lemma C.15. Under assumptions in Section 2.1 for the linear setting, suppose || A. - B, B₁ llp + BellBe S St 1 and S 10A7' if ||Asllop S 2 and |Bel op 2 we choose c &satisyying Eq. (29), then with probability
1 - 2C' (CCeep((-ee N) for universal constant C > 0, we have
| S /
|B S2VA
Proof. Inspired by Soltanolkotabi et al. (2023), we recall the stacked iterate Z, defined in Eq. (12) and construct an anti-iterate
A Z, = - B
Additionally, we define a perturbation matrix
(5- La R
Odxd
R Σ - Ia)
0 D
Then, we can reformulate the recursion of Z as
Ze+1 =
- Z.-1i
12,2% - 2 - r) Z, + n7,2
(Izd ,1Z.Z,) Z, + 7Z, Z. rT 73
where I is defined
Oaxd A
A
r ==
Ok
Then, by the triangle inequality, with probability 1- 20 (e2- for universal constant C > 0, we have
1Z++ llop < || (Iad - -,Z,Z,) Ze op
+1
+,|2 Z, || + b olp + 1 Ze
< 1 -1-1) 17 Zz )
|Zzll.p
[by simultaneous diagonalization]
+1 Z,Z, Z
+ [ Z op
+ 17 | Zi lop
op
< (1 Ze (2))
|| +1
- Z.
|| Zz op + niii | Zz lop + m7 Re || Zz 1lpp
op
33

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
where the last inequality follows from the fact that
7 op = | lop
r op >i:
E op = [ La) R
S Relloo
w.h.p. - 20 exxx(_=eNN)
Using the assumption
Z Z
+ +cl| = || A, B, Billop + c c S Xi,
op
then 12+1 llop can be further bounded by
Zc++ S <11-1122 lap + +1//1 Zeeloop
(37)
Denote x = || /ellopp and [(x) = -(1 - 1122 + 22//1i I, we have (2) + 21j - -370² and (")(x = -61z. Then, we
know ('(x") for 0 attained at x= V 1+2n2ī
3n V 3q A111- As pick 1/ S 1027 then x² 2 A1. which
implies the maximum of f()) attained at x= = 2//27 A7 over x E [0, 22i] since Z Zillop s ///2 and
f(2 2(1 4ndi + 2n = 2 - 4n 22
which directly implies || | olop S 22 By consequence, || |1 2 if |ello
|B /2
since A.41 and B,+1 are sub-matrices of Z,+1-
Based on the above results, we are ready to present the intermediate results M., A.. B,, and
A op
Lemma C.16. Under assumptions in Section 2.1 for the linear setting, with spectral initialization (Spectral-init), we take in data concentration as
1
S min
2x
32x(32Xg + 128
and set the step-size as
1
(1 - e/w)
1 S min
128kA;
1152X;
then with probability least with probability
1-2 -2Cexx( exepppee2 for universal constant C' we have that V
X
M, s
(38)
2
max {IlAellop
|BBll I 22
(39)
> V X
Ar. (A,), (B))
(40)
4VK
32ke /
A
<
(41)
lop
X;-
Also, can obtuin
M41 1 (1 -1--1
Aj. Me.
(42)
64x
34

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. Inspired by the matrix sensing technique from Xiong et al. (2024), we develop an inductive approach to prove the claims on our settings. First,
0, Eq. (38)-Eq. (41) can be adopted from Lemma C.11. Next, we assume Eq. (38)-Eq. (41) hold at é 2 1, recall Eq. (34), by the triangle inequality, we have
| Uh
5 5 7 (BK))
-
+ ME |Be olop
BB llop
lp
op
$ (1 Va (B ))
A
+ Anj.Mev A7
op
(A;-)²
||
+ 2nc2 v2
1 16k
op
(2;-)2
32ke v AY
27e,, vAg
17 16
1x-
32kE / xi
which proves the Eq. (41) at t + 1. Next, by Lemma C.14, we have
1 1 2,:
R2 op S
Mr
SK
+ S + 21/² M 64 3301" + 128 Xi,M: + 128me2
+ 64n2² 22x²² X, + 87
M, + >M, 2
(1 11 Sx
Xr- M₁
+ 1) 1Ged1 + Gdex² + 21/111;- + (4172;. + 2; + +811) + 12862x²
+ 17 (128me²² "x;; + 8 ()x:)) + 12871 Nrx j.M:
Xi.
Mr + 27 16622 + 64ck= + 2n212;. ).M
11 8K
[due to the order dominance]
X
Mr + 21/ 16c>2 + +61en² M
(b n S dENAT
!1 16k
X
Me
x,- by c S 16x(32A + 128K2
32k
where the order dominance from the second inequality follows from the fact that r and e are sufficiently small constant such that the terms in O(nc), 0(2), 0112)) Onr²²) are significantly smaller the terms in O(n) and O(c).
35

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Similarly, we can obtain
R 1lo s (11
17 Xr-- Mr
since Amin (A 2 0
16k
mp Seli + 21/2121- + 41 + +4ex22 + +22mee X,. + +64men N7-- + 12862x²
2 2048c n² 1x.
+8ne2t
+ 64ne222 + 128nE" j..
Xx-
1 11 16k
Mr 2n
Seli + 21/iddr- + 6den= j..
[due to the order dominance]
11 32k
X
MM 27
881i + 64en² MM
[by 1 S 128NA1
X
1
Me:
2 byes S 32m(322 + 128m2)
64k
which proves the Eq. (38) at t 1. Therefore, we can conclude that
(1- Xr-
M+1 1
Mc.
64x
Next, assume Eq. (38)-Eq, (41) hold at / 2 1, we have
A441 B,+1 - (4 A. B,B,) n² (A,B, - A) EΣ (A, B,
+ 7 Σ (A,B, - A) (A,B. - -A) SAA.
Accordingly, we can derive
[| (A]41 BetB (A] A. BaBB)l
(A] - (AT B,
211 11111 op
11 max { 11 1
72, PM
[by Eq. (29)
1+1
2(1-1)
1872
1 - 17
(X;-)2Xx
64kc
1=1
80
1) A
28
< (1;)(();))))))))
64k
1=0
64k
< 187)
11527A;
(1-e/m) byn n S T152X]
< (1 (1-e/mh) (1-e/k)Ai
Since A0 - BBBB) lop = 0 due to the spectral initialization (Spectral-init), by triangle incquality,
[AT,, A(41 - B1+1 I (1 - e//n) Next, by Lemma C.15, we can obtain
|A s 2 |B+tillop < 2 /
36

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
which proves the Eq. (39) + 1. Lastly, assume Eq. (38)-Eq. (41) hold / 2 by Weyl's inequality, combine with
M S 2 we have
A BY 1"11pp 2 x A,- (A B = Ay- (A B 2 1r-
2
Again by Weyl's inequality and the Eq. (39) at time f. 1 we can get
X
V AVF
2 2/1 Ap. (B(11) 2 Ai (A (B(+1) 2 Ap (A B
+ Ar.
2
Besides, (A follows similar derivation. We prove all the claims.
Theorem C.17. Under assumptions in Section 2.1 for the linear setting, with spectral initialization (Spectral-init), we take E in data concentration
X 2xx 32x(32A; + 128k2)
< S min
and set the step-size
1
(1-c/1)
7 S min
(43)
128KA
1 1152X;
then with probability least with probability
2C ( universal constant C' we have that V
1 17 Xp-
A. B: - All S 22r
(1-1
A
64k
Proof. By Lemma 1616, with probability at least with probability 1 2C cxp(-ceX -c2N for a universal constant C > 0, we can obtain the linear convergence of generalization risk
A.B. All S V2 A.B:
A op
TRnk(A,B.) =r by Lemma C.12 and Rank(A) r*)
S 22
1
11
64k
which is independent of the choice of LoRA rank ifr r..
Remark: The above convergence rate is independent of the choice of LoRA rank r if 2 r". achieves an c-risk in O (x* (1/e)) iterations. The linear convergence rate heavily depends on k.
C.3. Preconditioned Gradient Descent under Spectral Initialization
The convergence rate in Theorem C.17 will become slow if the downstream feature shift A is ill-conditioned (i.e., k is extremely large). This motivates us to add preconditioners, which is a key technique to accelerate convergence in matrix factorizationnsensinng (Tong al., 2021; Zhang et al., 2021; 2023; Jía et al., 2024). The preconditioners are derived from Riemannian metric in Mishra al. (2012) which originally formulated (AAT)-I and (B B)-1, Also, there is an efficient variant, i.e. (AAT + AI -1 and (B" B + -1, which commonly used in practice for numerical stability via the preconditioning parameter X 2 0.
In the over-ranked setting (r > r*), B,B, and A, A, are not necessarily invertible. Hence we add the following preconditioners to vanilla GD (4)
A A,
X
X (W: + A,B,) 8) B. (B.B.)'
N
(Prec-GD)
B B. - 7
(A) A.) X X (W (W Y):
where t denotes the pseudo-inverse of a matrix M. Such modified preconditioners are also considered in Li et al. (2025).
37

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
In the following proofs, we will prove that the LoRA fine-tuning can achieve faster linear convergence which is independent of condition number under (Spectral-init) and (Prec-GD). Similar to Lemma C.12, the dynamies of B₁ are still limited to the r' -dimensional singular subspace V of A under (Spectral-init). We can verify this fact by the following lemma.
Lemma C.18. For any natural number 2 0, under assumptions in Section 2. for the linear setting, with (Spectral-init) and (Prec-GD), we have
B.V. (x-(k-)))
Proof. For t O, recall the SVD of GP, ie. UG c VT: in Eq. (28), we have
Bo
1/2
V
-- 1/2 [1r)
ΣAV
-x-(krr))
[[r]]
Assume B,V, 1 = 0dx(d-r-) holds for any natural number / 2 1, then
B = B.V N (AT..) A. (x (WW+ - V.
= 3 AA) AA - V
Orx(k-rt))
[by our inductive hypothesis]
which proves the claim.
We can re-formulate (Prec-GD) to be
A A, 1 (B. (B.B,)',
(44)
B++1 B: B1 A.) A, - -A).
(45)
Before we start our main proofs, we first define the following notations
SVD of product matrix A,B, = VT where Ue E RRxx"" S2 E Rr'""", and V. E Rixr"". Notice that here we employ rank-r" SVD A, since Rank (A, S r" due to Lemma C.18 and Xp (A,B,) > strictly which we will obtain from Theorem C.21.
The left compact singular matrix of A, as UA, E Rdxr.
The right compact singular matrix of B, as VB, E Bix""", Notice that here we take the top-r* right singular subspace of B, due to Lemma C818.
By the pseudo inverse theorem and Jia et al. (2024, Lemma 14), we can obtain
A (ATA)) A, = UA
(46)
(B.) (B.B.) B = VB, VB.
(47)
(B.) (B.B.) ' (A,A,)) AT = V.S. u7
(48)
Lemma C.19. Denote R, := A.B, A, E = E La. under assumptions in Section 2.1 for the linear setting, with
(Prec-GD), then we have
R41 R. UA-NUAUAA R VI
RR
VI U ER,
38

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. With Eq. (44) and Eq. (45), we can construct
R&11
A++1 B++1 - A A,B: -
- -nAi (A A.) (A - A)
- (A, (B)) (B.B.)* BB
(A,B - (B.) (B.B.) (ATA.) (A - A)
RR UUAAUUA ER, E VB, VB.
[by Eq. (46) and Eq. (47)
+ + -UEUS--UUE5 ER,
[by Eq. (48)
R UUA, R TR.VB. - UA, - - V2, 1r ER.
which proves the claim.
In the next, we aim to estimate the signal part R. - AUA, U R. TJ VB, V.. Lemma C.20. Recall R. A,B. A. under assumptions in Section 2.1 for the linear setting, with (Prec-GD), then
R UUA R. - R, V. IF S - 17) Relle
Proof.
R. - UU A, R R, VI,
Rs (V.. Vn + +I VB. U R (VB, Vú, + -L-VB.V VA. - B, B, |F
R& VI, - UUA R.V.. - BB. VB, lF since R (I+ VB. VT.) by Lemma C.18)
=I(Aa 77 (Ia+ UA., R.V..
La - 1) (Ia + UAA 1
R.V VE.1II
<(1 11 |R
In UA < n, since UA UA, rank-r projection matrix
which concludes the proof.
Finally, we have the following linear convergence under (Prec-GD) and (Spectral-init).
Theorem C.21. Under assumptions Section 2.1 for the linear setting, with (Spectral-init) and (Prec-GD)) we choose
1
S min
12/T"
and set 1) E (0, (1+e)=) then with probability at least
20 excpppp2 22 for a universal constant 0, we have
|A,B. A S 1 ( (1-1) 7) X
2 2
Proof. We prove it by induction. We suppose the following two inductive hypothesis
Ay- (A,B,) 2 x
(49)
X- Ao A 1p
(50)
39

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Starting from 1 0, under (Spectral-init), with probability least 1 2C exexp(-E2N) for universal constant C 0,
have
A.. A
G A|1F
[ Ia
A
[by Eq. (28)]
||A
x cv /v**|AAlop
[since Rank (4) r*]
X
$ 2
since e S $1/2v/=H
Then, by Weyl's inequality, we have
A,- (A) - A,- Bo) S A... - Allop S Bo - AFIF
which implies
Ap- (An 2 X
(51)
Therefore, we verify Eq. (49) and Eq. (50) at O. We assume Eq. (49) and Eq. (50) hold at t = -,,3,.. then by Lemma C.19, with probability at least with probability - 2C' exp(-ceN) for universal constant C' > 0. we have
1R+
R. AU-UUA,,, R. B. VT, 1r
+ 17 UA U ER, +n ER, B. V VR ll. + 12 S..,,, ÊR,
F
$ (1 - 1) || R
[by Lemma C.20]
R 11 + 176 | U, Re Relly + m/e R.VB. 11 (1 +c² d,- (A,
by = S c
$ (1 - --1) |Ril lI
+ nC Relle + ne Relle + 27 + c)² [Re
(since Eq. (49) and Eq. (50) hold at
(1 (1 -2c) + 27 + c)2) |Rll
1 ) ) RellF
0.5-2x taking 7 S
[1+e]3
This implies Eq. (50) time /. By consequence, we can obtain Eq. (49) time + 1 again by Weyl's inequality.
Remark: The convergence rate is independent of the condition number of k. The choice of stepsize rj is upper bounded by 0.5-2c E (0, 0.5), which a decreasing function of e. Therefore, if the condition number x is very large and thus is chosen [I+ej²
as sufficiently small, then can reach 0.5 and we still have a fast convergence rate independent of This is particularly useful in practical fine-tuning tasks, where the adapted matrix can be highly ill-conditioned when its rank increases. We can empirically observe the ill-conditioned issues in real-world benchmarks, as shown in Appendix G.5 for more discussions.
D. Proofs for Nonlinear Model
We deliver the proofs for nonlinear models in Section 4 here. The problem setting and results for lo A |F presented in Appendix D.I. In Appendix D.2, we present the proofs of Theorem 4.2 as well proofs for smoothed GD.
D.1. Problem Settings and Spectral Initialization
Recall the pre-training model from Assumption 2.1
Spre (x) (e wa) ER W w
(52)
40

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
and the downstream teacher weights from Assumption 2.2
W = W + A E Rd
with W = [.j. u2.
20
The empirical loss of LoRA fine-tuning is defined
(A) = 2N 11o (x(W3= A,B.)) - [xW:
Next, we can derive the empirical gradients for A, and B, respectively.
V VAL (A., B₂) = N
x
0 X + A(B.) - a ( Oo' [X(WA+ + +,BB)) B
o (X(Wi)) - (W o' (XW,) Bi
x
[denote W. == W A,B,]
1 X (W:) o' (WW)
1 _ a (WW) od (W:) B,
(53)
N
B.
[denote Jw, := F1. - T2d
where the matrix operator Jw RRdx Badk is formally defined as (by denoting W, = W + A,
X (o (xW+)= (x(w))] o (XW).
Jw W-
(54)
Similarly, we can compute
BBL B,) N
1
A A (o (X(W))- -o (xW:)] foooooooo oo (XW.)
= Jw.
For full fine-tuning, we consider the following empirical loss function over K E Bdxk
L (KK)
0 (XK)
(XW:) |
2N
The gradient w.r.t. K is
7L VL(K)
X 0 XK)
(XW:)] o' [K]
Next, we can define the one-step negative gradient of full fine-tuning in the nonlinear case as
G VL (W²)
1 X [o (Xwi)- -o (xwi)] oo (
N
Jw:.
[by definition of Jw in Eq. (54))
Additionally, we define
1 X a (XW:)
r
@ o' (XW:)
N 1 N
(55)
r X
(XW3) σ (XW:).
In this section, we aim to analyze the initial properties of low-rank adapters under (Spectral-init) in a nonlinear context. The high-level proof strategy begins with examining the spectral properties of the one-step full gradient matrix, G Unlike the linear case, the presence of nonlinearity prevents direct analysis. To address this, we first establish the concentration of the

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
empirical full gradient, leveraging the fact that the empirical gradient approximates its expectation closely when the sample size is sufficiently large.
Subsequently, we utilize tools from (Brutzkus & Globerson, 2017) to derive useful properties of the expected gradients. These properties are then transferred back to the empirical gradients through concentration results. Finally, since low-rank adapters under (Spectral-init) represent the best r-rank approximation of G we apply matrix analysis techniques to derive the desired results. Also, concentration results this part can serve as an important component for the later convergence analysis.
D.I.I. COMPUTATION OF FULL POPULATION GRADIENTS
First, we simplify Fie and 2, which defined in Eq. (53) to be
1 N
Σ
o
x wj) o' ( w(,1
o (,ل tot o' (अ) wr
N 1m1
and
N N
T 2.
a wr,1)
o') wr,1)
(#. wx d' (r, wr,,)
where Wim is the m-th column of W = W + A,B, and Th is the m-th column of W The following two lemmas provide the columnwise expectation F1 and 2, respectively.
Lemma D.I. Under assumptions in Section 2.1 for the nonlinear setting, for any S ki have
w ll2
Bj To' (z Wi.m) a w
sin 0 Wrm W m + (x O(wi, wxh)) i
(56)
2x
and
Ex
woo' (H we. (e we.)) = Wt.
(57)
Proof. First, for any 1 < m I ki we have
F
[o' @ w..))
(x wi x wmm) (3t whh
Ex
(wt
Ow
sin (mm th De m + (Tr w) w
[by Lemma E.4]
2m
[
The proof for Eq. (57) is the same as that Eq. (56), i.e.
Eg
o' (H Wem a (x wem)) x
1
|uermlle sin Oww.. We. W (T - O(wr.m We,m)) Wt.m
[by Lemma E.4]
2xt
|
1
2 Wt.m
Next, we can obtain the expected full gradients via the following lemma.
Lemma D.2. Recall W := W + A,B, suppose ||,,B, - Alp S phç- for some constant E (0, 1), under assumptions in Section 2.1 for the nonlinear setting and Assumption 4.1, then it holds that
1
Ex [Jw.]
2 (A,B, -) v((),
42

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
where (1) is defined as
(t) = (A, (AD()))) + W D1(t) - D2(t)) (I - Da(t))
with
sin [0(00 + re, md}o sin [((z)t Tr, mttt) }
(t(t) == Diag
(2 Diag
0(iij rt, i..).... ..... 0(00k + re,k wit)
| 1|2
ا ll2
(t(t)= ==
Diag
(58)
|100 + +r.111
| + re
Then have the following upper bound
V (1)l
50
A. Allp
KY
Proof. We first give some notations here. Let W be the m-th column of W E Ba w be the m-th column of W
Am as the mn-th of the low-rank shift A, [A, B:)m as the m-th column of A,
By Lemma D.I, we can derive m-th column of -E [Jw;] for any m = 1,2, ... as
Ex
0 D Wi. ( wmm) a' (z) Wt.m) x
En [(
(z Wi,m)
(nn)))
Wi, x
1
lla
(wi.
with)
sin f(we.. T wi 0(werm nitt, yan
2T
[l
1
| ll2 sin Ow wi )am. O(wi, with )
(IA.
Am)
(59)
2m
|
residual part R
Denote
Pt.m [A, Am
(60)
then we can write
Wi,m th + [A, Am wb + Pt.m
as a relative perturbed time-dependent vector. Next, we take back to the residual part in Eq. (59)
W ||2
R-
sin [e(wū) Tr,m w. ]] (ww) + Temm) - 0(20, + Ft.m W ww
|| 1 + remmll
sin [0 Ti.m w Tt.m
|| T + re.m ll2
|| | + rr,m 1|2 | ll2 sin [0( [0(w), + Te.m w wi
(w + Tt,ml 1|2
+ sin [0(w), Ti.m
whth)] wht 0(w), + Ti.m t W
(61)
Combining Eq. (59) and Eq. (61), we write -E [Jw.] in matrix form
Es [Jw.] (A,B, -) (A,B A) + Wt D1(1) D2(() (10)((44Dgp))))
43

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Additionally, we define
(1) = (A,B, - ()D(() W D1(1) D2(() - D1(1(++ -Da(1)
[A - Dt)) + W (D D2(()
W (D) - Ds(())
(62)
(()
(2(
(a())
For notational simplicity, we drop time & column index and denote w == w and r == Tim-- By condition |A, - S pA we have
||rll S |A,B, Allr p
Next, by Assumption 4.1, we can obtain
||||l | 2
pd.--
1
S
0
(63)
[ 2
KT*
Part Control the Angle f(w r, w)
We denote (r .w) and B
Irl
| then can derive
(m.ar) [ell
+
+
cos (ww + r w)
2
(r w))
VI+ 2a + 33
w
which can imply
B a²
||r||2
sin diw + r, w) = cos2 {(w + w
- (V3-a² a²
(64)
1 + 2a +
|| tw||2
By consequence, we can obtain
|| l|
0(w + w) =0
(65)
|| ||l|
Lastly, by the fact that sin x < aa if then we can have
|||
0(w + +r, w) sin 0(w + r. w7
0
(66)
Part II: Control the Ratio
w
w++rla
We compute
w 2
(1
$ (1-a a //
|||||
1
=
(67)
w + r||2
/1 +
2a + 3
|
where the second equality follows from the first order binomial approximation Vi+
A (1
4 if
<< 1 and we have
|||||
O Kr
by Eq. (63). By consequence, we can have
1 2
| [
5
51+0
(68)
Now, we can upper bound Eq. (62) in terms of Frobenius norm by triangle inequality, i.e.
(t)le
(6)l==
[22( [F)
44

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
For
| we have
[ sin
[0(2))
M2,1
2ith D
| [1)(( v max
|,A.B. AlE
ismgk
| + rr,mll2
|rmmmll
o ||r,mll2
max Ismsk
1 + | ll2
|| - A
[by Eq. (64) and Eq. (68)]
|oo l|
| ll (
1
max lsmmsk
||A,B, A
ll
S 0
|A, All
[by Eq. (63)
kr
For || (2 we have
(2(
< W 1lpp D1(t) D2((1)
W =|
1lop
sin
((wi + rem w.th
-O(uö + Tamm , w.m))
2
m==
| 12 | 12
W
op
[by Eq. (66)
m=
||r
W
opp
Σ o
|| ll
max 1sisk
||200] 18
[rr,i
[
max 1lop
||A, B. All
1sikk
|
llop minigiss ||000|
W
|
0
max 1sisk
||A,B, Al1p
| ||2
00
||A,B, All
[by Assumption 4.1 and Eq. (63)
(kr*)²
For (t(()) Ip: we have
(a(())
Ilon
D (1)( Da(I))llF
| lla
W
Σ
sin [0(wi. + Timm wmm)
Te.mell
m==
| 2|2
max Sm
x
sin²
[0(win + rem whh)
? + Te.mal12
||re,lll2
| 12
max Ssmc
O
[by Eq. (64) and Eq. (67))
@mm l|z
| ||3
m=1
||remllle [ ll2
Σ | ||rem 11
loo max Ismsk
O
[due to the positivity of [| [2]]
minj V5 || 113
W llop
| l|2 /wwh ||2
max Ismsk
I.A.B. - AllE
minngisst
|
1
<0
IIA.B. AAlle
[by Assumption 4.1 and Eq. (63))
45

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Combine the above upper bounds together, can obtain
(t)lle A, A
00
Kr
which completes the proof.
D.1.2. CONCENTRATION OF EMPIRICAL GRADIENTS
In this part, we aim provide the concentration of empirical gradient Jw, F1,t T2, R Recall W = W B, and wtmm the corresponding m-th column of W., denote Fi,y as the j-th element of H;, for notational simplicity, define each element of Jw, Ti, 2, as
x in Frobenius norm.
( wii.) of (a]] w Tag ERR for S Sk,IsiiN,,
(2:)
Then, we can write Jw, in an slement-wise way
cl (ai) 22
1
Jw. N
:
RRi
4,1
(+
(a++
and
d
N
2 a. (ã)
Jw. - En Jw.
G. (ã.)
N
F
m=1m=1 m=1
1=1
Next, we have the following lemma.
Lemma D.3. For 1 S. m S k, 1 J S d, under assumptions in Section 2.1 for the nonlinear setting, with probability at least 1 - 2C' cxp (-Ne) for a universal constant C >> 0 and c E (0,1), we have
N
1
a.
(i,) Es cm (i)
SC CC*K2c <
wi. liiml12:
N
for some absolute constant C* 0 and K /8/3.
Proof. Since Тs,j ~ NN 1) for any 1 m S and < I d, then we have that K := | = /8/3. By the Orlicz-based definition of subgaussian norm, subgaussian norm of random variable is identical to its absolute value. Then, for any R. we have the following moment generating function
a w (T. w wi.m)) o' ( w D]
E
exp
<
exp
2, : wpmm))
[by Lipschitz continuity of a and o']
( || (ūt, w. - wi, 1112 )
<B
exp
[by subgaussian property]
for some constant C* > O, which implics
(3) wtth) - ã i wii))) o' (o]
(Z. wi wi...) 11 s (Cr)) (2x,, w wi... || = (C* - w 11
where the last inequality follows from the fact that || if NO0, s²). Therefore, by Vershynin (2018, Lemma 2.7.7), this implies ci,.. (æ;) is sub-exponential with
Br. == ||c,,m (Ø) ll.. < | 1o ( (zx, whh) - x wi 0' य w 11,, S CCK -wiml22 (69)
46

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Then, let ftm C* e t
mllll for (0, 1), apply Bernstein's inequality for sub-exponential variables
Vershynin (2018, Corollary 2.8.3)
P
x2
En
2. (z)
ct,m
$2C exp
-N min
6t
for some constant 0) [by Eq. (69) and e E (0,
B
B
20 exp (-Ne2).
Theorem D.4. Suppose E (0, 1), under assumptions in Section 2.1 for the nonlinear setting, then with probability at least 1 dC exp (-Ne²) for a universal constant C O, we have
Jw. - Ex [Jwi]
SCKKVVde AAle,
for some absolute constant C* > 0 and K = 88/3.
Proof. By union bound argument and Lemma D.3, with probability at least 1 - 2Cdk exp (-Ne²) for universal constant
C > 0, we have
d
N
2
Jw:
Es
Jww.l
(3,)
Ex
c. (z)
FF
t,
d
(C*
13*
-wi w
j=1 m=1
d()* -
which implies
Jw. - Es [Jwi]
CCe A|IF, A|IF,
which finishes the proof.
Lemma D.5. Recall G = L(W')) = Jw,, under Assumption 4.1 and assumptions in Section 2.1 for the nonlinear setting, with (Spectral-init), suppose e S K2qv//rr= for some positive constant p > and we set 7 = 2, then with probability at least 2Cdk exp(-Ne2) for a universal constant >> 0, it holds that
A, - Allp S pAp
Proof. We start with decompose A - A llop into three components, i.e.
|| A Bo
Allop < - G llop +T||Ga - Eg [ [G*]llop
-
+
E [G
A
Allop
(70)
low-rank upproximation error
concentration eavor
population error
First, for the population error, we can use similar technique from Lemma D.2, since for 1 < m < k. we can obtain
A la O 1
(
(_1
by Assumption 4.1
G
F
<0
|
|F
kr*
47

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Next, for the concentration error, following Theorem D.4, we replace W, with W and then obtain the following concentra- tion with the probability least 1 exp(-x Ne²) for universal constant C > 0, we have
PVr*IAA
PAZ--
|| Ex [I
(71)
3 3
3/2r*
where é S
20 20* K2nr/dr"K for > 0.
Lastly, we can upper bound the (r* + 1) singular value of G (with scale parameter 7) which acts as the low-rank approximation error. Due to the randomness contained G?, we decompose the (r" + 1) singular value into two components, i.c.
+d (G*) S 7d (G*) A
- (r [G*]]| A (71E (G'
concentration crror
population ertor
First, for the concentration error, we can obtain
41 (G) Ar++1 (> [G*]]
1 G Ex [G"]
[by Weyl's inequality]
5> | En G llp
I p\--
[by Eq. (71)
3
Second, we can obtain the population enror as
1
Ar++1
[
[G+])=
+1
EE
[G']]
2 +1 (4)
[since Rank(A) r*]
[G
A
[by Weyl's inequality]
1
Eg
G
A
2
1
50
||
I|r
Kr*
Now we can have
p\2-
||A|| ky*
yd (G <
+
O
O
pA...
(72)
/
Vp*
Therefore, combine everything together, recall Eq. (70), we can obtain
1
Bl Allop 5 (G9) + + E3
|G
+
Eg
[
A
F
pA;- +
1
pd;-
0 Vr
pA;-- + 3/
pt
-
(73)
Vr
/2r*
Since work in the exact-rank case Rank (A,B,) <T= r" with Rank(A) rr" then Rank A, A) s 2r",, this can imply
|| A Allr S V2r l.o - Allop 2 S
which completes the proof.
48

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
D.2. Preconditioned Gradient Descent under Spectral Initialization Recall the loss of LoRA fine-tuning:
(Ai, B,)
1 2N
o X (Wi (.B))
- (xW:)
Then, we employ the following preconditioned gradient updates for LoRA fine-tuning
A A. + n.J. B, (B,B.)
(74)
and
B = B, B. J..
(75)
Similar to the linear case, we define the following notations
SVD of product matrix A,B, == U.S.VT where Ue E Rdxr, S RR" xr, and Vi Rxxr
The left singular matrix A₁ as UA E Rdxr.
The right singular matrix of B, as VB, R
Lemma D.6. Under assumptions in Section 2.1 for the nonlinear setting, we update At and B, via Eq. (74) and Eg. (75)
under spectral initialization (Spectral-init), then we have the following recursion
A441Br11 A= (1 - - (A,B, A)VE, V.
(1-n/22 (Ia U (A,B, A)VB, VE,
+ +(1 +11- n - A) VB, VB,
(Ia UA,UA) (A,B. - VB,
E, VB, VE, + UUAAUU E. 17 Jw, V.S, Jw
(76)
and
E. Jw. (A,B. A)
Then, by choosing E (0, 1), we have the associated upper bound in Frobenius norm
| AlLF
(77)
<(1 11-m)) |UA (A,B. - A)VA, VIJ lF
+ +11-1/2) 7/22 [1 UA, (A,B: A)VB, B. -U (A -) Vn, V
+ -I((Ta - -UA,UA.) (A,B:
- (A,, A VB, VT. II
+ 2n | n [w. Jw.
(78)
49

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Proof. By the preconditioned update in Eq. (74) and Eq. (75), we can construct
A A A,B, A
TJJ. B (B.B,) B, - A (.)A.) Jw.
+ Jw, Bi A.) Jw.
A,B, A
- - (B.B.) B. + r (B.B.) -1 B.
+)+ + +n, A.) -1 A,E. Jw, Bi (B,B ) J.Jw.
-n/2A, (A) (A,,
A,B, -A
-n/2(A,,, A)VB VB. V B.
UX, (A,B; A) + AU _ + Jw, ViST Uu Jw.:
[by pseudo inverse theorem and Jia et al. (2024, Lemma 14)]
from our choice on E, = Jw. (A,B. A). We can continue to expand
B A = (Ia UUA + UA (A,B, - A) (Ia VB, VT + VB, VB.)
-1/2 - UA + UAA
-A, A)VB, VB,
-n/2UA,,A (AB. (I. VB, Vn, +BB VA.)
+ +B/.V V, U - 1 Jw, V Jw.
= (1 - A) (AB
- ( 1/22 (Ia UA (A,B; A) VT. + (1 - 1
V..
() -VB VB.))
+ (Ia - UA. (A,B: - () - -V..
u Jw + E VA, VA, + UA U. 1/ Jw:
Based on the above formulation, suppose 7) E (0, 1), we can derive the following upper bound by triangle inequality
1 B ALL
(79)
< (1 - UX, - A)VB, VI
+ [[(1-//2) (Ia AA (A,B: - A)VB, B, |F
+ [[(1 A - A) BB, VE.)
+ +II(Ta - UAA (A,B, - AT VV. VT.) l|e
+1 E.V F. +1 | UU, =. ||5 +1²
Jww, V.S, Jwallp
[by triangle inequality]
5(1 11-m) UU, (A,B, - A)VV, VI lIF
+ (1 - 22) (Ia UA. (A.B, A)VB, VT. UAA - A) (1. VB, VT.) Ir
+ ||( UA. (A,B: AI -B. VT.)
+ bn | n V.sV '4, Jwilln
(since 7) E (0,
which proves the claim.
In order to derive the convergence rate of ||4 B.+1 AlE g in the above terms, we need to provide the estimation of the
50

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
following four terms
E 1pp |ww, V, IwilF:
[I U.) (A,B; A)VB, VT, + UA A, (A,B, (T VB, VI.lI
[I - (A, A) (I+ VB. V. lF
which are important elements in Eq. (77). We firstly prove the upper bound for |E and |[ V,S, 'i, Jw. ll since they are relatively straightforward. After that, we will handle with the remaining three terms which are the most technical part. All of these three terms rely on the condition A. ALLF I pd and we will prove it by induction finally in Theorem D.10.
30- (Prrv//dr"x with = 2, assume ||, All S ,X under assumptions Section 2.1 for the nonlinear setting and Assumption 4.1, then with probability at least
Lemma D.7. For a positive constant E (0, 1), suppose S
1 2Cdk exp (-22) for universal constant C O, have
E IP S 0
1 kr
+
+C=K3
|A,B, - AllF
Proof. Recall E, : Jw, 1/B (A,B, A) from Lemma D.6, then with probability at least 1 2C exp (-e2N) for
universal constant C' > 0, we have
2 (A,B, - A) Ex Jw.l + Ex [Jw.] Jw:
F
2 B. - A) Ez Jwill
Rg [Jw.] Jwill 1l
concentration
population error
(t)llF
|A,B, Allr + +1EE Jwil Jw.llr
[by Lemma D.2]
A B,
A F
+ V=*
|A.B, All
[by Lemma D.2 and Theorem D.4]
15
which completes the proof.
Lemma D.8. Under assumptions in Section 2.1 for the nonlinear setting, suppose || || Allp S pNr, for positive
constant p > 0, with probability at least 1 - 2C' exp (-e2N for some constants C > 0, it holds that
J. V,S, 14 Jw. S (1++)) 1-p A,B. Allp
Proof. First, with probability at least 1-2 22'eexp exp (-E2N) for some constants C > can derive
Jw. V.S, '4, Jw. llp
|Jw. 112 |V,,,'u4, llop
X a (X(W A.Bi) - (XW:) Ap (A,
@ o' (X(Wi A.B.))
X X(A -2)
F
[by Lipschitz continuity of o']
d (A,B))
N (H
1
||A,B, 1
Ar (A
< <1 + e)2
||A,B Alle
[by concentration of operator norm]
51

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
where the last equality follows from r* and
A (A, 2 An (A) - |, - A 21 (1-)))
With Lemma D.7 and Lemma D.8, now we can prove for the other three terms.
Lemma D.9. Suppose || || B, - A S pxzx with positive constani E 1/41, then it holds that
(L. UA A ( VB. V. S
A, All
V1 8p2
and
I AVR.
- VB, Vo. S I, - Alp
Proof. First, we recall
A B
A B
Z
Z,
and define a preconditioned operator P and symmetrized downstream feature shift matrix Á as
A. A. A11 B; (B. ) 1 P(Z.) = A. A,AAR A,) B. (B.B. )-1
Oaxd
P())) =
Â =
0k
Next, we observe that
1
0 Odxd
A,B, - A
2122 - -1Z - -
2
[A.B. A)
0k
leading to
ZZ.z - -Z,Z,)
Á
|,
Aloo
| (Z.zz 2.2.)
- = A. AAF
2
lop
Based on the compact SVD of in Eq. (1), we can write out the eigendecomposition of Å as
S
xr
U
U
$
$
where $
(80)
0
0
S*
/2
V
v2 -V
Notice that we can also obtain the SVD À as
À = ÚSVT
T
S*
O x
(81)
0
xr
S
VT
Notice that À is a low-rank matrix with 2 because of Rank (A) = r"
I Z,Z, -Z,Z recovers Á, this
indicates that the top-2r" subspace
Z, Z
Z,2,
will align to perfectly. Next, we can derive the projection
matrix for the top-2r" subspace of 1 - Z,Z,
First, we have
A, A, - AT
A( B, -B,
Z,PT (Zi)
(A) -
52

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
which can imply
1
AA A
A.
A A,B, B.
2 Z.P'
(Zi)ZzZ
2
B (A A.)
A
(B
B, A
1
AAA] A.B.
B, Z.Z.
2
A
Similarly, we can derive
1
A,A,
B
Z,P" (2,)
B, B, 2 Z.Z.
2
2
B, A
Additionally, we have
1 Z (Z)) =0(44))((4+))
Z,P' 21) Z,Z (d++))((++))
2
Base on the above identity, we can obtain that the subspace of Z is orthogonal to the subspace of Z, 2, Since Rank (2.2,) S r and Rank Z S r, then we have that Rank 7 2, S 2r since r = r'. Therefore,
construct valid projection matrix
P == (==Z(T((z)) + -,++ (Z,):
(82)
which satisfies
1 Pt 2 Z,ZI) = 2
1 2 Z.Z.)
2
and
2 (Iu+x- PP)) (Z1Z] - -2,2,) = =((++))((++B)-
(83)
Also, we can verify that P, is symmetric and P,P, = P,. Therefore we can conclude that P, is the projection matrix which maps matrices vectors to the top-2r subspace of ] ZZ,Zz Z,Z,) For notational simplicity, here fix the timestamp and denote
Z.Z. - -2,2, 2
F =
which means
F
||A,B, Allr S pr,..
(84)
22
Next, we define P, = LL E ((d+))x(d++) with
== UA
Odxr
E gR(++)xxrr
Okxr
VB,
and (Iatk- - -P:) I where
UA Oxx(d-r)
(ax(k-r) V
E (R(++))x(d+8-22)
L
then we have
L
F
F
L
L.]
22
22
F
IF
LL FL
LT /2
L
-IT
Á
L1
/
[by Eq. (83)
L L
LT
L1
V22
I FF
1
LT ÅL
1
AL
2
L
2
(85)
V2
2
F
53

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Since l4t P. L1 L then we have
[I(Ta UA
VB,
VD.
L
AL
(86)
V2
2
Next, by Eq. (85), we have
F
2
AL
+
2
L
Å 1
leading to
F
F
L
D
ALL
2
2
(87)
|
A
2
L
Å
+ 1
L
AL
V2
F
F
The technical part is lower bound based on Eq. (81), i.e.
|
ÅL 2
and
L
|
2
we will rely on the following decomposition which
AL
=
F
(I]
(1 (
((TV8)) (1T (ITOSI)) (IVSV))) tr
tr (L (LVSS) (I] (IIōSV))
LI (S LL VS (S L L[
Notice that $1/2 LL V and $V20 L,L, US//2 are two positive semi-definite matrices, then by lower bound of trace of product of positive semi-definite matrices, using Weyl inequality, we have
2 A2r- ( LL VS
S-0 LL USUU
F
(v LL
0 L L. 03//22
2 X x A2.
N Aar
(V
L. 1,0000 L L
where the last equality follows from
5 | L L. US122
= tr L -
= = L (1.1,ÜS L L.E.) LIL
= -"(1110000LL L. (L.) (L.E(USULLE))
= - L
Similarly, we have
L ÅL
> Xr. Aer \ (
Next, we can derive
L L 0SO L LT |
Li L $ + + L IT
[by Eq. (80)
-IIII, + | 1 L. LI
+2 L 2 L 1 1.1
54

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
and
VSV L L
L1 L LiL
@
+
+55 L 1
[by Eq. (80)
F
1111 +
1 05 L
L 45' ILD.).
Also, we can obtain
L | - LL L L
| (
25 LILl F
[by Eq. (80)
= L L 2
L
I + LL L 45'
( L L. D L.
Notice that the matrix inner product term is the inner product of two positive semi-definite matrices, then by trace inequality for positive semi-definite matrices, we can obtain
L 2S L
LLL $ L_ L_ET) tr (LLL
os. ( 1 (1E,)) 20.
Then, we can claim
L LIC L
LLE L D
(88)
Next, we can obtain
D
Xr. A2r (O [LE
+ Xr x Aer (V'LL ( 11L0U01
2 Xr min A22> (U
.ar. (V
LLLIVSVLLLL L | + 1 (l)
2 min { A2r. (U A2r (i (v L) | ÀL lg
[by Eq. (88)
Then, combining the above inequality and Eq. (87), we have
1
L
AL
L JAL F
2
F
A
22,- min { /2rr- (ÜTLLT) Aer- (VILLTV) |
F
v2
2X,- min Aer- (ÜLLTŌ) .P2M (vLLLT
Next, we will focus on the lower bound of Agr identical to each other, so here we only prove for Agr- (ÚLLLU) Ú). First, Aer
Ü LUTLLT Ū and Agr
LV
Due to symmetry, the technique
Ü ( = =22 (LTŪ)
since
O LL Ô is symmetric. Next, have
2z ( = =1- 1,0
55

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
where |
can be upper bounded by Wedin's sin(@) theorem, here we use a variant from in Chen et al. (2021b,
oo Theorem 2.9) to obtain
2
F
A
2/2 F V2
A 22
I
v2
op
op
by
F
A v2
S
Nr-
22r0
v2)
which implies
(11)21-8)
21 2188pp
(89)
Therefore, we have
A
0²())-)² p² 1²
(
F
V2
L 2
A
L
[by Eq. (85)
2
2
X min A2r
Ü ÜLLU
(LLLT))
ااايا|
2 X
LIAL
[by Eq. (89) and 1/4]
which implies
D AL
< =222.
Xp.
Finally, combining Eq. (86), we can obtain
2
/² 1- 8p2
[I UL VB
LT
AL
S
A, All
Notice that
1
1
2
1
(Tatk - P.)l +
2 1 (T4k- P.) ÁP, |
AL
+
L
AL
2
F
2
F 2
2
=I((La UA, AVB. B, 1|2 |UA, A (I+ VB, VT, B, ll
II UAA, AV,, VB. +UA VA, V 11
then by the decomposition Eq. (85), we can obtain
(I UAU AVR A VB, VI. < F
|A,B; - A
V2 F
which completes the proof.
Based on the above estimation, we are ready to deliver the linear convergence rate of |A, All
Theorem D.10. Suppose e sc=K==vvsr-> for positive constant p S 1/2 we take ? for (Spectral-init), set E (Cg 1) where G7 > 0 is small constant, under assumptions in Section 2.1 for the nonlinear setting and Assumption 4. I, then with probability at least 1 - 2Cdk exp (-e²N) for a universal constant C > O, we have
A, - A - 11 2 PA:..
56

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficienlly
Proof. We prove it by induction. The following hypothesis holds by Lemma D.5, i.e.
A.B, - p S pA
We suppose it also holds for at time t, then the conditions of Lemma D.7, Lemma D.8, and Lemma D.9 are fulfilled. By consequence, show that
Ay (A, 2 - p)
[by Weyl's inequality]
Next, by Eq. (77) from Lemma D.6, under initial conditions from Lemma D.5, for time t 1, we can derive
1 All
<(1 - 1) |UA U (A,B, A)VB, V.llI
+ +1 (1-1/2) (Ta - UA, (A,B: - A)VB, VE, - (A,B - - VB, VB.) |
+ +II(E,- ((, (A,B, AI VB, 1lr
+ 2n |Bll + " Jw. V.S, 'n, Jw.llr
S(1 - 7) ||A,B; - All
+
1 11/2 +
P
|A AllF
[by Lemma D.9]
VI 8p2
2m (0 (0(1)
+ +C* ||A.B. - A
[by Lemma D.7]
+ 17 (1 + +)2
P
||A,B - AlF
[by Lemma D.8]
1-p
2 1n/2222 +
P
|A,B; - A
VI 8p²
2p
+1
||A,B. AIL
3/
2
+ 7?
(L+ 30* /2 2dr
P
|AAB. AllF
since
1-p
30
with probability least 2Cdk exp (- for universal constant > Since we take p S 20. and p
8-
monotonically increasing, then there exists a constant Cy > O such that Vn E (Cyy 1), we have
| - A| s (1- 2) |A, -
Then, we can obtain the inductive hypothesis at + 1 and prove the claim.
E. Auxiliary Results for Proofs
In this subsection, we present some auxiliary results that are needed for our proof. First, we present the estimation of the spectral norm of random matrices. It can be easily derived from (Vershynin, 2018) and put it here for the completeness.
Lemma E.I. (Vershynin, 2018, Adapted from Theorem 4.6.1) For random mub-Gaussion matrix X E N xd whose rows are i.idd. isotropic sub- -aubsiassian random vector with aab-Gaussian norm K, then we have the following statement
P |
x
La
20
2exp (-CN min (8,,))
for a universal constant C depending only on K.
57

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Lemma E.2. (Vershynin, 2010, Adapted from Corollary 5.35) For a random standard Gaussian matrix S E Rdxr with [8] (N 1), if 2r, have
vd S | & (2Vd+ VT),
(90)
with probability al least 1 C' Ceep(-d) (d) some positive constants C.
The following results are modified from the proof of Stöger & Soltanolkotabi (2021, Lemma 8.7).
Lemma E.3. Suppose S Radx is random standard Gaussian matrix with [S] ((0,1) and U E Rdxx" has orthonormal columns. I 2 2r", with probability at least 1 - C' ('exp(-r) for some positive constants C, we have
Amin (U S)
S < 2r", by choosing Ç 0 appropriately, with probability at least (CE) exp(-r) for some positive constants C, C², have
Amin (U 2
Lemma E.4. (Brutzkus & Globerson, 2017, Lemma 3.2) Define 0(w v) = cos cos-1 (
(w .a)
then we have
[wllal wlla
θ h(w, v) E .t) ((w, a)) ((v .))}
||||
sin 0(w w)w + (a 0(w, )()),)))))))))))))))))))))))))
Ou
2x
||/2
F. Detailed Comparison with LoRA-GA
LoRA-GA proposes the following initialization strategy
2/4
42/4
Ao
U
Bo
Va
[.1r)]
[+++1:22]]
W
W
Ay
(91)
Vr
where & is the output dimension, iss user-specified hyperparameter in constant order. They propose to recover the one-step full gradient G to the largest extent after the first LoRA update, i.e., under gradient descent with stepsize the adapted weight becomes
W off + A, := = + AgBo +
a VF
AJ
off
V
W +
B. B BJ A
fr
update in the full paraneter: apace
Then, Ag and By in Eq. (91) can admit the best rank-2r approximation of G# in terms of full parameter update as they drop the term. However, this scheme has structural limitations in various perspectives.
First, pointed by our theory, B, will align to the right-side rank-r* singular subspace of G under random initialization. That means, due to the way LoRA-GA chooses the (r + 1) 2r-th singular values for Bo the iterate B, does not lie in the desired subspace and may not escape undesirable subspace.
Second, for common LoRA-based algorithms including ours, the global optimization problem is to solve
min
AB
A
(92)
58

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
which achieves the global minimum at the best rank-r approximation of A. However, under LoRA-GA in Eq. (91), the modifications to the pre-trained weight, i.e. W lead to an unfavorable optimization problem, i.e. to solve
min A.B
W
+
a
(AB - Bo
W
Vr
2
AB Ao +
min A,B
which achieves the global minimum at a biased best rank-r approximation of A, i.e. A + A, no matter what initialization it is. This upward bias scales in the order of ((V)) as they propose the scaling to be vk/c² for stability and can be dominant it has stronger signal than downstream feature A.
Lastly, since they ignore 1j²-term in the illustrative analysis, this imposes latent assumption that the best rank-2r approximation of G in terms of full parameter update only holds if the stepsize n, se 0. This restriction is consistent with ablation results from Wang et al. (2024) that LoRA-GA is not robust under moderate/large stepsize.
In contrast, LoRA-One aligns well with our theory under correct subspace specification. We do not modify the pre-trained weight so the optimization problem remain the same as Eq. (92). Also, our method is robust to the choice of stepsizes and can undertake large stepsize to achieve faster convergence as shown in the Appendix G.2.
G. Experimental Settings and Additional Results
In Appendix G.I, we firstly provide the experimental details of small-scale experiments in our text. Experimental settings of various NLP tasks in the main text are given by Appendix G.2, Appendix G.3, and Appendix G.4, respectively. Finally, we visualize the spectral properties both the pre-trained weights and the difference weights after fine-tuning in Appendix G.5 to justify the validity of Assumption 4.1. All scall-scale experiments were performed on AMD EPYC 7B12 CPU. All experiments for T5 base model and Llama 2-7B were performed on Nvidia A100 GPUs (40GB).
G.1. Scall-Scale Experiments
Here we give the experimental details of Fig. I(c), Fig. 2, Fig. 3, and Fig. 4, respectively.
Details for Fig. 1(c) The experimental settings are sourced from Meng et al. (2024). We use 10000 odd-labeled data from MNIST (LeCun, 1998) for pre-training and 1000 even-labeled data for fine-tuning. The learning rates for Full Fine-tuning, LoRA, and LoRA-One are set to 5 x 10-4
Details for Fig. 2: We initialize Ao and B, via (LoRA-init) over variance a² E [1,0.1. 0.01, 0.001, 0.00011. We examine for dimension d = k = 100 and d= & = 1000. We set N = 16d, r" = 4, and r = 8. We construct A = UVT where U E IR 100x4 and V E 100x4 are obtained from the SVD of a matrix whose elements are independently sampled from (N00 1). We set learning rate 7 = 64 We run 1500 GD steps for each case.
Details for Fig. 4: We take d k 100 and N 12800 in common. For: 1) under-ranked case r 4, 8, 2) over-ranked case where U E TR. 1000x and V E 1000xr" are obtained from the SVD of matrix whose elements are independently sampled from ((0 1) and the diagonal values of S is the first r* elements of the dictionary {40, 30, 20, 10, 1, 1, 1,, 1.5.). For LoRA-GA defined in Eq. (91), we use learning rate 1 = 0.5 and stable parameter 16. For LoRA-SB and LoRA-One, we use learning rate = 0.5 and scaling parameter 1.
from. 8, r' = 4. We sample each element of W independently from (0 1). We construct A = USV
Comparison on GD trajectories of Fig. 3: Here we conduct a toy experiment to intuitively compare the GD trajectories under (Spectral-init) and (LoRA-init). We fine-tune simple pre-trained model y x w on downstream data generated by y I (w" + w), where xT . wh w R²² and y, y E R. We propose to use LoRA to fine-tune this model by y = x h + ba) where a = [a az] E R²² and b E IR. Without loss of generality, we set अ = 0 and w [2 1J. The set of global minimizers to this problem is {aj 2/t, az 1/t, b" = =t E R}. We generate 4 data points (~1, , Z. a.) whose elements are independently sampled from .o 1) and calculate for (#1 3/2 M/ ./4. We use the squared loss EE= (7+- ho a)2. For (LoRA-init), we initialize each element of ag from N/O. 1) and by = 0. Notice that the variance I follows from the Kaiming initialization (He et al., 2015). For (Spectral-init), we first calculate the one-step full gradient, g²
i.e. g² = 1 b क. Accordingly, we initialize ag-
and by v [lgg Next, we run GD to train and for
Vlo l
59

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
1000 steps with learning rate ry = 0.1. For each initialization strategy and data generation, we for 2 different seeds.
G.2. Natural Language Generation
The common hyperparameters are presented in Table 6. Next, we present in the order of (MetaMathQA, Code-Feedback, Alpaca]. We search the best learning rate over {5 10 , 10-4, 1 x 10-4, 10 $, 2 10-5, , 10-9) and batch size over {16, 32, 128]. The optimized learning rate and batch size are presented in Table 7. Additionally, the scale parameters are set to {128, 16, 128) for LoRA-One and {64, 64, 64) for LoRA-GA.
Furthermore, we employ the gradient approximation approach proposed by Wang et al. (2024) to replace the full-batch full gradient with stochastic full gradient using a smaller sampled batch from training data and denote the corresponding sample size as Gradient Batch Size. According to the ablation studies on spectral properties and performance under various gradient batch sizes in (Wang et al., 2024), larger gradient batch size only can yield marginal improvement, indicating that it sufficient
smaller batch size for computational efficiency.
Table Common hyperparameters for fine-tuning LLaMA 2-7B on MetaMathQA, Code-Feedback, and Alpaca.
Epoch
Optimizer
(31,32)
LoRA Precision
Weight Decay
AdamW
(0.9, 0.999)
1 x 10
FP32
0
Warm-up Ratio
LoRA a
LR Scheduler
Max Length
#Runs
Gradient Batch Size
0.03
16
cosine
1024
8
Table 7: Optimized hyperparameters for LoRA, LoRA-GA, and LoRA-One.
Batch Size
Learning Rate
LoRA LoRA-GA
22,,32,32
[2 x 10 x 10 -,004 ,5x 10
(32,32,32) 32]
[5 x 10 x0x 10-5, 1x 10-5)
LoRA-One
{32, 322,22166)
{2 4 4,5x 10 10-40 2x 10 10
G.3. Math Reasoning on Full Data and Multiple Epochs
We present the detailed values of Fig. 5 in Table 8. The common hyperparameters are same as Table 6. We search the best
learning rate over (5 x 10 10-4, x 10 x 10 10-4, 4,5x 10-5 2 x 10-5 1 x 10-53 and batch size over {16, 32, 64, 128). The optimized learning rate and batch size are presented in Table Additionally, the scale parameter are set to 128 for LoRA-One and 64 for LoRA-GA. The imbalance parameter for LoRA+ is set 16. The results of LoRA, LoRA+, and LoRA-GA are taken from (Wang et al., 2024) since their optimized hyperparameters align with our search.
Table 8: Performance comparison across different methods and epochs
Epoch
Epoch
Epoch 3
Epoch
LoRA
55.19
58.37
59.28
58.90
LoRA+
56.37
59.21
59.93
59.97
LoRA-GA
56.48
58.64
60.16
60.88
LoRA-One
57.54
60.84
62.62
63.80
G.4. Natural Language Understanding
In Section 1. we have presented the experimental comparisons between Algorithm 1 and typical LoRA based algorithms. We follow the configuration of prompt tuning as Wang et al. (2024). The general hyperparameter settings are provides in Table 10. To ensure fair comparison, we tune the learning rate via grid search over {1 x 10- 5 10 2x 1x 10-1).
60

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
Table 9: Optimized hyperparameters for LoRA, LoRA+, LoRA-GA, and LoRA-One.
Batch Size
Learning Rate
LoRA
128
1 10-4
LoRA+
128
5 10 10-5
LoRA-GA
128
5 10-5
LoRA-One
128
2 10
Additionally, the scale parameters for LoRA-One are set to be {128, 16, 128, 128, 643 for MNLI, SST-2, CoLA, QNLI, and MRPC.
For Section 6.1, the learning rates for one-step gradient update with gradient batch size 2048 are set be (0.1, 1.0, 0.05, 0.1} for SST-2, CoLA, QNLI, and MRPC. The learning rates for low-rank update 8) with gradient batch size 8 be {1 x 10 0.1, 5 x 10 2,5 10 10- We omit results for MNLI since the test accuracy remains at 0.0% for the first dozen steps in both full and LoRA fine-tuning, likely due to substantial structural discrepancy between pre-training and downstream tasks. Table 10: Common hyperparameters for LoRA fine-tuning on T5-base model.
Epoch
Optimizer
(31,82) 32)
Batch Size
Weight Decay
LR Scheduler
AdamW
(0.9, 0.999)
10-8
32
cosine
Warm-up Ratio
LoRA Alpha
#Runs
Sequence Length
Precision
Gradient Batch Size
0.03
16
3
128
FP32
G.S. Empirical Verification of Assumption 4.1
We perform full fine-tuning for the pre-trained T5 base model (Raffel al., 2020) SST-2 dataset from GLUE (Wang al., 2019) to approximately access the downstream feature matrices. To ensure better convergence, take the hyperparameter settings which are presented in Table 11.
To validate the part i) of Assumption 4.1, we inspect the spread of the metric values presented in Fig. 7. We can clearly observe that the ratio between the operator norm of fine-tuned weight and minimum norm of neuron within each layer is bounded.
Fine-tuned T5 Base on SST2
35
30
25
Frequency
20
15
10
5
10
15
25
30
W
min
|[
Figure 7: Histogram of the metric values defined presented on x-axis for all fine-tuned weight matrices.
61

---

One-Step Full Gradient Could Suffice for Fine-Tuning LLMs, Provably and Efficiently
To validate the part ii) of Assumption 4.1, we collect top-32 singular values for each pre-trained layer W of pre-trained
model. After training, we collect top-32 singular values for each difference weights, i.e. AW W fine tuned W The results are shown in Fig. 8, We observe that, across all layers, the singular values of the pre-trained weights are significantly larger than those of the difference weights. For example, the layer on the right has pretrained operator norm exceeding 200, while its downstream operator norm is only around 4. Moreover, the singular values decrease drastically as the index increases, indicating an ill-conditioned behavior during fine-tuning.
Singular values of W
Singular values of AW
200
3.5
175
3.0
2.5
125
2.0
100
75
1,5
50
1.0
3 6 12 15 18 24 24
6 9 15 18 21 27
Layer
Layer
Index
Index
Figure 8: Left: top-32 singular values for each pre-trained weight matrices W Right: top-32 singular values for each difference matrices AW W fine-tuned W after full fine-tuning. The Index is ranked from the largest the smallest singular values.
Table 11: Hyperparameters for full fine-tuning on T5-base model used for Appendix G.5.
Epoch
Optimizer
(31, B2)
Batchsize
Weight Decay
LR
LR Scheduler
Warm-up Ratio
10
AdamW
(0.9, 0.999)
10 10-8 8
32
0.1
1 10-4 4
cosine
0.03
62


Reviews:
[{"review_id": "2m8K3KnkV5", "reviewer": "Reviewer_4cE1", "summary": "This paper focuses on the learning dynamics of Low-Rank Adaptation (LoRA) and proposes improvements in initialization and gradient preconditioning.\n\nThe authors analyze both linear and nonlinear matrix factorization cases, where the objective is to minimize $\\|\\tilde{X}(W+AB)-\\tilde{Y}\\|_F^2$ using gradient descent. In the linear case, they show that the factors $(A, B)$ in vanilla LoRA align with the top singular vectors of the he first iteration's full gradient. To accelerate this process, they propose spectral initialization based on the full gradient, and prove linear convergence under such initialization. To further address ill-conditioned targets, they introduce preconditioning, leading to a condition-number-agnostic convergence rate. They also extend these results to nonlinear settings. The paper provides insights into why introducing full-gradient information benefits fine-tuning.\n\nFor practical applications, the authors propose LoRA-One, a fine-tuning method incorporating spectral initialization and preconditioned GD, achieving performance gains on the GLUE benchmark compared to LoRA+, P-LoRA, and Galore.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer's effort and constructive comments on this work. We fixed the typo on \"algin\" and address your concerns as below.\n\n>**Q1** Simplistic model\n\n**A1** Our formulation **can be not regarded as the matrix factorization (MF) problem**, which is strictly defined as $\\min_{A, B} ||AB-\\Delta||_F^2$. \n\nFor linear models, our problem reduces to MF only when the downstream data matrix $\\tilde{X}$ is an identity matrix (i.e., square and symmetric). However, our setting is more general: we assume $\\tilde{X}$ to be random sub-Gaussian (e.g., bounded entries) and allow it to have arbitrary dimensions, including rectangular and asymmetric shapes. This makes our framework more flexible than standard MF.\n\nOur setting for nonlinear models is structurally different to MF due to the nonlinearity of ReLU.\n\nThere are some prior works that use MF to study LoRA but the practical validity of their theory should be concerned. In a recent paper [1], authors analyze the dynamics of MF using Nystrom initialization and apply it to LoRA. However, such initialization in MF needs to know the complete ground truth $\\Delta$ as the prior knowledge, which is unknown in real-world fine-tuning. So there is a large gap between their MF theory and practice. The MF is also considered in [2] and applied to LoRA, however the convergence and generalization guarantees are missing.\n\nOur work theoretically studies LoRA under realistic assumptions, demonstrating the subspace alignment between $(A_t, B_t)$ and one-step full gradient, and then develops a theory-grounded algorithm to achieve promising performance in practice.\n    \n---\n\n>**Q2** LoRA and FFT optimizes differently (e.g., paper). Existence of \\Delta.\n\n**A2** We thank for the reviewer to point out this paper and the empirical findings in this paper are very interesting and inspiring. In our updated version, we will cite the paper above and add a discussion.\n\nOur theory follows the classical data generation process in machine learning for generalization guarantees, i.e., the label of (downstream) data is generated by a linear/nonlinear target function, corrupted by some noise. Under this setting, $\\Delta$ represents a part of the target function and is therefore unique. Then different fine-tuning strategies (e.g., full FT, LoRA) can still lead to different solutions that achieve similar performance in practice, as suggested by the aforementioned arXiv paper. This phenomenon is common in deep learning.\n    \nNote that, if we focus solely on optimization guarantees, our theory remains valid regardless of whether $\\Delta$ is unique or not.\n\n---\n\n>**Q3** Skeptical about introducing full-gradient info into PEFT\n\n**A3** We understand the reviewer's concern and would like to provide more details to make it clear. \n\nAll experiments are conducted on a single A100 40GB GPU. Due to memory limitations, full fine-tuning is not feasible. However, our implementation employs a memory-efficient approach [3,4], allowing the computation of the first full gradient. Notably, this approach cannot extend to full fine-tuning.\n\nIt's true that using one-step full gradient information requires additional (and marginal) time cost but significantly improves the accuracy. Here we provide a table (displayed in the anonymous link: https://imgur.com/a/qiB59c2) for comparison between vanilla LoRA and LoRA-One on LLaMA 2 7B. It takes additional 10 mins but the accuracy is improved with more than 5%.\n\nIn fact, using additional gradient information from full fine-tuning is a trend and empirically considered in [3,5] as well.\n\n---\n\n>**Q4** LoRA-one leads to better alignment on NLP tasks?\n\n**A4** Based on Algo. 1, LoRA-One achieves perfect alignment with first-step gradient at initialization but LoRA needs to take long time (e.g. one epoch) to achieve a weak alignment, e.g., fine-tune on MRPC via LoRA with r=8 for one epoch, using principal angle defined in Thm 3.1 & 3.2, we have\n\n| | Avg. | Min | Max |\n| - | - | - | - |\n| A | 0.399 | 0.199 | 0.854 |\n| B | 0.267 | 0.156 | 0.393 |\n\nThanks to alignment, LoRA-One surpasses the first-step GD accuracy at the start and continues growing better (displayed in anonymous link https://imgur.com/a/BG5DZca). But LoRA needs a long time to achieve the first-step GD accuracy (Table 2). Also, LoRA-GA mismatches singular subspace with first-step gradient which is the potential reason that we consistently outperform theirs on all datasets even without precondition (Fig 3, Table 10&11).\n\n---\n\nWe expect that our clarification would help you have a better understanding on this work.\n\n---\n\nReference:\n\n[1] On the crucial role of initialization for matrix factorization. ICLR25.\n\n[2] Compressible dynamics in deep overparameterized low-rank learning & adaptation. ICML24.\n\n[3] Lora-GA: Low-rank adaptation with gradient approximation. NeurIPS24.\n\n[4] Full parameter fine-tuning for large language models with limited resources. ACL24.\n\n[5] LoRA-Pro: Are Low-Rank Adapters Properly Optimized? ICLR25.", "claims_and_evidence": "The paper presents both theoretical analysis and empirical validation, including:\nAlignment of LoRA updates with the first-step gradient (Fig. 2), GD trajectories of LoRA-init vs. spectral-init (Fig. 4), and practical performance improvements on GLUE benchmarks.", "methods_and_evaluation_criteria": "Characterizing LoRA as a single-layer matrix factorization problem seems somewhat unrealistic. While LoRA is often viewed as a low-rank approximation of the full fine-tuning update, its optimization behaves differently in practice (e.g., https://arxiv.org/abs/2410.21228 shows that LoRA updates are nearly orthogonal to full fine-tuning updates). This raises questions about whether a unique $\\Delta$ exists in such cases.\n\nMoreover, I am skeptical about introducing full-gradient information into parameter-efficient fine-tuning (PEFT). If computing the full gradient is feasible, then full fine-tuning—given proper hyperparameter tuning and regularizations—usually serves as a strong benchmark. From my perspective, the most effective approaches in practice still tend to be either full fine-tuning with careful optimization or vanilla LoRA.", "theoretical_claims": "The theoretical analysis is conducted under a single-layer matrix factorization setting, focusing on its optimization and generalization. The theorems are mostly correct and intuitive.", "experimental_designs_or_analyses": "The authors validate their findings through numerical experiments on matrix factorization and further demonstrate the effectiveness of their method on modern NLP tasks, as outlined in the **Summary**.", "relation_to_broader_scientific_literature": "This paper may be in the interest of the general area of improving the efficiency of large models.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "See **Summary** and **Methods And Evaluation Criteria**.", "other_comments_or_suggestions": "A minor typo: Line 16, \"algin\" → \"align\".", "questions_for_authors": "I would like to see that if LoRA-one actually leads to a better alignment to the first step gradients on the practical NLP tasks?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rzVDnNCrcB", "reviewer": "Reviewer_cf1y", "summary": "The paper provides a theoretical analysis of LoRA fine-tuning by showing that a single full gradient step naturally aligns the LoRA updates with the top singular subspace of the full gradient, and by introducing a spectral initialization strategy, it can effectively recover the downstream low-rank target before iterative training even begins; moreover, by incorporating preconditioned gradient descent, the method eliminates dependence on the condition number of the feature shift, thereby accelerating convergence, and these insights culminate in the development of the LoRA-One algorithm, which achieve empirical improvements over traditional LoRA methods on various benchmarks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer’s efforts and the positive support.\n\n---\n\n>**Q1** *The theory is detailed and rigorous, although it is unclear how generalizable these insights are to the practical setup of LoRA fine-tuning deep transformers.*\n\n**A1** We are greatly thankful for the reviewer to point out the potential extension to transformer-based models. The attention module and depth will be very challenging. Currently, we believe our techniques can be extended to single linear attention module which considered in a variety in-context learning theory papers [1], i.e.\n\n$$\n\\hat{y} = \\left\\langle \\frac{\\mathbf{W}\\mathbf{X}^\\top\\mathbf{y}}{N}, \\mathbf{x}_{\\tt query}\\right\\rangle,\n$$\n\nwhere $\\mathbf{W}$ is the attention matrix and $(\\mathbf{X},\\mathbf{y},\\mathbf{x}_{\\tt query})$ are data. By a reparametrization, we can admit the following matrix inner product form\n\n$$\n\\hat{y} = \\left\\langle \\frac{\\mathbf{X}^\\top\\mathbf{y}\\mathbf{x}_{\\tt query}^\\top}{N}, \\mathbf{W}\\right\\rangle.\n$$\n\nTreating $\\frac{\\mathbf{X}^\\top\\mathbf{y}\\mathbf{x}_{\\tt query}^\\top}{N}$ as a random measurement matrix, we can characterize the above model as a special variant of matrix sensing model [2] and LoRA will be a factorization approach. With this characterization. we believe our analysis can also apply. In the updated version, we will add a discussion on the possible extension to linear attention module.\n\nThe original attention module in transformer are more sophisticated due to the existence of softmax activations and multiple tunable weight matrices. First, handling the nonlinearity brought by softmax needs to incorporate more theoretical tools, such as [3,4]. Second, the simultaneous training of all layers (Q,K,V) in attention is quite challenging, we might seek  tools from dynamical systems to deal with the coupling between parameters.\n\nWe believe the theory can be extended to deep transformer if the single attention module can be understood thoroughly.\n\nFrom an empirical perspective, in our experimental part, we choose LLaMA 2 7B and T5 base model which consist of multiple attention layers and we achieve promising results (Table 2 & 8) on various NLP tasks using our theory-grounded method, which can empirically justify our insights from theory.\n\n---\n\n>**Q2** *I'm not very convinced of the theoretical merits of this work: the theorem statements are not the cleanest one can hope for*\n\n**A2** Our theory demonstrates the subspace alignment between $(A_t, B_t)$ and one-step full gradient, see Section 3.1, and motivates us to design one certain initialization to achieve this subspace alignment, which is for convergence acceleration. According to the reviewer's feedback, we will use the simplified version for better illustration. For example, Theorem 3.2 (alignment between $A_t$ and $G^{\\natural}$) can be simplified as\n\nTheorem 3.2 [Simplified]. Under standard random initialization for LoRA, after training for $t^*=\\mathcal{O}(\\ln d)$ steps, then we have the subspace alignment between $G^{\\natural}$ and $A_t$\n\n$$\n|| U^\\top_{r^*,\\perp}( G^{\\natural}) U_{r^*}\\left( A_{t^*}\\right)||_{op} \\mbox{~is small}, \\quad w.h.p.\n$$\n    \n---\n\nReference:\n\n[1] How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression? ICLR24.\n\n[2] Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing. COLT23.\n\n[3] Training dynamics of multi-head softmax attention for in-context learning: Emergence, convergence, and optimality. COLT24.\n\n[4] On the convergence of encoder-only shallow transformers. NeurIPS23.", "claims_and_evidence": "This work makes two claims, one theoretical and one empirical.\n\nTheoretically, this work analyzes LoRA fine-tuning applied to a 1-layer neural network without and with a non-linearity. The theory is detailed and rigorous, although it is unclear how generalizable these insights are to the practical setup of LoRA fine-tuning deep transformers.\n\nEmpirically, this work proposes the LoRA-One method, which is inspired by the 1-layer analysis, and apply it to various fine-tuning tasks (on deep transformers). The results outperform LoRA-type methods in terms of generalization performance on several benchmarks.", "methods_and_evaluation_criteria": "The evaluation criteria seems to be sound.", "theoretical_claims": "The proofs justifying the theoretical claims seem to be sound.", "experimental_designs_or_analyses": "The experimental design seems to be sound. The experiments are done thoroughly, and the gains over other competing methods are consistent.", "relation_to_broader_scientific_literature": "The prior work seem to be appropriately referenced.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "I'm not very convinced of the theoretical merits of this work: the theorem statements are not the cleanest one can hope for and it is unclear if their insights generalize to deeper architectures.\n\nHowever, the experimental demonstrations are thorough and convincing. In the end, the idea of initializing the LoRA modules according to the subspaces defined by the full gradient seems like a very nice insight.", "other_comments_or_suggestions": ".", "questions_for_authors": ".", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mNNP4TtmOx", "reviewer": "Reviewer_4x8e", "summary": "This paper investigates methods to enhance the performance of Low-Rank Adaptation (LoRA). The authors make two key discoveries: (i) LoRA tends to align with a specific singular subspace in a single step, and (ii) the use of preconditioners significantly improves convergence in high-rank scenarios. Building on these insights, the authors provide rigorous theoretical guarantees for the convergence of the proposed preconditioned gradient descent algorithm. These contributions advance the understanding of LoRA and offer practical improvements for its application in high-rank settings. \n\nThis paper is theoretically rigorous and presents highly interesting insights that are valuable to the community. The analysis is well-structured, and the findings contribute meaningfully to the understanding of low-rank fine-tuning and optimization dynamics. While the theoretical focus is on gradient descent and preconditioned gradient descent, the work lays a strong foundation for further exploration. Overall, it is a compelling contribution that will likely inspire follow-up research and discussions in the field.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer's efforts and constructive comments.\n\n>**Q1** *Appropriateness of title*\n\n**A1** The title is derived from our theory: under proper initialization from one-step full gradient, we can recover $\\Delta$ to large extent **at initialization**, see Prop. 3.3 (linear) and Lem. C.5 (nonlinear). This claim is also supported by our toy experiments (Fig. 4) and small-scale datasets (CoLA & MRPC): It achieves similar performance with LoRA but fewer time cost. We provide a table (displayed in anonymous link: https://imgur.com/a/i42zqCm) for comparison.\n\nWe are aware of the fact that one-step full gradient is not sufficient for large-scale datasets on LLMs. Hence we continue to run LoRA-One for one epoch. We provide a table (in anonymous link: https://imgur.com/a/qiB59c2) for comparison.\n\nWe will update the title according to the reviewer's feedback.\n\n---\n\n>**Q2** *Simplistic model*\n\n**A2** Extension to complex architecture, e.g., transformer, can be found in A1 of our responses to Reviewer cf1y.\n\nWe also remark that, prior works on the analysis of LoRA are generally limited to single-vector linear regression [1,2], matrix factorization [3,4], or strict assumptions [5] (rank-one shift, rank-one LoRA, frozen $B$). Unlike them, our model is more advanced, and settings (e.t.c. algorithms, rank) are standard in practice.\n\n---\n\n>**Q3** *Data assumption*\n\n**A3** The sub-Gaussian data assumption (e.g., bounded data) is commonly used in theoretical literatures [5,6]. Even for the linear model, our setting is different from low-rank factorization problem because we allow $\\tilde{X}$ to be random and can be any matrix shape (including rectangular and asymmetric matrices), which is more flexible.\n\nOur analysis can be extended to more complex data, such as structured data. For example, the linear model under subgaussian data with covariance $\\Sigma$, we can track the summary statistic $||\\Sigma AB - \\Delta||_F$ instead of $||AB - \\Delta||_F$ by reparametrization. Also, the nonlinear model can be extended to gaussian mixture data via incorporating Stein's lemma into Hermite analysis. We leave this extension as a future work.\n\nAccording to the reviewer's feedback, we will add a detailed discussion on the validity of our theory to practice and clearly clarify its scope and limitations in the updated version.\n\n---\n\n>**Q4** *GD vs Adam*\n\n**A4** We understand reviewer's concern about the gap between theoretical (GD) and practical (Adam) optimizers. Before studying Adam, understanding GD is the first step, commonly used for theoretical analysis [1-4,6] including LoRA and use Adam in practice. Our theory can serve as a conceptional motivation to design the practical algorithm and we achieve significant empirical improvements (see Table 2&8 in paper and Table in A1). Extending our theory to adaptive methods will be an interesting topic.\n\n---\n\n>**Q5** *Distinction with prior methods*\n\n**A5** In our submission, we have compared with gradient alignment based algorithms in Appendix E. According to the reviewers' suggestion, we add a detailed comparison with Zhang & Pilanci 2024 [2] & LoRA pro here and will include in the updated version.\n\nComparing to [2]:\n\n- We identify ill-condition as a convergence bottleneck in LoRA (Thm. 3.5) and demonstrate that pre-condition resolves it (supported by Thm. 3.6&4.3, Fig. 3, Tables 10–11). [2] employ pre-condition (originated from matrix sensing) for stability but overlook ill-condition.\n\n- Our method significantly outperforms theirs (Table 2), highlighting the importance of spectral initialization.\n\nComparing to LoRA pro:\n\n- We only need the **exact first full batch** gradient for initialization, but LoRA pro needs to **approximate** the **stochastic batch** gradient from full fine-tuning in **every training step** using more matrix operations. \n\n- LoRA pro adds $8dr^2+4kr^2+24.5 r^3$ **more** FLOPs for each $A\\in R^{d\\times r},B\\in R^{r\\times k}$.\n\n- We have **much more efficient** memory usage, e.t.c. for GSM8K, we provide a table (in anonymous link: https://imgur.com/a/yjZYUzC)  for comparison.\n\nThere is a series of empirical works e.t.c. LoRA Pro and LoRA-GA introducing gradient information, we are first to provide a theoretical foundation for these heuristic methods and wish we can inspire more theoretical works.\n\n---\n\nReference:\n[1] Lora+. ICML24.\n\n[2] Riemannian preconditioned lora. ICML24.\n\n[3] On the crucial role of initialization for matrix factorization. ICLR25.\n\n[4] Compressible dynamics in deep overparameterized low-rank learning & adaptation. ICML24.\n\n[5] Gradient dynamics for low-rank fine-tuning beyond kernels.\n\n[6] Implicit balancing and regularization: Generalization and convergence guarantees for overparameterized asymmetric matrix sensing. COLT23.", "claims_and_evidence": "1. The title of this paper, \"One-Step Full Gradient Suffices for Low-Rank Fine-Tuning,\" suggests that a single step of full gradient computation is sufficient for effective low-rank fine-tuning. However, the experimental results indicate that the performance of one-step fine-tuning is not only suboptimal but also significantly inferior to that of LoRA. This raises questions about the appropriateness of the title, as it does not accurately reflect the empirical findings. The authors should consider revising the title to better align with the actual results.\n\n2. The model presented in Eq. (2) appears overly simplistic and may lack practical applicability, particularly in the context of modern deep learning architectures. While the theoretical results derived for this model are insightful, it remains unclear whether they extend to more complex and widely used architectures, such as transformers.\n\n3. The authors assume that the input $X$ follows an isotropic centered sub-Gaussian distribution, which ensures that $\\tilde{X}$ in Eq. (2) is approximately orthogonal. Under this assumption, the linear case in Eq. (2) becomes nearly equivalent to minimizing $||W-W^b||$, thereby reducing LoRA to a standard low-rank matrix factorization problem of the form $||AB-\\Delta||$. In this simplified regime, one-step full gradient descent is indeed sufficient for convergence. However, this assumption on $\\tilde{X}$ is critical to the theoretical framework and results presented in the paper. It would be important for the authors to discuss the validity and practicality of this assumption in real-world scenarios, particularly when dealing with more complex data distributions or architectures, such as transformers, where such conditions may not hold. This would help clarify the scope and limitations of their theoretical findings.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "The theoretical results are well-established and sufficient for the simplified model presented in Eq. (2).", "experimental_designs_or_analyses": "The experiments presented in the paper are sufficient to support the theoretical claims within the scope of the simplified model and assumptions considered.", "relation_to_broader_scientific_literature": "Closely related.", "essential_references_not_discussed": "Not available", "other_strengths_and_weaknesses": "1. This paper primarily focuses on analyzing the properties and convergence of gradient descent (GD) and preconditioned gradient descent for low-rank fine-tuning. However, in practice, optimizers like Adam, which incorporate adaptive learning rates and momentum, are widely used for training parameters. This creates a notable mismatch between the theoretical analysis and practical implementation. Specifically, the first and second moment estimates in Adam significantly alter the learning dynamics of the parameters, leading to behavior that may differ substantially from the theoretical results presented in this paper. To bridge this gap, the authors should consider extending their analysis to include adaptive optimization methods like Adam or provide empirical evidence demonstrating how their theoretical insights translate to such practical settings.\n\n2. The proposed preconditioned gradient descent method appears to be identical to the algorithm presented in Zhang & Pilanci, 2024, and bears strong similarities to the LoRA_pro method introduced by Wang et al., 2024. The primary distinction lies in the spectral initialization proposed in this work. Given this overlap, the authors should place greater emphasis on discussing the novelty and significance of their initialization scheme within this learning regime. Specifically, they should clarify how the spectral initialization contributes to improved convergence, stability, or performance compared to existing methods, and provide empirical or theoretical evidence to support its importance. This would help better highlight the unique contribution of their work and its potential impact on the field.", "other_comments_or_suggestions": "Please refer to the previous section.", "questions_for_authors": "Please refer to the previous section.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eah0hlLutp", "reviewer": "Reviewer_TZFT", "summary": "This paper presents a theoretical analysis of Low-Rank Adaptation (LoRA) for efficient fine-tuning of large language models. The main contributions are:\n- Theoretical analysis showing that LoRA's gradient updates align with the singular subspace of the full fine-tuning gradient.\n- Introduction of a spectral initialization strategy and preconditioned gradient descent to improve convergence.\n- Proof of linear convergence rates for both linear and nonlinear models under certain conditions.\n- Empirical validation showing improved performance over vanilla LoRA and its variants on NLP benchmarks.\n\n## update after rebuttal\n\nThe authors provide a strong theoretical foundation with comprehensive proofs, which is crucial for the community. Therefore, I decide to raise my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate reviewer's efforts and constructive feedback.\n\n---\n\n>**Q1** More diverse experimental tasks\n\nWe extend our method to fine-tune T5 model on a subset of SuperGLUE [1] datasets, which is more challenging than GLUE and widely used in fine-tuning papers [2-5]. We use full fine-tuning, LoRA and LoRA-One with rank 8 for comparison. For a fair comparison, we search the optimal stepsize for each method via a large grid ranging from 1e-2 to 1e-5. The other settings are same as Appendix F.2. except the epochs of CB is set to 4 since it is a very tiny set. The final results are provided in the table below.\n\n| Data | BoolQ | CB |   COPA  |  RTE   |  WIC   | Avg. |\n| -------- | -------- | -------- | --- | --- | --- | --- |\n|     Full FT     |     $70.89_{\\pm 0.02}$     |    $89.29_{\\pm 0.00}$      |   $63.67_{\\pm 0.94}$  |   $75.33_{\\pm 0.34}$  |  $66.35_{\\pm 0.32}$  | $73.11$ |\n|     LoRA     |     $70.01_{\\pm 0.03}$     |     $85.12_{\\pm 0.84}$     |  $61.67_{\\pm 0.47}$   |   $70.88_{\\pm 0.17}$  |  $65.78_{\\pm 0.37}$  | $70.69$ |\n| Ours     |   $70.21_{\\pm 0.09}$   | $88.10_{\\pm 0.84}$  |  $65.33_{\\pm 1.24}$   |   $74.61_{\\pm 0.61}$  |  $68.29_{\\pm 0.49}$  | $73.31$ |\n\nWe also extend our method to fine-tune image classification task on Vision Transformer (ViT [6]). We fine-tune ViT using CIFAR10 and CIFAR100 [7], search optimal stepsize for each method to ensure a fair comparison. Since the convergence of LoRA is slow on CIFAR100, we also run two epochs for comparison. The results are provided in the table below.\n\n| Data (#epoch) | CIFAR10 (1) | CIFAR100 (1) | CIFAR100 (2) |\n|---|---|---|---|\n|Full FT| $98.48_{\\pm 0.03}$ | $89.31_{\\pm 0.18}$ | $91.73_{\\pm 0.08}$ |\n|LoRA| $97.91_{\\pm 0.08}$ | $76.46_{\\pm 0.22}$ | $80.23_{\\pm 0.12}$ |\n|Ours | $98.50_{\\pm 0.04}$ | $86.68_{\\pm 0.44}$ | $88.83_{\\pm 0.11}$ |\n\nIn both reasoning and vision classification tasks, we can observe that LoRA-One consistently outperforms LoRA and achieves comparable performance (even better) with full fine-tuning.\n\n---\n\nReference:\n\n[1] Wang, A., Pruksachatkun, Y., Nangia, N., Singh, A., Michael, J., Hill, F., Levy, O. and Bowman, S., 2019. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances in neural information processing systems, 32.\n\n[2] Zhang, Q., Chen, M., Bukharin, A., Karampatziakis, N., He, P., Cheng, Y., Chen, W. and Zhao, T., 2023. AdaLoRA: Adaptive budget allocation for parameter-efficient fine-tuning. The Eleventh International Conference on Learning Representations.\n\n[3] Meng, F., Wang, Z. and Zhang, M., 2024. PiSSA: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37, pp.121038-121072.\n\n[4] Kopiczko, D.J., Blankevoort, T. and Asano, Y.M., 2024. VeRA: Vector-based Random Matrix Adaptation. The Twelfth International Conference on Learning Representations.\n\n[5] Zhao, Z., Shen, T., Zhu, D., Li, Z., Su, J., Wang, X., Kuang, K. and Wu, F., 2025. Merging LoRAs like playing LEGO: Pushing the modularity of lora to extremes through rank-wise clustering. The Thirteenth International Conference on Learning Representations.\n\n[6] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S. and Uszkoreit, J., 2020. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.\n\n[7] Krizhevsky, A. and Hinton, G., 2009. Learning multiple layers of features from tiny images.", "claims_and_evidence": "The claims made in the paper are generally supported by clear and convincing evidence:\n- The theoretical alignment between LoRA updates and the singular subspace of the full gradient is demonstrated through mathematical proofs.\n- The effectiveness of the proposed spectral initialization and preconditioning methods is supported by both theoretical analysis and empirical results.\n- The linear convergence rates are proven under specific assumptions for both linear and nonlinear models.\n- Experimental results on NLP benchmarks show consistent improvements over baseline methods.", "methods_and_evaluation_criteria": "The proposed methods, including spectral initialization and preconditioned gradient descent, make sense for improving parameter-efficient fine-tuning of large models. The evaluation criteria, using standard NLP benchmarks like GLUE tasks, are appropriate for assessing the effectiveness of fine-tuning methods.", "theoretical_claims": "I checked the proofs in the main paper.\n- The alignment analysis in Section 3.1 shows that LoRA updates align with the singular subspace of the full gradient, which appears correct.\n- The convergence proofs for both linear (Theorem 3.6) and nonlinear (Theorem 4.3) models follow standard optimization proof techniques and seem valid.\n- The analysis of spectral initialization and preconditioning methods is mathematically rigorous.", "experimental_designs_or_analyses": "The experimental designs are sound:\n- The comparison against vanilla LoRA and other variants is comprehensive.\n- The ablation studies help understand the contribution of different components.\n- The results on multiple NLP tasks demonstrate the general effectiveness of the proposed method.", "relation_to_broader_scientific_literature": "This work relates to several key areas in machine learning:\n- Parameter-efficient fine-tuning methods like LoRA, Adapter, and Prefix-Tuning\n- Matrix factorization and low-rank approximation theory\n- Optimization for deep neural networks\n- Theoretical understanding of fine-tuning large language models", "essential_references_not_discussed": "The paper adequately covers relevant literature.", "other_strengths_and_weaknesses": "Strengths:\n- Strong theoretical foundation with comprehensive proofs\n- Practical algorithm with empirical validation\n- Clear improvement over existing LoRA variants\n\nWeakness:\n- The experimental validation could be expanded to more diverse tasks.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a theoretical and empirical study of Low-Rank Adaptation (LoRA), focusing on its optimization dynamics and proposing a new method, LoRA-One, which combines spectral initialization with preconditioned gradient descent. The theoretical contribution is that LoRA updates, under gradient descent, align with top singular directions of the full gradient. LoRA-One builds upon this insight to offer accelerated convergence and improved generalization. Empirically, LoRA-One consistently outperforms LoRA and its variants (LoRA+, LoRA-GA, LoRA-Pro) across NLP tasks (GLUE, SuperGLUE) and vision benchmarks.

There are some concerns about the scope of the claims and its overlap with existing work. The title and the framing could overstate the contribution; "one-step full gradient suffices" is only true under specific assumptions and in simplified settings. The authors acknowledged this and have agreed to revise the title. Also, the proposed LoRA-One is, in the end, very similar to a prior work of Zhang & Pilanci 2024, although the present paper provides a more thorough analysis, with a focus on the ill-conditioning aspect of the problem.

In the end, I found both the theoretical and empirical aspect of this paper to be compelling. The paper builds a deeper understanding of LoRA and uses it to inform the design of a practically competitive algorithm. This is a very worthy paper for inclusion in the ICML program.
Decision: Accept (oral)