LoRA Training Provably Converges to a Low-Rank Global Minimum Or It Fails Loudly (But it Probably Won't Fail)
Authors: Junsu Kim
Jaeyeon Kim
Ernest K. Ryu
Venue: ICML 2025 oral (2025)
Keywords: Low-rank adaptation
LoRA
deep learning theory
non-convex optimization
large language models
fine-tuning
post training
Primary area: theory->deep_learning
TL;DR: LoRA training works because there is a global minimizer near initialization and spurious local minima are far away.

Abstract: Low-rank adaptation (LoRA) has become a standard approach for fine-tuning large foundation models. However, our theoretical understanding of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on linearization arguments or consider highly simplified setups. In this work, we analyze the LoRA loss landscape without such restrictive assumptions. We define two regimes: a "special regime", which includes idealized setups where linearization arguments hold, and a "generic regime" representing more realistic setups where linearization arguments do not hold. In the generic regime, we show that LoRA training converges to a global minimizer with low rank and small magnitude, or a qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero-initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small-magnitude region of the parameter space—where global minima lie—thus shedding light on why LoRA training usually succeeds in finding global minima.

LoRA Training Provably Converges to a Low-Rank Global Minimum or It Fails Loudly (But it Probably Won't Fail)
Junsu Kim
Jaeyeon Kim
Ernest K. Ryu
Abstract
X spuoous
rank
T. < rank <
Low-rank adaptation (LoRA) has become stan- dard approach for fine-tuning large foundation models. However, our theoretical understand- ing of LoRA remains limited as prior analyses of LoRA's training dynamics either rely on lin- earization arguments or consider highly simpli- fied setups. In this work, we analyze the LoRA loss landscape without such restrictive assump- tions. We define two regimes: "special regime", which includes idealized setups where lineariza- tion arguments hold, and a "generic regime" rep- resenting more realistic setups where linearization arguments do not hold. the generic regime, we global
Xapurious
rank S r.
X
X Xqurrios
Figure 1: In LoRA fine-tuning, under the assumption that the global minimum X, has low rank and small magnitude, show that spurious local minima xpurious may exist, but they have high rank and large magnitude.
show that LoRA training converges minimizer with low rank and small magnitude, qualitatively distinct solution with high rank and large magnitude. Finally, we argue that the zero- initialization and weight decay in LoRA training induce an implicit bias toward the low-rank, small- magnitude region the parameter space-where global minima liu-thus shedding light on why LoRA training usually succeeds in finding global minima.
where & min(m, n2), A T and B Raxr, The low-rank factor matrices A and B are respectively initialized as a random Gaussian matrix and zero matrix, leading to X = 0 initialization. By training fewer parameters, LoRA fine-tuning significantly reduces memory usage, making fine-tuning feasible on GPUs with limited GPU memory.
The broad of LoRA has spurred theoretical works aimed at understanding its efffcctiveness. One line of work fo- cuses on analyzing LoRA's training dynamics, exploring why optimizers like SGD or Adam successfully find effec- tive low-rank updates despite the significant non-convexity introduced by the factorization X ABT, as well as the in- herent non-convexity of neural networks, by utilizing some degree of linearization. Specifically, Malladi et al. (2023) studies LoRA under complete linearization, effectively holding A fixed during fine-tuning and viewing the train- ing as a convex optimization problem. A subsequent work (Jang et al., 2024) presents a more refined analysis lineariz- ing with respect to the product = ABT, retaining the non-convexity arising from the interaction between A and B. Beyond linearization, Dayi & Chen (2024) analyzes two- layer ceacher-studet setup for rank-1 LoRA. In this work, we carry out theoretical analysis without any linearizations and any restriction on layers or LoRA rank.
1. Introduction
With the recent explosive trend of scale, fine-tuning a pre- trained foundational model to target downstream tasks has become dominant approach deep learning. Low-rank adaptation (LoRA) (Hu et al., 2022) is parameter-efficient fine-tuning method freezing the pre- trained weight matrix Wo E Rmxn and training a low-rank update X = ABT to it using
W- = Wo +X = Wo + ABT,
'Department of Mathematics, Seoul National Univer- sity DDepartment of Computer Science, Harvard University 'Department of Mathematics, University California, Los Ange- les. Correspondence to: Ernest K. Ryu <eryu @ math.ucla.cc edl..
Proceedings of the nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Contribution. We analyze the loss landscape of LoRA fine-tuning and show that in the "generic regime", more practical setup where linearization arguments do not hold, local minimizer is either (i) global minimizer with small rank and small magnitude or (ii) a spurious local minimizer with high rank and large magnitude. We further argue that the zero- pero-initialization and weight decay in LoRA training in- duce an implicit bias toward the low-rank, small- -aggiitue region of the parameter space, where global minima lie. Altogether, we shed light on why practical LoRA training effectively converges to global minima.
min XERmx Sr
s(X)
has been extensively studied in the optimization literature, including matrix sensing (Recht et al.. 2010) and matrix completion (Candès Recht, 2012). Rather than directly optimizing over the space of low-rank matrices, it is often preferred to employ the Burer-Monteiro factorization (Bu- rer & Monteiro, 2003), which formulates the problem by parameterizing X as X = UVT, U E IRmxr, V E R
As the Murer-Monteiro factorization introduces nonconvex- ity, a large body of work has identified conditions under which this approach avoids spurious local minima (Bhojana- palli et al., 2016; Ge et al., 2017; Park al., 2017: Zhang, 2021). Further studies extend these results general set- tings (Ha et al., 2020; Zhang, 2024). In our work, we utilize the framework established in these studies with novel tech- niques to extend its boundary to optimization guarantees in LoRA training.
Our key assumptions, formally defined and justified in Sec- tion 2, are the existence of a low-rank global minimizer for full fine-tuning, restricted strong convexity, and restricted smoothness. Notably, our analysis does not rely on any linearization arguments, making it more applicable to practical fine-tuning setups compared to prior work.
1.1. Prior works
PEFT methods and LoRA Parameter-Efficient Fine- tuning (PEFT) methods have emerged as effective ap- proaches for fine-tuning large language models on down- stream tasks while reducing computational and storage re- quirements. Among numerous proposed methods (Ben Za- ken et al., 2022; Li & Liang, 2021; Lester et al., 2021), Low- Rank Adaptation (LoRA) (Hu al., 2022) has become a predominant approach by decomposing weight updates into low-rank matrices. Several variants such as LoRA+ (Hayou et al., 2024), rsLoRA (Kalajdzievski, 2023), PiSSA (Meng et al., 2024), and MiLoRA (Wang et al., 2025) have been built upon the LoRA framework, addressing the discrepancy with full fine-tuning in optimization and performance.
1.2. Notation and preliminaries
Matrix notation. For X E Rmxn denote its singular values as 01 (X) 2 (x 2 ... 2 o, (X) 2 0. For matri- ces A and B, let ||A||, = (A) denote the spectral norm, || AlI. = ( the nuclear norm, || = VE (A)² the Frobenius norm, and (A, B) = tr(AAB B) the matrix in- ner product. For a tuple of matrices = (A()).... A())), denote ||||| = Σ [| for any matrix norm and rank A = maxigisz ran)(
Neural network. Let f(::-): P X' K be a neural network where P is the parameter space, X' is the data space, and RRKK is the output space. Assume the model is pre-trained to O, E i,i.e the pre-trained model is f()
Theoretical foundation of LoRA. Existing theoretical works on LoRA focus on the expressive power and the training dynamics of LoRA. Zeng& & Lee (2024) demon- strates that a certain LoRA rank suffices to express given fine-tuning function. Jang et al. (2024) proves that under the NTK regime, LoRA with rank 22( ((vN) can express the global minimizer of the original model. Malladi et al. (2023) argues that the LoRA fine-tuning dynamics are nearly equiv- alent to the kernel regression. Under this framework, Jang et al. (2024) proves LoRA fine tuning loss has no spurious local minima when the rank is O(VN). Beyond the kemel regime, Dayi & Chen (2024) analyzes two-layer teacher- student setup for LoRA and explains why SGD leads to convergence to a global minimum in this context. Zhang et al. (2025) also identifies the training dynamics in a 2-layer setup, proving LoRA will align to a singular subspace of one-step gradient of full fine-tuning.
Fine-tuning loss. Let Wo -w:(3) W(O) C O be the pre-trained value of the weights W that we choose to fine-tune. We wish to fine-tune the pre-trained model f(Oa: on a downstream task with data distribution v) ~ D. With slight abuse of notation, write f(W ;.) to denote f(O :)), where all parameters of θ excluding W are fixed to their corresponding values in Og. Let
X = (XX(1. x(O)
be the change of W during the (full) fine-tuning. The true objective one hopes to minimize
C Full ( = E ( [e(f(Wo +.:Wo+X;z)v/))) X)z)
with some loss function f(..)). We assume f(r, y) is non- negative and wwice-iifferentabbl with respect to x for any y. In practice, we have access to finite dataset ((,, 1
Low-rank optimization.
The low-rank optimization prob-
lem
2

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
so we minimize the empirical risk
Theorem (Informal, Theorem 1 of Ge al. (2015)). Stochastic gradient descent with noise differentiable strict saddle functions (i.e., every stationary point is either a local minimum or a strict saddle) does not converge to strict saddle points with high probability. Lee. if stochastic gradient descent with noise converges, converges to an SOSP with high probability.
twice-
N
Sulll (X)
eU(Wo + x;))),))
N
LoRA. Low-rank adaptation (LoRA) uses rank-r param-
eterization for each update matrix
In the context of our work, the implication is that LoRA training converges to SOSPs. The question we address is whether such SOSPs are global minima or whether it is possible converge to a bad local minimum.
X (BU)T E TM
with A E R and B() E R for l I.....L L. Denote
1.3. Weight decay and nuclear norm regularization
A = (1
()) B = (B() ...... B(()
Let À 2 0 and
and
Dora A, (A, & LIo + 2 (1A + [BI))).
AB = (B()T A (B(O)T))
Practical LoRA training typically employs weight decay (Hu et al., 2022: Dettmers et al., 2023) and applying SGD with weight decay on Loora equivalent to minimizing Clora without weight decay. other words, the effect of weight decay is equivalent adding f2-regaularization Let
Under this parametrization, we define the empirical LoRA risk as
foo (A (A B) A é (A"ll(ABB)
We adopt the standard initialization (Hu et al., 2022), re- spectively initializing each A and B as random gaussian and zero, leading ABT = 0 initialization.
D (X) A Iull (X) X|XII
From the prior literature on low-rank matrix sensing and Burer-Monterio factorizations (Recht et al., 2010, Lemma 5.1). known that minimizing this Ez-regularized problem in A and B is mathematically equivalent min- imizing the nuclear-norm regularized loss in the product = AB subject to rank constraint. In other words
Second-order stationary points. Let L: R" +
twice-continuously differentiable. We say R" is a (first-order) stationary point if
VL(X) =0
minimize Ex subject to rank(X) S
minimize A.B
(A,B)
We say X E R" is second-order stationary point (SOSP)
if
VL(X) 0, V²L(X)[U, UJ] 20 0,
When using LoRA, we hope to match the performance of full fine-tuning. We expect this to be feasible if the full fine- tuning problem (with nuclear norm regularization) admits global minimizer whose rank is at most r, since the LoRA update ABT cannot represent updates of rank larger than Therefore, as we discuss further in Section 2. we conduct our analysis under the assumption that Stull has low-rank global minimizer.
for any U E RR. Lastly, we say X E R" a local minimum there exists an open ball B that contains X and
L(X) < L(X')
for any X' E B. It follows that a local minimum is an SOSP. If a local minimum is not global minimum, we say it is spurious local minimum.
Nuclear norm regularization. Nuclear norm regulariza- tion a popular technique that promotes low-rank solutions in matrix optimization. As the convex envelope of the rank function the unit ball (Fazel et al., 2001), the nuclear norm penalty provides tractable alternative to directly min- imizing rank. Its effectiveness in yielding low-rank solutions has been demonstrated both theoretically and empirically across various fields, including matrix sensing (Recht et al., 2010), computer vision (Cabral et al., 2013), nonconvex
Prior works have established that stochastic gradient de- scent applied to twice-continouully differentiable functions (regardless of convexity) roughly converges to SOSPs.
Theorem (Theorem 4.1 of Lee et al. (2016)). Gradient descent on wwice-differeniabbe functions with random ini- tialization, almost surely, does not converge to strict saddle points. Le.if gradient descent converges, it converges to an SOSP, almost surely.
3

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Table 1: Global minimizer rank Fank(X,) as a function of weight decay value X.
for any X E R such that ||X - X.I D and rank(X) S r. We denote the largest a such that f is (a, T, D)-restricted strongly convex about X, as the
Max Rank
749
107
5
3
1
SST2
(r, D-RRC constant of f about X..
0.0
0.001
0.005
0.01
0.1
Max Rank
752
23
12
4
1
We say a wwice-differentiable function f: R=xx + RRi (8, r, -restriied smooth about X, if
CIFAR100
A
0.0
0.0005
0.001
0.003
0.005
V(X(XXXX + XV,UX + XV S B1UX + XVI
optimization (Hu et al., 2021), deep learning (Kobayashi et al., 2024), and LoRA (Jang et al., 2024). Collectively, these prior results make the assumption that Z admits a low-rank minimizer more natural.
for any [X E RR such that ||x X+Ilr S D and rank(X) S rl. IU E Rm such that |1 = || 1] and rank = 1J, and (V E B such that || and rank(U) = rV = 1]. We denote the small- est 3 such that f is (B,r, D)-restricted smooth about X, (or 3 oo if there is no such finite value) as the (r, D)-RSM constant of f about X,.
2. Main assumptions
In this section, we define and quickly justify the main as- sumptions used in our analyses of Section 3.
In this work, we consider the case where a > 0 and B < co. Although deep learning objectives are typically neither strongly convex nor have small smoothness con- stants, the restricted notions of strong convexity and smooth- ness are valid in many practical fine-tuning scenarios as we empirically demonstrate in Section 4. Finally, this current definition treats f as function of single ma- trix X. In Section 3.2, we generalize the definitions to X = (x("), x.").... (14)) with multiple matrices.
2.1. Existence of low-rank minimizer
Throughout our analysis, we assume that there exists rank r. global minimizer of full fine-tuning loss Chull and that our LoRA module uses rank 2 r..
We argue that there is sufficient conceptual and experimen- tal justification supporting the assumption. Initially, LoRA (Hu et al., 2022) was proposed based on the insight that learned over- ree-parrameteiieed models lie in low intrinsic dimension (Li et al., 2018; Aghajanyan et al., 2021), mak- ing them amenable to low-rank updates during fine-tuning, Moreover, as discussed in Section 1.3, training LoRA with weight decay is equivalent to nuclear norm regularization in full fine-tuning, thereby strongly biasing the solution toward low rank. As shown in Table and further discussed in Sec- tion 4, we experimentally verify the low-rank assumption in few setups. Finally, the extensive empirical literature demonstrating the success of LoRA with small rank r fur- ther justifies this assumption.
3. Spurious Local minima of LoRA
In this section, we analyze the loss landscape of LoRA fine- tuning and show that in the "generic regime". second-order stationary point (SOSP) is either (i) a global minimizer with small rank and small magnitude or (ii) spurious solution with high rank and large magnitude.
Section 3.1 starts by presenting the result in the simpler setup of fine-tuning single matrix when a low-rank global minimizer exists. Section 3.2 extends the result to the setup of fine-tuning multiple matrices. Section 3.3 extends the theory to work when an approximately low-rank global min- imizer exists. The extensions of Sections 3.2 and 3.3 slightly complicate the notation, but the qualitative conclusion is maintained. In Section 3.4, we discuss why first-order op- timizers with zero-initialization and weight decay, are un- likely to converge to the spurious local minimizers.
Nevertheless, it may sometimes be more realistic to assume that the global minimizer of full fine-tuning is only approxi- mately low rank. We address this issue in Section 3.3, where we generalize the analysis to the case where the global min- imizer of full fine-tuning is not exactly low rank.
2.2. Restricted strong convexity and smoothness
Our analyses also rely on the assumptions of restricted smoothness and restricted strong convexity, which are weaker assumptions compared to the smoothness and strong convexity assumptions commonly used in optimization.
3.1. LoRA converges to a global minimizer or fails loudly
We now state our main result.
Theorem 1. Let X 2 O. Assume the full fine-tuning loss Shill has a rank-r, global minimizer X.. Respectively denote the (r, D)-CC and (r, D)-RMM constants of Efull about X, as a and B. Assume a > 0 and B < co. Assume we use a LoRA module with rank r 2 T.. Then, every SOSP (A, B) of D) with X = ABT and ||xa X.IIr S D
We say a diteeedifferreniable function f: R (a, r, restrritt strongly convex about X, if
R + IR is
(V)(()- V)())) X X.) 2 2a| - X2
4

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
satisfies the following.
noticeable, and, as the experiments in Section show, gener- alization will be poor. In this sense, we describe this mode of failure to be "failing loudly."
1. 2f > 3 (special regime), Xn is a global minimum.
Relation to prior work. Interestingly, Theorem I com- pletely includes the prior loss landscape analysis of (Jang et al., 2024), which considers a linearized loss in the NTK regime with an e-perturbation. This perturbation ensures 2a 22 > 3 e. placing the loss objective in the special regime. Then, with Theorem we conclude that any SOSP is global minimum.
2. 2f S B (generic regime), one of the following holds. (i) X- is global minimum.
(ū) X= is not a global minimum, rank( = r with
of (Xn) 2 20
3 or. (Xr), and
Hr (1
1X0 12
2000.
3or
3. PROOF OUTLINE OF THEOREM 1
where Trankrr, (Xo) is the projection of Xo onto the set of matrices of rank r. or less.
Our proof technique takes inspiration from the low-rank optimization literature. In fact, the analysis in the special regime 2a 2 8 naturally extends results from matrix sens- ing (Zhu al., 2018; Ha al., 2020). On the other hand, the analysis on the generic regime 2a < 3 is novel re- sult of ours. In the matrix sensing setting, showing that local minimizers near the solution are global minimizers has limited meaning since there is not a good estimate of the global minimizer, so such results were not pursued. On the other hand, in the LoRA fine-tumi fine-tuning setup, 0, the pre-trained baseline, is good estimate of the global minimizer.
To clarify, when we say X is or is not a global minimum, is with respect to Cilll.
We denote 2a > B as the special regime, as the loss objec- tive should be very well-conditioned to fall in this regime. Most practical setups would fall into the generic regime with 9 2 2a, thereby being the regime of primary interest.
The global minimizer X, of the full fine-tumine tinett loss Zfill is assumed to be low rank, and we intuitively understand that X. should have small magnitude since we are fine-tuning. Theorem I states that in the generic regime, there may be additional spurious local minima, but those will have high rank and will be far away from the global minimizer X..
We defer the full proof of Theorem I to Appendix A. providing brief outline here. For notational simplicity, write f(X) = Efull (X) and g(A, B3 = Clar( (A,B) B), and X = Xn- (So is assumed to be an SOSP.) Denote the compact SVD of X as
RJx and σs, We v, as the i-th (largest) singular value of X and the corresponding singu- lar vectors. From the first and second-order optimality of g(A, B), we acquire the following properties:
The following corollary restates Theorem 1 in alternate form that clarifies its main conclusions.
Corollary 1. Consider the setup in Theorem 1. Further assume the strict inequality Ta. Let (A, B) be a SOSP of Dooss with Xo = ABT and [ X. F S D. Then,
= 0 = V B+
2. 0 = Vng(A, = VS(X)T A+
(i) If (Xa) < 20,, (Xo), then Xm is global mini- mizer.
V V B)(UU, VU, V), = UVVT( +
v'((X)[AVT + UBT, AVT U + +UB + ACUUUT + [V 2 0 for any (U.V).
(ii) If (xo) > 20 or, (Xu), then X is a spurious solu- tion, and further X has large magnitude with
Properties and imply V V((X) can be represented as
Et (X0o ||
VS(X) = - -Xxx R3 + S, L = SRx =0 (1)
| >
200rs
for some matrix S. Furthermore, plugging in (U, = (-u,u,A, D into property and using the B- restricted smoothness of ʃ. where (u,, v.) are the top sin- gular vectors of S, we find
LoRA training converges global minimizer or fails loudly. As discussed in Section 1.2, Lee et al. (2016) and Ge et al. (2015) imply that LoRA fine-tuning with SGD converges to a SOSP. In Section 3.4, we argue why it is likely that the SOSP we converge to is global minimizer.
|| S X + Bo,
However, if LoRA tine-tuning does converge to a spuri- ous solution, its high rank and large magnitude would be
From (1), (2) we can induce there exists a subgradient e (|I)) such that ||g + V)(X)]l 2 & Bar.
5

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Denoting Z = (X( and dre, see || xx2| || < So," or, and therefore the top r. singular vectors of X- R7 coincide with those of X. Thus, by the Eckart-Young- Mirsky Theorem, we have
for any X such that |X - X, lI & D, E R Rm such that rank(U)) and ||0|| | = 1 V E R such that rank(V) = 1 and ||V || 1. We denote the tuple 3 of the largest z such that f is (S,r. r. D) resstriict strongly convex about X, as the (r, D)-SMM constant of ʃ about X.. Here V,, Vi, refers to the gradient and Hessian respect to the /th matrix (
Sr. (X) argmin J| (X kZ)
rrnk
Since rank(X.) ,,
Next, under this extended notion of restricted smoothness and convexity, we present the natural extension of Theo- rem below. The proof follows the same reasoning as in Theorem I and is detailed in Appendix A
||X X 1B S X - X x2ll3
which is again equivalent to
| - 1I < |X - 1 + 2x, - x,
Theorem 2. Let A 2 O. Assume the full fine- tuning loss Ctull has a rank-r, global minimizer X. (x!") .x!")) Respectively denote the (r, D)-SC and (r, D) -MSM constants of Etull about X, as a = Assume a11) (L) > O and g .......(1)) < DO. Assume we use LoRA modules all with rank 2 Tx. Then, every SOSP (A, of Lx with Xn = ABT and ||xn X.Ile S D satisfies the following.
Now a-restricted convexity at X, implies
(X X-X.. (/((xX--X((X)) (X.)) 2 a a X X.
(a) a and B = (3(1)
(B( .....,,(L),,
By the global optimality of X., from Mordukhovich & Shao (1995, Theorem 3.1) we have - fX.) E I(l] and thus the subgradient property implies
(X., 9 - (X))( 20
Summing up the three inequalities, we have
1. 2a 2 300 for a L (special regime), Xu is a global minimum
(2ka x) -X + || X1 < 0 Therefore when 2ka > 1, X. X. and when 2na the LI
2. If (0 < <g for some I = 1.... L. [. (generic regime), one of the following holds.
inequality of the theorem holds.
3.2. Extension to fine-tuning multiple matrices
(i) X is a global minimum.
X (1) is exactly rank
For the sake of notational convenience, Theorem I was stated for the case of fine-tuning single weight matrix. In this section, we generalize the result to the case of fine- tuning multiple matrices.
(ii) X is not global minimum,
r with (x
(x2
2a
B00 Tr (x and
1 11 1lankrr (X 11
| (1)
x!0 x01
First, we extend the definition of restricted smoothness and strong convexity to the multiple matrix case.
gis.
x Rm Xne + R be twice dif- ferentiable. Let X (X(1), x(2) X(L)), a (a (2)), and B (8(1),... B(4)).
Let f: R xn1
for some 1= 1,... L. where Hranker (x is the projection of X (1) onto the set of matrices of rank r. or less.
We say is (a, D) restricctd strongly convex about X. if for each 1 2 L,
To clarify, when we say X is or is not a global minimum, it is with respect to Chull.
(V.((X.) V (//(X))VV.ill
x0= xx! 2 x0I 1...
3.3. Extension to approximately low-rank minimizers
for any X such that |X- X+IIF < D. We denote the tuple of the largest (0)s such that is (a, resttrictte strongly convex about X, as the (r, RSCC constant of f about X..
In Sections 3.1 and 3.2, we assumed the nuclear-norm regu- larized full fine-tuning loss 2 has a low-rank minimizer, but this assumption may be unrealistic especially when the weight-decay parameter À is too small. In this section, we relax this assumption and consider the case where D'ull has an approximately low-rank minimizer. As in Section 3.1, we present here the result for the single matrix case. In Appendix A, we provide a 'Mateer Theorem that combines the generalizations of Theorems 2 and 3.
We say a twice-differentiable function f: mm + RR is (8, r, De-rstrricted smooth about X, for each 1 < < L,
(U)(()) + XVV, Ux()) + +
5,800 10 x VI
6

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
We say ) is a 8-global minimizer of full fine-tuning if ||x'
When weight decay is used, we can make this argument further quantitative. The global minimizer X, satisfies
- -X.IIF 5
[(X+)+ +XX.11 S E(0) + A|[0[|+:
for some X, that exactly minimizes [
thus ||xX < Z00) Here, Σ(0) the loss corresponding to
Theorem 3. Let E 0 and À 2 Assume the full fine- tuning loss Cx has a rank-r, 6-global minimizer x(") with 5 ==of )( Respectively denote the (r, D)-RCC and (r, D)- RSM constants of Gtull about Xx as o and? B. Assume 0 < a and? B < c0. Assume we use a LoRA module with rank r 2 r+- Then, every SOSP (A, B) of Ciara with X = ABT and ||x X.II D satisfies the following.
directly applying the pre-trained model to the fine-tuning task, so (() should not be inordinately large when the fine-tuning task is not too different from tasks seen during pre-training.
On the other hand, spurious local minima exist only outside neighborhood of zero, as argued in Corollary 1. Because the SGD or Adam optimizers used for LoRA training are initialized at 0, the optimization is biased towards smaller- magnitude solutions near the starting point, which are the global minima. In Section 4, we experimentally test this theory by fine-tuning LoRA with non-zero initialization; indeed, we find there is an instance in this scenario, where the fine-tuning gets trapped in spurious local minima.
1. 2f 2 B(1 + e) (special regime), Xa is an e-global minimizer.
2. 2f < () (generic regime), one of the following holds.
(i) X- is an g-global minimizer.
(ii) Xn is not an E-global minimizer, is exactly rank r with (Xn)
Xn
Weight decay implicitly biases the optimization towards low-rank matrices. Practical LoRA training typically em- plays weight decay (Hu et al., 2022; Dettmers et al., 2023), and it is shown in prior theoretical work that weight decay induces an implicit bias toward low-rank matrices. This makes it more likely for the LoRA training to converge to the low-rank global minimizer, rather than to spurious local minima with high rank being o, (X) 2g o, (X).
2a max ((1+e) Fr. (Xo); 2RVF . -e,, and either 20
a
or (Xn) s
B or. (Xn)
or
|0o Brankrr (Xo) 1| ca
| - X,lla
200r.
For deep linear networks, this implicit bias is characterized somewhat precisely.
Bar
where Hranksr, (Xo) is the projection of X onto the set of matrices of rank v. or less.
Theorem (Informal, Theorem 3.2 of Wang & Jacot (2024)). When training a deep linear network with positive weight decay, sufficiently small learning rate, and ground-trutth teacher model with low effective rank, there positive probability of jumping from high-rank critical point to lower-rank one, but the probability of jumping back is zero.
To clarify, when say Xn is or is not an e-global minimizer, it is with respect to Cfulll.
3.4. LoRA training probably won't fail; it probably won't converge to spurious local minima
While the theory of Wang & Jacot (2024) does not imme- diately apply to general deep (non-linear) neural networks, it does provide meaningful insight into the implicit bias towards low rank. In the more general setup, Galanti et al. (2024) argues for similar implicit bias. Adapting their ar- guments to LoRA training, we get the following statement.
In the analysis of Section 3.1 and its subsequent generaliza- tions, we showed that in the generic regime, spurious local minima may exist, but if the training converges to them, this will be very noticeable (failing loudly), as the spurious solutions have high rank and large magnitude. In this sec- tion, we argue that the standard LoRA fine-tuning procedure induces implicit biases that make it unlikely for the LoRA training to converge to these spurious local minima.
Lemma 1. Consider LoRA training with SGD with batch size b, learning rate μ. and weight decay X >> 0. For any low- rank update X- = ABTT of a weight matrix in the network, if the sequence of X X-values throughout training converges to matrix X. then X is approximately low rank in the sense that for any 0, there exists some W with
Zero-initialization biases the optimization towards min- ima with smaller magnitude. LoRA fine-tuning is initial- ized with B= = O, leading to X = ABT = 0 at initialization. This choice comes from the intuition that fine- fine-tuning should not change the model too much i.e., that X, should be small, so the initialization should be at 0.
x
b blog(s/4)
w/
<E,
rank(W) <
|
log(1 - pA)
We provide the proof of Lemma in Appendix A.4.
7

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
1e-1
5.0:
Zero Initialization Random Non-Zero Initialization
G
Training Loss
Test Accuracy 6
1.0
Zero Initialization
Random Non-Zero Initialization
5
1e3
8-
Zero Initialization
Random Non-Zero Initialization
7.
Rank
Norm
2
2-
Zero Initialization
Random Non-Zero Initialization
0
3
2
5
le5
1e5
Figure 2: LoRA training converging to global minima with zero-initialiization vs. spurious local minima with random
non-zero initialization. (y-axes) Training loss, test accuracy, minimizer rank, and minimizer norm. (x-axes) Training steps.
4. Experiments
Table 2: RSC and RSM values for different ranks.
In this section, we validate our theory through real-world experiments. First, we verify our assumptions outlined in Section 2. Then, we present both the success and failure modes of LoRA fine-tuning, where training either converges to low-rank, small-magnitude global minimizer or stuck on a high-rank, large-magnitude local minimizer.
Rank 8
16
32
64
B/a
8.0249
18.7032
320.82
N/A
a 0.0061
0.0029
0.0002
-0.0445
B 0.0492
0.0539
0.0726
0.3371
and X 0.01. Table 2 presents the a and / values for the largest B/a value across weight matrices. We see as increases, decreases and increases. In fact, when is as large as 64, the requirement a > 0 breaks, and our theory no longer applies. These results demonstrate that our assumption of a > 0 and B < oo is plausible using a low LoRA rank r. This also suggests that reduced memory footprint is not the only benefit of using small r; the a, f-- values that determine the loss landscape also become more favorable with small r.
Experimental setup. We conduct experiments on two tasks in NLP and vision. For the NLP task, fine-tune RoBERTA-base model (Zhuang et al., 2021) on senti- ment analysis task, using the SST-2 dataset (Socher et al., 2013) from the GLUE benchmark (Wang et al., 2018). For the vision task, we fine-tune& a vision transformer (Doso- vitskiy et al., 2021) on the CIFARIO0 100 dataset (Krizhevsky, 2009). Both models have 12 attention layers, and we tune the query and value weights of each layer, following the prescription of Hu et al. (2022). We describe further details in Appendix C.
Results: Validating main theorem. Finally, we verify our main result through an illustrative example for the SST2 task with prove that spurious local minima may not exist, but when they do, they exhibit high rank and large norm, being read- ily distinguishable from the global minimum and thereby avoidable through zero initialization. We present in Figure 2 that such spurious local minimum found by large random initialization indeed fails loudly in the sense that it has high rank. large magnitude, and poor generalization performance. We further demonstrate in Appendix C that spurious local minima isn't found in any smaller initializations.
0.01 and r 8. To clarify, our results
Results: Verifying low-rank global minima exist. First, we validate our assumption of low-rank global minimum. We perform full fine-tuning the nuclear norm regularized loss objective Z5td with varying values of weight decay X. The results of Table exhibit a clear decreasing trend on the rank of the global minimum as a function of A. Notably, when À is set to values at least 0.001, the resulting rank is lower than typical LoRA ranks (4, 8, or 16).
Results: Verifying RSC and RSM. Next, we verify our assumption of restricted strong convexity and smoothness. As it is infeasible to exactly compute a and 33 values, we estimate them by Monte-Cart Carlo sampling with 1000 samples within rank bound r = 8, 16, 32, 64, distance bound = 5,
5. Conclusion
In this work, we theoretically analyze LoRA fine-tuning and obtain a new type of result: that second-order station-
8

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
ary point is either a global minimizer with low rank and small magnitude or is spurious solution with high rank and large magnitude. Unlike previous analyses based linearization, our approach relies on general condition of restricted strong convexity and smoothness, which are con- ditions the experiments of Section confirm to be practical. We further argue that zero-initialization and weight decay in LoRA training induce an implicit bias toward this low-rank small-magnitude region, explaining why LoRA typically converges to global minima in practice.
References
Aghajanyan, A., Gupta, S., and Zettlemoyer, L. Intrin- sic dimensionality explains the effectiveness of language model fine-tuning, Association for Computational Lin- guisties, 2021.
Ben Zaken, E., Goldberg, Y., and Ravfogel, S. BitFit: Sim- ple parameter-efficien fine-tuning for transformer-based masked language-modee models. Association for Computational Linguistics, 2022.
While the primary focus of this work is on establishing the theoretical convergence of LoRA, our framework possesses broader practical relevance. The properties of spurious local minima that we characterize may be used to diagnose and monitor the fine-tuning process. Furthermore, as our frame- work relies solely the low-rank decomposition structure of LoRA and few minimal assumptions, our theory applies to many LoRA variants, including LoRA+ (Hayou et al., 2024), rsLoRA (Kalajdzievski, 2023), PiSSA (Meng et al., 2024), and MiLoRA (Wang et al., 2025) as well.
Bhojanapalli, S., Neyshabur, B., and Srebro, N. Global optimality of local search for low rank matrix recovery. Neural Information Processing Systems, 2016.
Burer, S. and Monteiro, R. C. A nonlinear programming algorithm for solving semidefinite programs via low-rank factorization. Mathematical Programming, 95:329-357, 2003.
Cabral, R., De la Torre, F. Costeira, J. P., and Bernardino, A. Unifying nuclear norm and bilinear factorization ap- proaches for low-rank matrix decomposition. Interna- tional Conference Computer Vision, 2013.
Our results open several avenues for future work. One is to perform more rigorous analysis of the implicit bias induced by weight decay and zero initialization. Another intriguing insight is that the restricted strong convexity and smoothness constants a and B improve as the LoRA rank decreases, suggesting that smaller-rank parameterizations enjoy more favorable optimization landscapes. This obser- vation contrasts with the modern wisdom of deep learning theory that overparameterization helps training and aligns with recent results indicating that overparameterizztion can slow down training (Xu & Du, 2023; Xiong et al., 2024). Exploring this phenomenon further is another promising direction.
Candès, E. and Recht, B. Exact matrix completion via con- vex optimization. Foundations of Computational Mathe- matics, 55(6):111-111 2012.
Dayi, A. K. and Chen, S. low-rank fine-tuning beyond kernels. arXiv preprint arXiv:2411.15388 2024.
Gradient dynamics for
Dettmers, T., Pagnoni, A., Holtzman, A., and Zettlemoyer, QLoRA: Efficient finetuning of quantized LLMs. Neural Information Processing Systems, 2023.
Acknowledgments
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenbom, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations, 2021.
EKR was supported by the Samsung Science and Technol- ogy Foundation (Project Number -SST-BB2101--- and the National Research Foundation of Korea (NRF) grant funded by the Korean government R00-S-2002-0042
JK acknowledges the support from the Ilju foundation schol- arship. We also thank Jaesung Park, Uijeong Jang, and Sunwoo Kim for providing valuable feedback.
Fazel, M., Hindi, H., and Boyd, S. P. A rank minimiza- tion heuristic with application minimum order system approximation. American Control Conference, 2001.
Impact statement
Galanti, T., Siegel, Z. S., Gupte, A., and Poggio, T. A. SGD and weight decay secretly minimize the rank of your neural network. NeurlPS 2024 Workshop on Mathematiccs of Modern Machine Learning, 2024.
This paper advances the understanding of Low-rank Adapta- tion, contributing to the broader field of Machine Learning. There are many potential socictal consequences of our work, none of which we feel must be specifically highlighted here.
Ge, R., Huang, F., Jin, C., and Yuan, Y. Escaping from saddle points - online stochastic gradient for tensor decomposition. Conference on Learning Theory, 2015.

---

LoRA Training Provably Converges to Low-Rank Global Minimum or Fails Loudly
Ge, R., Jin, C., and Zheng, Y. No spurious local minima in nonconvex low rank problems: A unified geometric analysis. International Conference on Machine Learning, 2017.
Meng, F,, Wang, Z.., and Zhang, M. PISSA: Principal sin- gular values and singular vectors adaptation of large lan- guage models. Neural Information Processing Systems, 2024.
Ha, W. Liu, H., and Barber, R. F. An equivalence be- tween critical points for rank constraints versus low-rank factorizations. SIAM Journal on Optimization, 30(4): 2927-2955, 2020.
Mordukhovich, B. S. and Shao, Y. On nonconvex subdif- ferential calculus in banach spaces. Journal of Convex Analysis, 2(1-2):211 1-227, 1995.
Parikh, N. and Boyd, S. Proximal algorithms. Foundations and Trends in Optimization, 1(3):127-239, 2014.
Hayou, S., Ghosh, N., and Yu. B. LoRA+: efficient low rank adaptation of large models. In International Conference Machine Learning, 2024.
Park, D., Kyrillidis, A., Carmanis, C., and Sanghavi, S. Non- square matrix sensing without spurious local minima via the Burer-Montein approach. International Conference on Artificial Intelligence and Statistics, 2017.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adaptation of large language models. International Con- ference on Learning Representations, 2022.
Recht, B., Fazel, M., and Parrilo, P. A. Guaranteed minimum-rant solutions of linear matrix equations via nuclear norm minimization. SIAM Review, 52(3):471- 501, 2010.
Hu, Z., Nie, F., Wang, R., and Li, X. Low Rank Regular- ization: review, Neural Networks, 136:218-232, 2021. ISSN 0893-6080.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng. A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. Em- pirical Methods Natural Language Processing, 2013.
Jang, U., Lee, D., and Ryu, E. K. LoRA training in the NTK regime has no spurious local minima. International Conference on Machine Learning, 2024.
Tomihari, A. and Sato, I. Understanding linear probing then fine-tuning language models from NTK perspective. Neural Information Processing Systems, 2024.
Kalajdzievski, D. A rank stabilization scaling factor for fine- tuning with LoRA. arXiv preprint arXiv:2312.03732, 2023.
Wang. A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, GLUE: multi-task benchmark and analy- sis platform for natural language understanding. EMNLP Workshop BlackboxNI.P. Analyzing and Interpreting Neu- ral Networks for NLP, 2018.
Kobayashi. S., Akram, Y., and Oswald, J. V. Weight decay induces low-rank attention layers. Neural Information Processing Systems, 2024.
Krizhevsky, A. Learning multiple layers of features from tiny images. Master's thesis, University of Toronto, 2009.
Wang, H., Li, Y., Wang. S., Chen, G., and Chen, Y. MiLoRA: Harnessigg minor singular components for parameter- efficient LLM finetuning. Association for Computational Linguistics, 2025.
Lee, J. D., Simchowitz, M., Jordan, M. I., and Recht, B. Gra- dient descent only converges to minimizers. Conference on Learning Theory, 2016.
Wang. Z. and Jacot, A. Implicit bias of SGD in L2- regularized linear DNNs: One-way jumps from high to low rank. International Conference on Learning Repre- sentations, 2024.
Lester, B., AI-Rfou, R., and Constant, N. The power of scale for parameter-efficient prompt tuning. Empirical Methods in Natural Language Processing, 2021.
Xiong, N.,
Ding, L., and Du, S. How peeeraaramttiiiotioon slows down gradient descent in matrix sensing: The curses of symmetry and initialization. International Con- ference on Learning Representations, 2024.
Li C., Farkhoor, H., Liu, R., and Yosinski, J. Measuring the intrinsic dimension of objective landscapes. International Conference Learning Representations, 2018.
L. and Liang. P. Prefix-tuning: Optimizing continuous prompts for generation. Association for Computational Linguistics, 2021.
Xu, W. and Du, S. Overr-aaraeeterization exponentially slows down gradient descent for learning a single neuron. Conference on Learning Theory, 2023.
Malladi, S., Wettig, A., Yu, D., Chen, D., and Arora, S. A kernel-based view of language model fine-tuning. Inter- national Conference on Machine Learning, 2023.
Zeng, Y. and Lee, K. The expressive power of low-rank adaptation. International Conference on Learning Repre- sentations, 2024.
10

---

LoRA Training Provably Converges to Low-Rank Global Minimum or Fails Loudly
Zhang, R. Y. Sharp global guarantees for nonconvex low- rank matrix recovery in the overparameterized regime. arXiv preprint arXiv: 2104. 2021.
Zhang. Y. Improved global guarantees for the nonconvex Burer-Monteiro factorization via rank overparameteriza- tion. Mathematical Programming, pp. 1-30, 2024.
Zhang. Y., Liu, F., and Chen, Y. LoRA-One: One-step full gradient suffices for low-rank fine-tuning, provably and efficiently, arXiv preprint arXiv: 2502.01235, 2025.
Zhu, Z., Li, Q. Tang, G., and Wakin, M. B. Global optimal- ity in low-rank matrix optimization. IEEE Transactions on Signal Processing, 66( (3):13133433668 2018.
Zhuang, L., Wayne, L., Ya, S., and Jun, Z. A robustly optimized BERT pre-training approach with post-training. pp. 1218-1227, 2021.

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
A. Omitted theorems and proofs
A.1. Full proof for Theorem
Here we present the complete proof of Theorem For simplicity, we write [(X) as f(X), Clora (A, B) as g(A
Set (A, B) E R x R to be SOSP of fx, ABT as X, the compact SVD of X as Lx Ex RJ, and o; as the i-th (largest) singular value of X. From the first and second-order optimality of Lx (A, B), acquire the following properties:
1. 0 Vag B) (A(BTT) B AA
2. VBg(A, B) V A AB
3. VAg [U,
= - + + 2 0.
Y(U. V) E R x Rexr
By equations I and 2,
- AVVJ(ABT)B- = = ABTB,
so if > 0 then ATA = BTB. Therefore by Lemma I. we can set A = LX B = RxEx2 for some
orthogonal matrix W. Plugging A, B back into properties and gives us
V Rxx -Axx, VS(X)T Lx -Rxx.
If X 0, then apparently Vx(x) Rx = 0 and Vf(X)T Lx = 0, so the equation above holds regardless of X. Now by
Lemma B.2, (X) can be represented as
VS(X) = -Xx RX + S, S = E
for some diagonal matrix Σx and some Lx, Rx that Lx Lx] and Rx Rx] are orthogonal. Now we will show that the spectral norm of S is bounded by X + Bo, We first consider the case where rank(X) = r, or or (X) > Setting u. v, as the top singular vectors of S, and plugging in (U, V) (-u. A, o B) into property 3, we achieve the following inequality.
v (()) -uu, X, Xu.e. ",u + A (101)> 11VIE) 2 2(VS(X), u.u,
2((-AXx + S, u, u.N.l.
-20,(-Alo Rx + S, u11
= =20,(S, un+:)) = 20, ||
Where
| + =|[1 tr(u,uJAAA u.A. + BBT. tr(uuu] Lx Ex I u,,) + tr(eee] Rx u. o, (Ir(u.u)) + br(www))) = 20,
Now the restricted smoothness of f yields the following inequality.
V=f(X)[Xe,e] u,u,X, XU,uI u. u.u,x] S 81| - u, u, u,
2 |1.1l unu? 12. - 230
Combining the two inequalities results in
||||| I 3a, +
Now we see the case where rank(X)= r' < or o, (X) = 0. Since rank(X) <r, A and B are also rank deficient, so we can find a unit vector w that Aw = 0. Now
Aw = > AT Aw
> BT Bw = 0= wB Bw 0 + |Bw]]
12

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
so we have Bw 0 as well. Now for any unit vector IRm, ,",00 E IR", plugging in (U, V) (uwt, uu into property results in
(Vf()), uo S Since this holds for any unit vector u, v, we have |V I X. Since vf(X) = RXx + S. this implies ||5||2 S as well.
Now that know |||| Bo, + X for all cases, we will find what this implies. Consider the subgradient O(1) which we know the explicit form as
{ { GGRRmxn G= Axx + WW, W =0, O, WRx O, 1IW SA
Since S also satisfies L = 0 and SRx = 0, there exists an element g E (AI that llg - Vf(())l2 S 80, Now let Z = (g + VX()). For any positive real number R that &. x 8a, S dr,, the singular vectors of K are orthogonal to those of X and ||x2]|2 I or. so the top r, singular vectors of X xZ coincide with those of X. Therefore, by the Sokart-Younge Mooung-Mirsky theorem
X
argmin - (XX-(xx221 rank(Y))r.
where X a projection of X onto the set of matrices of rank r. or less. Since rank(X..) = r., we can plug in X, into Y here, resulting in
| X + S S| X + xZ
which is again equivalent to
| - X
Z) -
XXx
XI + 2k(X, X,
(3)
Here we actually know that (X** - X, 1) = 0, since the singular vectors of X and Z are orthogonal. Now by restricted convexity at X., have
- XX-X, V)(x) V.(x.)) 2a X
(4)
Since X, is the global minimizer of fx(X) f(X)+A ||X]], and the nuclear norm is lower semi-continuous, from Theorem 3.1 of (Mordukhovich & Shao, 1995) we have -VX(X,) (..l).. We know by definition that E |Ila) so the property of the subgradient implics
--V 20
(5)
Summing up the inequalities (3).(4)(5)) we have
(2ka 1))|X, x1I? + | - -x13 0
Since this inequality holds for any positive real number x that & x Ba, S or,, if a. = O we can select an arbitrarily large
resulting in X. = X. If or f O, we can plug in onz resulting in X = X, if 1 > 2ka and
| - 11 XX - XII
Σ E.sr.+1 +1 o3
1 - 2na
1 - 2ka
otherwise. Therefore our statement is proven.
A.2. Proof of Theorem 3
Here we present the proof of Theorem 3.
Proof. We can proceed with the identical reasoning and notations with the proof of Theorem in A.1, up to defining Z = g + (X) with || |Z||2 S Bo,.. Now denoting the true global minimizer as X., we have a rank r. approximate global minimizer X with |X_ x < 8. We first prove that (X) 2 e a if X is e-spurious local minima, which requires independent reasoning, and then we prove the remaining results analogously to Theorem 1.
13

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Step 1. Fix some constant e >> and define r' min(7 (X) <; < S... Now setting positive real number that fā, < c. By the Eckart- Young-Mirsky theorem we
argmin ||Y (X - = = (X KZ))
Srn
where (X- - =2)"* is the projection of X - kZ onto the set of matrices of rank r, or less. Since ||x2|12 < by definition of r' the 1-th to r-th singular vectors of X n.Z coincide with those of X, while the subsequent singular values are all smaller than c. This implies
X - - -Z) s max -r. 0r
r. <r', we are done, so we can consider only the case where Tx - 2 0. Combining the two relations, we have
|| - - -xx==2]]F= - | - - -k)"* ||5 + ||(X -xZ) - -
-x
- I Scv r+ - + + || - (x--x2 -KZ)llF
and squaring each sides, this expands to
| I S (r. r') 20 - - (x-x2)]l= + |x - X12 2x( -x", %)
Now in the same way with A. due to the restricted convexity and the subgradient property we have
2ak| -x,1x111 - 2 X.,
Adding the two up, we have
(2ax - 1)]l x. 1 11
- X1i Se (r, + r, - 11 - (X-- - -KZ)ILF + -1IXT- - X -||X, - x
+ 2k? X) 21:(X - XX", .Z
which again simplifies to
(2amx 1)11X X.1li- 25 x 33.6 r') + 2000
or equivalently
8
|X - Xx11o S
2ox + 20x )2 + 3r - r²) + 200 -x.-
Therefore, setting c -v=ーp and 8a, (or any large number if o, = 0), if (x) < 3. then 8 = = implies ||X- X.IIF < e, making X an e-global minimizer. Thus if X is a spurious local minima, then o, 2 25
Step 2. Similarly to Theorem 1, define as positive real number that k :ar < or,, implying ||xZ|| S or. Then
X E argmin |Y (X k2)
(Y)r
and rank(x = r,, so
| X + xZ P
(x(10) - X + xZ12
which is equivalent to Now the relations (4).(5) from the proof of Theorem still apply here so adding these we have
| X 2 E - X + 2k -X,)
- 11 S x(X Z)
so adding the two inequalities, we obtain
2xaIXX X 113
|
X:
+|xT.
|
x12 S 2k(X7- Z).
14

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Here we can upper bound (XT X,, 2) by 20,, ||x X+Il. by the duality of the nuclear norm and spectral norm
and |x2212 I Tr.. Dr, is again bounded by D' since we are looking at ||X X. S ||X F + ||x, S 2D, and
yr.
[xx -. I vr Tx Ix x S 6 x vr-r., r. by the Cauchy-Schwarcs inequality. Therefore we have -r-ra
- x 11 - X x ||3 + || - x1e 1 2dD
T+
We can expand this inequality, using X x (X X.) (X, (X.-X,- xx*)
I 2 - X,, X, - + 5D -- 26 X,11r + +22D (2na - 1)|| x1 + - x1 -
r Tx
rx
Solving the quadratic inequality about || X F|, 2ka - ->> E the discriminant is
8 + 4(2a 1) 28D V
--
1 - x1|?
r+
Now know that if X is an e-spurious local minima then ||X since 8 = o(s²) and therefore there would be no spurious local minima. 2fa 1 < O, we would see
X||F F 2 0 2 20 E, so the discriminant is negative
|x - XI - (26D
r-T,
82 1-2ka
V
|X,, X S
-2xcx
1 2kc
resulting in
8 X12 e3
|x.- - xIIE S
E²
2ka
given 8 = o(e²)
A.3. Theorems for the multiple matrix case
In Section 3, we presented most theorems as a result for tuning a single matrix for clarity. Here we explicitly portray the natural extension the multi matrix case for the theorems and method of proof. First, we provide the proof of Theorem 2, the multi matrix case without error.
Proof. As in the proofs above, we write ((X) as f(X), LAT B) as g(A, B) for simplicity. Set (A,B) to be a SOSP of fx. and ABT as X (x ,..., .(.)). From the first and second-order optimality of (A B), we acquire the following properties:
1. VA) = V/(ABT) B AA
2. 0- Vng(A, B) V/(ABT) A
3. V2 (U, (U, V)) UP/K AVT UBT] + + 2 0
Recall that for tuple of matrices
C C= (=C..,...,, ,DD))...,( D(O)
we write the product of the tuples as
CD ("(0)))))..,,,()((()))))0))
the scalar product as
kCC = (kO(1), (icot,),..,,KCC (C=((CC))...,KCCL))
15

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
and the Frobernius norm
||
|C
We interpret the gradient V (ABT) as tuple with the same shape with ABT, and the Hessian v²f(X) as (m171 + + mI)L) x (m1n1 +... + mtnr) block matrix consisted of m,n, x myny blocks (X)
By looking at only the lth matrices from the equations 1,2, we have
1. 0 V Aang(A)) B0) Vtf(ABT) B + XA
2. 0 V (og((")) B = V.J(ABB)T A + AB
Furthermore, inputting U, V with
U U UU......,,....
- = (o....,,,....
v vw,
for some U00), V()) we see
3. 2 /TTp://www000/////////////000000
Now we can proceed similarly with the single matrix case. By the new versions of equations 1 and 2,
-(AD))//ABBBB = X( A = A(B())BB")
so if X > 0 then (A4))T A (B())B(O) Now denote the SVD of ( as 1 (RQ)) and o." as the i-th (largest)
singular value of x(),, By Lemma 1. we can set A - (2:²)
matrix W. Plugging A(), BB() back into properties 1 and 2 gives ) 1/2,, Bl0) = =R (2 w for some orthogonal
Vif(X)-
R - Vf(X). L -
If X 0, then apparently v.)(X) L 0 and VIJ(X)T 1 0, the equation above holds regardless of X. Now by Lemma B.2, (X) can be represented as
ViS(X) - (1) Rel +S S L xxx (O)
A
for some diagonal matrix Σx and some 1. 2 that [100 (1)
D and R R are orthogonal. Now we will show that
the spectral norm of S is bounded by X + ++8(()((1)
We first consider the case where rank rank(X(() (xx()) = r, or or (x()) > O. Setting u. v. as the top singular vectors of S, and plugging in (U7), v()) (-u,u) uJ (t) U. u. B()) into property 3, we achieve the following inequality.
(i,f(X)(XX (9 I - u. u.u.x()) X I - u.u. x1 + > (100011 + |100)))) 2 2(Vif(x u ".
= =2(-Ar 1 R + uuuf S.u,, u. (
= (2 R(O + S.u.
=20 (S, v.uI) = 20 ||
Where
(/00) 2 F + +||v00|1
2 F
= Ar(uuj A (1) (A(0) + ir trlo,o (BB() (B())),,uu))
tr(u,, (1) (1) (9)
-X u, + tr(ue (1() R& v,0o)
of!" (tr(u.u!) + tr(eee))) = (0
16

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Now the restricted smoothness of yields the following inequality.
( -u, 1000000000000 1, -.,,x((] s 3W0 | |x
= = o00? |l - u. - - 23
Combining the two inequalities results in
||||| S 80) + X.
Now we see the case where rank((X()) = 7' < r. or o, = 0. Since rank(x(") < r, A and B are also rank
deficient, so we can find a unit vector w that A0 = Now
A の = 0 > (A = 0= = (B B0w = 0 + w/ w(B())BB") w = 0 - 1B0c0 nc =0
so we have B(0 = 0 as well. Now for any unit vector u E Rm u E R", plugging in (U,V) = (uwt, -uw-) into property 3 results in
(Vif(X), u( ك X Since this holds for any unit vector u, v, we have [Vv la S A. Since Vif(X) = 2 TT + S, this implies ||Sl|| S well.
Now that we know ||S]|2 S g00 + A for all cases, we will find what this implies. Consider the subgradient ((1l (1) [-), which we know the explicit form as
G (R wW, (L - O, WRY = O, I 5
0(AII((11--- = GE Rmxn
Since S also satisfies (T = 0 and 2 = O, there exists an element E = (I.) that I| Vi/ S 00 t Now let Z (g- + (x)). For any positive real number orthogonal to those of ( and ||^2]|e S of so the top r, singular vectors of x - KZ coincide with those of X Therefore, by the Eckart-Young- You-n--uungyyyyy theorem we see
that k x 8(0) o I or. the singular vectors of RZ are
(t)
(1)
(x(0)' E argmin ||Y - (Xx0-x)) rank(Y)
where (X is a projection of (1) onto the set of matrices of rank v, or less. Since rank(X!!) S Tx. we can plug in x(()) into Y here, resulting in
1(x x0) xZl1 < ||x x0 +21
which is again equivalent to
(1xx0''- -x0 22x((x000) x ",,)) < x - x 11 2x - xx) ()
(6)
Here we actually know that ((x x(0) = 0, since the singular vectors of (1) and Z are orthogonal. Now by
restricted convexity at x."), we have
(x x!!) VI((X) - v./(x.)) 2 a0 -100 x
Since X, is the global minimizer of fX() = f(X) + ||XX| and the nuclear norm is lower semi-continuous, from
Mordukhovich & Shao (1995, Theorem 3.1) we have (x.( E ((l)x!")1.. We know by definition that g e ((A[[x(()))) [I.), so the property of the subgradient implies
(x - x!! .9 - (-Vis(X.)) 20
(8)
Summing up the inequalities (6).(7)(8)) we have
(2na (0) 1)]|x! x09 - 50
17

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Since this inequality holds for any positive real number a that R x g00 (
S oo,, if o = we can select an arbitrarily
of
_ large K, resulting in x = X If of F O, we can plug in k 20 resulting X = x(! if 2 2na!") and
|| x0 N 1(x X
1 2ka
otherwise. Therefore, if or (x()) X X.. If not, there must exist some satisfying the inequality above. Therefore statement is proved.
2007 -r (X()) for all 1 S S L then x!" x for all 1 S S L,, resulting in
Now present Master Theorem, applicable for the most general case with multiple matrices and an approximately low rank solution. We first analogously define ò-global minimizer for the multiple matrix case:
We say x a 8-global minimizer of full fine-tuning if
[1xx!))(0) x! 15 1 I
where X, is an 'exact' global minimizer of Chull Based on this definition, we present the analogous result. Theorem A.1. (Master Theorem) Let 0 and X 0. Assume the full fine-tuning loss E has exact global minimizer X., and a rank-r. 8-global minimizer x(6) with 5 = o(s²). Respectively denote the (r, D)-RCC and (r, D)-RSM constants of Liull about X. as (a (((),....(.))) and 3 (B(1), ,.... ). Assume (! > 0 and B(I) 00 for each
1 <1 L. Assume we use LoRA module with rank T.- Then, every SOSP () of Dara with X ABT and [X - X+I1r 11 S D satisfies the following.
1. 1/2a(0) 2 (1 + f all I = 1,..., L (special regime), Xn is an e-global minimizer.
2. 1f < (1 + e) for some 1....., I (generic regime), one of the following holds.
(i) X is an s-global minimizer.
20 (ii) X is not s-global minimizer, x is exactly rank with o, ox( max B[i+e] Sr, (X 2B -e), and either
xx(xx() < B
2a or, (x(1
or
Iranksr. (x 2
x.
200r.
Sa,
where Branker. (x() is the projection of X onto the set of matrices of rank r, or less.
To clarify, when we say X is or is not an e-global minimizer, it is with respect to E
Proof. We can proceed identically with the proof of Theorem 2 up to defining Z such that || S ar.
We first prove that (x) > a if ||x(0) - x || 2 E, which requires independent reasoning, and then prove 2B
the remaining results analogously to Theorem
Step 1.
Fix some constant 0, and define r' = min(y (x,() <,, S Sr.. Now setting x as a positive real number that Bo, < c. By the Eckart- Kokart-Young-Mirsky theorem we see
argmin ||Y (x(() kZ) (x(1) - =Z)"
Sr,,
where (X =Z)r. is the projection of ( KZ onto the set of matrices of rank r. less. Since ||*2||2 c, by definition of rf the 1-th to r-th singular vectors of (!) kZ coincide with those of X while the subsequent singular values are all smaller than c. This implies
1(x00'' - (x11) xz s c² maxfr= - -
18

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
If frr. <T', we are done, so can consider only the case where r, rr' 2 0. Combining the two relations, we have
11(x0y- - (x =Z)]lF= = 11( (xx1) +z)) ||5 + ||(X(1 =ZZ) (xx) =z)IlFF
I Sevr.- - + + (((x (x(1 kZ)
and squaring each sides, this expands to
IIX0 - S
F+ (x
22
Now in the same way with 1, due to the restricted convexity and the subgradient property we have
2axl|xx(0) - 12 - 2x(x0) - x! .).
Adding the two up, we have
(2ax - 1))xx x! th 12 + - x3 - S dr - r) + rr - -x(() - -RZ)lle
+ (1((x!))) - x1(xx(/ - x()))
(2( - x!", + 2A(x()) - (x(0) 22)
which again simplifies to
- (2ax - - xx.' - 532 - -1)) + 2r
or equivalently if 2ax - 11>11100 0.
l| |x | 11 S 2ak -
2an - 22ax-11 (r + 2c r.
8
Therefore, setting .Viー and = Do, (or any large number if or = 0), if (x(1) < 3 then d = =(=²) implies
||x(0 xI < e. Thus ||x0 x." || 2 e, then (x))) 2 2B E for some I.
Step 2. Similarly to Theorem 1, define x as positive real number that k. Bo, < <o,,, implying || S or,, Then
(x E argmin ||Y - -xx(0) - -xx0-x2))1 mn)
and (0)) = so
1(X0) x09 xZ1B3 < ((x(5)))) x00 + xZ
which is equivalent to Now the relations (4),() from the proof of Theorem still apply here so adding these we have
-x < - 11 + - -xx Z).
x(),
al|x@ - -xx'))"000 z xxxx - x(.92)
so adding the two inequalities, we obtain
2na (x0 (x(") 1 11x - (x! -x
< 2k((((
_!") Z).
Here we can upper bound ((x! (x)(0)(0) (0)00 x!", Z) by 20,, ||[ -x=0) 1 by the duality of the nuclear norm and spectral norm and || < or.. Tr, is again bounded by vs. since we are looking at || |F < D. and ([(x())) x <
D
Vr-r.- (()) x! < V -r by the Cauchy-Schwaan inequality. Therefore we have
2kal/ x11) - (000 - 1Ix - (1 + 1(x(0)' x 8D /
r.
19

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
We can expand this inequality, using x - (xx2)) (x x!") (x!") - (x) as
(2xa - 1)]1 + 11(x0)----- s 2 x,",,."' - (x!3))0) + 8DD
r- T
s 220|| x!'1| + bDV r T+
Solving the quadratic inequality about ([(x,))))) x if 1 > & the discriminant is
8 + 4(2ka 1) (D " - 1 T+
Now we know that if X is an E-spurious local minima then [(x(0)-- - x 2 or 2 2/ e for some 1, so the
a
discriminant is negative since o(=2) and therefore there would be no spurious local minima. If 2xa
0, we would
|X 11 S 1(x(0) 1 - (dD r-re
8 2xsxx
resulting in
[(x(1) |(x (0) E3
||x -xx0-_X0 -x01 S
1 1 2xtc
given 5 = o
A.4. Proof of Lemma 1
Proof. Denote the low rank matrices at the tth iteration as A, E Rmxr and B, E Rnxr, the weight update X, = A, and the mini-batch used at the ith iteration as S.. With slight abuse of notation, denote the mini-batch loss for batch S, at iteration as
Es (X.) b
Σ
((S(Wo X)
(2.2)ES.
Denote the input vector to the layer E R where ut -Wo + ABT) is inputted to the next layer. Here see that
VALs, (X,)
1
Σ
+ +BT;
b
Now the gradient of the loss expandable by the chain rule as
V Val(f(Wo ABT:2),)) @e(f(Wo + ABT:z)..) de(f.v 0
Of(Wo + AB);z)
al(f,y) 0f
Of JuT A) )
0A
0A
Therefore V F v) is rank-1 1 matrix, as it is product of scalar and two rank-1 matrices. Thus, Es, (X1). as sum of b rank-1 matrices, has rank at most b.
Now we can characterize the SGD process with weight decay as
A = A; A (V - 2ndA.
B = B, BR-WVBBC. (Xi) 24AB.
or equivalently,
Att1 = =(1 (1-) Ar WV (X..
B (1-2u))B, NV DV (X.).
20

---

LoRA Training Provably Converges to a Low-Rank Global Minimum or Fails Loudly
Recursively applying this process times, have
A. = (1 - 222)" A Σ (1 SIsAp-LSss,,C (X(-))
==1
Therefore denoting Ut, == 11 D)=1( 22A2 LEs, (Xe=z) we see A,, Br are approximately close to Ut,s Ven by
| - Uz.1 || - (1 - 11 | - Vi,ll S (1--2u))*|BA,
for analogously defined Vi,. and rank(U,,n), rank(Vi,n)) S nb. Therefore,
X A. (Ut,n + (1 -1-2241 221A)" A A.-n) ) (VT, + (1 2μa)" BT,
=U,,n + (1- -222) B (1 2nA) A. V.nn + (1- 22d))an A
Since (V f. + +(1 - 2 B (1 2 Ac-n Vinn are both matrices of rank at most b, we can define a matrix W with
X₁ |x|||
W (1 2222222
|xll
and rank & 2nd. Since we assumed X₂ converges to X, when is sufficiently large we can assume for any that (1 2n e/2, there exists some matrix W with
X |l |
< 2. Thus
X |x,||
log(e/2)
W <,,
rank(W) S b log(1 (1 2μAA
B. Matrix lemmas Lemma B.I. If matrices A E mRxr, B Ruxr satisfy ATA = BTB,
A= U2W,, B = V
where UEVT is singular value decomposition of ABT, and W is orthonormal matrix.
Proof. Take the compact singular value decompositions of A and B,
A = A=UΛEAV
B UB.
Then we have
VV = -T
Here VA. Vn are both orthogonal matrices and Σ Σ3 both diagonal matrices, by the uniqueness of the singular value decomposition. by sufficient reordering and sign flipping of the singular vectors can have VA VB, ΣA ΣB. Therefore inputting this into ABT = x(),, we have
U EX = UADAUT
which implies again Σイ = 1/2 UA U Up v up to reordering and sign flipping. This proves the given statement.
Lemma B.2. For orthonormal matrices U E xr and V E Rax if E Rmxxn satisfies XV = and XTU V then X can be expressed as
U ŪEFT
where U E (m-) is an orthogonal basis for the orthogonal complement of the column space of V (n-(n-r) an orthogonal basis for the orthogonal complement of the column space of V, and E (n- is a rectangular diagonal matrix.
21

---

LoRA Training Provably Converges to Low-Rank Global Minimum or Fails Loudly
Proof. First fix arbitrary Ú and V, that makes [U U] and X can be expressed as
V
V]
orthogonal. Since orthogonal matrices are invertible,
A B D
V VT
x = U
C
for some A E R B E Rrx(ne)) C' p(m-t)xr D ((m--)x(m-r), Expanding this expression we can write X again as
x UAV + UBVT ÜCCT+
Therefore XV = UU, XTU = V implies UA+ÚC= VA VB U, respectively. Now the orthogonality of U, Ü and V, V implies A = L and B, C' = 0. Finally, let the singular value decomposition of D be D = UDEVT Then
+ (VVVD(
and ÜUD, VVD are still orthonormal matrices in the same space, statement is proved.
C. Experimental Details
C.1. Verifying low-rank global minima exist
Experimental setup We perform the experiments by fine tuning RoBERTa model for the SST-2 dataset, and fine tuning a Vision Transformer model for the CIFAR-I00 dataset. To maintain only low rank updates, follow Malladi et al. (2023). by first training the linear classification head only, and fine tuning on the resulting model. This way of training is in fact quite reasonable, as prior studies report it is beneficial to LP-LoRA (Tomihari & Sato, 2024), i.e. perform linear probing first and then apply low rank adaptation. We tune a total of 24 matrices for each experiment, with the hyperparameters as below.
Table 3: Hyperparameters for verifying low-rank global minima exist
HyperParameter
SST2
CIFAR100
Task
SST2
CIFAR100
0, 0.1, 0.05, 0.01, 0.005, 0.001 0.005
0.01, 0.005, 0.003, 0.001, 0.0005 0.001
Learning Rate
Scheduler
Cosine Annealing Proximal SGD
Cosine Annealing Proximal SGD 128
Optimizer Batch Size
128
Epochs
150
150
Optimizing nuclear norm For verifying the existence of a low rank global minima exists, we optimize over the loss 2) 2full (X) =(x) + X| Here the nuclear norm hon-differentiable, so instead of the standard stochastic gradient descent, we apply proximal gradient method, well known to converge to a global minimum convex objective (Parikh Boyd, 2014):
X1+1 = prox wXIlII (Xt - ,Vī(X,))
where
1 2p
X11 + AIIx)...
prox nA]-II- (X) argmin
X
X
Computing the rank To effectively compute the rank considering numerical issues, we use the torch.linalg.svd function to obtain the singular values of the normalized weight matrix, and truncate values below 10 10-4, As we consider approximately low rank matrices Section 3.3, our theory robustly holds for this truncated approximate rank as well.
22

---

LoRA Training Provably Converges to Low-Rank Global Minimum or Fails Loudly
250
600
200
--000
A- 0.000s
0.005
Rank
400
1.0.0
Rank
150
0.0.003
1.0.5
0.05
100
200
2
6
2
3
4
1e4
1e4
Figure 3: Rank of the first value matrix throughout training. (left) SST2, (right) CIFAR 100
Results The final ranks after training presented well in Table Here present the change of the ranks throughout training. For simplicity, we present the rank of the first value matrix, which had the largest rank among all matrices in general. The results presented in Table are the ranks of the weight matrix with the highest rank for each X.
We also present the test accuracy curves for each task. We see that weight decay have contrastive impacts for sst2 and cifar100, where weight decay doesn't harm the generalization performance as long as it is not too big in SST2, while sigmificantly impacts performance for CIFARI00. Therefore we do use CIFAR100 for the following experiments, as the weight-decay equipped solutions function of X, therefore successfully verifying assumption.
the ones of our interest. Nevertheless, clearly the decreasing trend of rank
0.8600
6600005
0.9
0=00001
0.0.00
Test Accuracy
Test Accuracy
0.8575
A=
0.8
0.005
0.8550
0.7
0.8525
0.5
0.8500
0.6
0.8475
MWV
www.
0.8450
2
8 1e4
1e4
Figure 4: Test accuracy throughout training. (left) SST2, (right) CIFAR100
C.2. Verifying RSC and RSM
Precisely computing the RSC and RSM constants a deep neural network is generally infeasible, and therefore we perform monte-carlo sampling for 1000 samples find an estimate for these values. Specifically, we compute
(V))X,,) (X(,)),, X, x
min I
|1X!0 x x!911
(X X t + + UXX + +
max 800
15t5n
||]|=== Mil===1 rank(U) raak)))
for samples X X with ||X!0 x D. rank(X!")) S for each S < N. Due to the intense computational bottleneck of the Hessian, limit the computation only the last layer of W and W.. following (Jang al., 2024). The a, 3 values we present in Section 4are the values for W. which had larger values than W,, for each case.
23

---

LoRA Training Provably Converges to Low-Rank Global Minimum or Fails Loudly
C.3. Validating main theorem
In Section we present the training results in two cases: zero initialization and Large initialization, which demonstrate global minimizer and spurious local minimizer, respectively. We additionally present here that in smaller initializations, and in other setups, LoRA training always converges to the global minimum, indeed validating argument that spurious local minima must be unfeasible if they exist.
5
Int: 1
Initiaiiation
Int:
Init 3 Hittlliission
Test Accuracy
Train Loss
0.8
0.7
0.6
Zero Intaaizztion Int
Largo Intalizatin
0.25
0.50
0.75
1.00
1.25
1.50
1.75
0.25
0.50 0.75 1.00 1.25 1.50 1.75
2.00
1e5
1e5
Figure 5: LoRA training on SST2 with varying initialization (left) training loss, (right) test accuracy
We present the specific initializations below. Here the initialization for the A matrix in the zero init follows the standard Kaiming uniform initialization, while other initializations are random gaussian initializations.
Matrix
A
Zero Initialization
u( V
15 15 768
O
768
Initialization 1
N (0, ())
N (0, 1)
Initialization 2
N (0, 1
N' (0, +
Initialization 3
A/ (
1 TO 20
N ( 1 T
Large Initialization
N (0. !)
N (0, !)
Table 4: Initialization values for matrices and B
24


Reviews:
[{"review_id": "XhZadiYm2E", "reviewer": "Reviewer_ni32", "summary": "The authors have shown that a low rank and low magnitude initialisation in LoRA models results in convergence towards a global minima. Conversely, larger rank models with larger initialisation variance results in convergence towards spurious local minima with high probability. Whilst this is a result observed in experimentation in previous literature, this is largely a novel result for theoretical confirmation.\n\n# update after rebuttal The rebuttal discussion was informative and helped to understand some aspects of this paper better. I will maintain my original score", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Common Response (Repeated in all responses)**\n\nFirst of all, we thank the reviewers for their positive and constructive feedback. We are excited to see that the reviewers are appreciative of our theoretical contributions. Below, we address each of the reviewers' comments individually.\n\n### **Individual Response**\n\nWe thank the reviewer for their meticulous and constructive comments, and we are pleased to hear that the reviewer recognizes the novelty of our theoretical contributions.\n\n**Discussion with Table 2**\n\nWe appreciate the reviewer’s encouraging remarks on the results of Table 2 and its contrast with over-parameterization. We agree this point is interesting enough to merit further emphasis, so, following the reviewer’s suggestion, we will highlight this discussion more prominently in our revised paper.\n\n**Correction of typos**\n\nWe have corrected all typos pointed out by the reviewer. Once again, we sincerely thank the reviewer for their thorough reading of our manuscript and their detailed feedback.\n\n**Question on alternative reasons for the failure case**\n\nWe thank the reviewer for raising a thoughtful point. Indeed, bad (large) initializations can lead to poor performance due to issues such as activation function saturation (for tanh activation functions) or exploding gradients. In our setup, this phenomenon indeed occurs if the initialization variance is further increased to $\\mathcal{N}(0, \\frac{1}{2])$, as demonstrated in the figure in the link below.\n\nhttps://drive.google.com/file/d/14ui2E-hmg7E7LvaUp2X0hBPx9Zb6AQZp/view?usp=sharing\n\nHowever, it is unlikely that the behavior observed in Figure 2 of our main text is caused by these issues, as the gradient norm remains stable and nonzero throughout training, as demonstrated in the figure in the link below.\n\nhttps://drive.google.com/file/d/1w7k0dGcjFvBebFtw8OCqaKZb3BP-1180/view?usp=sharing", "claims_and_evidence": "As mentioned in the summary, there is a theoretical claim that low rank low magnitude initialisation results in convergence towards global minima. This is supported by a mathematical proof. Some experimental evidence is present. \nThe results in Table 2 maybe even understated – an interesting insight which both highlights and limits the application of this theory to LoRA models (that convexity assumption $\\alpha$>0 does not hold for larger rank). \nContrasting over parameterisation is very interesting, could be worth discussing that more prominently.", "methods_and_evaluation_criteria": "Use of CIFAR 100 is a reasonably complex dataset for showing differences in initialisations during low-rank adapted training. The paper presents results for sufficient number of steps and there is a good use of metrics like rank (which was particularly novel for readers/reviewers) for showing convergence differences.", "theoretical_claims": "I have attempted to review the theoretical correctness of the proofs.  I have validated Proof A.4 and the lemma proofs in appendix B. I was not able to comprehensively verify the results of the main theorem as it is quite dense and complex. \nThe assumptions of restricted convexity and restricted smoothness appear reasonable and are empirically backed. However, they do break down at larger rank in Table 2.", "experimental_designs_or_analyses": "Experimental design does seem to be a bit 'rushed' but does validates and supports mathematical proofs. Table 2 results are of interest, maybe worth discussing more.", "relation_to_broader_scientific_literature": "This work builds on many existing works and confirms existing empirical results in literature as well as 'common knowledge' of LoRa convergence.", "essential_references_not_discussed": "None noted", "other_strengths_and_weaknesses": "Discussed in other sections. None", "other_comments_or_suggestions": "* Page 2, line 76, column 2: “for any any matrix” -- “for any matrix” \n* Page 3, line 152, column 2: “Collectively, these prior results make the assumption that L admits a low-rank minimizer more natural.” -- “more naturally”? \n* Page 6, line 296, column 1: Missing subscript *, currently reading as a dot product where one of the components is 0. \n* Page 11, line 561, Property 3: ∇f(X)UV⊺  -- ∇f(X),UV⊺ {needs comma} \n* Page 11, line 571: “regarless” -- “regardless” \n* Page 12, line 658: “then, which requires independent reasoning, and then ..” {doesn’t need both then’s} \n* Page 17, line 926: Additional epsilon ϵ used in notation that isn’t referenced anywhere else. Currently assuming this is a typo of ε considering the context of the proof.", "questions_for_authors": "With the experimentation, is it not true that the use of large variance in initialisation can cause poor performance for alternative reasons? (i.e., initialising with kaiming gives you better results than having a large initialisation in a simple feedforward neural networks)", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qHMHBCgUB3", "reviewer": "Reviewer_wCMH", "summary": "This paper provides a theoretical understanding of the training dynamics of LoRA (i.e. Low-Rank Adaptation) of transformers. The authors first establish the equivalence between the low-rank form of loss and the rank-constrained optimization problem. Then the authors state their main results that the LoRA results can be split into two regimes, and in the second regime (\"generic regime\") LoRA can lead to \"failed\" results (i.e., does not converge to a global minimum and stuck in a spurious local minimum). The authors further demonstrate that standard LoRA practices, such as zero initialization and weight decay, and extend to multiple matrices fine-tuning.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Common Response (Repeated in all responses)**\n\nFirst of all, we thank the reviewers for their positive and constructive feedback. We are excited to see that the reviewers are appreciative of our theoretical contributions. Below, we address each of the reviewers' comments individually.\n\n### **Individual Response**\nWe are happy to hear that the reviewer found our claims to be well supported by theory and experiments and the contribution distinct from prior works. We agree with the reviewer's view that “this paper could be useful to identify/design better low-rank adaptation algorithms,” and we outline how our theory can be applied to several LoRA variants in our response to Reviewer 9M1X.\n\n**Regarding the question of whether multiple experiments share the same global minimum**\n\nWe thank the reviewer for the insightful observation and suggestion. Yes, our theorem indeed implies that multiple experiments should converge to the same global minimum (the same product $X=AB^\\intercal$, but the low rank factors $A$ and $B$ are determined only up to rotations). To demonstrate this empirically, we extended the experiments originally presented in Figure 5 of the appendix to directly test whether training trajectories with multiple random seeds converge to the same limit. See the figure of the link below:\n\nhttps://drive.google.com/file/d/1vU2oZT2qAHxcY2gnIjA5CIBUwevxNJhs/view?usp=sharing\n\nIn the figure, we plot the ‘total variation' of the training trajectories with multiple random seeds, defined as: $\\sum_{1\\le i<j\\le N}  \\|| \\Theta_i^{(t)} - \\Theta_j^{(t)} ||$ where $\\Theta_i^{(t)}$ is the parameters of the $i$ th model at the end of the $t$th epoch. We see that the total variation converges to 0, indicating that multiple experiments share the same global minimum. (The total variation starts at 0 because the $B$ matrix is always initialized to be 0, so the product $X=AB^\\intercal$ starts out at the same value of 0, even though $A$ is initialized differently.)", "claims_and_evidence": "I think the claims made in the submission are well supported by both theory and experiments.", "methods_and_evaluation_criteria": "The authors consider a few cases (both NLP and CV tasks) and conduct experiments to verify their main theorems. More specifically, the authors use two different initialization methods, one of which lead to global minimum and another converges to spurious local minimum. However, it would be helpful in my opinion to add another experiment to show that the multiple experiments can converge to the same global minimum.", "theoretical_claims": "Although I have not got solid theoretical background, I tried my best to check the theoretical claims. Did not identify issues.", "experimental_designs_or_analyses": "Yes. In my opinion the experiments only account for a very small part of this paper, and are mainly served as a validation of theory results. The authors come up with two settings and demonstrate the results fit the expected outcomes.", "relation_to_broader_scientific_literature": "I am not familiar with relevant literature, but I checked the referenced works and I believe the authors' work has distinct aspects compared to those existing ones.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "This paper could be useful to identify / design better low-rank adaptation algorithms since it provides a principled way to identify if the fine-tuning could work or not. It can also be used as a monitor to tell if the training / finetuning is healthy or not.", "other_comments_or_suggestions": "There are some occasional misuse of \\cite and \\citep. It would be great if the authors can fix them. Example: \"following the prescription of (Hu et al.,2022).\" at line 378.", "questions_for_authors": "- It would be helpful in my opinion to add another experiment to show that the multiple experiments can converge to the same global minimum. Correct me if I am wrong but if the claims hold, should the solutions of low rank matrices be similar?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WrRZz8ZQ59", "reviewer": "Reviewer_9M1X", "summary": "This paper provides a theoretical analysis of Low-Rank Adaptation (LoRA)  loss landscape (near the global or local min). \nThe main contributions are as follows:\n\n(1) The authors identify two regimes—special (well conditioned) and generic (more realistic). In the generic regime, second-order stationary points (SOSPs) are either (a) low-rank, small-magnitude global minima or (b) high-rank, large-magnitude spurious minima. \n\n(2) The authors shows that zero-initialization and weight decay are biased toward low-rank, small-magnitude solutions, which partially explains LoRA's empirical success despite non-convexity.\n\n(3) The paper provides some experimental results confirming the existence of low-rank global minima under nuclear norm regularization and validate restricted strong convexity/smoothness assumptions. They also illustrate that the failure cases (high-rank solution) may results in from large initialization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Common Response (Repeated in all responses)**\n\nFirst of all, we thank the reviewers for their positive and constructive feedback. We are excited to see that the reviewers are appreciative of our theoretical contributions. Below, we address each of the reviewers' comments individually.\n\n### **Individual Response**\nWe are excited to hear that the reviewer found our main theoretical results interesting. We are also grateful for the detailed feedback provided by the reviewer, and we respond to them individually in the following.\n\n**Regarding the relation with other LoRA variants** \n\nYes, as the reviewer points out, our theory is indeed applicable to the more modern enhanced LoRA variants. We discuss a few popular LoRA variants, including the ones pointed out by the reviewer.\n\nLoRA+ (Hayou et.al., 2024) and rsLoRA (Kalajdzievski, 2023) share LoRA's objective and initialization while modifying hyperparameters like the learning rate and scaling factors to accelerate and stabilize training. Therefore, our theory assures these variants can also successfully find the global minimum, hopefully, faster and more stable.\n\nPiSSA (Meng et.al., 2024) and MiLoRA (Wang et.al., 2024) modify the training objective by decomposing the pretrained weight matrix into the sum of a principal ($A_pB_p$) and minor($A_mB_m$) component. PiSSA optimizes the objective $f(A_{lora}B_{lora} +A_mB_m)$, initializing $A_{lora}B_{lora}$ with $A_pB_p$. In contrast, MiLoRA optimizes $f(A_pB_p+A_{lora}B_{lora})$, initializing with $A_mB_m$. While appearing very similar, our theory introduces a contrasting view for the two methods.\n\nPiSSA initializes the fine-tuning weights with $A_pB_p$, which inherently contains 'principal' pretrained model information. In this setting, applying weight decay to $A_p$ and $B_p$ is counterintuitive; rather than regularizing the fine-tuned model to remain close to the pretrained model, weight decay here would bias the model toward losing essential pretrained characteristics. Thus a low-rank solution isn't naturally expected, and the model would be required to be very well-conditioned (i.e., within the special regime) to guarantee convergence to the global minimum under our framework.\n\nConversely, the use of weight decay in MiLoRA is more sensible as $A_mB_m$ inherently represents a minor component. Therefore, we can more naturally apply our results to MiLoRA to guarantee convergence to a global minimum, in contrast with PiSSA. Investigating the practical implications of this theoretical insight is an interesting direction of future work.\n\nWe note that variants like LoRA-GA (Wang et.al., 2024), which significantly alter training dynamics, do fall outside our current scope and offer promising directions for future work.\n\n**Q) Why the restricted smoothness …**\n\nOur definitions of restricted convexity and smoothness are motivated by the proofs requiring only a \"directional form\" of strong convexity and smoothness. In contrast, the conventional definitions require strong convexity and smoothness for all possible directions. While our definitions may feel more complicated, they are weaker and more realistic assumptions compared to standard strong convexity and smoothness. In particular, instead of requiring the standard smoothness $\\nabla^2 f(X) [A, A] \\le \\beta ||A||_F^2 $ for all $A \\in \\mathbb{R}^{m\\times n}$, we only require it for the directions$A$ in the space $\\\\{UX+XV | rank(U)=rank(V)=1 \\\\}$, a much smaller, restricted space.\n\n**Q) Why X_box can be full rank**\n\nWe apologize for this confusion. By “full rank” meant rank $r$, since the low-rank update $X_{box}=AB^T$ can have rank at most $r$. However, we now see that this was not the correct language, so we will update it to say rank $r$ instead of full rank.\n\n**Q) In Proof 1, the authors used kappa …**\n\nAs in line 253, $\\kappa$ is defined as $\\frac{\\sigma_{r*}}{\\beta \\sigma_r}$. Thus, the case $2\\kappa \\alpha >1$ in the last line of the proof corresponds to the special regime and case (i) of the generic regime, while the converse corresponds to the case (ii) of the generic regime. Thank you for pointing this out, we will make this distinction clearer in our revision.\n\n**Q) ||X||_\\star, is this the nuclear norm? but … Q) equivalence between the unconstrained…**\n\nWe apologize for the confusion. The L2 regularization applies to the LoRA fine-tuning objective, while the nuclear norm regularization applies to the rank-constrained full fine-tuning objective. As we note in Section 1.3, these two objectives are equivalent, as shown in Recht et.al., 2010, Lemma 5.1.\n\n**Q) Ge et al. 2015 is somewhat different from the standard SGD … Adam.**\n\nWe appreciate the reviewer’s precision on this point. While we believe that the qualitative point we derive through this reference remains valid, we agree with the reviewer that, strictly speaking, Ge et al.’s result does not imply global convergence for all optimizers. We will clarify this point explicitly in our revised paper.", "claims_and_evidence": "The claims are supported by the theoretical proofs and experimental validation.", "methods_and_evaluation_criteria": "The methods makes sense.", "theoretical_claims": "The main theoretical results (Theorem 1 and Cor 1) are interesting. I checked the proofs (not every details) and most parts look correct to me. I have not identified any serious flaws, but some claims and notations requires some clarification.\n\n(1) Why the resticted smoothness is defined in this form (it is somewhat different from the standard smoothness assumption). In particular, why U and V  appear here. More explanations are needed.\n(2) In theorem 1, 2(ii), why X_box can be full rank? (since X_box is defined to be AB^T)\n(3) Proof 1. In the proof, the authors used kappa to distinguish several cases of the theorem. However, it is unclear to me how the condition of kappa corresponds to the condition of alpha and beta. In particular, which case is the special regime and which case is the generic regime (this is not mentioned in the proof at all)?", "experimental_designs_or_analyses": "The main contribution of the paper is theoretical and the experimental section is relatively small.\nI checked the experimental setting and the results, which look reasonable to me.", "relation_to_broader_scientific_literature": "LoRA is part of a broader trend in machine learning towards developing parameter-efficient methods for adapting large pre-trained models. The contribution of the paper is useful in understanding the loss landscape (near the global/local min) of LoRA optimization.\n\nThe paper also discusses the implicit bias of zero-initialization and weight decay. This connects to a growing body of work on the implicit regularization effects of optimization algorithms like SGD in deep learning models.", "essential_references_not_discussed": "There are a flurry of recent works on improving the vanilla LoRA (by tuning the learning rate or changing the initialization etc.), such as LoRA+, rsLoRA, LoRA-GA, PiSSA. Does these work change the conclusion of the paper? Some papers propose to change the initialization of LoRA (e.g.,  LoRA-GA, PiSSA). Do these results contradict the result on zero-initialization?", "other_strengths_and_weaknesses": "Strengths:\n(1) Theoretical Contributions: The paper provides a rigorous theoretical analysis of LoRA's loss landscape. The helps explain the success of LoRA in practice.\n(2) The new theoretical analysis doesn't rely on linearization arguments.\nWeaknesses:\n(1) Limited Experimental Section: The experimental validation, while sufficient to support the theoretical claims, is relatively small in scope. It focuses on fairly small-scale specific tasks (SST-2 and CIFAR-100).\n(2) Some assumptions and proofs requires clarification.\n(3) Several recent important improvements on LoRA are not mentioned at all.", "other_comments_or_suggestions": "1. page 3. section 1.3 ||X||_\\star, is this the nuclear norm? but the sentence before this line says \"adding l_2 regularization\" which is confusing.\n\n2. page 3. section 1.3 line 129. the equivalence between the unconstrained minimization problem and the constrained minimization problem. Are they always equivalent? Is there any condition or assumption?\n\n3. Information theorem line 153. The algorithm used in Ge et al. 2015 is somewhat different from the standard SGD. Moreover, the standard algorithm for training LoRA is Adam. So there is a discrepancy between the theorem cited and the conclusion (line 160) drawn from it.\n\ntypo:\n(1) page 5, line 251\n(2) page 11. line 571.", "questions_for_authors": "I have asked questions in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5GE6Txl1hI", "reviewer": "Reviewer_Rhr4", "summary": "The authors investigate the landscape of LoRA fine-tuning under assumptions of restricted strong convexity and smoothness. In particular, the authors prove a characterization of second-order stationary points for problems with regularization, showing that spurious high-rank local minima are bounded away from the global minimizers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Common Response (Repeated in all responses)**\n\nFirst of all, we thank the reviewers for their positive and constructive feedback. We are excited to see that the reviewers are appreciative of our theoretical contributions. Below, we address each of the reviewers' comments individually.\n\n### **Individual Response**\n\nWe are happy to hear that the reviewer found our results sound and illuminating. Below, we address the reviewer’s main concern regarding the scenario with near-low-rank updates.\n\n\n**Q) Theorem 1 and Corollary 1 … spurious local minimizes of higher rank far from the global one, it doesn't eliminate possibility of spurious local minima with numerical low-rank**\n\nWe thank the reviewer for the thoughtful question. We clarify the claims of Theorem 1 and Corollary 1 as follows.\n\nCorollary 1 states that if $X_\\square$ is a spurious local minima, then $\\sigma_r (X_\\square) \\ge \\frac{2\\alpha}{\\beta} \\sigma_{r*} (X_\\square)$. This provides a substantive lower bound on the $r$th singular value, making $X_\\square$ not numerically low rank. (Recall that $r> r_\\star$ in Corollary 1 and “low-rank” refers to rank $r_\\star$.) Moreover, this lower bound on $\\sigma_r (X_\\square)$ means that $X_\\square$ and the global minimum $X_\\star$ cannot be close, since $\\sigma_r(X_\\star)=0$. Therefore, Corollary 1 does eliminate the possibility that a spurious local minimum is low-rank or is close to $X_\\star$.", "claims_and_evidence": "The claims of the authors sound reasonable and the proofs look correct under the authors' assumption. However, I fail to recognize the quantitative part of the result, in fact Theorem 1 does not exclude the case of numerical low-rank spurious local minima. See \"questions\" for more details.", "methods_and_evaluation_criteria": "I believe the experimental evaluation lacks both a controlled setting to show quantitatively the authors' claims.", "theoretical_claims": "The theoretical claims and the proofs provided by the authors are correct as far as I am concerned.", "experimental_designs_or_analyses": "The experimental evaluation is not satisfying in relation to the theoretical claims. I would have appreciated a more controlled numerical setting (e.g. we know the global minima) to show convergence and the possible absence of spurious local minima around.", "relation_to_broader_scientific_literature": "LoRA variants are abundant in literature, and their popularity exploded in the last years given their effectiveness. The authors' work focuses on showing that the landscape of these problems is in some way \"well-behaving\", with low-rank local minima and higher rank spurious minima which are bounded away from the global one.", "essential_references_not_discussed": "As far as I am concerned, all the relevant literature was discussed.", "other_strengths_and_weaknesses": "The results in the paper are sound and shed light on why zero initialization works well in practice for LoRA fine-tuning.", "other_comments_or_suggestions": "See \"Experimental Designs Or Analyses\".", "questions_for_authors": "While qualitatively the result in theorem 1 and corollary 1 state that there might be spurious local minimizes of higher rank far from the global one, it doesn't eliminate the possibility of spurious local minima with numerical low-rank (therefore, potentially close to $X_*$. Can the authors comment on this?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: **Paper summary**

This paper studies the effect of initialization with less restrictive assumptions. Based on the reviewer's comments, the author theoretically analyzed that a low rank and low magnitude initialization in LoRA models results in convergence towards a global minimum, while larger rank models with larger initialization variance result in convergence towards spurious local minima with high probability. 

**Recommendation Justification** 

- *Reviewers quality and reviews quality*: Most of the reviewers are familiar with LoRA, either in theory or in more practical terms. It appears that the author was able to convey their message well. The reviews are generally positive with 3,3,3,4 scores in the end. Most of the reviewers also comment on the author's rebuttal and at least acknowledge it. So, I think this evaluation is fair. 

I also think this paper is quite interesting in terms of its analysis result. It deserves to be published at ICML this year.
Decision: Accept (oral)