AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
Authors: Xin Yu
Yujia Wang
Jinghui Chen
Lingzhou Xue
Venue: NeurIPS 2025 poster ()
Keywords: Parameter Efficient Fine-Tuning
Large Language Models
Low-Rank Adaptation
Primary area: deep_learning

Abstract: Low-Rank Adaptation (LoRA) has emerged as an effective technique for reducing memory overhead in fine-tuning large language models. However, it often suffers from sub-optimal performance compared with full fine-tuning since the update is constrained in the low-rank space. Recent variants such as LoRA-Pro attempt to mitigate this by adjusting the gradients of the low-rank matrices to approximate the full gradient. However, LoRA-Pro's solution is not unique, and different solutions can lead to significantly varying performance in ablation studies. Besides, to incorporate momentum or adaptive optimization design, approaches like LoRA-Pro must first compute the equivalent gradient, causing a higher memory cost close to full fine-tuning. A key challenge remains in integrating momentum properly into the low-rank space with lower memory cost. In this work, we propose AltLoRA, an alternating projection method that avoids the difficulties in gradient approximation brought by the joint update design, meanwhile integrating momentum without higher memory complexity. Our theoretical analysis provides convergence guarantees and further shows that AltLoRA enables stable feature learning and robustness to transformation invariance. Extensive experiments across multiple tasks demonstrate that AltLoRA outperforms LoRA and its variants, narrowing the gap toward full fine-tuning while preserving superior memory efficiency.

First Author. AltLoRA: Towards Better Gradient Approximation in Low-Rank Adaptation with Alternating Projections
Xin Yu f Department of Statistics The Pennsylvania State University State College, PA 16803 xmhy5152@psu.cr
Yujia Wang
College of Information Sciences and Technology
The Pennsylvania State University
State College, PA 16803 y yjw5427dpsu.eduu
Jinghui Chen
Lingzhou Xue Department of Statistics
College of Information Sciences and Technology
The Pennsylvania State University State College, PA 16803
The Pennsylvania State University
State College, PA 16803 jzc59170psu. edu
1zxue@psu.
Abstract
Low-Rank Adaptation (LoRA) has emerged as an effective technique for reducing memory overhead in fine-tuning large language models. However, it often suffers from sub-optimal performance compared with full fine-tuning since the update is constrained in the low-rank space. Recent variants such as LoRA-Pro attempt mitigate this by adjusting the gradients of the low-rank matrices approximate the full gradient. However, LoRA-Pro's solution is not unique, and different solutions can lead to significantly varying performance in ablation studies. Besides, to in- corporate momentum or adaptive optimization design, approaches like LoRA- -rrro must first compute the equivalent gradient, causing higher memory cost close full fine-tuning. key challenge remains integrating momentum properly into the low-rank space with lower memory cost. In this work, propose AltLoRA, an alternating projection method that avoids the difficulties in gradient approx- imation brought by the joint update design, meanwhile integrating momentum without higher memory complexity. Our theoretical analysis provides convergence guarantees and further shows that AltLoRA enables stable feature learning and robustness to transformation invariance. Extensive experiments across multiple tasks demonstrate that AltLoRA outperforms LoRA and its variants, narrowing the gap toward full fine-tuning while preserving superior memory efficiency.
Introduction
Low-Rank Adaptation (LoRA (25) has emerged as leading approach for parameter-efficient fine- tuning (PEFT) 24 38 35 of large language models ([5) 51 61 40). Building prior work investigating the intrinsic dimensionality of neural networks (2 36 LoRA assumes that fine-tuning updates be effectively captured in low-rank subspace. Specifically, for pre-trained model with weight matrix Wo E Raxd LoRA reparameterizes the weight update AW via a low-rank
Correspondence to: Lingzhou KuedIzxueppsu.eda
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

decomposition as Wo + AW = Wo + =BA, where B E Rixr, A E Rxxd and = is scaling factor. Here, < min(k. d) the rank of the update. Thanks to its substantial m and computational savings 25 LoRA has enabled scalable adaptation across diverse applications, including reinforcement learning from human feedback (RLHF) 57 23 diffusion models 43 77
and mixture-offexpers (MoE) architectures [67 37].
This gap has fueled growing interest in optimizing LoRA via hyperparameter tuming under stable feature learning 21 (0 and optimizers that preserve transformation invariance [79]. Formally, if we denote the loss function as L, full fine-tune tuning will utilize the full gradient Ww L E R Rxx for backpropagation. In contrast, the gradients in LoRA for B and A are given by (Vw L)A" and B (Vw L), respectively (see Section 2). This reparamcterization significantly alters the gradient flow during training [88 by restricting it to the low-rank space.
Despite its parameter effficincy, LoRA often underperforms full fine-tuning (13 25
promising direction to fill the gap between the gradient dynamics is to ensure that the equivalent gradient established by LoRA approximates the full gradient (166) 65] 50)). However, two key challenges in the gradient approximation for low-rank adaptation remain unaddressed. First, LoRA- Pro [66] depends an auxiliary variable that impacts the performance significantly. Depending on the choice of this variable, the evaluation score varies from 31.74 to 57.57 on the GSM8K datasets (see Appendix D.1 in (66) ). Obtaining a unique solution requires solving Sylvester equation. which introduces additional computational cost and relies on standad assumption. Second, as LoRA-Pre -rro accelerates the equivalent gradient with full- pllaaaemetter learning. it requires memory cost like full fine-tuning with space complexity (kkd) as shown in Table [ In contrast, LoRA maintains a more efficient space complexity of O(rr + rd). Under such memory constraints, how to incorporate momentum properly within the low-rank structure is largely unexplored.
In this paper, close the performance gap between LoRA and full fine-tuning, address the two key challenges outlined above and propose a novel PEFT method, AltLoRA, based on Alternating updates to the Low-Rank Adaptation. AltLoRA properly approximates the full gradient by alternately pro- jecting it onto low-rank subspaces and B. Building on this injectioonbbsel gradient approximation, we further introduce a new mechanism to optimize momentum effectively within the low-rank space, while strictly adhering to the memory constraints of LoRA (25). Without allowing full tull-parameter learning, AoRA is the first work in the literature to properly optimize both gradient and momentum over the low-rank subspaces, while achieving stable feature learning and transformation invariance, as summarized in Table
Table I: Comparison with Existing Work
Methods
Gradient Approsimation
Stable Feature Learning
Transfornation Invariacee
Time Complexity
Space Complexity
LoRA
25
x
x
Olky dr")
Oiky dr)
LoRA=
O(ky d
Oikr dr)
ScaledAdam
81
x
O(ky des)
Olkr dr)
LoRA-Bite
O(kr* dr3)
Ojkr dr)
LoRA-Pro
66
((kdr))
(lkd)
AlILoRA
V
(kr +
O(kr +
Our main contributions are summarized as follows:
We propose AltLoRA, novel PEFT method that efficiently approximates the full gradient via alternating projections onto the low-rank subspaces A and B. Moreover, we design a new momentum mechanism that operates within LoRA's memory constraints, enabling effective optimization of momentum within the low-rank space.
Theoretically, we prove that AltLoRA ensures stable feature learning in the infinite-widtt neural network regime and, more generally, maintains transformation invariance, even when incorporating momentum. We also provide convergence guarantees for fine-tuning overparameterized two-layer ReLU networks.
Empirically, we show the effectiveness of AltLoRA through extensive experiments on tasks including natural language understanding. dialogue generation, mathematical reasoning. and code generation. AltLoRA consistently outperforms existing LoRA-based methods.

---

2
Preliminary
Let us first revisit the optimization paradigm of LoRA (25] If we denote the loss function as L, i...
L(A, B) == L(W +BA), we can derive the gradient w.r.t A and B as follows:
0L aL ow
aL ÖL ow
VAL: = 0A aw 0A sB" LVw VBL = dB dw dB = s)V" (1)
Here, as the full gradient multiplied by the low-rank matrices to constitute the gradient of LoRA, implicitly compresses the full gradient into the low-rank spaces. Suppose update A and B, then the model parameter in the (t + 1)-th iteration is:
gradient descent to
W e++ Wo + Al41
x + +o++B,, ( B. sn (v., A. (V.L(V
W. Wn
Here, we omit the term related to r/². Compared with the full gradient update - V. LoRA's gradient can approximate the full gradient as long as sB(V + LL) is close to Vw L. With a similar motivation, some previous work analyzes the approximation based on the Frobenius norm ( (65 66 50).. Noticeably, LoRA-Pro (66) achieves gradient approxiation by adjusting the gradients of matrices A and B based on the following solutions:
(VwL)) XA,9 - B B" -BB,
where X E Rexr denotes ancillary matrix and its selection is crucial and challenging for LoRA- Pro. As shown in their ablation studies, the selection of X would vary the performance of the evaluation significantly. Besides, obtain unique solution for X, LoRA-Pro imposes additional uncommon assumptions to solve Sylvester equation. However, even selecting a unique X, the equivalent gradient(sBgr) B s A) established by LoRA-Pro is independent of X, which implies that X is only used to distinguish the gradient of A and B when jointly updating and doesn't 't influence the model update. It motivates the development of more efficient alternating and eliminates the influence of X. To circumvent the ambiguity and inefficiency introduced by this joint updating strategy, we propose an alternating update strategy that approximates the full gradient long as sB(V xL) or s(V a A is close to V L.
the following notation describe the asymptotic behavior as the width grows. Given sequences on E R and d, E R*, we write e₂ = O(d,). resp. c, = [dd,), to refer to On Rdn, resp. Cn >> kdn, for some constant σ. >> 0. For vector and matrix sequences, the notation is applied entry-wise. Additionally, we use @ and @ to denote element-wise matrix multiplication and division, respectively. [P] denotes the set of indices (1,.... P).
Notation. Hereafter,
3 Methodology
3.1 Alternately Approximating the Full Gradient via Low-Rank Adaptation
We propose alternating update scheme, where update A first and then update B based on the new A. Define the low-rank modules as A, and B, at the t-th iteration, and the approximated gradients as A L and V BL, respectively. We begin by obtaining the optimal scaling gradient of A by solving
min [Vs L) T.I. L
where || 1 denotes the Frobenius norm squared- sum squares of all entries in the matrix. Then by gradient descent, we can update A and the full model as
A < A, V
W44 +- W, BB I.),
(5)
where we update the full model (1 + 1/2)-th iteration to keep consistent with the joint update (66
(update A and B in one iteration). In our experiment, without any ambiguity, we treat the update A or
3

---

After doing backpropagation w.r.t A, the gradient of B doesn't t approximate the full gradient time since the full model has been update to the state of (t + 1/2). Then we minimize the discrepancy between the full gradient at W W and the approximating gradient constructed by Be as follow
B as a single step (see Algorithm[] D.
min [|s(VB,))AHH1 VW 413 n.
(6)
Then by gradient descent, we can update B and the full model as
Bi+1 + B, BVBL W4+1 + W LV
(7)
The following theorem gives the closed-form solution of Problems and 6
Theorem 1. Assume B. Rg and A, Brxd are full rank for any i, i.e. rank(B,) rank At)
r. Solving Problems and (6) yields the unique closed-form solutions
1 (B, Bc) Vo
A. L [b] B₂) B. (Vw.
(8)
1 V L (Vw.. L) (AL g2
1 V
where V "eL and VB,L L are the gradients of LoRA defined in Equation i
Theorem shows that both problems admit unique optimal solutions for V A. and V L, which only requires full rank. Therefore, it offers a new gradient approximation with less computational cost and promotes more efficient updating strategy. Besides, instead of accessing the full gradient like full fine-tuning, the optimal gradient approximation only requires the standard gradient of A or B by backpropagation at each step and calculating the inverse of small matrix with size
Theorem | requires that the matrix B, and A, are full rank, but in the over pacemaaramteri cases, the assumption is hard achieve. To alleviate it, if we penalize the Frobenius norm of these two approximated gradients, i.e., weight decay, the condition can be eliminated (see Corollary [T. For simplicity, in the rest of the paper, focus the modified gradient in (8) for analysis. The closed-form solution in (8) yields the following full model update(wid) gradient descent)
W4+1 = Wo+y 1 (VV B, A)
Wt++ - -v(V w + A -AAe1
Wi nnBB
A. V/( 1) (A411 1
= W, - (B, BB)))-----11 (Vw, - T (V L) A Ar41
= We - m/Proj((BB)). (Vw, - 17( V W L) Proj((
Interestingly, the proposed solution for gradient approximation in (8, consistent with the literature work [59] 83 73
30 42 called scaled gradient descent [46] 45 in low-rank matrix estimation (54).
Therefore, the view of gradient approximation would provide a novel interpretation of applying scaled gradient descent within the broader context of low-rank matrix decomposition. As optimizing LoRA
with momentum for acceleration is a standard way in the literature [|[2]] 21], we will discuss how to properly design momentum within the low-rank space inspired by gradient approximation.
3.2 Proper Momentum Design within the Low-Rank Subspaces
For LoRA 255 and its variants 21 86] without allowing full- aill-parmmtter learning, the parameterization restricts both the gradient and the momentum updates to low-rank subspaces the memory cost is Okr dr). As we have shown, the optimal gradient approximation under this constraint is obtained by projecting the full gradient onto the low-rank subspace. This insight naturally motivates the need to also align the momentum optimally within the same low-rank space, in order to fully leverage momentum-based acceleration under low-rank constraints.
Since the momentum evolves throughout training, it is essential to dynamically optimize it. For simplicity, focus the optimization paradigm for B and develop method inductively. Given

---

the aligned momentum M, within the low-rank space A, at time t, the alternating update strategy proceeds by updating
A4+1 and then aligning M with the new low-rank space A,+1. To this end, we first recover M. to the full-dimensional space, and then project it onto the new subspace spanned by A like gradient approximation. The following theorem formalizes this key idea.
Theorem 2. Assume A4+1A4+1 is full-rank, i.e., rank Ac+1 A =r. If M has aligned with the low-rank space As the 1-th iteration, by minimizing the following problem
min | A, MM Ac+1 MP
(10)
We can find M₁ = MM A (A 1 which makes the momentum aligned with the new low-rank space A++1 optimally.
Theorem shows that it is only necessary to store two small matrices so that we can optimize momentum properly. Similarly to Section 3.1 here (see Corollary 2 In contrast to LoRA-Pro with full-parameter learning (Space Complexity Olkd)), we aim to strictly satisfy the space complexity O(kr + dr) for parameter efficiency and keep momentum adaptively aligned with the low-rank spaces as the gradient approximation does.
also remove the assumption of full rank
where down-projection and up- projection matrices are employed to transfer compressed gradients across low-rank spaces. In contrast, we derive the optimal alignment directly within the low-rank subspaces preserve gradient information. In Section 4.2 we theoretically demonstrate that aligning momentum with low-rank space guaranties formation invariance, whereas LoRA [5 and its variants 21 86
A similar notion of momentum design is explored in 18
have misaligned
momentum that undermines this robustness [79].
After analyzing how to efficiently optimize both the gradient and momentum under limited resource constraints, we summarize our proposed algorithm, AltLoRA, in Algorithm [T] Unlike the joint update strategy, AltLoRA A updates only one of the low-rank matrices, either A or B, at each step, based on the scaled gradient and momentum presented in Theorems and [2] The number of trainable parameters at each step is reduced by half compared to the joint update. Designed as a practical PEFT method, AltLoRA can be scamlessly integrated into existing libraries such as Hugging Face [69 (sec Appendix C.I for implementation details). To further accelerate and stabilize the training paradigm of AltLoRA, we introduce AltLoRA+, an enhanced variant that naturally incorporates second- moment estimates similar to A (see Algorithm 2 for details).
Algorithm 1: AltLoRA: Gradient Approximation via Alternating Projection with Proper Momentum Design under LoRA's Memory Constraint
Input: Momentum states Mn. M, scaling factor R; learning rate 77 momentum coefficient B1; total steps T'; weight decay y
Output: Final matrices Ar and Br
for = 0,..., 1 do
if mod 2 = 0 then
Update A:
Only backpropagate w.r.t. A, and obtain V
V L (B, B,) V A
MC =B,T
B B--
MfA < M, + +(1 -3.
A A n(M, +nAi)
else
Update B:
Only backpropagate w.r.t. B, and obtain V.,L
V L = 112 V. L(A41
MB = A,ATA (A41AAT))
M BB + +(1 B.).
Br+1 - B, nM, +7B))
Time Complexity and Space Complexity. When & min (ki d), the time and memory cost of AltLoRA and AltLoRA+ is similar to the standard LoRA and more efficient compared with LoRA-
5

---

Pro. The additional computational cost takes ( time, and since is very small, this overhead is negligible when compared with the back propagating time. In the experiment, we will show that the delay time compared with LoRA is mild even when the rank r increases. (see Table
4
Theoretical Analysis
4.1 Stable Feature Learning
Given the current trend of increasing model sizes (176) 75)) it raises a lot of attention to analyze the asymptotic training behavior of neural networks the number of neurons approaches infinity (56 19, 74). There is line work in LoRA ([21) 20 81) considering the infinite-width NN setting. To achieve stable feature learning (see Definition in Appendix D.ID. they propose a fine-grained choice of hyperparameters in the original LoRA, like the learning rate [21]. the initialization ((00),
and the optimizer (181). The core idea is that the update increment over the loss function or parameter should be of constant magnitude, which ensures that neither the NN predictions nor the increments explode or vanish as the NN size increases, thereby leading stable training dynamics. First, demonstrate that our method achieves stable feature learning on toy model in Appendix [D. We then prove that this stability extends arbitrary LoRA ranks and holds for AltLoRA and AltLoRA+, which we formalize in the theorem below. For clarity of presentation, we omit the scaling factor in the subsequent theorems and analysis.
Theorem 3 (Informal). Assume that, with the input x. BAx has dimension (n). In Algorithm orr Algorithm 2 if we use the same learning rate O(1) to update A and B, it would achieves stable feature learning. Moreover, without momentum in AltLoRA or AliLoRA+, the model update achieves stable feature learning as well with
W+41 W,- (PProjaBB - WV.,.. Proir(am)))
(11)
where
PrrojaBB) (ww, L), (V,., L.W.W./PPRR... Proj((())))) E O()).
However, when doing joint update (181), the update will introduce additional across term I2(V [[A] (A,A7) '(B, B,) order term w.rd T3, but is same magnitude as ((V...))rojj.an and (Proje(B,)( (V w. L) in infinite-width width NN setting.
E O()). The across term is indeed the second
In Theorem 3. AltLoRA and AltLoRA+ achieve stable feature learning. Morcover, as the joint update would introduce the cross term with unignorable magnitude (especially v) is ((1) instead of ((1/n) nn) in the toy model), joint update with scaled gradient descent (81) breaks the clean interpretation of projecting the full gradient onto low-rank subspaces and degrade the performance as our experiment studies show later.
4.2 Transformation Invariance
With the motivation that an optimizer should yield the same update to the full model regardless of the specific factorization, transformation invariance, sufficient condition for stable feature learning, proposed by LORA-RITE [79]. Here, we will prove that our designed gradient and momentum in Algorithm D would be inherently robust transformation invariance.
two pairs of LoRA matrix (Ai, B1), (A2, B2) can represent the same finetuned weight = Wo + Bi Ai = Wo + B2A2. An optimizer exhibits transformation invariance its updates, (δAi 8B,) and (6A2 8B2) sutisfy
Definition 1. If there
Wo + (B1 + 8B, +AA) Wo + (B2 8B2) 5.A2)
(12)
[ + (A) + 6A,) = -B2 + (B) (A2++66)))))))) 6A2).
LoRA-RITE 79 notices that, after combining scaled gradient descent with element-wise Adam in 81 the ScaledAdam can't 't preserve transformation invariance. As the momentum is optimized properly, we will analyze how A o keeps transformation invariance naturally, especially when incorporating momentum.

---

Recall the definition of projection matrices in Equation ( Proje(n)) == B, B. (or
Projr(A)) == A (A,44)) At). The following lemma provides insight into how Algorithm | achieves transformation invariance.
Lemma 1. If any two pairs of LoRA factors( (A,, B1), (A2, B2) satisfying
= Wi - B1A - Wo + B2A2, Proje(B)) Proje(B2)) (ro)(())) Projr(Aa)-
(13)
then
Even though the full model update can be decomposed into different pairs of low-rank adaptations, within each pair of LoRA factors, the column space of B (or the row space of A) equivalent the column space (or the row space) of the full model update. Therefore, the projection matrix would be preserved invariant over the pairs of low-rank adaptation.
Theorem 4. AltLoRA in Algorithm is transformation iinnfformationnimaarinnn invariant.
Building on the insight from Lemma I we leverage the invariance of the projection matrix to the low-rank subspaces to approximate the full gradient via the gradient and moment information. As result, with the goal of gradient approximation without full-pane aall-parmmeter learning, our method achieves transformation invariance inherently. LORA-RITE (79 is also aware of the equivalence of low-rank spaces, but they do not notice or exploit the invariance of the projection matrix. Instead, they design an unmagnified gradient requiring polar decomposition each iteration, which introduces additional computational overhead. In contrast, our method avoids polar decomposition, contributing to its superior efficiency (see Table 3p LoRA-Pro [66] also achieves transformation invariance but does so without adhering to LoRA's memory constraint. AltLoRA in Algorithm I by comparison, strictly follows the memory budget of LoRA while preserving transformation invariance through a more efficient design. While Algorithm 2 does not currently maintain transformation invariance under second-order momentum, this opens an exciting avenue for future research. In Appendix [ we provide detailed discussion on why extending our first-order momentum design to the second order poses fundamental challenges. Despite this, AltLoRA+ achieves substantial empirical gains over LoRA and its variants, demonstrating the practical strength of our approach even when we only keep the cerssfoomationnvaaiiaaa up the second momentum.
4.3 Convergence Analysis
Following 81, we provide a convergence analysis of AltLoRA (or AltLoRA+) without momentum within the over- peeepparameteiieed two-layer ReLU NN tuning problem (see Appendix D.3). In Theorem 7 we show that the convergence is independent of the condition number of the data matrix. In contrast to [8]] we impose fewer assumptions establish the convergence analysis. Notably, we don't 't require the extended spectral initialization in Definition 7.3 [1I. In our experimental study,
AltLoRA (AltLoRA+) can achieve superior performance with the variant of initialization used by LoRA and its variants (see Appendix E.3.2) which supports our insight empirically.
5 Experimental Results
This section empirically shows the effectiveness our approach across various model architectures and datasets. Section 5.] summarizes the experimental settings and results supervised fine-tuning (SFT) benchmark tasks, and 352tion|3.2prooviicee details of the setup and results for natural language understanding tasks. Finally, ablation studies from multiple perspectives are presented in Section 5.3 The code for our project available at https:///gthuu..omm/ www./ocasXiin Yu/AltLoRA
5.1
Experiments on SFT of LLM: Natural Language Generation
Training Details. We assess our methods on dialogue generation with the WizardLM dataset
mathematical reasoning with the MetaMathQA dataset [80], and code generation with the CodeFeedBack dataset [90] using the LLama-3.1-4 1-B and Llama-3-8B models [17] (see Appedix
(72.
We compare AltLoRA and AItLoRA+ with the pretrained model, full fine-tuning, LoRA PisSSA [4), A LoRA+ D AdaLoR (86). LORA-GA 65 LoRA-Rite
25
79 and A-Ro-Prof6e To ensure fair comparisons, we closely follow the experimental protocol
established by (66). Unless otherwise stated, we fine-tune models using default hyperparameters (if
7

---

used): 51 - 0.9, B2 = 0.999, and zero weight decay, We adopt a cosine learning rate schedule with warm-up-ae ratio 0.03. LoRA adapters are applied to {Q, K, V, V] layers. By default, we set the rank to r = 8 and the scaling factor to 32 for dialogue generation tasks, and r = 8, the mathematical reasoning and code generation tasks. We carefully grid search the learning rates ? To obtain reliable estimate of model performance, we perform three runs with different random seeds and report the average and standard deviation of the results.
16 for
Evaluations. We evaluate the baselines similar to [66]. Specifically, for the dialogue generation task, we use the MT-Bench dataset [89] with GPT-4o, with scores ranging from to 10. We report the score from the first turn as our metric. For the math task, we evaluate the model on the GSM8K test set [T] using the LLM Evaluation Harness [[6], and we report the exact match accuracy. For the code generation task, we evaluate on the HumanEval dataset [6] and report the PASS@I metric.
experimental results, which demonstrates AltLoRA superior perfor- mance. With a rank of 8, AltL achieves noticeable improvement over the original LoRA: 0.5 on MT-bench, 8.38 on GSMSK and 3.1 on HvmanEval using 1.ama-3.1-8B Notably, AltLoRA achieves significantly higher scores on MT-Bench compared to LoRA-Pro and Full FT. In addition. A yields improvements over LoRA-Pro on both GSM8K and Hvma and AltLoRA+ obtains better performance in mathematical reasoning than Full FT. These further demonstrate the effectiveness of the new design gradient and momentum. The additional study on Llama-3-88 model (see Table S Appendix E.1 also demonstrates a clear advantage over baseline methods.
Results. Table 2 presents
in
Table 2: Comparison of different LoRA variants on MT-Bench, GSM8K, and HvaanEval benchmarks on
Bold indicates the best result, underline represents the second-best one.
Method
MT-Bench
GSMSK
HumanEval
PreTrain
5.93=0.08
11.34+:..38
16.97
Full FT
6.31
73.31+0032
50.81+1.100
LoRA
6.06+-0.22
66.11+1..3
40.31++.34
PISSA
0.15
6..78:1.111
1..44::1.11
LoRA
6.10=0.06
02:44.44
43.91+1.44
LoRA+
6.40=0006
1.33
44.10
DoRA
6.08=0003
08.33+0088
42.13+1.31
AdaLoRA
6.08=0.05
1..63+1.45
42.21+2.66
LoRA-GA
6.00=0.09
70,33+0.91
42.01+1.211
LoRA-Pro
6.19-00.3
73.12400.56
43.13.11.45
LoRA-Rite
6.10
74.10+10.31
43.1220.551
AltLoRA
6.56=0004
74.49+00.57
45.91
AltLoRA (rank=32)
6.39-0004
73,24+-0.29
46.87+11.49
AltL.oRA (rank=128)
6.27+0001
74.11+.0.21
45.41+
AltLoRA+
6.1620.02
76.91110.11
10.10111.35
Altl.oRA+ (rank=32)
+0.02
76.32+0.29
49.97+1.522
AltL.RRA+ (rank=128)
6.07-0003
77.08+0.83
49.77+1.58
Memory and Time Consumptions. In Table 3 we also compare the memory cost and training time of our methods with Full FT, LoRA, LoRA-Rite and LoRA-Pro Llama-3.1-80 mode. Without full- parameter learning, we have a comparable memory cost and training time close to LoRA. After taking higher rank of LoRA, the memory cost and computation cost won't 't increase significantly. However, as LoRA-Pro requires storing the full size first-order momentum and second-order momentum, incurs additional calculations like polar
leads an unignorable cost like Full FT. As decomposition, it also increase the computation time.
*See Appendix E.1 for details of learning rate grid search. We set the sequence length to 1024 and the macro batch size to 4 for math and code tasks, and macro batch size to 8 for dialogue generation. All experiments are conducted NVIDIA A100 and NVIDIA A6000 GPUs
8

---

Table 3: Comparison of memory usage and training time across different fine-tuning methods.
Method
Memory Cost
Training Time
Full FT
> GB
23min
LoRA
22.26 GB
2h 13min
LoRA-Rite
25.39 GB
2h 44min
LoRA-Pro
40.12 GB
4h 5min
AltLoRA
22.56 GB
2h 34min
AltLoRA(rank=32)
23.11 GB
2h 41min
AttLoRA(ran==123)
25.11 GB
52min
AltLoRA+
23.16 GB
2h 38min
AltLoRA+(rank=32)
24.98 GB
45min
1ltLoRA=+rank=128))
27.76 GB
2h 56min
5.2 Experiments Natural Language Understanding
Training and Evaluation Details. We assess our methods natural language understanding on subset of GLUE benchmark dataset with fine-tuning T5-base(32] model. We compare AltLoRA and AltLoRA+ with the full fine-tuning, LoRA (25), PisSSA 44], rsLoRA[31] LoRA+ DoRA[1I] (86), LoRA-GA[6S], and LoRA- Prr 66 We fine-tune the T5-based model [52] with our methods and the baselines on subset of GLUE datasets 633 MNLI, SST2, CoLA, QNLI, and MRPC. We use the accuracy as the evaluation metric. To ensure fair comparison, all experiments are run three times with different random seeds, and we report the mean and standard deviation of the results. Due to space constraints, additional experimental details are provided in Appendix E.
Results. As shown in Table 4 AltLoRA+ outperforms the baselines on average. In particular, it achieves the highest score on MRPC, the heihhet CoLA, MNLI, and SST-2 datasets.
Table 4: Performance of fine-tuning T5-Base on 5 sub-tasks of the GLUE benchmark. Bold indicates the best result, underline represents the second- beot one, and # marks results reported from 65
Method
MNLI
SST-2
CoLA
QNLI
MRPC
Average
Full 86.29
93.97
80.87
93.02-10.03
86.89110.13
88.21
LoRA
85.32:10.01
93.76.10.05
81.31
92.96.10.09
86.03-10.24
87.88
RSLoRA
85.23110.01
93.96110.06
81.21
93.12-10.09
86.27-10.24
87.96
DoRA
85.58
93.65+00.06
81.16:10.04
93.04:10.06
86.14:10.12
87.91
LoRA+
85.32:10.06
93.92::.111
81.21
92.97=00.03
86.25:10.16
87.93
PiSSA
85.87++0.04
93.84++0.06
81.90+00.05
93.16++0.09
86.644-0.12
88.28
LoRA-GA"
85.70+L0.09
94.11+
80.57+00.20
93.18 0.06
85.29-
87.77
AdaLoRA
85.45+00.11
93.92+00.09
80.31+00005
91.66++0.05
86.16++0.60
87.50
LoRA-Pro
85.70+00.11
93.92+0..10
78.42+00.33
93.15+00.3
86.54+0.50
87.55
AltLoRA
85.26.10.04 93.87100.05
80.44100.99
91.5611.011
86.60:10.99
87.55
AltLoRA+
85.81=0.03
94.03++0.12
81.44+0.30
92.99+0.03
87.25+11.12
88.30
5.3 Ablation Study
Figure T presents an ablation study the learning rate and the scaling factor a for LoRA, AltLoRA and AlILoRA+, using the LLaMA 3-B model on mathematical reasoning tasks. The results show that our proposed methods are robust in learning rate and the scaling factor with consistent superior performance. Moreover, it shows that a 16 obtains overall better performance compared and a = 32. The influence of increasing rank is reported in Table (see Appendix E.3 of the results on Llama-3-8B model). Besides, studying the choice of hyperparameters, in Appendix E.3.2 we present additional ablation studies on the Llama 3.B model as well. To evaluate the effectiveness of alternating strategies, we compare them against the joint update method. the approaches of multiple LoRA modules, such as in the mixture of LoRA experts, has gained popularity (37 70), we also assess the impact of varying the number of experts in LoRA layers. Finally, further validate the robustness of our method with respect to initialization, discussed in Section 4.3 study

---

Accuracy
50.0
Lountng 1e5
Leamrr Rate
40
74.49
76.91
$5.00
66.50
74.12
75.08
-000
60.00
32
Alri lo)
Aptu (M)
Appho tc)
Figure 1: Evaluation Accuracy of LoRA, AltLoRA and AlRA+ for various learning rate 1) and scaling factor a combination on the GSM8K datasets using 1-88
different initialization strategies. These ablation studies collectively demonstrate that our method is robust hyperparameter variations and applicable more complex model architectures.
Conclusion
We propose AltLoRA, memory-efficient fine-tuning method that alternates updates of low-rank matrices to dynamically project both the gradient and momentum within low-rank subspaces, By leveraging efficient closed-form gradient approximation and a principled momentum design, AltLoRA operates entirely under low-rank constraints while ensuring stable feature learning and transformation invariance without requiring full parameeer learning. Extensive experiments across diverse tasks demonstrate the superior performance AltLoRA and its enhanced variant, AltLoRA+, over LoRA and its variants, narrowing the gap to full fine-tuning while retaining memory efficiency.
10

---

Acknowledgements
The work of X. Yu and L. Xue was supported by the U.S. National Science Foundation under the grants CCF-2007823 and DMS-2210772 and by the U.S. National Institutes of Health under the grant IR01GM152812. The work of Y. Wang and J. Chen was partially supported by the National Science Foundation under Grant No. 2348541. The views and conclusions contained in this paper are those of the authors and should not interpreted as representing any funding agencies.

---

References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr. Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 087744, 2023.
[2] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning arXiv preprint arXiv:2012.13255. 2020.
[3] Jiamu Bai, Daoyuan Chen, Bingchen Qian, Liuyi Yao, and Yaliang Li. Federated fine-tuning of large language models under heterogeneous language tasks and client resources. arXiv e-prints, pages arXiv-2402, 2024.
[4] João Carlos Alves Barata and Mahir Saleh Hussein. The moore-penrose pseudoinverse: A tutorial review of the theory. Brazilian Journal of Physics, 42:146-165, 2012.
[5] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph. Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprini arXiv:2107.03374, 2021.
[7] Shuangyi Chen, Yuanxin Guo, Yue Ju, Harik Dalal, and Ashish Khisti. Robust federated finetuning of Ilms via alternating optimization of lora. arXiv preprint arXiv:2502.01755, 2025.
[8] Yiming Chen, Yuan Zhang, Liyuan Cao, Kun Yuan, and Zaiwen Wen. Enhancing zeroth-order fine-tuning for language models with low-rank structures. arXiv preprint 1rX00:2410.07698 2024.
[9] Yiming Chen, Yuan Zhang, Yin Liu, Kun Yuan, and Zaiwen Wen. A memory efficient ran- domized subspace optimization method for training large language models. arXiv preprint arXiv:2502.07222, 2025.
[10] Cheng Cheng and Ziping Zhao. Accelerating gradient descent for over parameterize asym- metric low-rank matrix sensing via preconditioning. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7705-7709, IEEE, 2024.
[11] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10.1/100 /4168, 2021.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized Ilms. Advances in neural information processing systems, 36:10088- 10115, 2023.
[13] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 533::220-235, 2023.
[14] Shihan Dou, Enyu Zhou, Yan Liu, Songyang Gao, Jun Zhao, Wei Shen, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Xiaoran Fan, et al. Loramoe: Alleviate world knowledge forgetting in large language models via moe-style plugin. arXiv preprint mwXi:2231..09977 2023.
[15] Ke-Lin Du, MNS Swamy, Zhang-Quan Wang, and Wai Ho Mow. Matrix factorization techniques in machine learning, signal processing, and statistics. Mathematics, 11(12):2674, 20233
[16] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL hhttps://zente org/ records/ 12608602
12

---

[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2 2024.
[18] Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. arXiv preprini arXiv:2402.033925 2024.
[19] Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. International conference on machine learning, pages 2672-2680. PMLR, 2019.
[20] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The impact of initialization on lora finetuning dynamics, Advances in Neural Information Processing Systems, 37:117015-117040 2024.
[21] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402. 2tti:22022./2554, 2024.
[22] Yutong He, Pengrui Li, Yipeng Hu, Chuyan Chen, and Kun Yuan. Subspace optimization for large language models with convergence guarantees. arXiv preprint arXiv:2410. 1289, 2024.
[23] Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, et al. Improving neutral point of view text generation through parameter-efficient reinforcement learning and a small-scale high-quality dataset. arXiv preprint arXiv:2503. 03654, 2025.
[24] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efffcieet transfer learning for nlp. In International conference on machine learning, pages 2790-2799. PMLR, 2019.
[25] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, al. Lora: Low-rank adaptation of large language models. ICLR, (2):3, 2022.
[26] Chengsong Huang, Qian Liu, Bill Yuchen Lin, Tianyu Pang, Chao Du, and Min Lin. Lo- rahub: Efficient cross-task generalization via dynamic lora composition. arXiv preprint arXiv:2307.13266 2023.
[27] Qiushi Huang. Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. Hira: Parameter-efficient hadamard high-rank adaptation for large language models. In The Thirteenth International Conference on Learning Representations, 2025.
[28] Nam Hyeon-Woot Moon Ye-Bin, and Tac-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficee federated learning. arXiv preprint (...v:2208.06000 2021.
(29) Prateek Jain, Praneeth Netrapalli, and Sujay Sanghavi. Low-rank matrix completion using alternating minimization. In Proceedings of the forty-fifth annual ACM symposium Theory of computing, pages 665-674, 2013.
[30] Xixi Jia, Hailin Wang, Jiangjun Peng, Xiangchu Feng, and Deyu Meng. Preconditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent. Advances in Neural Information Processing Systems, 36:76202-76213, 2023.
[31] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.052. 2023.
[32] Siddhartha Rao Kamalakara, Acyr Locatelli, Bharat Venkitesh, Jimmy Ba, Yarin Gal, and Aidan N Gomez. Exploring low rank training of deep neural networks. arXiv preprint arXiv:2209. 13569, 2022.
[33] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF international conference on computer vision, pages 4015-4026. 2023.
13

---

[34] Dawid J Kopiczko, Tijmen Blankevoort, and Yuki M Asano. Vera: Vector-based random matrix adaptation. arXiv preprint arXiv:2310.1155. 2023.
[35] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameler-efficient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
[36] Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. arXiv preprint arXiv:1804, 08838, 2018.
[37] Dengchun Li, Yingzi Ma, Naizheng Wang, Zhengmao Ye, Zhiyuan Cheng, Yinghao Tang, Yan Zhang, Lei Duan, Jie Zuo, Cal Yang, et al. Mixlora: Enhancing large language models fine-tuning with lora-based mixture of experts. arXiv preprint MxXi:2444.151555 2024.
[38] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1eXiv:2101.00 100000,, 2021.
[39] Xutao Liao, Shaohui Li, Yuhui Xu, Zhi Li, Yu Liu, and You He. Galore +: Boosting low-rank adaptation for Ilms with cross-head projection. arXiv preprint arXiv:2412. 2024.
[40] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.14437, 2024.
[41] Shih-Yang Yang Liu, Chien-Yi Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Sinng Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation. In Forty-first International Conference on Machine Learning, 2024.
[42] Zhiyu Liu, Zhi Han, Yandong Tang, Hai Zhang, Shaojie Tang, and Yao Wang. Efficient over- parameterized matrix sensing from noisy measurements via alternating preconditioned gradient descent. arXiv preprint arXiv:2502.0044
2025.
[43] Simian Luo, Yiqin Tan, Suraj Patil, Daniel Gu, Patrick von Platen, Apolinário Passos, Longbo Huang, Jian Li, and Hang Zhao Lem-lora: A universal stable-diffusion acceleration module. arXiv preprint / 1.05556, 2023.
[44] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. Advances in Neural Information Processing Systems, 37:121038-121077 2024.
[45] Bamdev Mishra and Rodolphe Sepulchre. Riemannian preconditioning. SIAM Journal on Optimization, 26(1):635-660, 2016.
[46] Bamdev Mishra, K Adithya Apuroop, and Rodolphe Sepulchre. A riemannian geometry for low-rank matrix completion. arXiv preprint arXiv: /2/ 1550, 2012.
[47] Lorenzo Noci, Chuning Li, Mufan Li, Bobby He, Thomas Hofmann, Chris Maddison, and Dan Roy. The shaped transformer: Attention models in the infinite depth-and-width limit. Advances in Neural Information Processing Systems, 66:54250-54281 2023.
[48] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247, 2020.
[49] Mert Pilanci and Tolga Ergen. Neural networks are convex regularizers: Exact polynomial-time convex optimization formulations for two-layer networks. In International Conference Machine Learning, pages 7695-7705, PMLR, 2020.
[50] Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Pranceth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning. arXiv preprint 1rX5v:2411.19557. 2024.
[51] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages PmLR, 2021.
14

---

[52] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1(10)):1667 2020.
[53] Benjamin Recht, Maryam Fazel, and Pablo A Parrilo. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM review, 52(3)471-501, 2010.
[54] Angelika Rohde and Alexandre B Tsybakov. Estimation of high-dimensional low-rank m The Annals of Statistics, 39(2)8877-930, 2011.
[55] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[56] Samuel Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep informa- tion propagation. arXiv preprint arXiv:1611.0123., 2016.
[57] Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, et al. Parameter efficient reinforcement learning from human feedback. arXiv preprint arXiv:2403. 10704, 2024.
[58] Youbang Sun, Zitao Li, Yaliang Li, and Bolin Ding. Improving lora in privacy-preserving federated learning, arXiv preprint 2xXiv:2403112111 2024.
[59] Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150):1 63, 2021.
[60] Tian Tong,
Cong Ma, and Yuejic Chi. Low-rank matrix recovery with scaled subgradient methods: Fast and robust convergence without the condition number. IEEE Transactions Signal Processing, 19-2396-2409) 2021.
[61] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092 2023.
[62] Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:22/0.07555; 2022.
[63] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07444 2018
[64] Hongyi Wang, Saurabh Agarwal, Yoshiki Tanaka, Bric Xing, Dimitris Papailiopoulos, al. Cuttlefish: Low-rank model training without all the tuning. Proceedings of Machine Learning and Systems, 5:578-605, 2023.
[65] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approxima- tion. Advances in Neural Information Processing Systems, 37:54905-544 559331, 2024.
[66] Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? arXiv preprint 2rXiv:2407.1822. 2024.
[67] Zihan Wang, Deli Chen, Damai Dai, Runxin Xu, Zhuoshu Li, and Yu Wu. Let the expert stick to his last: Expert-specialized fine-tuning for sparse architectural large language models. arXiv preprint arXiv:2407.01906 2024.
[68] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. arXiv preprint 240::2200000976, 2024.
[69] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Huggingface's 's transform- ers: Sf art natural language processing. arXiv preprint arXiv:1910.0377.. 2019,
15

---

[70] Xun Wu, Shaohan Huang, and Furu Wei. Mixture of lora experts. 0rX1v:2404.36628, 2024.
arXiv preprint
[71] Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of lora: Efficient fine-tuning of language models via residual learning, arXiv preprint arXiv:2401 0.1111, 2024.
[72] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qing- wei Lin, and Daxin Jiang. Wizardlm: Empowering large pre- pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, 2024.
[73] Xingyu Xu, Yandi Shen, Yuejie Chi, and Cong Ma. The power of preconditioning in overpa- rameterized low-rank matrix sensing. In International Conference on Machine Learning, pages 38611-38654. PMLR. 2023.
[74] Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian pro- cess behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint arXiv: arXiv:1902.04466, 2019.
[75] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522, 2020.
[76] Greg Yang and Edward J Hu. Tensor programs iv: Feature learning in infinite-widhh neural networks. In International Conference on Machine Learning, pages 1 11727-11737. PMLR, 2021.
[77] Yang Yang, Wen Wang, Liang Peng, Chaotian Song, Yao Chen, Hengjia Li, Xiaolong Yang, Qinglin Lu, Deng Cai, Boxi Wu, et al. Lora-composer: Leveraging low-rank adaptation for multi-concept customization in training-free diffusion models. arXiv preprint 1rXiv:2403.11627, 2024.
[78] Can Yaras, Peng Wang, Laura Balzano, and Qing Qu. Compressible dynamics in deep overpa- rameterized low-rank learning & adaptation. arXiv preprint arXiv:2406.0u 041 12, 2024.
[79] Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsich, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. arXiv preprint arXiv:2410.2063 2024.
[80] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. arXiv preprint arXiv:2309. /2284, 2023.
[81] Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation models. arXiv preprint arXiv:2402.02347, 2024.
[82] Feiyu Zhang, Liangzii Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. Incretora: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv preprint arXiv:2308. /2043, 2023.
[83] Jialun Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned gradient descent for over- parameterized nonconvex matrix factorization. Advances in Neural Information Processing Systems, 14:5985-5996, 2021.
[84] Jialun Zhang, Richard Y Zhang, and Hong-Ming Chiu. Fast and accurate estimation of low- rank matrices from noisy measurements via preconditioned non-convex gradient descent. In International Conference on Artificial Intelligence and Statistics, pages 3772-3780. PMLR, 2024.
[85] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memooyy-eficient low-rank adaptation for large language models fine-tuning. arXiv preprint arXiv:2308.03303, 2023.
[86] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning, arXiv preprint arXiv:2303.10512 2023.
16

---

[87] Yuanhe Zhang, Fanghui Liu, and Yudong Chen. One-step full gradient suffices for low-rank fine-tuning, provably and efficiently. arXiv preprint arXiv:2502.0122 2025.
[88] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient IIm training by gradient low-rank projection. arXiv preprint arXiv:2403.03507, 2024.
[89] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595-446633, 2023.
[90] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xucling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Openccdeiinteppreter:: Integrating code generation with execution and refinement. arXiv preprint arXiv: 14658, 2024.
[91] Bojia Zi, Xianbiao Qi, Lingzhi Wang, Jianan Wang. Kam-Fai Wong, and Lei Zhang. Delta- lora: Fine-tuning high-rank parameters with the delta of low-rank matrices. arXiv preprint arXiv:2309.02411,222 2023.
17

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We have summarized our contribution at the end of the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
be expected to generalize other settings.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
we discuss how to reduce the assumption of full-rank with weight decay, which makes our theory applicable in practice. As our algorithm involves the matrix inverse, we discuss the computational cost the experimental study (see Section
Justification: In Section 3.1 and 3.2,
5.1.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and prool's
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer: [Yes]
Justification:
Due to the page limitation of the main paper, we put the proof Section 3
and 4 into Appendix B and D respectively. For clarity, summarize the key steps for
establishing the proof at the begining of Appendix and D
Guidelines:
The answer NA means that the paper does not include theoretical results, All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
we discuss the experimental setup, like hyperparameter choices and datasets, and provide the link to our code repository.
Justification: In Section 5
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We have already provided the link code repository in Section S
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. ec/
gublic/guidess heaia//gguidee//eeeubbmiiss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips cc/ wttpppullllllppieeee http://oooeooooosssooooooooeeeee
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We present the experimental setup for training and evaluation in the main text. Additional details about the datasets used for each task are provided Appendix E.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yes]
Justification: We report the evaluation score with mean and standard error for each experi- ment. The randomness of our experiment is discussed in Appendix E.T
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Experiment Setup in Section 5.] and 5.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps://eeurjgs htcppp.ccpppuue httpp//ccccttticcccceeeee.e..
Answer:[Yor Yes]
Justification: We keep the code to preserve anonymity.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our theoretical analysis would provide new insight for the broader context of low-rank matrix estimation (see Section 3.T). And for the practical impact, our work would be applied another parameter-efficent setting easily, as we discuss in Appendix C.1
Guidelines
The answer NA means that there is no societal impact of the work performed.
21

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We have gotten the permission to use the pretrained models, like Llama-3-8B and Llama-3.1 1-BB via Hugging face and have cited their work properly.
Guidelines:
The answer NA means that the paper does
existing assets. The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
22

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: We release new code implementing the proposed method. The code is docu- mented and includes instructions for reproducibility. An anonymized version is provided for review.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve any crowdsoureing experiments or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve research with human subjects or crowdsourcing, and therefore no IRB approval was required.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
23

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https://neurl)) http:p//eeerrrppccc//ooooeeeeeer :ctee///eeeeeeeee/////////////// for what should or should not be described.
24

---

A Related Work
Low-rank adaptation(LoRA) 25 has been the subject of extensive research in foundation
models([5]] S 1 33 55 611, with numerous variations and improvements ([34 28 32 78 12
27 9. One line research focuses on dynamically adjusting the LoRA rank during training. This
includes DyLoR (62). IncreLoRA A [82], and AdaLoRA [86). Another line of work involves enhancing
LoRA performance through the addition of extra scaling matrices, which include DoRA [4]] and DeepLoRA (78). These directions orthogonal to work. Regarding the optimization of LoRA, we find that the following topics are close to our work.
Stable Feature Learning Under the infinite-width NN setting [19 56)), LoRA+((1)) finds that the standard LoRA inefficient and they propose to use different learning rates for and B. To provide careful choice of hyperparameters for efficient use of LORA, a line of work analyzes LoRA under efficient learning ( (20) 8T). Noticeably, [8]] introduces preconditioners under a Riemannian metric (45)) and updates LoRA by using scaled gradients of A and B simultaneously. While their method aims to improve stability and efficiency, is important to note that their goal is not to approximate the full gradient. This approach does not yield an optimal approximation to the full gradient update. Moreover, (79 proposes an adaptive matrix preconditioning method preserving transformation-invaaiin a sufficient condition for stable feature learning.
Approximation full-tuning or full gradient To fill the gap between LoRA and full fine-tuning, there are two lines of work with different motivations. The first class of work focuses on the initialization, like [66]. It proposes to make the initialization of LoRA align with the full-finetuning directly. However, after the first step, how difference between LoRA and full-tuning is unknown. The second line of work focuses on optimizing LoRA properly over the optimization trajectory( 50 87) Noticeably, [66] proposes to optimize the gradients of A and B together to approximate the full gradient. But the optimal approximation is hard to find under practical conditions and aligning momentum towards the full gradient requires storing full-size matrix (k x d) in their algorithm. These challenges also exist in later work ((50).
Gradient Projection in LoRA Motivated by the view that LoRA updates can be viewed as performing random projection from the full gradient, F-LoRA( [8] achieves high-rank updates by resampling the projection matrices. There are also some approaches that propose training networks with low-rank factorized weights from scratch ([44 [2 ). Random projection also applied in Ga-LoRA(88) and
following work [39 () but they need to access the full model and can't 't store the low-rank adapter in the end. On the contrary, without full-parameter learning, we use gradient projection to keep the gradient best preserved in the low-rank spaces.
Alternating Update To the best of our knowledge, we haven't 't found the existing work of updat- ing LoRA alternately in the centralized setting, but in the decentralized setting, i.e., Federated Learning, we notice 77] used the alternating strategy to address the challenge of inaccurate model aggregation( 68
3 58) with computational and communication efficiency. Besides, in the centralized setting, (85] proposes to freeze A and update B, which would be regarded as a specific case of our work to do alternating minimization.
Scaled Gradient Descent Our proposed methods are also closely related scaled gradient de- ScenteScaled GD) in traditional low-rank matrix estimation under over-parameterizzation and ill- conditioning (59) 60) 29 46)). Notably, (59] shows that the scaled GD) would keep the convergence independent of the condition number. Different variants of scaled GD have been proposed and studied
in work ( ((73 g3 10 84) ). For the alternating scaled GD, [30] finds that it would enable faster con- vergence with larger step sizes compared with scaled GD. And 42] provably shows that alternating
scaled GD would achieve a linear convergence rate, starting from arbitrary random initialization.
B The Proof and Details in Section 3
In this section, we provide the formal proofs and detailed discussions supporting the results presented in Section 3. Specifically, Appendix B. presents the proof of Theorem removes the full-rank assumption in Corollary [ via weight decay. Appendix B.2 contains the proof of Theorem 2 and demonstrates how the full-rank assumption can similarly be relaxed using weight decay in Corollary 2
25

---

B.1
The Proof in Section 3.1
B.1.1 The Proof of Theorem
Proof. The first-order condition of Problem yields
sB AA Vw, 0,
(14)
where is a positive scaling factor. Then we can reorganize it and obtain
sB B, B.V B Ww,L.
(15)
As we assume the matrix B is full rank, it yields
VA - ( B.) (Vw, L.)
(16)
Furthermore, recalling the definition of the gradient of standard LoRA in we obtain
(BB (Vw, z²
B.) B A.
1
(17)
Similarly, we can obtain the closed-form solution of V W,,4 L in
B.1.2 Corollary I and Its Proof
Corollary 1. For B e Rexr and A Brxd, solling problems in (18)
min VA
|BB A, L.)
Vw, 1.
2 | 1
(18)
min
B, B. VW L + || V B. 113.
yields the unique closed-form solution
V A. L 1 [B3]" B. + Allxx) B, (Vw.L) = (B} B₁ + Alxxr) -1 A.A.L
(19)
VB, L 1 (V L)
+ (Ixxr) L A AIxx))
where lexe is the identity matrix and 0.
Proof. For the first line problem in (8). the first-order condition yields $B8 (s A...A.L - Ww, L.) + As V A. = 0,
(20)
where a positive scaling factor. Then we can reorganize it and obtain s(B B, AI)
B² Ww.L.
(21)
To keep (BF B, + XI) invertible, we only require that À isn't 't too small and yields
L B B, AI) (V,
(22)
Furthermore, recalling the definition of the gradient of standard LoRA in we obtain
V T. 1 (B3 B, - B w. L.)
1 (B) B. AT) - L.
(23)
$2
Similarly, we can obtain the closed-form solution of VB, L in 19 V VB, holds with the fact that W ++4 Wo B.A
Noticeably, the result
(Vw., L)A[1
In Corollary the hyperparameter À can be small enough (Ie in our numerical studies) and we don't tune the hyperparameter overall. For more discussion about the selection of X in the over- parameterized setting for low-rank matrix estimation, please refer to APGD 42)), ScaledGD (73)) and (8))
26

---

B.2
Proof of Section 3.2
B.2.1 Proof Theorem
Proof. The proof is similar to Theorem thus we omit it here.
B.2.2 Corollary and Its Proof
Corollary 2. If we assume Mi has aligned with the full gradient in the t-th iteration, by minimizing the following problem
A + |M
min M
M, A.
M Ar.
(24)
2
we can find the unique solution M 1 + AI) which is the best approxi- mation current full gradient.
Proof.
The proof is similar to Corollary T thus we omit it here.
C Appendix for Algorithm 1
C.1 The Implementing Details for Algorithm
AltLoRA, novel PEFT method, be seamlessly integrated into popular libraries such as Hugging Face Transformers [69]. The key engineering modifications are as follows:
To enable alternating optimization of LoRA parameters, we ex- tend the existing Transformer architecture by introducing a control argument within the training_ster function. This argument identifies the current update phase and selec- tively disables gradient computation for parameters named "1ora_A" or "1ora_B" thereby facilitating an efficient alternating update mechanism.
Alternating Updates:
Custom Optimizer Integration: Similar to prior LoRA variants that incorporate new optimizers [8 66). AltLoRA can be easily adapted by implementing new optimizer class. This allows flexible modification the optimization dynamics tailored alternating update strategy. It would provide broader impact to incorporate with other parameter- efficient structures, like MoE or RLHE, when using low-rank adaptation.
C.2 AItLoRA+
With the goal of approximating the full gradient under the memory constraint of standard LoRA, propose AltLoRA in Algorithm to properly optimize the training paradigm of LoRA. Furthermore, the ultimate goal methods, like LoRA((5)), and the full model fine-tuning. Therefore, witnessing the success of incorporating the second momentum for accelerating and stabilizing the optimizing paradigm (25), we propose a variant of AltLoRA, called AttLoRA+ (see Algorithm to help accelerate our optimizer with second momentum. The increasing memory cost for storing second momentum O(kr + dr), AltLoRA+ won't t require storing the full size matrix (kkd) like LoRA-Pro [66].
fill the gap of performance between the existing parameter-effcient fine-tuning
27

---

Algorithm 2: AltLoRA+: AltLoRA with Second Order Momentum
Input: Momentum states MR. Mp. V and ViP scaling factor a learning rate f) momentum coefficient 01 and 82, total number of steps T, weight decay coefficient ? and constant c
Output: Final matrices Ar and Br
for t .,.... T do
i mod 2 = 0 then
Update A:
Only backpropagate w.r.t. A, and obtain VA,L
V P.L (B, VV)
Mí = B.) B.-IM
MÁA ByMM +1--3)VA,,
V +
+11-))(VA, V.
Ac+1 + A. n (MA @ (VV. +7)+
else
Update B::
Only backpropagate w.r.t. Br and obtain B,
V n. L 1.2 V B. (A441AA44)
M =
(Act1AAA
MP -BI +1--3:) +1--3:VV,,
VB 32V. (1-32)(V B, L,
Bu+1 + Bi B (M, @ (VV, + +c)
D Proof and Details of Section 4
In this section, we will start to analyze the training paradigm of AltLoRA in Algorithm and AltLoRA+ + in Algorithm In Appendix D.I we first give the formal definition of stable feature learning in Definition 2 Then we will analyze our methods without momentum on toy model in Appendix (tpendiix.....]] Furthermore, Appendix D.I.2 we provably show that Altl or AltLoRA+ with arbitrary RA ranks achieves stable feature learning in the infinite dimension NN setting. Then, we provably show that AltLoRA would achieve transformation invariance. Finally, in Appendix D.3 within an peeepparamtteiieed two-layer ReLU NN tuning problem, prove that AltLoRA or AltLoRA+ without momentum would converge linearly without the requirement of spectral initialization.
in Appendix D.2
D.I Appendix Section 4.1
First, let's recall the definition of stable feature learning below.
Definition 2 (Stable Feature Learning (Definition A.1.(81)) Consider any general LoRA layer BAx with B E R and A E Igrxa being LoRA parumeters. Denote A. = W. W B,A, B1-1A,112 for fine-tuning step We say that LoRA mdoel achieves Stable Feature Learning when x, Ax, BAx E O(1) for alll LoRA layers and D E (1) for all fine-tuning step t.
D.1.1 Analysis on A Toy Model
Following LoRA+ 21), let's consider the simple linear model first
f(x) = (W ba )æ,
(25)
where W IR/ Ixn the pretrained model weight and b R, a Rd are trainable LoRA parameters. Consider the quadratic loss function C(a, b) = (f(x) - y)2/2 with some scalar label y. We adopt Gaussian initialization
NN, o LL,)) N (0, o2) Conventionally, ba" is initialized at zero
for LoRA, and we thus consider setting of 0, of O(1)
28

---

For simplicity, assume AltLoRA or A without momentum updates with learning rate 7) (n)) for some E R. Since the training process involves only elementary algebraic operations. the quantities there should be of powers of n. If we treat updates A and B each time as a single iteration, in iteration the feature update is given by
Af+1 : = f(-1(z) - f((x))
= (niak - -n/b( a. L)" - (V ()a x bla
(26)
-n(f(()) ,)|1=111 - 1 (F+2 (x) - y)llae+11
where f14((2) = (W + b(a We denote 87 (f(((())) y>|1= 87 () (x) y). To achieve stable feature learning, it requires 67,63 E O(1) and fur- ther f((x)) ((1) 1L > 0. Thus, we have the below modified linear constraints.
c c+1=0
(for of = (()), (for 0f = (()). (for f (2) (()),
c 27107 71 a+t11+11 = [[bb+1] + + =
(27)
where, for the sake of notational clarity, we introduce new notation 7 such that (n captures the polynomial behavior for any t.
Solving the equations in (7 we can derive 1. With O(n = we get 7|6,] = > [b9] 0 and yaaf a]] Tnybo "1|==| || Recursively, we can derive b,, d2 of of E O(1) for all t. Therefore, we obtain f: E (1) and Af E O(1). The above toy model illustrates that our proposed method achieve stable learning with learning rates for A and B of the same order of magnitude.
D.1.2 Proof for Theorem 3
In this part, extend the analysis above to general neural architecture with LoRA layers. We show that the conclusion from the analysis on the linear model hold for general neural architecture.
Assumption (Assumption in (21). We assume that the gradient processing step by AltLora in Algorithm 7 (or AliLoRA+ in Algorithm 2) satisfies g'A = ((n) for all where g'A is the processed gradient of A by AlILoRA (or AliLoRA+ +) in I-th update.
Lemma (Lemma A.3. in (1).. For any matrix A E BR where m being powers of n, such that A A is invertible and 7 Ai ffr all (i, have
> I(A ((ATA)) A) -
with being any column of A.
Now, we state the formal version of our Theorem 2
Theorem 5. Let of and of denote the processed gradient of A and B, respectively, in Algorithm 1 or Algorithm 2 Assume Assumption I holds for the gradient processing of AliLoRA or AliLoRA And ge and ot E O(1) after the gradient processed. Further assume Ax has dimension of (()). Then the following results hold:
(1) AltLoRA (AltLoRA+) achieves stable feature learning with 17 = O(1.
(2) If we consider AltLoRA AltLoRA+ without momentum, the update yields
W441 W, - nProjo.,, (w,,))n(Vw... -V.. L)ooo(((am))
(28)
where 1pProjeBB.) www, L)..((VVVVV VVW L)PTO((()))) E O(1.. However, when doing joint update, the update will introduce additional across term op (ww, L) (A(AA))-((B)) (B(( - (Vw, L) e ((1) The across term is indeed the second order term w.rl n, but is same magnitude as mProja(B.)
(BoiiiBB)/WW.. (Vw.L) L) and n(V (V Projr(())) in infnite-width NN setting.
29

---

Proof. (Part 1) First, we will prove AltLoRA (AltLoRA+) can achieve stable feature learning. The technical lemmas and assumptions used for proof are also well-adapted in [21 81.
We will alternately update A first then update B. If we treat update A frist then update B as a single iteration, it could yield the update of the full model W
A
B,A By- B1-1A
B₁ArI - BIT + 1,T - Bi-1A--1
(29)
(B, Br-1)AxT + Bi-1( (A, A(-1)
- (A,A,) 1- BB,-1 1 B 9'
Then will denote these two parts of the update in the R.H.S of (29 as
8: = nB (B, (B, B:-1) 1-1
(30)
8/ = no's 1-1 (A,A,)- AAT. -
Following Assumption are equivalent to
we know 9
9 x E (n)). Thus the conditions 8/. 82. 11- E O(x)
{n] 1 (B B.-)
(31)
jn/ + [A,,, + [A,a]
For gradient update, we have
A& = A - 1/ (B,
Bz -2 1-1
9A
(32)
B₁ = Bt-1- - 1191 1-1 ( (ArA,)
thus we have
y[Bi] max +,
[[A[a]= max ( 7 71 -1) +1]
Note A1 = Ap. the recursive argument of 6 and o7 e O(1) is the same as [81]. Therefore, we find that AltLoRA AltLoRA+ achieves stable feature learning with O(1). We conclude that our algorithm would achieve stable feature learning with the same order of y in contrast to the standard LoRA ((1)
(Part 2) When removing the momentum in our methods, under Assumption I would achieve stable feature learning as Part has proved. Then the update of the full model W is
W41 W, - P (V L) -n(V nVV
L) Proi((A))))
(33)
L),n where nProjeB, (V L), n( (V L)r__LL)Praie (Ao))))) E O().
However, when doing a joint update with scaled gradient descent (81)), the update the full model W is
W+1 = W. nProjec w.,, L)PojCA)))
(34)
+ +1 (Vw, (AAA (A.A7) (B. B.) - B (Vw. L)
where the additional cross term 172 (V L) A) (B BB (Vw, is of order O(1). While this term is second-order with respect to v, shares the same magnitude as first-order terms //Projj(B.) (Vw; I.) and 7)( ww: P.ojroj((A) under the infinite-width neural network setting. straightforward explanation that the embedding dimension contributes quadratically to the cross term's 's effect, matching the overall scale of the first two terms.
30

---

D.2 Proof of Section 4.2
First, let's restate Lemma again and prove it. Lemma 3. If any two pairs of LoRA (actors(AA,,a Bi), (A2, B2) satisfying
Wo - B1A1 Wo B2
(35)
then
Proje(B)) Proje(Ba)
(36)
Proje(A)) (rojr(As)
where Projet-) and Proje()) is defined in ?
Proof. we know the column spaces of B1 and B2 are equivalent, as both them span the column space of W Wo. Thus, the projection matrices to the column spaces of B₁ and B2 are the same, i.e. Proje(B,) = Proje(B.)) where Proje()) defined in (9.. Similarly, row spaces of A and A2 are equivalent. And the projection matrices to the column spaces of A₁ AND A2 are the same, ie., (ro)((A)) = Proje(A2).
Lemma D tells that if two pairs of low-rank adaptation would get the same full model update, the projection matrix would preserve invariant over the pairs low-rank adaptation. Next, we will restate Theorem here and start prove the theorem.
every term is consistent across all equivalent LoRA pairs. Consequently, Algorithm 7 is transformation-mnwee -iivariannt..
Theorem 6. In Algorithm 7
Proof. Now we will use an inductive argument to prove it. Let's denote (B1,t, A1.), (2,,), as two pairs LoRA adaptation in the t-th interaction statisfying
Wo + B1, = Wo + B2,
(37)
For the first pair BB,,, A1,)) we denote MA and MA as the momentum used for AI, in Algorithm Let's assume, for the (t 1)-th iteration, we have the equivalent decomposition
B1.--Au1111 B1,1-1,
(38)
Besides, we assume it transformation invariance (1 - 1) iteration, then
B1,1-2M,,-22
(39)
M1,-2A11-22 = 1aa-----M,.2- 111-2A111-22
(40)
which implics that the historical information invariant over the pairs of (B) and (B2, A2).
Then for the f-th iteration, we need to prove
- = B2,t-1M5..
(41)
M M M1,-11221-11
(42)
holds well, and the update is transformation-innaaiint B1, B2,
First, we will focus on the update of A and prove B1,,- = B2.-1 Recalling the definition of MA, is the cumulative gradient the time in Algorithm [] yields
BI.-IMA.
= B1,--1 [1(B
+ (1-13))
B1. (1 31) B (B. ALLIL
-. (1- 31) Proje(BL.-.) VWi
(43)
31

---

where the last line uses the results in ① and WL.-1 == Wo + B1,--1A1.-1 Next, under the assumption for induction 411 and Lemma] [ it yields
B1,--1MA B1 Proje(BB - (1- Projo(Br....) W
B)Projc(Ba, B2, M2, (1- (1- 1
Proje(Ba,.-) V (44)
B2,--1 12,1-1221-1
After updating A, we can find the update of the full model as
B1,c-1A1 = B1,,-1(A,,--1 - m,M,, 1,
1,- = -B2,--1A21-1- - -
-B1,t-1A11--1 1, M
(45)
= =B2,1-122.1
where the second-to-last line uses the results (38) in (t 1)-th iteration and the results in 44 Again, reapplying Lemma 1 we can find that Projo(AA, Proje(A2.5)
Up to now, we have shown that the update of A is transformation-inaaiiae and B1,t-1M
1,t-1- M
With similar argument, can prove I A MB p - A1.-1 and
A1, = A2, Therefore, with the inductive argument, we prove the update of Algorithm I is transformationiinvariant
In contrast to the prior work 779.. our analysis centers on Lemma to establish the proof of Theorem Leveraging the alternating update strategy Algorithm ! we analyze the contributions of and B to the full model update separately, allowing us to rigorously demonstrate transformation invariance. In comparison, [9] adopts a joint update of A and B, which introduces cross term A that is ignored in their analysis, resulting in an inexact form of transformation invariance. Our alternating approach provides principled direction toward achieving exact transformation invariance.
Discussion With our newly designed momentum mechanism, the first-order momentum terms remain consistent across all equivalent LoRA pairs, thereby ensuring that AltLoRA robust to transformation invariance. In contrast, Alti ++ does not preserve this invariance. Motivated by this observation, further attempt to design second-order momentum mechanism that aligns optimally within the low-rank space under memory constraints. Although the second-order momentum terms are individually consistent across equivalent LoRA pairs, their combination with the first-order momentum leads to inconsistencies, ultimately breaking transformation invariance. To address this issue, employing unscaled gradients and momentum, demonstrated by LoRA-Rite [79], could be viable solution. However, as this approach diverges from our primary focus, we leave it for future work.
D.3
Convergence Analysis
D.3.1
Set Up
Following the previous work (81)), we provide a convergence analysis of the proposed algorithm within the over over-varanen parameterized two-layer ReLU NN tuning problem. For data matrix X E gaxd and and any arbitrary vector u, we consider set of diagonal matrices (diag([Xx 2 0)) which take value 1 or 0 along the diagonals that indicate the set of possible arrangement activation patterns for the ReLU activation. Let the distinct elements of this set be denoted as DI...., Dr (see [8I] for more details). The constant corresponds the total number of partitions of IRd by hyperplanes passing through the origin that are also perpendicular to the rows of X [49]. Intuitively. P can be regarded the number of possible ReLU activation patterns associated with X. [49] explains that two-layer ReLU problem shares the same optimal objective with the convex problem
Rd.,
P Σ
min PE
W -
(46)
32

---

As we focus on fine-tuning given a pretrained model with model weights (Wi)en we can do low-rank adaptation and rewrite the problem (46) as
min A, 1..........
E D + B,A,) - Y
(47)
where X E Raxd A, E Rexo B, Raxr and Y E TRaxc We consider the response model Y= Ei D + B,A)). We define X == -. A, are fixed and unknown matrices. Let's denote o-(-) the r-th largest singular value. First let's introduce the definition of Restricted Isometry Property (RIP).
Definition 3. (Restricted Isometry Property, 1531) The matric C' E &gnxd is said to satisfy Restricted Isometry Property(RIP) with parameters (r, 5,) there exists constants 0 S 6, S 1, for any matrices M E Raxe with rank the below holds
(1- & |CMMI S (1 1.
(48)
RIP is widely used condition in the filed of compressed sensing 42 15 53 73 that the operator C' approximately preserves distances between low-rank matrices. In the absence of noise, we can establish direct relationship between the loss function and the recovery error. If denote C, == D,X, Problem (47) is equivalent to the problem below up to a change of labels
which states
min A,,
Lc A)
Σ
C,(BAA, x.)
(49)
where B = (Bi,..,, ]p] and A= (Ai,...
Ap}.
Notation Inspired by the previous work 42 corresponding dual norms for a matrix kxr
83
84
we introduce two local norms and their
PA: = A((A))
[W]]p = [WP
||]]p 43
= IIWP |l.
A
A
(50)
Pe: = (B)) B,,
[W]]p [|WP |F,
[[]|e 4: = | ||r.
Here, we assume A; and Bj are of full rank for any i. If they aren't of full rank, we can replace them with the Moore-Penrose ivverse[4)). Now we are ready to establish the convergence analysis.
D.3.2 Useful Lemma
For the f-th iteration, let's denote B, = ....... ,B"' and A. {A),- ...,,............... A If we apply AltLoRA AltLoRA+ without momentum for Problem (49), for any E [P], the alternating update rule as we proposed can be written as
A/41 A - (BB
(51)
B B v/V Bj Lo (B, A++1)((A(4))) 11
First, we will list assumptions used in analysis.
Assumption 2. Suppose that C, D; obeys the r-RIP with a constant o for each i.
145, Assumption 3. Suppose that ||C C, ||2 XXT D' D,X S P(P-1)
Assumption 2 and 3 also adopt in [81] analyze their optimizer for LoRA. For matrix X with i.i.d Gaussian entries N (0, 1/d Dollo). D.X satisfies RIP for a constant ò, when [| |lo is on the order of r(d+ + /) (dō²). Note || DI D2Xl|2 =|X X|12 for all (i.j)'s. Thus bounding || | DJ D,X
amounts to bounding the largest singular value of the empirical covariance.
Lemma 4. For given i [P], the gradient of Problem 449 are
V L(,,(A
EUB X.)
(52)
P Σ -
V (B,
33

---

Proof. For any given i and i, yields
0 aA; V (B) =
Σ CB -x)
EEB XX). (53)
Similarly, we can derive the V L(B,
(.B,A A) as shown 522
Lemma S. Suppose Assumption 2 and 3 holds, then we have
Lo(Br, A++1) & Lo (B) max
v A; Lo Br, Ar)
(54)
Le A411) & Le (B,, A(t1) a max V DILLo Br, Act1)
where 01 = P(n - +1
Proof. Using the update rule in 5T we have
P
1 CCBAA XX.)
Lo Art1)
2
1
ΣG,
c, "VA - xx.)
2
1
Σ CB -X+)
2
(55)
P
7
Σ BB AB As)
P Σ C.BBBA| X.). Σ
T2
For Ti, recalling Lemma 4 then we have
2
Ti B A.)
Σ
17
Σ
A.)
LV Ai))
i71
(1
+ 6x)
(56)
P
max
V AagLo Al
2
n/2
+
max || Cill2P(P 1) max | Le Bi. AD)
2
($)
(b) 7/
+ 8,
+
} P Pmxx / 4l La (B, A "g
where (a) uses Cauchy Inequality, Assumption 2 and the fact that ((B)) Bj) - = (b) uses the assumption that maxişj ||C] C5Il2 < P(P-1)
(1+
34

---

For T2, using Lemma 4 again, we have
P
T2 -
Σ C xx),
Icc(B: [G At Ac)
Σ
Trc( [Bc A1)
(57)
Σ V Lo Bt. A0)1
- 1) max V A Toc( Bi: Ar)llp
To sum up, yields
r/² + 6, + p)
Le LB, A Lo(B,,
P Pmax Va Ai)
(58)
2
Similarly, we can induce
(B(1, A+t1) S (B. A+t1) (1] -1- 17 17(1+0 8r + 7))
/
P Pmax
V By Lo Br. Act1)
2
(59)
Lemma 6. Suppose Assumption 2 holds, then, for any i [P]. we have
| AV Le Be A mt 2 2(1 L.) B,, AA)
(60)
L. A 1 I
2 2(1 Lo (B., Ac+1).
Af+1
Proof. See Lemma 6 in 42 for the detailed proof.
Theorem 7. Assume for any i E ig) the matrix C₁ = D,X satisfies the rank r-RIP with constant o (Assumption 2) and 0 < 1 1 then AliLoRA AlILoRA+ without momentum solves the over-parameterized problem leads to
(o AH1) S (B. A.)
(61)
and
2
2
1 + 8,
Σ B, -X.
S
ne) 2t
Σ
BiAb -X,
(62)
8,
F
F
where ne = 2P(1 - 5,) (n) (1+++
Proof.
172(1 6, p))
Lo411, Act1) S Lo(B,, Art1) Aot1
P Pmax Vy A
172 + 6, + 111
$ LBC(
2P (P(118r)Lad 8,)) Act1)
< <1 11-2P11- -d,) < (1 3 L. Lo(B,, Ai),
172 8r + +1 ))
La Art1)
2
(63)
35

---

2P(1 6r) ())
7'448, R
where we apply Lemma and and
Moreover, under
2
Assumption
2
have
P
2
+ 8r (1 - 1/c) n=)21
p
2
Σ
A X.
Σ
BB X
(64)
1 8
E
Appendix for Expirments
E.1
Details and Results for Supervised Fine-tuning
For the experimental setup, follow the configuration used in LoRA- [66] and summarize the key description here. As the experiments involve randomness from initialization and optimization, all results are averaged over three different random seeds.
Dialogue Generation Task We fine-tune large language models on a 52k subset of the WizardLMM dataset [72) and evaluate it using the MT-Bench dataset (89). GPT-4o is used to asses the quality of the model's response and we report the first-turn score as the metric.
Math Task We fine-tuning large language models on 100k sample from the MetaMathQA dataset 80]. The model is then evaluated on the GSM8K test set [1]] and we report the accuracy as the metric.
Coding Task We tine-tuning large language models on a 100k subset of the CodeFeedBack dataset 900
and test it on the HumanEval dataset [6], reporting the PASS@ 1 metric.
For the choice of learning rate, perform grid search for LoRA, its variants, and AltLoRA+ le-5, 4e-5, le-4. Since AltLoRA does not use second-moment estimates, we conduct an extended grid search le-2, le-3, le-4, 4e-5, le-5. We observe that AltLoRA performs better with higher learning rates, and therefore report results using le-2, le-3, le-4 in the main evaluation. We set the iteration number to be and the max step is 3000 for each experiment.
E.2
Additional Results
In Table 5 we compare our method with existing approaches across the three tasks described on Llama-3-8B model. Our method further bridges the performance gap between LoRA and full fine-tuning.
Table 5: Comparison of different LoRA variants on MT-Bench, GSM8K, and Human Eval benchmarks (accuracy in %) on Llama-3-8B-Basse
Method
MT-Bench
GSM8K
HumanEval
PreTrain
5.63
49.96-t 0.38
34.76-t0.37
LoRA
6.20
62.11+:0.13
37.71+0.12
AltLoRA
6.05
64.39.10.23
40.81
A
6.34
67.38+00.1
43.81+0.31
In T we report the training loss of AltLoRA and its baselines on GSM8K using Llama-3-8B- Base model. Our method not only converges to a lower training loss than full fine-tuning, but also does more rapidly than most baselines.
Base Llama-3-8B base and Llama-3.1 base model, we also consider the instruct tuning model- Hlama--8B-Instrus model. We fine-tune it on the MetaMathQA dataset and evaluate it on the GSM8K test set, reporting accuracy as the primary metric. In Table 7 our results demonstrate that even when using stronger pretrained model, low-rank adaptation can still yield performance gains. More importantly, both AltLoRA and oltLoRA+ consistently outperform the standard LoRA baseline. We will add the full comparison in the revised revision.
36

---

Table 6: Training loss on GSM8K (Llama-3.1-80 across training steps (lower is better).
Method
step 500
step 1000
step 1500
step 2000
step 2500
step 3000
Full Fine-tuning
0.51
0.28
0.24
0.18
0.18
0.18
LoRA
0.57
0.30
0.26
0.23
0.22
0.22
LoRA-Pro
0.51
0.31
0.24
0.22
0.21
0.21
LoRA-GA
0.45
0.24
0.20
0.18
0.18
0.18
LoRA-Rite
0.55
0.31
0.24
0.22
0.21
0.21
AltLoRA
0.48
0.25
0.21
0.19
0.18
0.18
AliLoRA+
0.45
0.23
0.19
0.18
0.17
0.17
Table 7: GSM8K accuracy (%)
and its fine-tuned variants.
GSM8K Llama3-8B ---istttttttttttt
Method
61.24
Full FT
83.26
+ LORA
81.12
AltLoRA
82.60
+ AlILoRA+
83.88
E.3 Additional Ablation Study
We conduct additional ablation studies to further demonstrate the practical effectiveness of proposed methods. In Appendix [E.31]] we evaluate the performance of our methods under varying hyperparameter settings the LLaMA 3-8B model. Furthermore, in Appendix [E.32] beyond the learning rate, scaling factor a, and rank examined in Table we perform comprchensive ablation studies for both AltLoRA and AltLoRA+ on the LLaMA 3. 1-8 model.
E.3.1 Additional Ablation Study for Llama-3-8B Model
We further conduct ablation studies on the LLaMA 3-8B model to evaluate the robustness of our method under varying hyperparameter settings. As shown in Figure we compare the performance of LoRA, AltLoRA, and AltLoRA+ on the GSM8K dataset across different learning rates and scaling factors a E {8, 16, 32). + consistently outperforms
baselines across all configurations, demonstrating both higher accuracy and stronger robustness to hyperparameter variation. We also have that methods have better performance using a 16.
LoRA
AttLRA
AttLoRA-
Goining Bat
62.11
64.39
66.28
64.12
65.0
60.88
62.77
63.33
12.5
64755
64.38
02.31
16
32
32
50.0
32
Alhh
Aphh
Alpha
Figure 2: Evaluation Accuracy of LoRA, AltLoRA and AltLoRA+ for various learning rater and scaling factor combination on the GSM9K using Llama-3-8B.
E.3.2 Additional Ablation Study for Llama 3.1-8B Model
Training Epoch rule out under-training as a potential factor for empirical performance, we increase the training budget to 5 epochs under the same setup as Table 2 Results are reported in Table (i) More epochs generally improve all methods; (ii) old and AltLoRA+ retain and strengthen their advantage-Allee A reaches 766% GSM8K and approaches Full FT on
37

---

Table 8: Comparison of different LoRA variants on MT-Bench, GSM8K, and HumanEval benchmarks on Llama-3, 1.taa-3.---- 3.1--8--B-B.. 88-Base (5 training epoches). Bold indicates the best result, underline represents the second-best one.
Method
MT-Bench
GSM8K
HumanEval
Full FT
6.24+10.03
75.31
50.42:11.20
LoRA
6.19220.2
66.7811.14
41.54=1.01
PiSSA
6.11
67.3910.61
42.12:11.45
TsLoRA
6.18+0.02
6..98+11221
44.31=1.31
LoRA+
6.21+0.02
74.00+0.61
44.32+1.33
DoRA
6.23+0003
67.12+00.4
45.12=1.32
AdaLoRA
6.05+0.00
68.29
42.58
LoRA-GA
6.24+0.05
73.44+1.110
4..56+11.48
LoRA-Pro
6.22+-0.2
73.24++0.79
43.21+1.500
LoRA-Rite
6.20++003
73.48+00.65
1..2
AltLoRA
6.26
74.52:10.37
1.12
A
6.18+
76.01-10.34
50.01
HumanEval; (iii) compared with standard LoRA and its variants (LoRA-Pro, LoRA-Rite, LoRA-GA), our methods benefit more consistently as the training budget grows, indicating that the gains are not a short-iteration artifact but persist under stronger training.
Ablation study on the updating strategy Table 9 contrast to joint update with scaled gradient descent [8]] AltLoRA can optimally approximates the full gradient with alternating update and obtain better performance in evaluation. Interestingly, we find that the alternating update scheme-wher matrix B is updated before A c-coosistently yields better performance. One possible explanation that, under the standard initialization where B set to zero, updating A first does not lead to meaningful descent.
Table 9: Performance comparison of LoRA, AltLoRA and AltLoRA+ on the GSM8K and Llama 3.1 8B with different updating strategies.
GSM8K
LoRA
AltLoRA
AltLoRA+
Alternating (A first)
66.11
74.49
76.91
Alternating (B first)
67.66
76.31
76.97
Joint Update
66.43
74.21
76.56
Ablation study on the number of LoRAs As low-rank adaptation comes to be a popular parameter- efficient technique for fine-tuning, it's well applied more complicated scenarios (43 57 23
70).. Notably, a very significant application is to improve the structure of the mixture of experts
with parameter efficiency 70 37 ). handling multiple tasks simultaneously [8 26 and addressing
catastrophic forgetting ([14). Following the work ((70). we explore the performance as the number of LoRAs varies and utilize the gating balancing loss. Additionally, we compare AltLoRA and standard LoRA on the GSM8K dataset using the Llama 3. 1-88 model(see Table 10). In our experiments, the number of LoRA experts is set to {1, 4, 8}. and the entropy regularization weight 0.0001. We observe that increasing the number of LoRA experts enhances the capacity of the language model, leading to improved performance.
Table 10: Comparison of the mixture of experts model, with different expert numbers on GSM8K and Llama 3.1--B-Basc
Expert Num
LoRA AdaLoRA
LoRA+
AltLoRA
AltLoRA+
1
66.11
72.63
72.33
74.49
76,91
4
67.43
71.71
71.27
75,01
77.33
8
67.89
70.34
71.44
75.33
76.94
38

---

Ablation Study on Initialization. To further validate the robustness of our method with respect initialization, as discussed in Section 4.3 we conduct an ablation study using different initializa- tion strategies. "Gaussian" refers to the standard random initialization used in the original LoRA framework [25] "Kaiming" denotes the widely adopted Kaiming initialization, which is designed to maintain variance stability across layers. "Spectral" represents initialization strategy based on spectral decomposition, where we perform singular value decomposition (SVD) on the pretrained weight matrix and construct the low-rank components using the top-r singular vectors, like the initialization proposed in [88] In Table we can see that with different initialization strategies, our method would achieve superior performance over the standard LoRA. Without spectral initialization, using Kaiming initialization for A and setting B to be zero would achieve the best performance. Besides, ensure the initial update of BAA is zero, Notably, setting B while using small initialization for A yields better performance compared the reverse setup. This finding consistent with observations in existing literature 20
of the matrices must be initialized to zero.
Table 11: Comparison of the initialization strategies on GSM8K and Llama 3.1-8B-BB BBsBase
Initialization Strategy
LoRA AltLoRA
AltLoRA+
Gaussian
zero
66.37
73.13
76.87
zero
Gaussian
66.18
72.13
76.50
Kaiming zero
65.11
74.49
76,91
zero
Kaiming
67.10
74.03
76.88
Spectral zero
67.63
74.67
76.60
zero
Spectral
67.10
74.61
76.37
E.4 The performance and efficiency trade-off'
We summarize the memory-accurage trade-off of our method against strong LoRA baselines on GSM8K with the 1.ama-3.1-83 backbone in Table 12 Under comparable memory budgets, AltLoRA consistently improves accuracy over standard LoRA and LoRA-Rite, and AltLoRA+ achieves the best overall accuracy while maintaining memory footprint close to standard LoRA. This demonstrates that our optimization yields more favorable efficiency frontier: for a given memory cost, AitLoRA variants deliver higher accuracy; for target accuracy, they require less memory.
Table 12: Memory cost (GB) accuracy (%) GSM8K (fine-tuning Llama-3.1 3.1-8B), AltLoRA and AltLoRA+ improve accuracy while preserving LoR A-level memory efficiency.
Method
Memory Cost (GB)
Accuracy (%)
Full FT
> 48
73.31
LoRA
22.26
66.11
LoRA-Rite
25.39
74.10
LoRA-Pro
40.12
73.12
AltLoRA
22.56
74.49
AltLoRA+
23.16
76.91
39


Reviews:
[{"review_id": "inKvWto6Ui", "reviewer": "Reviewer_tXEi", "summary": "This paper propose a novel method called AltLoRA for aligning full finetuning gradients. It change the gradient update jointly on A and B (like lorapro does) to iteratively update A then update B respectively. Based on lorapro theory, they derive simpler solution then lorapro.  Numerical experiments demonstrate that AltLoRA outperforming vanilla LoRA and its variants on several benchmark datasets.", "questions": "See weakness", "limitations": "The theoretical framework contains flaws (W1), and the optimization method appears unreliable (W3). The authors evaluate all baselines using only 1 epoch, which is insufficient for convergence for most baselines (W2).", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### Strengths\n- This paper provides a novel perspective on the initialization of LoRA from the gradient updation aspect.\n- The paper is theoretically comprehensive, well-written, and easy to follow.\n- The method demonstrates promising performance and exhibits robust behavior across different hyperparameter settings, particularly for the sensitive parameters like Alpha and learning rate in LoRA.\n\n### Weaknesses\n- W1. The theoretical framework contains flaws: as established, B is zero-initialized in LoRA, but in Algorithm 1 at $t=0$, updating A requires computing the inverse of $(B^TB)^{-1}$, which leads to undefined behavior. Furthermore, after several update steps when $t>0$, B contains minimal values (due to small learning rates), causing $(B^TB)^{-1}$ to become extremely large. This consequently makes the gradients for A excessively large and corrupts the weight of A. Since we typically use fp16/bf16 mixed-precision training in practice, these large backpropagated updates may exceed numerical ranges. These two factors may cause the loss to diverge and potentially lead to training collapse.  \n- W2. The authors evaluate all baselines using only 1 epoch (L891), making convergence speed the primary determinant of reported performance. This evaluation protocol may disadvantage LoRA-based methods, as faster initial convergence does not necessarily indicate better final performance, slower-converging methods may ultimately reach superior solutions. For reliable comparisons, experiments should run for at least 5 epochs (or more for hard-to-converge tasks) to obtain reliable performance measurements.\n- W3. While standard LoRA jointly optimizes both variables A and B (M1), AltLoRA decouples this into sequential optimization - first A, then B (M2). However, the convergence properties for LoRA with M1 optimization remain unproven (D3 only discusses a 2-layer ReLU NN case without providing proofs for general LoRA scenarios. Furthermore, practical training typically involves more complex neural architectures). Empirical evidence suggests that alternately freezing layers during training often leads to training instability rather than performance improvement. Furthermore, since M2 is essentially a degenerate case of M1 with stricter convexity requirements, the local minima found by M2 should theoretically be inferior to those found by M1. Therefore, the observed superior performance of M2 over M1 in Table 2 and Table 4 presents an unexplained anomaly that warrants further investigation. \n- W4. In Table 1, the performance seems degrade as rank increasing, which is strange. \n- W5. Given the theoretical concerns raised in W1, AltLoRA’s superior performance over all baselines appears unexpected. Table 8's finding that zero initialization of B outperforms Gaussian initialization contradicts the issue identified in W1. I personally consider it may related with W2. Due to these issues,  I currently maintain reservations about the methodological validity and choose to assign a low rating temporarily.  Upon receiving adequate explanation of these points, I am happy to revisit my assessment and raising the scores.", "was_revised": true, "final_justification": "I raise my rating to a weak accept, in recognition of the extensive experiments and the interesting improvements that (aligned) alternating updates can bring. However, I maintain reservations regarding the flawed theoretical analysis and the imprecise descriptions in the paper.", "rebuttal": "The authors thank the reviewer for insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.\n---\t\n**W1: Computing $(B^TB)^{-1}$**\n\nSorry for the confusion. Please note that we have clarified at ***L121*** and ***L149*** of the original submission that we didn't directly compute $(B^\\top B)^{-1}$ but used a regularized inverse instead. We use $(B_t^\\top B_t)^{-1}$ in Algorithms 1 and 2 just for notational simplicity.\n\nTo be more specific, the full-rank assumption for $B$ is typically not satisfied during training according to our analysis. Thus, following the reasoning in our gradient-alignment analysis (see *Corollaries 1 and 2*), we introduce weight decay during the gradient alignment and compute the well-defined regularized inverse:\n$ (B^\\top B + \\lambda \\mathbb{I})^{-1}.$ \nThis formulation ensures numerical stability and avoids undefined behavior during updates. In all experiments of the original submission, we have used $\\lambda=1e^{-5}$. \n\nWe will revise the algorithm descriptions to replace $(B_t^\\top B_t)^{-1}$ with $(B_t^\\top B_t + \\lambda \\mathbb{I})^{-1}$ to avoid any ambiguity.\n\n---\t\n**W2: Increasing training epoch**\n\n\nThank you for the helpful suggestion. For fairness and direct comparability, our original experiments followed the evaluation protocol used in *LoRA-GA* (and *LoRA-Pro*), which trains for ***one*** epoch, and we reported the corresponding results. \n\nTo provide a more comprehensive evaluation, as the reviewer suggested, we have ***re-run all experiments in Table 2 using five epochs***. The full results are reported below. We observe that our method continues to consistently outperform all baselines under this extended training regime, demonstrating that the improvements are robust beyond early convergence and hold across longer fine-tuning schedules. We would be glad to include these results in the revised manuscript for completeness. \n\n\n\n| Method                     | MT‑Bench | GSM8K | HumanEval |\n|----------------------------|:--------:|:-----:|:---------:|\n| Full FT                        | 6.24±0.03 | 75.31±0.47 | 50.42±1.20 |\n| LoRA                           | 6.19±0.02 | 66.78±1.14 | 41.54±1.01 |\n| PiSSA                          | 6.11±0.03 | 67.39±0.61 | 42.12±1.45 |\n| rsLoRA                         | 6.18±0.02 | 67.98±1.21 | 44.31±1.31 |\n| LoRA+                          | 6.21±0.02 | 74.00±0.61 | 44.32±1.33 |\n| DoRA                           | 6.23±0.03 | 67.12±0.84 | 45.12±1.32 |\n| AdaLoRA                        | 6.05±0.02 | 68.29±0.98 | 42.58±1.84 |\n| LoRA‑GA                        | 6.24±0.05 | 73.44±1.10 | 45.56±1.48 |\n| LoRA‑Pro                       | 6.22±0.02 | 73.24±0.79 | 43.21±1.50 |\n| LoRA‑Rite                      | 6.20±0.03 | 73.48±0.65 | 45.46±1.21 |\n| **AltLoRA**                   | 6.26±0.04 | 74.52±0.37 | 46.28±1.11 |\n| **AltLoRA+**                   | 6.18±0.04 | 76.01±0.34 | 50.01±1.01 |\n\n---\t\n**W3: Why AltLoRA outperforms standard LoRA**\n\nWe respectfully disagree with some of the reviewer's opinions and aim to clarify the reasons behind the improved performance of our method.\n\nFirst and foremost, unlike full fine-tuning, the gradients in standard LoRA are confined to low-rank subspaces, which limits its effectiveness. Prior work, such as LoRA-Pro, has identified this limitation and attempted to align LoRA gradients to approximate the full gradient. However, LoRA-Pro does not yield a unique update solution and demonstrates unstable performance. In contrast, our alternative optimization strategy addresses this issue by ***better aligning the low-rank gradient with the true full gradient***. This alignment underlies the superior performance of AltLoRA compared to standard LoRA.\n\nRegarding theoretical analysis, we emphasize that our work is not purely theoretical or solely focused on convergence proofs. Instead, our goal is to develop an effective LoRA algorithm ***supported by theoretical insights*** that justify our design choices. Specifically, we have demonstrated that our approach achieves stable feature learning and transformation invariance—property not present in standard LoRA but crucial for non-asymptotic training behavior. Furthermore, the convergence guarantee, albeit established on a simplified two-layer ReLU network, provides additional evidence of the method’s design soundness.\n\nFinally, we note that the strategy of freezing layers, which may appear as an *empirical heuristic*, fundamentally differs from our alternating optimization approach. Alternating optimization is a well-established technique with decades of study. Recent works—[1] in LoRA and [2] in matrix factorization—also show that alternating optimization can be more stable and effective than joint optimization in certain settings.\n\n---\n**W4: Performance as rank increases**\n\nIn Table 2, we observe that performance degrades as the rank increases. A similar trend appears in the original LoRA paper (Tables 15 and 18 in [3]). We hypothesize that larger ranks introducing more trainable degrees of freedom can easily lead to overfitting when learning from a small sample size. Consistent with this view, in the five-epoch setting, the performance of AltLoRA and AltLoRA+ with higher rank improves and exceeds the default setting (rank = 8).\n\n\n| Method                     | MT‑Bench | GSM8K | HumanEval |\n|----------------------------|:--------:|:-----:|:---------:|\n| **AltLoRA**                   | 6.26±0.04 | 74.52±0.37 | 45.41±1.11 |\n| **AltLoRA** (rank=32)    | 6.13±0.03  | 74.92±0.67 | 46.82±1.34 |\n| **AltLoRA** (rank=128)      | 6.28±0.04  | 74.71±0.56 | 46.88±1.38 |\n| **AltLoRA+**                   | 6.18±0.04 | 76.01±0.34 | 50.01±1.01 |\n| **AltLoRA+** (rank=32)         | 6.18±0.05 | 77.69±0.41 | 50.21±1.43 |\n| **AltLoRA+** (rank=128)        | 6.14±0.04 | 77.21±0.67 | 50.44±1.22 |\n\n---\n**W5: The superiority of AltLoRA**\n\n\nBy computing $(B^{\\top}B + \\lambda\\mathbb{I})^{-1} $ instead of $(B^{\\top}B)^{-1} $, we eliminate the theoretical concern raised in *W1*. As a result, the downstream argument in *W5*, which relies on *W1*, no longer applies. To examine whether the superior performance of zero initialization over Gaussian initialization for $B$ is related to *W2*, we revisit the same setup in Table~8 under a 5-epoch training schedule. We find that, even with more steps, zero initialization of $B$ yields a higher accuracy (74.52) compared to Gaussian initialization (74.31), which shows a consistent result with our manuscript.\n\n---\nThank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.\n\n\n---\n[1] Chen S, Guo Y, Ju Y, et al. Robust federated finetuning of llms via alternating optimization of lora. arXiv preprint arXiv:2502.01755, 2025.\n\n\n[2] Jia, Xixi, et al. Preconditioning matters: Fast global convergence of non-convex matrix factorization via scaled gradient descent. Advances in Neural Information Processing Systems 36 (2023): 76202-76213.\n\n[3] Hu E J, Shen Y, Wallis P, et al. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2022."}, {"review_id": "qj0hW8m4Jw", "reviewer": "Reviewer_4vyq", "summary": "This paper proposes AltLoRA, a new PEFT method that narrows the performance gap between LoRA and full fine-tuning.\n\nThe key idea is to update the low-rank matrices $A$ and $B$ via alternating projections onto their respective subspaces, with each step having a closed-form solution. A momentum mechanism is also introduced, aligned to the low-rank space, avoiding the memory overhead of full-size momentum.\n\nThe authors provide theoretical analysis (stability, invariance, convergence) and show strong empirical results across language tasks. Both AltLoRA and its variant AltLoRA+ outperform prior LoRA methods and approach full fine-tuning performance, while retaining LoRA’s efficiency.", "questions": "See weakness.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength\n1. Novolty: The alternating update scheme for low-rank matrices is an elegant and original solution to the gradient approximation problem. It avoids issues with non-uniqueness in joint methods and naturally leads to a principled momentum alignment strategy that is both effective and memory-efficient.\n2. Soundness: The paper offers strong theoretical guarantees and extensive experiments across models (Llama-3, T5) and tasks (math, code, dialogue, NLU) show that AltLoRA and AltLoRA+ consistently outperform prior LoRA methods and approach or exceed full fine-tuning, while retaining LoRA’s efficiency.\n\nWeakness\n1. Writing Quality: The paper has several presentation issues. For example, Table 2's caption is below the table, unlike the others. Also, in line 601, the description of LoRA-Pro ([66]) is inaccurate—its main contribution is modifying gradients during training, not just initialization ([65]). This is likely a typo, but it could confuse readers.\n2. Missing Ablation on Momentum Alignment: While the momentum alignment mechanism is theoretically sound, the paper lacks an ablation comparing it with standard momentum updates. It would be helpful to know whether standard momentum leads to divergence or if it performs similarly, to better assess the necessity of the alignment.\n\n\nOverall, I think it's a good paper if the authors can fix the weakness in the camera-ready version.", "was_revised": true, "final_justification": "The author resolved my concern during the rebuttal. I have decided to keep my score unchanged and recommend acceptance.", "rebuttal": "The authors thank the reviewer for insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.\n \n---\n**W1: Writing Quality**\n\nThank you for bringing these presentation issues to our attention. We will relocate the caption of Table 2 to the correct position in the revised version. In ***L601***, the reference should be “***LoRA-GA***[65]” rather than “***LoRA-Pro***~[66]”. LoRA-GA indeed introduces a novel initialization approach that more closely aligns LoRA with full fine-tuning at the first step.\n\n\n---\n**W2: Missing Ablation on Momentum Alignment**\n\nWe include ablation studies to evaluate the effect of momentum alignment. The ***AltLoRA w/o MA*** variant differs from AltLoRA only in that momentum is not projected onto the low-rank subspace (momentum alignment). Across the tasks of natural language generation, AltLoRA outperforms AltLoRA w/o MA, achieving higher accuracy and lower variance. This demonstrates the advantages of aligning momentum with low-rank spaces.\n\n\n\n| Method                      | GSM8K | HumanEval |\n|----------------------------|:-----:|:---------:|\n| AltLoRA                       | 74.49±0.57 | 45.91±1.14 |\n| AltLoRA w/o MA                | 72.93±0.88 | 44.02±1.42 |\n\n\n---\n\nThank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns."}, {"review_id": "vJ9Czg2VmY", "reviewer": "Reviewer_rKDu", "summary": "This paper proposes AltLoRA which alternatively updates LoRA's two trainable components to simulate full matrix updates. Specifically, while fixing matrix $B$ and training matrix $A$, the authors consider an optimization problem that finds a proper subspace gradient $\\nabla_A$ so that $sB\\nabla_A$ has the smallest error to estimate $\\nabla_W$. AltLoRA directly provides the closed-form solution for the problem.  Based on it, the authors derived the update strategy for the momentum matrix. The overall AltLoRA algorithm works better than other PEFT baselines in the experiments, enjoying a comparable or better training performance than full fine-tuning. Besides, this paper provides analysis on stable feature learning, transformation invariance, and convergence analysis on a two-layer ReLU NN tuning problem.", "questions": "i) I wonder whether the models are fine-tuned on specific tasks for only one epoch or more than one epochs. I'm asking because I did not find the number of epochs the models are trained on, which I believe is highly correlated with the meaning of accuracy results. Maybe I have missed it, but it should be included in the experimental details.\n\nii) Is it a common choice to use LLaMA3-8B-Base as a pretrained model in the literature? I'm asking because in my experience, LLaMA3-8B-Instruct can be fine-tuned to perform much better using vanilla LoRA, and results based on the instructive model should better reveals the practical fine-tuning ability.\n\niii) In Line 611, it is stated that \"Random projection is also applied in Ga-LoRA([88])\", however, the method proposed in [88] uses a deterministic projection strategy based on top-K + SVD, and its name should be \"GaLore\" instead of \"Ga-LoRA\". In the literature, a variant of GaLore using random projection is called GoLore, which is proposed in [arXiv:2410.11289].", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\ni) The proposed method appear noval and makes sense for me. The underlying intuition is quite natural and the presentation is easy to follow.\n\nii) The empirical results are promising, which I believe can address the potential of the proposed method.\n\niii) The theoretical result of transformation invariance makes the method more appealing to me.\n\nWeaknesses:\n\ni) The convergence analysis is relatively simple. It only considers a toy model and deterministic gradients. Noting that PEFT is for large-scale training tasks, it is better to consider stochastic gradients in both the algorithmic design and the analysis.\n\nii) The performance-efficiency trade-off is not clear. In my understanding, AltLoRA achieves better convergence result while being less memory- and computation-efficient than LoRA. It is recommended to present a trade-off curve regarding both perfomance and efficiency, which I believe can help better address the effectiveness of AltLoRA.\n\niii) To better illustate the effectiveness of AltLoRA, I expect to see training curves regarding the training or validation loss. However, the experimental results only include accuracy tables, leaving it unclear whether AltLoRA benefits from faster and more accurate convergence or it benefits from better generalization.", "was_revised": true, "final_justification": "Since most of my concerns are addressed, I keep my score as is.", "rebuttal": "We thank the reviewer for the insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.\n\n---\n**W1: The convergence analysis and stochastic optimization**\n\nOur key contribution is to efficiently align both the gradient and momentum with the full gradient. The theoretical guarantees we establish provide insights into why the alignment intuition helps to improve performance in numerical studies. In contrast to the baseline work *LoRA-Pro*, which only proposed the theoretical motivation, our work provides formal guarantees on *stable feature learning, transformation invariance, and convergence*, thereby deepening the understanding of the importance of gradient alignment.\n\nIn particular, for the convergence analysis, [1][2] also analyze LoRA on an over-parameterized two-layer ReLU neural network tuning problem. Our convergence analysis demonstrates that the proposed *alternating scaled gradient descent* method for LoRA achieves the same convergence rate as joint optimization (as established in [1] using the same toy model). More importantly, we show that the convergence rate of alternating scaled gradient descent is ***independent of the condition number***, and that alternating updates allow for a ***wider range of learning rates*** compared to joint updates. These results constitute the core theoretical contributions of our convergence analysis for alternating projection in LoRA training.\n\nRegarding the stochastic gradient, thank you for raising this point. \n\nFirst, *for the algorithm design*, we do implement stochastic batch gradients in our fine-tuning process, as it is standard practice in PEFT. In the revised version, we will add the corresponding description to clarify this in our Algorithms 1 and 2. For example, for the update of $A$ in ***L167***, we will  explicitly note: \"Only backpropagate with respect to $A_t$ and obtain the stochastic batch gradient $\\nabla_{A_t}L$.\"\n\nSecond, while prior gradient alignment works also adopt stochastic gradients in practice, they typically do not analyze how stochastic gradients impacts the gradient alignment effect. Extending our theoretical analysis to explicitly incorporate stochastic gradient behavior is a promising direction for future work, particularly to better bridge the gap between LoRA and full fine-tuning from a stochastic optimization perspective.\n\n---\n**W2: The performance and efficiency trade-off**\n\nThank you for the helpful suggestion regarding visualization. We have reported the performance in Table 2 and the memory cost in Table 3. To better illustrate the memory-performance trade-off of our algorithm, we would like to include a scatter plot in the revised version of the paper. For each task, the x-axis will represent memory cost, while the y-axis will correspond to the performance metric (e.g., accuracy). This figure will visually demonstrate how our method improves over baselines. Here, we provide a table to describe the results for the GSM8K task by fine-tuning the Llama3.1-8B model, which supports that our algorithms achieve better performance than LoRA with similar memory efficiency.\n\n\n| Method                      | Memory Cost(GB) | Accuracy |\n|----------------------------|:-----:|:---------:|\n| Full FT                   | > 48  | 73.31 |\n| LoRA                      | 22.26  | 66.11 |\n| LoRA-Rite                 | 25.39 | 74.10 |\n| LoRA-Pro                  | 40.12 | 73.12 |\n| AltLoRA                   | 22.56 | 74.49 |\n| AltLoRA+                  | 23.16 | 76.91 |\n\n---\n**W3: Training Curves**\n\n\n\nWe provide the training loss in the table below. As shown, our method not only converges to a lower training loss than full fine-tuning, but also does so more rapidly than most baselines.\n\nThe *accuracy metrics* reported in Table 2 are evaluated on the corresponding test sets, as described in Appendix E.1. Our method achieves higher scores, highlighting its superior generalization performance.\n\n\n| Method                     | step 500 | step 1000 | step 1500 | step 2000 | step 2500 | step 3000 |\n|----------------------------|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n| Full Fine-tuning           |     0.51     |     0.28     |    0.24      |    0.18      |    0.18     |     0.18     | \n| LoRA                   |     0.57     |     0.30     |     0.26     |     0.23     |   0.22      |   0.22       | \n| LoRA-Pro              |    0.51      |     0.31     |     0.24     |    0.22      |   0.21       |    0.21      | \n| LoRA-GA             |     0.45     |   0.24       |    0.20      |     0.18    |   0.18       |     0.18     | \n| LoRA-Rite             |     0.55     |    0.31      |    0.24      |    0.22      |     0.21     |     0.21     | \n| AltLoRA                 |     0.48     |     0.25     |     0.21     |    0.19      |   0.18      |       0.18   | \n| AltLoRA+                |    0.45      |    0.23      |     0.19     |     0.18     |    0.17      |    0.17      | \n\n\n---\n**Q1: Epoch number**\n\n\nWe have reported the number of training epochs in the experimental setup to avoid confusion. As reported in ***Line 897***, the iteration number was set to 1, which corresponds to a single training epoch. \n\n\nWe agree that the number of epochs is an important training design, especially given its potential correlation with accuracy metrics. To further support the validity of our experiment, we add an ablation study for the training epoch with ***5 training epochs*** under the setup of Table 2. With the increase in training epochs, our approach would outperform baselines as well.\n\n| Method                     | MT‑Bench | GSM8K | HumanEval |\n|----------------------------|:--------:|:-----:|:---------:|\n| Full FT                        | 6.24±0.03 | 75.31±0.47 | 50.42±1.20 |\n| LoRA                           | 6.19±0.02 | 66.78±1.14 | 41.54±1.01 |\n| PiSSA                          | 6.11±0.03 | 67.39±0.61 | 42.12±1.45 |\n| rsLoRA                         | 6.18±0.02 | 67.98±1.21 | 44.31±1.31 |\n| LoRA+                          | 6.21±0.02 | 74.00±0.61 | 44.32±1.33 |\n| DoRA                           | 6.23±0.03 | 67.12±0.84 | 45.12±1.32 |\n| AdaLoRA                        | 6.05±0.02 | 68.29±0.98 | 42.58±1.84 |\n| LoRA‑GA                        | 6.24±0.05 | 73.44±1.10 | 45.56±1.48 |\n| LoRA‑Pro                       | 6.22±0.02 | 73.24±0.79 | 43.21±1.50 |\n| LoRA‑Rite                      | 6.20±0.03 | 73.48±0.65 | 45.46±1.21 |\n| **AltLoRA**                   | 6.26±0.04 | 74.52±0.37 | 46.28±1.11 |\n| **AltLoRA+**                   | 6.18±0.04 | 76.01±0.34 | 50.01±1.01 |\n\n---\n**Q2: Llama3-8B-Base**\n\nUsing the ***Llama3-8B-Base*** model is a common practice in the literature. Among the most relevant baselines, **LoRA-Pro** and **LoRA-GA** adopt either *Llama2-7B* or *Llama3-8B-Base* as the pretrained model. Furthermore, several recently published works [4][5][6] on LoRA also use *Llama3-8B-Base* as the standard backbone.\n\nFor the *Llama3-8B-Instruct* model, we fine-tune it on the MetaMathQA dataset and evaluate it on the GSM8K test set, reporting accuracy as the primary metric. Our results demonstrate that even when using a stronger pretrained model, low-rank adaptation can still yield performance gains. More importantly, both AltLoRA and AltLoRA+ consistently outperform the standard LoRA baseline. We will add the full comparison in the revised revision.\n\n\n| Method                     | GSM8K | \n|----------------------------|:-----:|\n|Llama3-8B-instruct | 61.24 |\n| + Full FT           | 83.26 |\n| + LoRA              | 81.12 |\n| + AltLoRA           | 82.60 |\n| + AltLoRA+          | 83.88 |\n\n\n---\n**Q3: Random projection and GaLore**\n\nWe appreciate the clarification regarding the naming: it should be “GaLore“[5] instead of ”Ga-LoRA“. Additionally, we acknowledge that the variant using random projection is introduced in a separate work, ***GoLore***. And we will include GoLore into our related work as well. We will revise the text accordingly to correct both the description and the naming to avoid confusion.\n\n---\nThank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.\n\n\n\n---\n\n[1] Zhang, F. and Pilanci, M. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models.  In International Conference on Machine Learning (pp. 59641-59669). PMLR.\n\n[2] He, Y., Li, P., Hu, Y., Chen, C., and Yuan, K. Subspace Optimization for Large Language Models with Convergence Guarantees. In Forty-second International Conference on Machine Learning. 2025.\n \n[3] Zhang, J., Chiu, H.M., and Zhang, R.Y. Accelerating SGD for Highly Ill-Conditioned Huge-Scale Online Matrix Completion. Advances in Neural Information Processing Systems 35 (2022): 37549-37562.\n\n[4] Liao, X., Li, S., Xu, Y., Li, Z., Liu, Y. and He, Y. GaLore+: Boosting Low-Rank Adaptation for LLMs with Cross-Head Projection. arXiv e-prints, pp.arXiv-2412.\n\n[5] Zhang, C., Lianhai, R.E.N., Cheng, J. and Li, Q. From Weight-Based to State-Based Fine-Tuning: Further Memory Reduction on LoRA with Parallel Control. In Forty-second International Conference on Machine Learning. 2025.\n\n[6] Huang, Q., Ko, T., Zhuang, Z., Tang, L., and Zhang, Y. HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models. In The Thirteenth International Conference on Learning Representations. 2025.\n\n\n[7] Recht, B., Maryam F., and Parrilo, P. Guaranteed minimum-rank solutions of linear matrix equations via nuclear norm minimization. SIAM Review 52.3 (2010): 471-501."}, {"review_id": "eKhC3ZGewq", "reviewer": "Reviewer_uqXj", "summary": "The paper proposes AltLoRA, a novel method for Low-Rank Adaptation (LoRA) that aims to bridge the performance gap between LoRA and full fine-tuning by introducing an alternating projection strategy for gradient approximation and momentum optimization within low-rank constraints. The authors provide theoretical guarantees for convergence, stable feature learning, and transformation invariance, supported by extensive experiments across multiple tasks (natural language understanding, dialogue generation, mathematical reasoning, and code generation). The method claims to outperform existing LoRA variants while maintaining memory efficiency.", "questions": "1. Could the authors elaborate on why second-order momentum breaks transformation invariance? Is there a principled way to align it with low-rank subspaces, or is this a fundamental limitation?", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "- Strengths\n1. The alternating projection approach for gradient approximation and momentum design is a fresh take on improving LoRA, addressing key limitations of prior work like LoRA-Pro.\n2. The experiments are comprehensive, covering diverse tasks and benchmarks. The results consistently show improvements over baselines, including LoRA-Pro, and the ablation studies add depth to the evaluation.\n\n- Weaknesses\n1. The paper acknowledges that AltLoRA+ loses transformation invariance due to second-order momentum (Appendix D.2). This is a significant limitation, yet the discussion is brief.\n2. While AltLoRA narrows the gap, it doesn’t consistently match full fine-tuning (e.g., Table 2 shows Full FT outperforms AltLoRA on HumanEval). The claim of \"narrowing the gap\" could be tempered.", "was_revised": true, "final_justification": "I consider maintaining my score.", "rebuttal": "We thank the reviewer for the insightful and constructive comments. Below, we provide our detailed responses to each point. We hope these clarifications help address your concerns.\n\n---\n**W1: The discussion about AltLoRA+ and transformation invariance**\n\nWe thank the reviewer for the comment. ***AltLoRA+ aims for accelerating AltLoRA instead of achieving transformation invariance***. That said, since our primary goal of AltLoRA+ is not to develop a fully transformation-invariant optimizer, we believe this limitation does not significantly impact its main contributions. In AltLoRA, our core contribution lies in inducing optimally aligned gradients and momentum within low-rank subspaces, which already demonstrates better performance compared to baseline methods. Besides the algorithm design, our theoretical guarantees, including stable feature learning, transformation invariance, and convergence analysis, provide a deeper understanding of the theoretical significance of gradient alignment in optimizing LoRA. To further ***accelerate convergence***, we propose AltLoRA+, which incorporates element-wise second-order momentum in a manner similar to AdamW (for the explanation of why element-wise second-order momentum will break transformation invariance, please refer to the answer of Q1). This enhancement leads to additional performance gains over the baselines. The improvement from AltLoRA to AltLoRA+ can also be regarded as an ablation study showcasing the benefits of momentum acceleration, which highlights the significance of AltLoRA+.\n\n---\n**W2: Narrowing the gap**\n\nOur work aims to better understand the limitations of optimization within low-rank subspaces and to design more efficient algorithms that align low-rank updates closely with full-gradient directions. While we do not claim to completely close this gap, our empirical results demonstrate that our method brings low-rank adaptation closer to full fine-tuning in empirical performance. Following the reviewer's suggestion, we hope to revise our claim like \"it narrows this gap toward full fine-tuning in two out of three natural language generation tasks and in the majority of natural language understanding tasks.\"\n\n---\n**Q1: Analysis on transformation invariance and aligned second-order momentum**\n\n\nFirst, we will clarify why the second-order momentum in AltLoRA+ breaks transformation invariance. Here, we will prove AltLoRA+ breaks *scalar scale invariance* (See Def 2 in [1] for the formal definition), as a weaker version of transformation invariance. For two pairs of low-rank adaptation $ B_1 A_1 = B_2 A_2$, if we assume $A_2 = s A_1$, $B_2 = \\frac{1}{s} B_1$, then\n\n$$\\delta A_1 = -\\eta \\frac{\\widetilde{\\nabla }_{A_1} L}{\\left(\\nabla\\_{A_1}L \\odot \\nabla\\_{A_1}L\\right)^{1/2}} = -\\eta \\frac{(B_1^\\top B_1)^{-1} B_1^\\top \\nabla_w L}{\\left(B_1^\\top \\nabla_w L \\odot B_1^\\top \\nabla_w L\\right)^{1/2}} = -\\eta \\frac{1}{s^2} \\cdot \\frac{(B_2^\\top B_2)^{-1} B_2^\\top \\nabla_w L}{\\left(B_2^\\top \\nabla_w L \\odot B_2^\\top \\nabla_w L\\right)^{1/2}} = \\frac{1}{s^2} \\delta A_2,$$\n\n\nwhere $\\widetilde{\\nabla }_{A_1} L$ is the scaled gradient. Therefore, the update would violate scalar scale invariance as $$B_1 \\delta A_1 = s B_2 \\cdot \\frac{1}{s^2} \\delta A_2 = \\frac{1}{s} B_2 \\delta A_2 \\neq B_2 \\delta A_2.$$ \n\n\n\n\nSecondly, it is possible to align the second-order momentum with the full gradient in a principled way, but doing so does not necessarily improve performance. Let $V_t^B$ denote the second-order momentum of $B$ at time $t$. For example, analogous to aligning first-order momentum in the low-rank subspace, we can formulate a similar problem for second-order momentum as \n\n$$\\min_{\\widetilde{V}^B_t} \\|\\|(A_t)^T V_t^B A_t - (A_{t+1})^T \\widetilde{V}^B_t A_{t+1} \\|\\|^2,$$\n\nwhere we assume $V_t^B$ is aligned to the low-rank space at time $t$. Here, the optimal solution is \n\n$$\\widetilde{V}^B_t = (P_{A_{t+1}})^T V_t^B P_{A_{t+1}},$$\n\n where $P_{A_{t+1}} = A_t A_{t+1}^T \\left( A_{t+1} A_{t+1}^T \\right)^{-1}.\n$ Then the updated second-order momentum for $B$ at time $(t+1)$ is $V\\_{t+1}^B = \\widetilde{V}^B_t + (\\widetilde{\\nabla}\\_{B_t}L)^T\\widetilde{\\nabla}_{B_t}L$. However, our numerical study shows that the accuracy of this approach does not lead to performance gains. With the same setup in Table 2, the accuracy on the GSM8K test set is only 56.12. Thus, while principled alignment is possible, it appears less effective in practice. In contrast, AltLoRA+, which focuses on aligning first-order momentum efficiently, demonstrates stronger empirical performance.\n\n\n\n---\nThank you for your constructive feedback. We will incorporate the new results in the revised version, and we hope our responses satisfactorily address your concerns.\n\n---\n\n\n[1] Yen J N, Si S, Meng Z, et al. LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. arXiv preprint arXiv:2410.20625, 2024."}]

Author rebuttal:

Meta-review: In this paper, the authors propose a new method for LoRA that aligns low-rank gradients with their full-rank counterparts, supported by theoretical analysis. The experimental results look promising. While reviewers initially raised concerns regarding the theoretical framework and empirical validation, the authors' rebuttal successfully addressed the majority of these issues. It is recommended that the authors incorporate the feedback from the rebuttal discussions into their revision.
Decision: Accept (poster)