RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
Authors: Yilang Zhang
Bingcong Li
Georgios B. Giannakis
Venue: NeurIPS 2025 poster ()
Keywords: Low-rank adaptation
parameter-efficient fine-tuning
optimization
Primary area: deep_learning
TL;DR: RefLoRA improves upon LoRA by dynamically selecting the optimal low-rank factorization per step. This leads to faster and stabler convergence, as well as superior performance on various NLP tasks, while incurring negligible computational overhead.

Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.

RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
Yilang Zhang Department of ECE University of Minnesota Minneapolis, MN 55414 zhan74530um edu
Bingcong Li Department of CS ETH Zürich 8092 Zürich, Switzerland bingcong. li@inf ethz. ch
Georgios B. Giannakis
Department of ECE University of Minnesota Minneapolis, MN 55414 georgi
georgios@umn, os@umn edu
Abstract
Low-Rant Adaptation (LoRA) lowers the computational and memory overhead fine-tuning large models by updating a low-dimensional subspace of the pre- trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adapta- tion (RefLoRA) method promotes flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and common- reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of state-of-the-art LoRA variants.
Introduction
Large language models (LLMs) have revolutionized wide spectrum of applications including chatbots (. code generation [8) and scientific discovery (55). Despite their success, adapting LLMSs to specific tasks remains computationally demanding. L.MMs are built on two-stage learning process, namely pre-training and fine-tuiing Pre-training is performed on massive, Internet-scale corpora. This endows LLMs with in-text comprchension and generation abilities, but results in models with billions to trillions parameters [T]. Though broad language capabilities are granted, pre-trained LLMs yet not tailored for specialized applications. To acquire domain-specific expertise, LLMs must be further trained on downstream tasks, what known as fine-tuming tinettunigg With the continually growing model size however, conventional full fine-tuning approaches (optimizing all model parameters) can be increasingly prohibitive due to the immense GPU memory and substantial computational capacity demands, rendering them impossible for individual users and organizations.
To tackle the computational bottleneck, parameter-efficient fine-tuning (PEFT) 22 has been investi- gated to enhance fine-tuning efficiency. As opposed to fully fine-tuning all parameters, PEFT methods either optimize a sparse subset [18] [54], or introduce additional lightweight trainable parameters 33 Among these approaches, low-rank adaptation (LoRA) 23 has gained popularity due to its low additional cost during inference. LoRA presumes that the parameter updates during can be captured by a low-rank matrix. Though effective, LoRA observed to suffer from challenges such slow and unstable convergence (41), inconsistent and unbalanced weight updates (67 20), and notable performance gaps relative to full fine-tuning 23 One key reason behind these challenges is the nonuniqueness of LoRA's low-rank factorization
while keeping the pre-trained ones frozen 22 46 36
29
lie low-dimen dimeeniiiionl manifold, and thus
67
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

To cope with these challenges, this work advocates "refactoring" low-rank adaption (RefLoRA), novel approach that commits the optimal factorization per step. Our contribution threefold:
We show that LoRA's inconsistent weight updates can be characterized by a symmetrie positive definite matrix. RefLoRA dynamically selects the optimal one by minimizing an upper bound on the loss, resulting in flatter landscape of the loss that facilitates stable and efficient optimization.
The optimal factorization is proven to have closed-form global solution; to yield consistent and balanced weight updates; and to bring about a lower overhead compared to SOTA approaches. Moreover, a simplified variant termed RefLoRA-S is developed to further reduce complexity.
Extensive numerical tests are conducted on matrix factorization, natural language understand- ing, and commonsense reasoning benchmarks with popular LLMs scaled up 8B parameters, demonstrating RefLoRA's faster convergence and consistent performance gain.
Related work. Building upon LoRA, several have been developed to ameliorate its performance. Some revise vanilla LoRA's architecture or dynamically adjust its hyperparameters. For instance, DoRA (5] decomposes LoRA weights into magnitude and direction components to improve learning capacity and training stability. AdaLoRA
70 and GeoLoRA [5T] allocates per-layer rank on-the-fly to prune less salient updates. FedPara (25 (a.k.a. LoHa) and LoKr [[66] respectively integrate the low-rank structure with Hadamard and Kronecker products for reduced communication cost and improved model expressiveness. Another line of work boosts LoRA's behavior in early fine- tuning epochs through well-designed initialization. PiSSA [1 leverages truncated singular value decomposition (SVD) to keep the low-rank initialization close to the pre-trained weights, whereas LoRA-GA [60] aligns the first update direction with full fine-tuning, Other variants refine LoRA's per-step optimization to promote empirical convergence. LoRA-Pro [61] redirects LoRA's gradient to match the weight update of full fine-tuning. LORA-RITE [67 substitutes the default Adam optimizer [26] with customized gradient calculation and moment estimation scheme. However, these aaatient-alteing strategies necessitate meticulous crafting to ensure stability and convergence. Our approach falls within the same family, but adheres strictly standard backpropagation rule, avoiding direct gradient manipulation and requiring less computational overhead. To meet constraints of space limits, additional LoRA variants are discussed in Appendix A
2 Preliminaries and nonunique low-rank factorizations
Consider a general weight matrix W E mm parameterizing a large model. With initialization being the pre-trained matrix Wo == WPt and t indexing iteration. conventional full fine-tuning updates the sizable matrix W, by backpropagating the loss function ((W.). Though straightforward. this approach requires an excessive memory footprint and major computational cost.
Aiming efficient fine-tuning, LoRA 23 freezes Wpt and presumes that the weight increment exhibits a low-rank structure W, = Wpt + A,B,, where A, E Tx B, E "kxr and r < mi, n is preselected constant. Consequently, LoRA's objective function boils down to
min L(A, B) = ((Wet AB
A.B
This reformulation reduces the number of trainable parameters to (m nnr < mn, thus effectively minimizing the associated memory and computation costs. As for initialization, LoRA draws entries of Ao from zero-mean normal distribution with small variance o² for numerical stability; and Bg to ensure W. W This choice gives rise imbalanced updates of A, and B,, and decelerates empirical convergence especially in early epochs. At the first iteration for example, the chain rule implies that AA, C(A,, B.) comparison, Bg 's gradient VB, L( Ap, Bn) (((Pit) Ao is generally non-zero. In fact, it turns out that BF < ||A+ |F and V A. C(A,, II < |/V C(A., B,)lI when is small [4]] and these unbalanced low-rank factors and updates can markedly impede the convergence of LoRA.
V F'(W + B, Ba, O, meaning no update to Ag. In
Additionally, the nonuniqueness of LoRA's low-rank factors leads to inconsistent parameter updates. While this issue has been previously recognized [3 67], its implications for optimization remain underexplored. Specifically, for any alternative decomposition (A,, B,) satisfying A,B," A,B,, the forward pass and loss remain intact, whereas the update of W, can differ significantly. For illustration, consider standard gradient descent (GD) iterations
A++1 A, + AA, A vVV((, B B, AB = B, Be-1 (e-n)e((W.)) AA, (L)
2

---

where 17 > 0 denotes the learning rate. The corresponding update to the weight matrix is
AW. = We+1 W, - A. B A,B, = A,AB, + A + AA,AB, Alternatvell,, GD can be also performed with the equivalent pair (A,, B.), yielding
(2)
A++1 A, AA, A. B,, B,+1 = B, B, AB, = B, n)v((W,) A. (3)
The resultant parameter update is thereby AW; A,AB, + AB,AT + AA,AB, which can remarkably deviate from A W. in (2) despite both factorizations representing the same W,. Further claboration this issue will be provided in the ensuing section.
The following notational conventions are adopted throughout the paper.
Notation, Bold lowercase (capital) letters denote vectors (matrices); || 12. 1I. 1| and (s,
stand for f2-, nuclear-, Frobenius norm and Frobenius inner product; Col(.), Co Null (-) and H. represent column space, null space and the i-th entry/column of vector/matrix; -f, (r(-), and rank(.) refer to the Moore-Penross pseudoinverse, trace, and rank; (-) and oy(+) are the i-th largest eigenvalue and singular value; $ indicates the set of r x symmetric positive definite (SPD) matrices; GL(r) denotes the general linear group of degree r (i.e., the set of r x r invertible matrices); and O(r) stands for the orthogonal group of degree (i.e., the set of r orthogonal matrices).
3 Low-rank adaptation with optimal refactoring
This section delves into the nonuniqueness of LoRA's factorization, and identifies the optimal one that minimizes the loss (WW + AW, is first demonstrated that all possible factors (A,, B,) can be characterized by an invertible matrix Pt, and the weight update W, fundamentally governed by an SPD matrix S, == P.P.T. Then, we will derive an upper bound of E(W, + AW,) as a function of S,, whose global minimum will be obtained in closed form. Building on these theoretical insights, we will optimally refactor LoRA's low-rank matrices per iteration to obtain more effective updates, that are at the center of our "refactored" low-rank adaptation (RefLoRA) approach. All proofs in this section are deferred to Appendix B
3.1 Characterizing LoRA's factorization and weight update
Our analysis begins with the following mild assumption, which has been utilized and validated on various realistic datasets [6]].
Assumption 1. rank(A, rank B.) r, Vt > O.
Assumption asserts that the tall matrices A, and B, maintain full column rank after the first iteration. Since LoRA seeks to approximate the full update of W, using a low-rank matrix, this assumption essentially reflects the effectiveness of LoRA's parameterization. Under Assumption [ the next lemma reveals that all equivalent factorizations can be captured by r invertible matrix P.. Lemma 1. With Assumption 7 in effect, holds that ((A,, Be) A.B. A,B.) (AP.P,,BPP..) P. E GL())) Moreover, if P. E O(r), then AW, AW.
Beyond characterizing the structure of equivalent factorizations, Lemma | iiipliis that consistent weight updates are preserved when (A,, B,) differs from (A,, B.) up rotation and reflection. This naturally raises the question: which factorization, or Pt, is preferable for effective optimization?
To answer the last question, we first present an important observation about AW, When setting A, A,P, and B, B,P," can be readily verified (see [14p of Appendix B.1 that
AW.- = AAP + AA,,
BT + AA,AB.
(5)
Letting P, V denote the SVD of Pt, it is clear that AW, is fully determined by S.. This implies that Vf" can be chosen arbitrarily from O(r). In other words, P. can be right-multiplied by any orthogonal matrix, without affecting AW, This observation agrees with the last statement of Lemma by replacing (A,, B.) and P. with (A,U, E, B, U.U. ) and V.. Consequently, identifying the optimal factorization boils down to selecting ideal S, E S__.
the SPD matrix S, P.P. U (""""""
3

---

3.2 Optimizing St via loss upper bound minimization
Our key idea is to select the S, that minimizes the loss ((W++1) = &(W + (S.). Unfor- tunately, directly minimizing this objective over S, requires exhaustive search due to the model nonlincarity, which is infeasible in practice. As an alternative, we derive a tractable upper bound on ((W + A (S:)), and minimize the bound to obtain an optimal S.. Our motivation stems from GD, which relies on the following Lipschitz smoothness assumption.
Assumption 2. The loss function E has L-Lipschitz gradient; i.e., [|Ve(W)- ve(W")lF S L/WW W |F, VW, W E R=xx.
Assumption 2 is equivalent to requiring Lipschitz smoothness of w.r.t. the vectorized weight matrix vec( W). which is fairly mild and common in machine learning [52 [6] and optimization 6].
Under this assumption, the loss admits the following quadratic upper bound
L (W. + AW,) S ((W,) + (Ve(W,), W/r + - AA..
(6)
Minimizing the bound yields the optimum A AW; = V )), thus recovering the standard GD used in full fine-tuning. Given that L typically unknown in practice, the optimal learning rate 1/L is replaced by a hyperparameter, whose value can be tuned via grid search on a validation dataset.
Although f is often Lipschitz smooth w.r.t Wt, its smoothness constants w.r.t. A, and B, can be unbounded due to the bilinear structure, unless one assumes boundedness or convergence of A, and B, [14]. Since these two assumptions are overly restrictive, they will be avoided in our analysis.
The nonlinear dependence of AW, on S, (cf. (5)) prevents an analytical solution when directly optimizing S, over the quadratic upper bound of & (W, + A A (S)). This motivates iterative solvers involving matrix multiplication with the sizable matrix V&(W,) () mxx. To mitigate the overhead, the next proposition relaxes the quadratic upper bound to factor out [[ thus decoupling S,'s optimization from Ve(W.).
Proposition 2. Consider GD update
3 with A, = A,P, B = BBP.T and S. == P,P.T Under
Assumptions 7 and2 2 follows that
&(W, + AW. < IIvEWW)IL
Ln²
(IA 1 I,S,, $11 Ln ) +O((T)))+ Const. (7)
where Const. refers to constants that do not rely on St.
The high-order term O (Ln)") originates from AB in (5). As is typically small (~ O(10-)),
this term negligible in practice 60, 67). I. As a consequence, the upper bound in is dominated by its first term. This leads to our RefLoRA objective
min (IA I|: + |BSS lB
1 Ln
(8)
S,ES",
Consider for convenience the variables Š, and Č. defined
S, = (A. (AAA,)) (ATA,) A, B (A A.) (ATAA) C := 2 1|-
(9)
based on which (8) can be solved in closed form as established in the following theorem. Theorem 3. Under Assumptions [ the global optimum of 8 satisfies
S1
if 2 c.L or or
S (C,Lm)
+ ((C.Ln)-2-11 1 S., i0 < < C.I
(10)
Theorem states if > 0 is not too small, then (8) admits a unique global optimum $, Otherwise, there can be multiple optima, while two can always be constructed by appropriately scaling S, In addition, 7 < 0 is included in (IO) for visualization purposes. We remark that S, in (9) known as the matrix geometric mean (A, A,) -1 # (B, B₂) of (A, A -- and (B, B,) [2]. which can be also written as S, (A, A.) [A_ A, B, b; cf. Lemma in the Appendix.

---

1.4
LoRA loss
LoRAA loes
1.
LoRA
1.3
LoRA bd.
1.2
Ref. upper hd.
1.2
Rat. upper bd.
Los ([W, kAW
AWW
1.1
1.1
0.9
0.05
0.05 Step size 0
Step slze 1T
(a) S S7
(b) S.
Figure 1: Visualization of loss ((W + AW,) and upper bound 77. LoRA corresponds to S, = I,, while our refactoring (ref.) optimizes St.
Figure plots the loss E (WW + A W.). and the upper bound 7 of a numerical example as a function of ry; also Appendix D.2 for details. LoRA corresponds to the non-optimized S, I, whereas 0 leads to a jump discontinuity for
our refactoring selects S S based on Theorem Notably, our refactoring. Figure Ia shows that by optimizing the upper bound (7). the associated loss becomes lower and flatter. This enables larger step size to achieve a lower loss, thereby accelerating the empirical convergence of LoRA; see Section for experiments corroborating this claim.
The flatter loss landscape arises thanks to balancing A, and B,. Indeed, the imbalance of A, and B, in vanilla LoRA necessitates different step sizes (20). Appendix [B33 proves that the balance A, A, = B, B, is guaranteed with S, = St, thus enabling a unified step size. With too small however, it is more beneficial (compared to balanced updates) to scale either A, B, to accommodate the small n, This corresponds to the two solutions in the 0 17 1/(Ć,L) case of (IO).
3.3 RefLoRA: Refactored low-rank adaptation
Having identified in Theorem 3 the optimal S; minimizing the loss upper bound, we are ready to introduce our refactored low-rank adaptation (RefLoRA) approach. RefLoRA substitutes LoRA's per- step update ( with the refactored version 3 where A, = A,P,, B, B.P, and P.P.T S As the right singular matrix V E O(r) can be arbitrary (cf. Section 3.1 p. one convenient choice is to simply set P, S/ Next, this subsection deals with two practical challenges facing the implementation of RefLoRA, and delves into several important properties of the resultant approach.
Smoothness constant L is typically unknown and difficult to estimate in practice, especially for LLMs. Thus, it is unclear when to switch between the schemes in (TO.. As remedy, one can either treat 1/ LL as a hyperparameter akin to GD, or, adhere to the balanced update by choosing S. = S, for Vn. The latter results in a continuous loss function and upper bounds as sketched in Figure Since this adjustment only affects the region where 1) tiny, still allows for larger 1) to improve convergence. For simplicity, the balanced update is adopted thereafter.
Adaptive optimizers such Adam (66 and AdamW [40] are default to optimizing large models, which adjust the update using the first moment and entrywise second moment estimated from the running average of stochastic gradients. When refactoring (A,, B,) to (A,P,, B.P, T). the first moment estimator can be transformed accordingly, while the second moment generally intractable due to the entrywise square. To tackle this challenge. we "refactor back" the low-rank matrices after the GD update in (3) by right-multiplying P, and P This gives an alternative update
Ac+1 := (A, + AAOP, = A, (VS
(l
B = (B, AB.)P = B, VV())) (OWW.TAAC A.S.
(11b)
whose axes align with the original (A,, B.), thus waiving the need to transform the moment estima- tors. This observation prompts the view of refactoring as GD with preconditioning matrices $. and S.. As side benefit, this also eliminates the need to compute P, from S..
5

---

Table I: Additional complexities introduced by LoRA variants
Method
Time
Space
LoRA forward/backward
(mmm)
2(mm)
LoRA-Pro 61
O(m²r + (m + +
Omm + (m + +)r)
LoRA-RITE 67
((m + n +r)r") O(m + n +r)=") O(m + +)))
O((m + n +)rr)
RefLoRA (Thm.
O(r²) O(1)
RefLoRA-S (Thm.
Interestingly, this reformulation offers an alternative interpretation of RefLoRA as Riemannian optimization over the quotient manifold M, = (Rmxr x xR,x*) /GL(r) [5], where the quotient is wrt equivalence relation (A, B) (AP, BP-T), P e GL(r), and the subscript denotes full-rank matrices. On this manifold, 11 can be equivalently derived from the Riemannian metric
J(A.B) ((G A: GB), (Z Zn) = SG ZA)F (G ZZ)F..
(12)
Denoting
9 as S(A, B), it can be verified that the metric (12) is congruence- innaarian
S(AP,BP-)) S(AP, BP )P- = (A, B)P-T. Moreover, it ensures the weight update always lies in the horizontal space altering the equivalence class, and never wastes efforts the vertical (gauge) directions moving within the same equivalent group; see proof in Appendix [B.7 As shown in the next section, this steepest descent on horizontal space brings about significant faster convergence.
Next, we present three key RefLoRA properties.
Balanced refactoring A A, B, can be achieved per iteration upon setting P.P. S as stated in Section 3.2 It is worthwhile pointing out that SVD-based initializations including inherently satisfy A, A B, B When is small, the balance tends to hold approximately even without refactoring, which partially explains why empirical convergence is fast during the early epochs. Analytically, balanced refactoring maximizes the potential loss reduction as formalized in the ensuing Theorem.
PiSSA [41] and LoftQ [4
Theorem 4. The solution S, = St given by ( minimizes the lower bound
0 V. AA,)
(V (W.))WW)))).))).. ABi) 2 V | |BS, 11:))
=A((W ( ((W i) with small 1)
The upper bound 0 stems from the low-rank nature of LoRA, which can be reached when V((W.) Null(B,)) ) and V((W)) Null(A,) (ATT)) )); i.e., the stationary points of A, and B, While this upper bound cannot be improved, RefLoRA minimizes the lower bound, and thus leads to a more effective descent in the loss.
Additional computational overhead induced by RefLoRA is as small as O(mm + n + +))²) in time and (()²) in memory, thanks to the decoupling of V((W,) in Proposition ② Compared to the forward/backwarr overhead (mm)) of LoRA, the extra complexity introduced by RefLoRA is relatively minimal. In contrast, -ro [61] suffers from O (m + n + r)r?) time and Om22 + (m + n + r)r) space, whereas LORA-RITE 67 requires O(m + n + r)r) memory for polar decomposition. Despite the low complexity of RefLoRA, further curtailing the extra cost can be beneficial for resource- -imited applications. The next theorem restricts S, to a scaled identity s,I, with 8 0, and derives result analogous to Theorem 3
Theorem 5. Consider
( confined with S, stIr, 8, E RR + Under Assumptions D and 2 it holds
B.lle A
in Z IA or <
B
(13)
ff < n 2||A B.ll
A 1
This simplified refactoring (Re(LoRA-S) enjoys further reduced complexity of O( + n)r) time and O(1) space; see Table for summary. Compared to Theorem 3 the scalar s; only has two global optima when 1y is too small. Note that this simplification only guarantees a weaker balance | = |B.. However, it can be directly combined with adaptive optimizers by scaling the moment estimators without the second refactoring.

---

109
|r BB
10
PP aiWal
100
Loon
Pobraine
(o-,0000001
100
LoRA (y=0.03)
ScaledGD 0000
(j=0.03
10
RefLoRA (y=0.0
ReiLoRA (g=0.03)
20
100
20
100
Irenation
Deration
(a) Loss vs. iteration
(b) Norms LoRA's factors and gradients
Figure 2: Comparison of LoRA, ScaledGD, and RefLoRA for matrix factorization
Consistent weight updates are always guaranteed for all equivalent factorizations.
Theorem 6. Under Assumptions
2 and for any A; A, let A AW; and AW, be the corresponding weight updates 5 with RefLoRA. It then always holds that AW; AW.
This consistency holds for either S, - S; In this context, we dealt with slow convergence and the non-uniqueness of low-rank factorization with the added benefit of balanced updates. The step-by-step algorithm of RefLoRA(-S) is listed in Appendix C Next, experiments are conducted to verify our findings.
S₁, but not for the lightweight version (1)
4 Numerical tests
This section evaluates the empirical performance of RefLoRA. All experimental setups includ- ing platforms, datasets, models. metrics, and hyperparameters are detailed in Appendix D Our implementation available at at/atps://githhu com/ hhw.gyyygann/gRelllR.....
4.1 Matrix factorization
The first numerical test considers low-rank matrix factorization min Y
ABI where E mmxx is given low-rank matrix [14] It can be viewed as applying LoRA to a single-layer model, and training with whitened data [2] 33 Figure 2a compares the training loss of RefLoRA with LoRA, and another popular approach dubbed ScaledGD [56] which is tailored particularly for low-rank matrix factorization, and has gained popularity among LoRA variants: see e.g., (69] Each method is tested with two learning rates 1) E {0.01, 0.03). While vanilla LoRA converges slowly with the lower learning rate and diverges with the higher one, RefLoRA remains stable and converges markedly faster under both rates, as corroborated also with observations in Figure | Notably, RefLoRA even outperforms ScaledGD, thanks to its balanced update. Figure depdccs the dynamies of LoRA (n = 0.01) by plotting the Frobenius norms of A,, B, and their gradients. It is observed that ||A|| and ||Be|lr are highly unbalanced across iterations, and !|r| as well as VB, fI|r exhibit a sharp change when is small. comparison, RefLoRA maintains A, A. = B, B., Vt, which guarantees that || = tr(ā, Aς) = tr, B.) = -BBBlle. This balance can afford larger learning rates, thus improving the empirical convergence.
4.2 Natural language understanding
Beyond matrix factorization the evaluation here starts with fine-tuning -eBERTaV3-baie (2], masked language model with 184M parameters, on the General Language Understanding Evaluation (GLUE) benchmark [59]. GLUE contains 8 datasets, providing general-purpose evaluation for natural language understanding (NLU) 59 The test setup follows from 23 70 where LoRA rank
r 8, reducing the trainable parameters 1.33M. We compare RefLoRA and its lightweight
variant RefLoRA-S against suite of LoRA variants, including SOTA methods DoRA 65 and
7

---

Table 2: Performance comparison using D bass on the GLUE benchmark dataset. The best results are depicted in solid lines. The score in the last column averages Matthews correlation coefficient (Mee), accuracies (Acc), Pearson correlation (Corr), and matched accuracy (M). The
results are obtained by averaging 5 random runs.
CoLA
SST-2
MRPC STS-B
QQP
MNLI
QNLI
RTE
All
Method
Params
Mcc
Acc
Acc
Corr
Aco/F1
M/Mm
Acc
Acc
Avg
Full FT
184M
69.19
95.63
89.46
91.60
92.40/89.80
89.90/90.12
94.03
83.75
88.25
BidFit
0.1M
66.96
94.84
87.75
91.35
88.41/84.95
89.37/89.91
92.24
78.70
86.20
HAdapter
1.22M
68.64
95.53
89.95
91.48
91.91/89.27
90.13/90.17
94.11
84.48
88.28
PAdapter
1.18M
68.77
95.61
89.46
91.54
92.04/89.40
90.33/90.39
94.29
85.20
88.41
LORA
1.33M
69.82
94.95
89.95
91.60
91 91,99/89.38 38
90.65/90.69
93.87
85.20
88.50
DoRA
1.33M
70.85
95.79
90.93
91.79
92.07/-
90.29/-
94.10
86.04
88.98
AdaLoRA
1.27M
71.45
96.10
90.69
91.84
92.23/89.74 74
90.76/90.79
94.55
88.09
89.46
LoRA-Pro
1.33M
71.36
95.76
90.20
91.92
92.19/89.60
90.23/90.19
94.29
85.56
88.94
LoRA-RITE
1.33M
69.55
95.41
90.93
91.79
92.02/89.42
90.22/90.33
94.42
85.20
88.69
RefL.oRA
1.33M
71.73
95.99
91.42
92.03
92.28/89.70 70
90.23/90.41
94.40
88.09
89.52
RefLoRA-S
1.33M
70.66
95.76
90.44
92.21
92.43/89.89
90.13/90.17
94.16
87.73
89.19
well as baselines falling in the same category with RefLoRA, i.e., LoRA-Pro 661 and LORA-RITE [67] The results can be found in Table is worth mentioning that these datasets are relatively small, so that full fine-tuning (FT) is prone to overfitting, thus leading to worse perfor- mance compared to PEFT methods. RefLoRA and RefLoRA-S outperform all competitors on out of 8 datasets, and present comparable performance to SOTA approaches the rest 3 datasets. Overall, RefLoRA achieves the highest average performance, demonstrating more effective optimization via refactoring. In spite of the simplified refactoring, ReILoRA-S maintains competitive performance, ie., only 0.33% lower than RefLoRA on average, while markedly reducing computational overhead.
AdaLoRA 70
4.3 Commonsense reasoning
which are autoregressive language models with 7B and 8B parameters. We tackle commonsense reasoning tasks following the setup 24 65 Training data are aggregated from 8 datasets listed in Table 3, and test sets remain separate for individual evaluation. These reasoning tasks are intended to push the model beyond pattern recognition, requiring commonsense and knowledge to make proper inferences. The baselines are chosen as DoRA (65), LORA-RITE 667. PrecLoRA [69], and NoRA+ BI. Note that the latter two approaches are variants of ScaledGD (56). sharing similar complexity with RefLoRA.
We further extend our numerical experiments to fine-tuning the LLaMA series [5
58 17
The accuracy comparison is summarized in Table 3 Both ooR and RefLoRA-S. consistently outperform other PEFT methods in out of 6 settings. Even under lower-rank configurations r 16, both RefLoRA and RefLoRA-S continue to lead or match rop-performing approaches, underscoring their parameter efficiency and robustness. These results demonstrate the effectiveness of the proposed RefLoRA(-S), and underscore the potential of optimal refactoring.
4.4 Subject-driven image generation with diffusion models
Akin to LoRA, RefLoRA can be seamlessly integrated into wide range of larger models beyond LLMs. Further numerical tests are conducted on a subject-driven image generation task (48) with Stable Diffusson v1.4 [45). The goal is fine-tune diffusion model using a few user-provided images so that it can generate the same object in various contexts. Specifically, the model is finc-tuned on set of images labeled "a photo of sks dog," and subsequently evaluated by generating images under the prompt "a sks dog eating nachos." LoRA adapters with rank r = 4 are attached to the U-Net component of Stable Diffusion, and experimental setups including hyperparameters are default to those in [48]. The fine-tuning losses for LoRA, LoRA-Pro, LORA-RITE, and RefLoRA are summarized in the table where average and standard deviation are calculated over random runs. It is seen that RefL ooR achieves 14.0%. 13.1%, and 9.5% improvements over LoRA, LoRA-Pro. and LORA-RITE, respectively. addition to quantitative gains in loss, we also observe noticeable improvements in image quality; see Figure g The generations from RefLoRA-tune models show
8

---

Table 3: Accuracy comparison using LLaMA series on commonsense reasoning datasets.
Method
Params
BoolQ
PIQA
SIQA
HS
WG
ARCe
ARCc
OBQA
Avg
ChatGPT-3.5-turbo
73.1
85.4
68.5
78.5
66.1
89.8
79.9
74.8
77.0
LoRA
0.83%
66.42
80.03
77.84
82.88
81.85
79.92
63.40
77.20
76.
PrecLoRA
0.83%
68.96
80.95
77.43
81.54
80.27
78.83
64.16
79.20
76.42
NoRA+
0.83%
69.85
81.83
77.38
82.09
80.03
79.67
64.25
78.60
76.71
LLaMA-7B
32
DoRA
0.84%
69.7
83.4
78.6
87.2
81.0
81.9
66.2
79.2
78.4
LORA-RITE
0,84%
69.82
82.75
78.55
84.72
81.69
82.15
66.23
81.40
78.54
RefLoRA
0.83%
69.60
82.48
79.53
88.25
82.56
81.57
66.64
80.20
78.85
RefLoRA-S
0.83%
70.18
82.48
78.15
87.41
82.08
81.52
65.36
81.60
78.60
DoRA
0.43%
70.0
82.6
79.7
83.2
80.6
80.6
65.4
77.6
77.5
16
RefLoRA
0.41%
69.66
82,43
79,43
87.38
81.22
80.68
65,44
78.60
78.11
RefLoRA-S
0.41%
67.65
81.50
79.07
88.28
81.77
81.23
64.59
78.60
77.84
LoRA
0.83%
69.8
79.9
79.5
83.6
82.6
79.8
64.7
81.0
77.6
PrecLoRA
0.83%
71.47
81.50
78.81
85.97
80.43
81.14
66.55
81.00
78.36
LLaMA2-7B
NoRA+
0.83%
70.52
81.94
79.07
87.66
82.24
82.70
67.06
80,20
78.92
32
DoRA
0.84%
71.8
83.7
76.0
89.1
82.6
83.7
68.2
82,4
79.7
LORA-RITE
0.84%
71.04
82.43
79.79
89.12
84.53
83.88
68.77
81.20
80.10
RefLoRA
0.83%
72.54
83.79
80.04
86.94
84.85
86.36
71.50
80.20
80.78
RefL.RA-S
0.83%
73.36
83.84
80.76
90.02
82.48
84.55
67.92
82.60
80.69
DoRA
0.43%
72.0
83.1
79.9
89.1
84.5
71.0
81.2
80.5
16
RefLoRA
0.41%
71.38
82.43
80.35
90.49
83.43
84.05
69.28
82,00
80.43
RefLoRA-S
0.41%
72.08
83.03
80.45
85.89
83.27
84.30
69.88
82.00
80. 11
LoRA
0.70%
70.8
85.2
79.9
91.7
84.3
84.2
71.2
79.0
80.8
PrecLoRA
0.70%
70.73
85.80
78.86
91.87
83.66
85.10
71.08
82.40
81.19
LLaMA3-8B
32 DoRA
NoRA+
0.70%
71.16
85.10
79.48
92.22
83.35
85.86
72.27
83.20
81.58
0.71%
74.6
89.3
79.9
95.5
85.6
90.5
80.4
85.8
85.2
LORA-RITE
0.84%
74.19
89.44
81.52
95.44
86.74
90.45
80.12
86.60
85.56
RefLoRA
0.70%
75.35
88.74
80.91
95.71
86.66
90.49
80.20
87.40
85.68
RefLoRA-S
0.70%
75.50
89.72
81.11
95.59
87.29
90.99
79.78
86.00
85.75
DoRA
0.35%
74.5
88.8
80.3
95.5
84.7
90.1
79.1
87.2
85.0
16 RefLoRA 0.35%
75.26
88.79
81.37
95.85
85.64
90.11
80.55
86.60
85.52
RefLoRA-S
0.35%
74.92
89.01
80.60
95.75
85.24
90.45
80.89
86.40
85.41
Table 4: Fine-tuning loss (1) for subject-driven image generation DrcamBooth.
Loss
LoRA
LoRA-Pro
LoRA-RITE
ReiLoRA
Avg
0.100 + 0.015 0.099 + 0.015 0.095 + 0.016 0.086 + 0.017
markedly clearer details and better object fidelity, particularly in the mouth and tongue regions, where features are often distorted in outputs from other three baselines.
4.5 Convergence and complexity comparison
Lastly, we evaluate the convergence behavior and computational efficiency RefLoRA comparison with LoRA (33. A-Pr (61), and LoRA-RITH (67]. These tests are conducted on the MRPC subset of the GLUE benchmark using DcBERTaV3- V--bae as the backbone model. For fairness, the learning rate is set to 7 4 x 10 across all methods. Figure 4a depicts the loss &(W,) over 10 fine-tuning epochs. The loss RefLoRA(-S) declines more rapidly and exhibits less fluctuation than the other three methods, which ultimately achieves the lowest value approaching 0. This confirms stability and convergence speed, on par with our theoretical insights in Section The sharper descent in the loss suggests improved optimization trajectories enabled by RefLoRA's principled refactoring.
The comparison computational overhead presented in Figure Time complexity reflected via the fine-tuning throughput (iterations per second; higher is better). while space complexity is measured in GPU memory occupation (lower better). For better visualization, the vertical axes start with non-zero value. The plot reveals that RefLoRA and RefLoRA-S respectively showcase

---

LoRA
LoRA-Pro
RefLoRA LORA-RITE
Figure 3: Images generated from Stable Diffusion fine-tuned with different approaches.
LORA
Throughout (ths)
Merreyy (MB)
0.7
LoRA-Pro
22
8866
3872
2.22
8975
LoRA-Rite
RefLoRA
0000
0.5
ReILoRA-S
8825
Lo89
1.6
0.3
1:30
8726
0.1
1.0
8700
0.0
LanA
Latt-Pa
Lott-Rbb
10
Epoch
Loss vs. epoch
Throughput ( and GPU memory usage (L)
Figure 4: Convergence and complexity comparison
88.5% and 98.7% throughput compared to LoRA, the additional memory of 132MB and < 1MB. In contrast, the throughput of LoRA-Pro and LoRA-RITE are 60.2% and 72.6% of LoRA, requiring 134MB and 140MB extra memory. This is consistent with our complexity analysis in Table I Extended comparisons scaling up to 27B models are offered in Appendix
Conclusion and outlook
This paper introduced refactored low-rank adaptation (RefLoRA), a principled LoRA variant that remarkably enhances the efficiency and stability of fine-tuning large models. By identifying the optimal matrix S, that minimizes the loss upper bound, RefLoRA addressed the key challenges in LoRA by providing balanced updates, improved convergence, and consistently superior empirical performance with affordable overhead. further facilitate scalability applicability large models, RefL-oRA- A-A-S leverages simplified refactoring to minimize the computational complexity. Our future research agenda involves analyzing the convergence rate of LoRA and RefL.ok ARA, and adapting RefLoRA to extensive model architectures such as vision transformers and diffusion models.
10

---

Acknowledgments
This work is partly supported by NSF grants 2220292, 2212318, and 2312547. B. Li is supported by Swiss National Science Foundation (SNSF) Project Funding No. 200021-207343.
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, al. Gpt-4 technical report. arXiv preprint arXiv:2303.08772 2023.
[2] Sanjeev Arora. Nadav Cohen, Noah Golowich, and Wei Hu. A convergence analysis of gradient descent deep linear neural networks. In Proc. Int. Conf. on Machine Learning (ICML), 2018.
[3] Dimitri P Bertsekas. Nonlinear programming. Journal of the Operational Research Society, 48(3):334-334, 1997.
Yonatan Bisk. Rowan Zellers. Jianfeng Gao, Yejin Choi, al. Piga: Reasoning about physical common- sense in natural language. In Pruc. AAAI Conf. Artif. Intel., pages 7432-7439, 2020.
[5] Nicolas Boumal. An introduction optimization on smooth manifolds. Cambridge University Press, 2023.
[6] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
[7] Daniel Cer, Mona Diab, Eneko Agirre, lñigo Semantic textual Semant. Eval. pages 1-14. ACL, 2017.
and Lucia Specia. SemEval-2017 task 1: and cross-lingual focused evaluation. In Proc. Inf. Workshop
[8] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03377 2021.
[9] François Chollet. On the measure of intelligence. arXiv: 191119991, 2019.
[10] Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 2924-2936, June 2019.
for
[11] René Descartes. La Géométrie. Jan Maire, Leiden, 1637.
[12] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms. Proc. Neural Information Processing Systems (NeurlPS), volume 36, pages 10088- 10115, 2023.
[13] Bill Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proc. Int. Workshop Paraphrasing, 2005.
[14] Simon S Du, Wei Hu, and Jason D Lee. Algorithmic regularization in learning deep homogeneous models: Layers are automatically balanced. Proc. Neural Information Processing Systems (NeurIPS), volume 31, 2018.
[15] Ziqi Gao, Qichao Wang, Aochuan Chen, Zijing Liu, Bingzhe Wu, Liang Chen, and Jia Li. Parameter- efficient fine-tuning with discrete fourier transform. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
[16] Ian Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT press Cambridge, 2016.
[17] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Llama 3 herd of models. arXiv preprint w:Xiv22407.2178. 2024.
2024.
[18] Demi Guo, Alexander M Rush, and Yoon Kim. Parameter-efficient transfer learning with diff' pruning. arXiv preprint arXiv:2012.0746: 2020.
[19] Yongchang Hao, Yanshuai Cao, and Lili Mou. FLORA: Low-rank adapters are secretly gradient compres- In Proc. Int. Conf. Machine Learning (ICML), 2024.

---

[20] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large models. In Proc. Int. Conf. on Machine Learning (ICML), volume 235, pages 17783-17806, 21-27 Jul 2024.
[21] Pengcheng He, Jianfeng Gao, and Weizhu Chen. DeBERTav3: Improving deBERTa using ELECTRA-style pre-training with gradient-disentanlled embedding sharing. In Proc. Int. Conf. on Learning Representations (ICIR), 2023.
[22] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly, Parameter-efficien transfer learning for NLP. In Proc. Int. Conf. on Machine Learning (ICML), volume 97, pages 2790-2799, 09-15 Jun 2019.
[23] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
[24] Zhiqiang Hu, Lei Wang. Yihuai Lan. Wanyu Xu, Ec-Peng Lim, Lidong Bing. Xing Xu, Soujanya Poria, and Roy Ka-Wei Lee. LLM-Adapters: An adapter family for parameter-efffciee tine-tuning of large language models. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2023.
[25] Nam Hyeon-Wowd Moon Ye-Bin, and Tae-Hyun Oh. FedPara: Low-rank hadamard product for communication-effieen federated learning. In Proc. Int. Conf. on Learning Reprexentations (ICLR), 2022.
[26] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Proc. Int. Conf. on Learning Representations (ICIR), 2015.
[27] Toshiaki Koike-Akino, Francesco Tonin, Yonglao Wu, Frank Zhengqing Wu, Leyla Naz Candogan, and Volkan Cevher. Quantum-peft: Ultra parameter-efficient fine-tuning. arXiv preprint arXiv:2503.05431 2025.
[28] Dawid Jan Kopiczko, Tijmen Blankevoort, and Yuki M Asano. VeRA: Vector-hased random matrix adaptation. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
[29] Brian Lester, Rami AI-Rfou, and Noah Constant. The power of scale for parameter-eflcient prompt tuning. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 3045-3059, November 2021.
[30] Bingcong Li, Liang Zhang, and Niao He. Implicit regularization of sharpness-aware minimization for scale-invariant problems. In Proc. Neural Information Processing Systems (NeurIPS), volume 37, pages 44444 44478, 2024.
[31] Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the crucial role of initialization for matrix factorization. In Proc. Int. Conf. Learning Representations (ICIR), 2025.
[32] Chi-Kwong Li and Roy Mathias. Geometric means. Linear algebra and its applications, 385:305-334, 2004.
[33] Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. Proc. Conf. Assoc. Comput. Linguist. Meet, (ACL),, pages 4582-4597, August 2021.
[34] Yixiao Li, Yifan Yu, Chen Liang, Nikos Karampatziakis, Pengcheng He, Weizhu Chen, and Tuo Zhao. LoftQ: seee--iiin--unnn--waar quantization for large language models. In Proc, Int. Conf. on Learning Representations (ICLR), 2024.
[35] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReLoRA: High-rank training through low-rank updates. In Proc. Int. Conf. on Learning Representations (ICI.), 2024.
[36] Zhaojiang Lin, Andrea Madotto, and Pascale Fung. Exploring versatile generative language model via parameter-efffcient transfer learning. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP). pages 441 --55, November 2020.
[37] Vijay Lingam, Atula Tejaswi, Aditya Vavre, Ancesh Shetty, Gautham Krishna Gudur, Joydeep Ghosh, Alex Dimakis, Eunsol Choi, Alcksandar Bojchevski, and Sujay Sanghavi. Svft: Parameter-cfficient fine-tuning with singular vectors. arXiv:2405.1959) 2024.
[38] Weiyang Liu, Zeju Qiu, Yao Feng. Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. et al. arrametee-efficieen orthogonal finetuning via butterfly factorization. arXiv preprint arXiv:2311.0622 2023.
12

---

[39] Ziyue Liu, Ruijie Zhang, Zhengyang Wang, Zi Yang. Paul Hovland, Bogdan Nicolae, Franck Cappello. and Zheng Zhang. Cola: Compute-efficient pre-training of Ilms via low-rank activation. arXiv preprint urXiv:2502.10940. 2025.
[40] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proc. Int. Conf. on Learning Representations (ICLR), 2019.
[41] Fanxu Meng, Zhaohui Wang. and Muhan Zhang. Pissa: Principal singular values and singular vectors adap- tation of large language models. In Pruc. Neural Information Processing Systems (NeurIPS), volume 37, pages 12 12103--12107,, 2024.
[42] Todor Mihaylov. Peter Clark, Tushar Khot, and Ashish Sabharwal, Can suit of armor conduct electricity? A new dataset for open book question answering. arXivv / 880990779, 2018.
[43] Theo Putterman, Derek Lim, Yoav Gelberg, Stefanie Jegelka, and Haggai Maron. Learning on loras: Gl-equivariant processing of low-rank weight spaces for large finetuned models. arXiv:2410.04207, 2024.
[44] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't 't know: Unanswerable questions for SQuAD. In Proc. Conf. Assoc. Comput. Linguist. Meet. (ACL), pages 784-789, 2018.
[45] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. Proc. Conf. Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[46] Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. AropterDrop: On the efficiency of adapters in transformers. In Prec. Conf. on Empiricul Methods in Natural Language Processing (EMNLP), pages 7930-7946, November 2021.
[47] Walter Rudin. Principles of Mathematical Analysis. McGraw-Hill New York, 3rd edition, 1976.
[48] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proc. Conf. Computer Vision and Puttern Recognition (CVPR), pages 22500-22510, June 2023.
[49] Keisuke Sakaguchi, Ronan Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 64(9):99- 106, 2021.
[50] Maarten Sap. Hannah Rashkin, Derek Chen, Ronan LcBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions. UrXiv:1904.09725 2019.
[51] Steffen Schotthöfer, Emanuele Zangrando, Gianluca Ceruti, Francesco Tudisco, and Jonas Kusch. Ge- oloRA: Geometric integration for parameter efficient fine-tume fine-tuning. In Proc. Int. Conf. on Learning Repre- sentations (ICLR), 2025.
[52] Shai Shalev-Shwartz and Shai Ben-David Understanding machine learning: From theory algorithmms Cambridge university press, 2014.
[53] Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 1631-1642, 2013.
[54] Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks, In Proc. Newral Information Processing Systems (NeurIPS), volume 34, pages 24193 20093-2225,,,, 2021.
[55] Ross Taylor, Marcin Kardas, Guillem Cucurull, Thomas Scialom, Anthony Hartshor, Elvis Saravia, Andrew Poulton, Viktor Kerkez, and Robert Stojnic. Galactica: A large language model for science. arXiv preprint (XX::22211.0000 2022.
[56] Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Mach. Learn. Res., 22(150):1 63, 2021.
[57] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models. arXiv preprint 2302. 2023.
[58] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
13

---

[59] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Oner Levy. and Samuel R Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Proc. Int. Conf. Learning Reprexentatins (ICLR), 2019.
[60] Shaowen Wang. Linxi Yu, and Jian Li. Lora-ga: Low-rank adaptation with gradient approximation. In Proc. Neural Information Processing Systems (NeurIPS), volume 37. pages 54905-54993 2024.
[61] Zhengbo Wang. Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. LoRA-pro: Are low-rank adapters properly optimized? Proc. Conf. on Learning Representations (ICLR), 2025.
[62] Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Trans. Assoc. Comput. Linguist.. 7:625-641, 2019.
[63] Adina Williams, Nikita Nangia, and Samuel R Bowman, A broad-coverage challenge corpus for sentence understanding through inference. Proc. Conf. North Am. Chapter Assoc. Comput. Linguist., pages 1112-1122, 2018.
[64] Zhengxuan Wu, Aryaman Arora, Zheng Wang, Atticus Geiger, Dan Jurafsky, Christopher Manning, and Christopher Potts. Reft: Representation finetuning for language models. In Proc. Neural Information Processing Systems (NeurlPS), volume 37, pages 63908 63962, 2024.
[65] Shih yang Liu, Chien-Yo Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-decomposed low-rank adaptation. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
[66] SHIH-YING YEH, Yu- fu-G Hsich, Zhidong Gao, Bernard B W Yang. Giyeong Oh, and Yanmin Gong. Navigating text-to-image customization: From LyCORIS fine-tuning model evaluation. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
[67] Jui-Nan Yen, Si Si, Zhao Meng. Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsich, and Sanjiv Kumar, LoRA done RITE: Robust invariant transformation cquilibration for loRA optimization. In Proc. Int. Conf. on Learning Representations (ICLR), 2025.
[68] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi. Hellaswag: Can a machine really finish your sentenee?
2019.
[69] Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned LoRA fine-tuning foundation models. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
[70] Qingru Zhang. Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for efffmeter-efficient fine-tuning. In Proc. Int. Conf, on Learning Representations (ICLR), 2023.
14

---

A Additional related work
LoRA variants. LoRA has been extended in several directions. For example, recent works further reduce LoRA's trainable parameters [28
37
19 27. As these methods can achieve performance 15
comparable to that of LoRA, they indirectly suggest that the expressiveness of LoRA has not been incorporates quantization in LoRA to reduce memory footprint and computational overhead. There are also works that broaden the applicability of LoRA to pre-training LLMs by e.g., sequentially chaining LoRA modules [35 39] As our method enhances the efficiency LoRA training, we expect that can be seamlessly integrated to such settings as well. Going beyond LoRA, there are also approaches for fine-tuning LLMSs in parameter efficient manner such as 33
fully exploited. Another line of work [2
B4]
29 38 64). Moreover, [30] shows that applying sharpness-awar minimization on LoRA promotes balance between A, and B, These approaches are orthogonal our work.
Broader Impact. The theoretical insights and algorithmic contributions of the current work are broadly applicable across a range of fine-tuning scenarios. Our RefLoRA enhances the efficiency and effectiveness of adapting language models downstream tasks, leading to improved performance in applications such as sentiment classification. This, in turn, can positively impact real-world systems including recommendation systems by increasing accuracy and relevance. However, caution should be exercised when deploying the method for generative tasks. In such settings, the outputs of language models should earefully reviewed, and proper safeguards, such as gating mechanisms, should be considered to ensure safety, reliability, and trustworthiness of the generated content.
Future directions. Due to limited computational resources, our evaluation currently deals with models having reasonably large scale, e.g., LLaMA3-SB. Our future work will include scaling RefLoRA even larger models, such as those with 30B parameters. Another promising direction is to integrate RefLoRA with sequentially chaining, namely [35]. This will further broaden the applicability of RefLoRA for pre-training LLMs.
B
Missing proofs
This appendix presents the proofs that were omitted from the main paper.
B.1 Proof of Lemma
Proof. First, we prove 4 by showing that the two sets in contain each other.
For any pair (A, B,P. it easy to see A,P,( T)T A,B.. Thus have
((A,, B,) A.B. A,B 2 ((A,P,, P. E GL(r)).
Next, we prove the opposite containing relationship. Let (A,, B.) be an arbitrary pair satisfying A, A,B,T It follows that
rank A, rank(A,B))) 2 rank At) + rank(B;) - -r =r rank( (A min(rank(A.)) (A.), rank (B.)) r..
We thus obtain rank( (A, = rank(A, A.B.) = r, and rank rank(B.) =
Since C.)(A, S Col(A.) and dim(Col(A, (A,B.)) rank rA, r rank(A.)) dim(Col(A,)). we have Col(A,B,) Col(A,); and likewise Col(A,B,. CCl((A))
Then, the condition A. A.B.T leads
Col( (A.) = Col (. = Col(A,, Col(Ar).
This suggests, there must exist an invertible matrix P; E grxr such that A. A,P
As a result, we have A,B.B, = A,B.T. Multiplying P- on both sides and taking transpose yield B, -.P.T This suggests
( ((A,, B.) A.B.T A.B, S (AA,P, P. GL(r))
15

---

which completes the proof of (4)
Additionally, it follows from the definition of A W, that
AW. = A + AA, + AA,A
A V() VV BB.B B. v((W.)
VV MAPPP. A, (W. VV B,P, PPP +
A.P. AB, + AA. (P.P. B AB,
(14)
where (a) uses 3 and that W." + A. = WT A,B; Wi, (b) is due to A, = A,P, and
B, B,P-T, and (c) leverages
Comparing with 2 it can be easily seen that P.P. I, results in AW, = AW.
B.2 Proof of Proposition
Proof. For 3 it follows from Assumption 2 that
(W, + AW.()) (S.) ((.) + (VZ(W,), AW, + I. " Wl)
((W)) (V.(.)) A. + AAS, B AA,AB,,),
2 A,S, + A B - AA,B lF
@ (Ve(W)) AS + AA, BT) / 2 A,S,AB 11 + |AA,S, B, 11+
L
2(A,S,A A B AAA + AA, BB. AA,B )P + Const.
g L V((W)) + A.S.
F + VE(W.) A BB
(15)
2 L
AA AA, B B 2(A,S,AB + AA, B, AA,BB )p + Const.
where (a) relies 14 (b) expands the squared Frobenius norm and merges terms independent S, of into Const.; and c) utilizes completing the square and merges the constant terms.
Next, we bound the four terms in 15)) Using the definition ( of AB, the first term is relaxed via
1 V((.)) A V((W
(C(W:)+ASSAN
VV 112
Tap Im A.S.A,
(16)
Likewise, the second term is bounded through
| V.(.)) AA.S, 'B. S <
- I, B,S Tnj
(17)
F
Again using the definitions of AA, and AB,, the third term in satisfies
( (A,S,AB] AA,S
BB), BB,)
7/2 (A.S.A) V((W)) (W.,)
< 7 A.S. (/W. 1lp
7² V (B. A.S.A,
||
B.S,
1|p (18)
where (a) follows from Cauchy-Schwarz inequality.
Regarding the last non-constant term in (15 it holds
SA, + AA,S, AA,B )
16

---

(AA. + V, B V. v((Wo))
VV(W, s (IA.S / ' | (B ve(:))l
S < I (| A|,A
+ || B B, [| lF = =
(19)
When performing Tine-tuning from pre-trained weight, both 7/ and [|V are observed to be As result, (99 is dominated by (18). and is neglectable practice; see also 60
tiny
Plugging
16
19
into
15
yields
2
Im A.S.A. L I Tnp
12 F + - B.S I.
(WW + AW.
< ((((),)
2
F
2
A 11 B.S, 1.5''BBll+ + +O(LTT) Const.
(
L
|V((W |A.S
2 2 (Im, A.S.A)) )F + | l1
Ln
2
(In, B, B + A 11 B.S, B [|p] +O(LnT) + Const.
Ln
(6)
L
IVE(W.))) [| 11 + |B,S, "B, 11p) Lnj
2
(1)
$ + B.S, * [1:] + +(())) + Const.
2n
[ (|)
+ + 11)) Ln
x
(I I |B.S, $ [1)) + O(T)))) + Const.
(d) 2 Wll ||A.S l| |B..s $ | Ln 1 O((LT²) + Const.
where (a) expands the squared Frobenius norm; (b) follows from (Im: A.S.A,) ) (,(A,S,AT) | /22 I|:: (c) because A,S,A, is SPD, thus |AA,A, || ,E, AA,,S, AASAA)
tr (A A,
= [Σ, (EA 1/1/2
S
E, (,A))) = tr(A,S,AT) ||A.S//2 [[:: and (d) utilizes completing the square.
B.3 Proof of Theorem 3
Proof. For SPD matrix S., the eigendecomppsitionn gives S, Q diag ) where Q
((r) and X is wlsment-wise positive. Thus, the objective (8) can be reformulated as
min (IA,0%% diag* (A. 2f x
ST
+
B diag
Ln
min co((())) -0
(IA.0f% diagà ) | diag
(255))
1 1n ((Q)x;))) (20)
In
min eO(r) 20
B Q B
Since f(Q/, X²) is continuous w.r.t. both Q E Or) and X > 0, its infimum can be determined
by checking the limit of f as it approaches the boundary of the open set sT and analyzing stationary points in the interior [7].
while (0, +00)" is open. It follows from 20 that, for any given Q E O(r), if some [X approaches 0 or +oo, the objective f(Q consequence, the minimum must be attained inside the interior of ST $
Notice that O(r) is closed
goes to +o0. As a
'Typically, O(10 and |Ve(W:)]la
[V((W.)
0 per matrix.
17

---

Next, the stationary points are investigated under the following two cases.
Case 1: 1) 2 1/(C,L) or 1 < 0.
Defining g(S,) := ||A | + [ [|. it will be shown that mins,es, g(S.) Ö₁ and the
corresponding minimizer is uniquely St.
From the stationary point condition we have
Vg((S)) A, A, S 'B B,S,"
S.A, A.S, = B, B.
(21)
A.) (AT
= (A) (AAA)
[A] [A=
B, (A, A.)
Since A,) (A, A,) is also SPD when S, 8 and rank AA)) r, its solution is uniquely
given by the positive square root
(A A.) A SA A,) [(A] A,) & B B. B (A A,)
Left- and higttyuuliplyiig (A, A.) 1/2 results in
(A, A,) [[A7 A.) A, B,(A, (A)
Given that (Sa)) approaches + on the boundary of $4 and has only one stationary point S,, its
minimum is reached uniquely at St, i...
min S.EST
g(S,) = ||A.S 1 + I $12 = (A)A, ir
A S, + tr, B B₂)
2tr A.S,)
A.A. 1 B
(22)
where (a) relies the eyclic property of trace. and (b) utilizes 21
Next, we prove the minimum value in (2 can be equivalently expressed as ||A,B] ||..
Let A, = U.E.VA AT be the economy-sized SVD. By the definition of $,, it follows that
A A | Lr
(A.(ATA.) (
[(A] A A B, B. (A
AT)
@ tr ([(A) A. BA A ) = Σ X ([(AT A.) (.
(AATA)) Σ (,,TTA,,))
(V B EOT BJ)
o. (A,
J.B.I.
where (a) relies the eyclic property of trace. and (b) is from the definition of nuclear norm.
From the condition E (- U (1/(C,L), we have g(S,) - 1/(Ln) 2 0, VS, E S thus
arg min S,ES;
(1 12
1
arg min (9(S.) S.53,
)
1 1 arg min ((,))
S.
Tmp
Tmp
S
Case 2: 2< <
18

---

For this case, it holds ming,es; - < O, hence the minimum of ( is reached when g(,)) 1/(). In particular, for any S₁ satisfying g(S,) (Ln),, it is possible to find scaling factor 7/2 > by solving a quadratic equation such that ((%) = (Lm)-
As an example, we consider scaling S, to attain g(%;S, = 1/()) worth emphasizing that, for any choice 1) (0, 1/C,L),, this solution universally valid because g(S,) < 1/(L)) always hold in this case.
By the definition of g(z)). the sought g(%),) 1/())) is equivalent the quadratic equation
7t A l +
[ 11 7t
Tn
Solving this equation gives
41 A,
B.S. |
7
+ C.Tan
12T2122
2||A,S l
which concludes the proof.
B.4 Proof of Theorem 5
Proof. Plugging S. syhr, 8t R++ into ( incurs alternative objective
min +1E3+
h(x)) (IA, + |
1 Ln
(23)
To solve this quartic optimization problem, we consider the following two cases.
Case 1: 17 A/ [B] or < O. In this case we have
| s B 2 ||I 2
Lij
Since (.)² monotonically increases on R4,, the unique global optimum of
23 is
arg min | F |B Ly
|
s,ER
Case 2: 17
The proof for this case relies on Descartes' rule of signs; ef. Theorem The gradients of the objective function h is given by
2
1 1((s:) = A| +1 2| Lnj
|
11 +
Ly
= 2s, (1 |
2
B, 11
2| 11
(24)
Ln
Ln
where the quartic polynomial in the parenthesis has coefficients with signs - +. -). Using Theorem g the gradient (4) has or positive roots. That says, (23 has either or stationary
point(s).
Notice that the objective only if ||A+]] + |B,
23
must be nen-ntgaeive, and its lower bound of 0 can be reached if and
s,
1L() O. Solving this quadratic equation over s, IR leads
to two global minimum
In +
4
12
B
1,
(25)
2
A
Hence, (23 must have 3 stationary points. As min..>0 |A 1| + B, S 21|
B.IlF
<
1/(Ln), the remaining stationary point is local maximum
B, F
A
Lastly, the objective (23 a continuous function of 8; E (0, +00), and tends to +00 when 8
and s, + +oc. We conclude it only has two global minimum given by (25).
19

---

B.5
Proof of Theorem 4
Proof. By the definition
A, and B,, it holds
(W).(W)).A.))) (Ve(W,) B,, B.) V.)
- V.( I1 <0
Likewise, have
(e,,((W),,B.)))) Vc A,S? 1 $0.
As consequence,
(VA, (W.), (A,), (VB, ( A (IV
- S /(((W,) A.S
M \V(( + B,S, $ 112).
follows from 22 that S, = S, is the unique global minimum of this lower bound.
B.6 Proof of Theorem 6
Proof. As A. = A.B.T = A.B. = A,B, Lemma suggests there exists Q E GL(() such that A = Q, and B' Q,Q, Further, to prove Theorem 6 it is sufficient to prove that Q (rr). Since A, has full rank, the condition Q (Or)) is equivalent to A,A,T = AAĀ,
By Lemma 7 this can be simplify as
A,A,T A,S, A,(A, (()A,TA)~-A AA'" A,B, A
Again using Lemma 1 and that A,B = A,B,, there exists P, E GL(r) such that A A,P, and B = B,P," which yields
A,A;T A,P, A.P. (P. " B.P, PPPT B
A,(ATA)) "P, (P. A. A,B. A
Notice that
[ T
(P. T A. A. B.P, TPPTT]" P, T (P. T)P A AB, B.
P. (P..A. PP.
A, B.P. T) = (A, A,B,
It then follows
A = A.) (A, AAAAA B.) A. @ A.S. = A.A.
where (a) applies 26 with X A,A, A, and Y B.B.
The proof is thereby completed.
B.7 Proof for Riemannian metric and gradient
We first show that the Riemannian metric 12 leads to update II The subscript is dropped for simplicity. By definition, the Riemannian gradient (G GB) should satisfy
IiA.B) ((G GB), (Z,, ZB)) = (V>((W), Za) (V (VBl(W), ZB)r, VZA E mXr ZB E Ruxr.
Using 12 the Riemannian gradient thus given by
SG A (V ZA)F, VZA + GA -= V
(G Zn)F = (Vn((W). ZR)F, VZ + G Vn((W) V((W)TAS.
We next show that our metric 12) guarantees the update of (A, B) is always on the horizontal space, and has no component on the vertical space. First, the vertical space at (A, B) is T =
(A.B)
((AX, BB X E Brxr], which can be easily verified via
(A + AA TB AAB +0(E))
20

---

Thus, for update (AA, AB) to be purely horizontal, we must have
J(A.B) ( -BX ") (AA, AB) = O, AA E R
From plugging in the update AA (W) and AB -I)((W) BS; gives
((A,))((A,, -BX 1 (AA
-AXS, -VVe + (-BX (-1X - -V(EW AS)
=0
which completes the proof.
B.8
Useful facts
Lemma 7. Under Assumption it holds
S; = (A) B. - A, B.J
A
Proof. It holds for any X, Y 4 that
x x YX
Taking square root on both sides and left-multiplying X-1 give
(26)
Letting X A,A, A, and Y B, B, 26 completes the proof.
Theorem 8 (Descartes' rule of signs (1). The number of strictly positive roots (counting multiplicity) of polynomial h is equal to the number of sign changes in the coefficients of h, minus a nonnegative even number.
C Algorithm pseudocodes
The Appendix provides the step-by-step pseudocodes for RefLoRA and its lightweight version RefLoRA-S. As our algorithms rely on Assumption but LoRA intialize B, 0 by default, one can either utilizes other full-rank initialization schemes [41 [34 60), or warm up the algorithm for l.p > 0 steps until A, and B, both satisfy full column rank. Without loss of generality,
pseudocodes WPt A,B.T
assume Ag and Bo already full-rank, and define W, = WPt A,B,
For RefLoRA-S, is worth noting that adaptive optimizers rely on the first and second moments of gradients V ((W.) = VE(W.)B, -
Vë, (W,(( and (B,((W,) = v((W,) A, Vs, VB, BB,&&WW ((W,). As consequence, one can correspondingly scale the gradient moments of A, and B, to render the gradient moments A, and B,, thus removing the need for preconditioning.
D Experimental setups and hyperparameters
This appendix provides the detailed setups well as hyperparameters used in our numerical tests.
D.I Platforms
All the experiments are conducted on a desktop equipped with an NVIDIA RTX A5000 GPU, and server with NVIDIA A40 and A100 GPUs. The codes for synthetical tests are written with MATLAB, and codes for M.eM-tel experiments are in PyTorch. In addition, our implementation of L.LM tests are based 24 70).
21

---

Algorithm 1: Refactored low-rank adaptation (RefLoRA)
Input: Loss L, pre-trained weight WPt, maximum iterations T,, and learning rate r7. Initialize: full-rank Ao and Bo-
for t o....,
do
2
Compute S. = (A, A
[[A] A.) (AA
adaptive optimizer then
Precondition gradients G. = B.S GB V((W,) A,S.;
Update Ac+1 = AdaptOpt A,, 77 GA, (), B,+1 = AdaptOpt GB,
else
Refactor A, = A,51/ B, = B,Š-1/2;
1/2 1/2
Update A,+1 A, -VV-VV((,,,,, Bu+1 B. -VVV)) A
end 10 end
Output: Ar and By.
Algorithm 2: Low-rank adaptation with simplified refactoring (RefLoRA-S)
Input: Loss f, pre-trained weight WPt, maximum iterations T, and learning rate 17. Initialize: full-rank Ag and Bo- for / o..., T do
Compute 6, = |B, F AclE: Refactor A / B. = 1/V3B,
if adaptive optimizer then
Update A,+1 = AdaptOpt/ A, 7. (., with first and second moments scaled by 1/v5 and 1/s,, and B.+1 = AdaptOpt( V((W,) A,,t) with first and second moments scaled by v and š;;
else
Update Act1 = A. - WB, B B. -V A.
end
end
Output: AT and By.
D.2 Setups for visualization in Figure 1
Figure D considers linear regression
min Y WX W
where E Bunk and E mR are random data matrices with entries generated from standard Gaussian distribution (0, 1). The corresponding LoRA objective is
min A.B
Y (WPt + AB T I
(27)
For simplicity. set = 2, r and WPt 0. Ao and Bo are randomly initialized from (0 from.N(O. 10) and (0 1))
Although it is well-known that linear regression has closed-form solution through least squares, we consider here because Lipschitz smoothness constant can be computed analytically as L = XXX 1|,, allowing us to track the upper bound (7).
D.3 Setups for matrix factorization
Matrix factorization aims to solve
min muin-ly- Y A.B
22

---

where Y E R is a given low-rank matrix. It can be viewed as a special case of the linear model (27) with n, X I,, and W O. This test utilizes m 128, 100, and The low-rank matrix Y is generated from standard Gaussian (0, 1), and then truncated to the largest singular values. Following the standard LoRA initialization, Ao is sampled from standard Gaussian A (0, 1). and Bo 0. Standard GD is employed in LoRA for Vt, while RefLoRA A and ScaledGD are applied when t 0.
D.4 Details of Datasets
Our evaluations carried out commonly-use datasets in the literature.
GLUE benchmark. The General Language Understanding Evaluation (GLUE) benchmark is designed to provide general-purpose evaluation of natural language understanding (NLU) [59]. Those adopted in our work include
MNLI 63
(Multi-Genre Natural Language Inference) tests model's ability to perform natural language inference across different genres text.
SST-2
53
(Stanford Sentiment Treebank) is a sentiment analysis dataset with binary labels.
MRPC 13
(Microsoft Research Paraphrase Corpus) focuses on paraphrase detection; i.e. identi- fying whether two sentences semantically equivalent.
CoLA 62] (Corpus of Linguistic Acceptability) requires models judge whether sentence is
linguistically acceptable.
QNLI [44] (Question Natural Language Inference) is a question-answering dataset converted to binary inference task.
OQ Question Pairs) contains pairs of questions and the task is determine if they semantically equivalent.
RTF [Recognizing Textual Entailment) consists of sentence pairs for textual entailment inference.
STS-B 7
(Semantic Textual Similarity Benchmark) evaluates the textual similarity of sentence pairs continuous scale.
These datasets present comprehensive benchmark to test general-purpose language models and are distributed under various permissive licenses. A summary of these datasets can found Table
Table 5: Summary of GLUE benchmark datasets
Dataset
Task type
ITriinl
TTssll
Metric(s)
MNLI
Natural language inference
393k
20k
Matched & mismatched accuracies
SST-2
Sentiment analysis
67k
1.8k
Accuracy
MRPC
Paraphrase detection
3.7k
1.7k
Accuracy, F1
CoLA
Acceptability judgment
8.5k
1k
Matthews correlation
QNLI
QA/NLI
105k
5.4k
Accuracy
QQP
Paraphrase detection
364k
391k
Accuracy,
RTE
Textual entailment
2.5k
3k
Accuracy
STS-B
Semantic similarity
7k
1.4k
Pearson Spearman Correlations
Commonsense reasoning. This category includes tasks that require models apply everyday knowledge and infer beyond explicit textual information. These datasets are vital for evaluating a model's ability reason about physical and social contexts. The considered datasets include
BoolQ 10 (Boolean Questions) is reading comprehension dataset of yes/no questions paired with Wikipedia passages, testing model's ability to extract and reason over text.
WG 49 (WinoGraadde) a challenging dataset designed to reduce annotation artifacts present in traditional Winograd schemas.
Data-att--aaaeeeaaa Dataset-Release. Reweeeeleeeeeee Beelease-Questionn-
httpe://paperssttccod..co http:///ttttctcooo/////ttt//ee/e
23

---

PIQA
4
(Physical Interaction QA) assesses knowledge of physical commonsense and intuitive
physics.
SIQA
50
(SOCIAL-I-QA) focuses social interaction and social commonsense reasoning.
HS 168
(HellaSwag) aims at evaluating grounded commonsense inference for multiple-choice sentence completion.
ARC
g (A12 Reasoning Challenge) contains grade-school level science questions, split into ARCe (ARC-easy) and ARCe (ARC-challenge), based on difficulty.
OpenbookQA 42 involves multiple-choice science questions that require integrating common- sense and scientific facts.
These datasets are drawn from multiple domains and present diverse reasoning challenges. All datasets used in our work are publicly available under open friendly licenses. Table summarizes these datasets.
Table 6: Summaay of commonsense reasoning datasets
Dataset
Task type
ITrainl
iTestl
Metric
WinoGrande
Coreference resolution
40k
1.3k
Accuracy
PIQA
Physical reasoning
16k
3k
Accuracy
SIQA
Social reasoning
33k
Accuracy
HellaSwag
Sentence completion
70k
10k
Accuracy
ARC-easy
Multiple choice QA
2.3k
1.2k
Accuracy
ARC-challenge
Multiple choice QA
2.6k
1.2k
Accuracy
OpenbookQA
Open-bood QA
5.0k
500
Accuracy
D.5 Details LLMs
We summarize the adopted language models in our evaluation. All model checkpoints are obtained from HuggingFace.
DaBERTaV3-bane (1 is transformer-based language model with 184 million parameters. The model checkpoin released under the MIT license.
GPT3-turbo is proprietary language model accessible via the OpenAl API. While the model weights are not publicly available, its tokenizer is open-sourced under the MIT license.
LLaMA-7B [57 is decoder-only transformer model, which is part of the LLaMA (Large Language Model Meta AI) series. The chekpoin 6 is intended for research use under a non-commercial license.
LLaMA2-7B 58) is refined successor to LLaMA. Its checkpoin is under permissive license for both research and commercial use.
LLaMA3-8B is part of the third generation of LLaMA series. The checkpoin is released under permissive Meta license for both research and commercial applications.
Stable Diffusion V1.4 455 is latent text-to-image diffusion model released by Comp Vis, Stability AI, and Runway. The checkpoin is made available under the CeaativeML-OpeRRAAe license.
D.6 Hyperparameters for fine-tuning L.MMs
GLUE setup follows from 23 70), where LoRA applied all linear modules. The results for
Full FT, BitFit, Adapters, LoRA, DoRA and AdaLoRA in Table 2 are taken from (70), while the remaining results are obtained from our experiments. We fix the LoRA rank as r = 8 and scaling
httpp///huuginnnseeecc/sseeaaaar

---

factor = 8, and search the optimal learning rate from {1 X 10 10 8,8 x 10 x 10 10-4). Batch size is fixed as 32 for all datasets. The default A 440 optimizer and linear learning rate scheduler are utilized for all tests. Other hyperparameters are gathered in Table 7 Note that RefL.oRA requires less fine-tuning epochs compared other LoRA variants due to its fast convergence.
Table 7: Hyperparameters for GLUE benchmark
Dataset
T)
Epochs
Warmup steps
Max seq. len.
Cls. dropout
Weight decay
MNLI
4 X 10 10-4
1000
256
0.15
0
SST-2
10-3
500
128
0.01
MRPC
4 X 10-4
10
50
128
0.01
CoLA
1 X 10-3
100
64
0.15
0
QNLI
x 10
500
512
0.1
0.01
QQP
1 10-3
1000
320
0.2
0.01
RTE
8 X 10
10
50
320
0.2
0.01
STS-B
1 10
10
100
128
0.1
0.1
where LoRA is attached to linear projections in transformers' self-attention and feedforward modules. The results for ChatGPT, LoRA, and DoRA in Table 3 taken from [65]. while the remaining
Commonsense reasoning setup is from 22 65
acquired through our tests. We test with LoRA ranks r E {16, 32} and scaling factor fixed as a = 2r. The learning rate is tuned from 10-4 10-4 1,3 x 10-4). The tuned learning rate per model in Table g The number of fine-tuning epochs is set to 2, and batch size is 16 for all tests. The default AdamW 40 optimizer and linear learning rate scheduler are utilized with 100 warmup steps. Dropout rate is 0.05. The remaining hyperparameters are set to the default values used in [24].
n (8 10-3
be found
Table 8: Learning rates for LLaMA models
LLaMA-7B
LLaMA2-7B
LLaMA3-8B
Rank
16
32
16
32
16
32
Learning rate f 2 x 10-4 2 10-4
3
10-4
2 10-4
8 10-5
1x 10-4
We also attempted to include LoRA-Pro [6];; however, it incurred runtime costs that exceeded
the limits of our resources. Similar scalability concerns have been reported by other users in the community; see e.g., issue #6 on the LoRA-Pro GitHub repository. For this reasons, we omitted it from our final results.
Subject-driven image generation utilizes the default setups in 48 Specifically, LoRA is applied to the to_k, to_q, to_v, to_out, add_k. add_k_proj. and add_v_pros modules of U-net with rank and scaling factor a = 4. The diffusion model is fine-tuned with batch size 1 and learning rate n 10-4 for 500 iterations. Adam W with 0.01 weight decay adopted as the optimizer.
Scaling to larger models
Scalability does not confine the applicability of RefLoRA. In larger models, the major runtime bottleneck is the forward pass and gradient computation. In comparison, the additional overhead of RefLoRA becomes negligible. compares the computational overheads of LoRA variants under various model sizes, where gradient checkpointing is turned on for Gemma3-27B 17mmaa3-7BB-a so that the model can fit within single NVIDIA H100 96GB GPU. Notably, the runtime and memory gap between LoRA and RefLoRA narrows as model size increases. These findings indicate that RefLoRA remains practical and efficient as LoRA, especially for larger models.
25

---

Table Throughput (it/st) and GPU memory consumption (GBJ) under various models sizes.
DeBERTa V3-base
LLaMA3-8B
Gemma3-27B-pt
Method
Tp.
Mem.
Tp.
Mem.
Tp.
Mem.
LoRA
1x (2.26)
8.73
1x (1.53)
36.21
1x (0.58)
64.28
LoRA-RITE
0.73
8.87
0.63
37.37
0.95x
64.28
RefLoRA
0.88
8.86
0.87x
36.35
0.98x
64.28
RefLoRA-S
0.99
8.73
0.99x
36.21
0.99x
64.28
NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims in the abstract and introduction clearly stated, which
supported by both theoretical analysis in Section 3 and experiments Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: The limitations RefLoRA are acknowledged the outlook of Section 5 and future directions in Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
26

---

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed
penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper includes clearly stated Assumptions and detailed proof's in Appendix [ for all theoretical results.
Guidelines:
The answer NA means that the paper does include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Appendix contains full details of datasets, hyperparameters, training setups, and platform specs.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm.
27

---

(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The models and datasets used are publicly available with links in Section Codes for reproducing the main results are provided as supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Sablic/guidess httpi//gguddee//ooeuubsssssss
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurlPS code and data submission guidelines (https ctfps..cc/pubbll httppppwwpppppppppppppppppaaaaep Stwp:iddee///eSSdmbmeeeeeeeee
ssioioolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed training settings including datasets, hyperparameters, and optimizers are documented Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
28

---

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: While results are averaged over multiple runs, error bars are not reported because the overall performance is computed across multiple datasets that differ in size and use heterogeneous metries, making it improper to calculate unified error bar. Additionally, space constraints in performance tables prevent the inclusion of error bars, and for fair comparison, follow prior works which also do not report error bars for their methods. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots. The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix D.I lists the compute resources (NVIDIA A40, A100, and RTX A5000) used for experiments. Section 4.5 compares the throughput and GPU usage.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: The paper does not involve sensitive data, human subjects, or unethical practices, and abides by NeurlPS ethical guidelines.
Guidelines:
29

---

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The broader impacts of this work are explicitly discussed in Appendix A
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have high risk for misuse (c.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This work does not release any new datasets pre-trained models that could pose risk of misuse. solely builds on publicly available models and benchmarks for evaluation purposes.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
30

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Original owners of codes, datasets, and models are explicitly mentioned via citations or urls in Section
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperssi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors are encouraged reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: No datasets models are introduced; the work only modifies and evaluates existing ones.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing or human participants.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
31

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects or participant-baa studies are involved.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing. editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: The paper focuses on the fine-tuning LLMs, whose usages are clearly described throughout the paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:e///oorrpp..ccooooffrerreee for what should or should not be described.
32


Reviews:
[{"review_id": "UXRcUzIkUe", "reviewer": "Reviewer_VpnN", "summary": "The paper introduces RefLoRA, a novel approach to improve Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning (PEFT) of large language models (LLMs). It addresses LoRA’s limitations, such as slow convergence and unbalanced weight updates due to nonunique low-rank factorizations, by optimizing the factorization process to minimize a loss upper bound, resulting in faster and more stable convergence.", "questions": "RefLoRA is tested only on transformer-based LLMs (DeBERTaV3, LLaMA series) for NLU and reasoning tasks. Can the authors provide evidence or analysis of RefLoRA’s applicability to other architectures (e.g., vision transformers, diffusion models) or tasks (e.g., text generation, RLHF)?\nThe paper omits LoRA-Pro and LoRA-RITE due to runtime and convergence issues (Section D.6). Can the authors provide a partial comparison, alternative evaluation, or explanation of why these methods failed, and how RefLoRA compares to other recent LoRA variants?", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The strengths of this paper are as follows.\nRigorous Theory: RefLoRA’s optimization of the SPD matrix $ \\mathbf{S}_t $ is supported by detailed proofs (Appendix B, Theorem 3), ensuring mathematical rigor.\nStrong Experiments: Outperforms LoRA, DoRA, and AdaLoRA on GLUE (88.50 vs. 88.15) and reasoning tasks (78.85 vs. 76.19) using DeBERTaV3 and LLaMA models (Tables 2–3).\n\nThere are some weaknesses in this paper:\nLimited Model Scope: Tested only on transformer-based LLMs, limiting generalizability (not addressed in limitations).\nIncomplete Baselines: Omits LoRA-Pro and LoRA-RITE due to runtime/convergence issues (Section D.6), weakening claims.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thorough comments and insightful suggestions. Our responses to the concerns raised are given next. \n\n1. **Applicability to other model architectures and tasks**  \nFollowing the reviewer's suggestion, further numerical tests are conducted on a subject-driven image generation task [1] with Stable Diffusion v1.4. The goal is to finetune a diffusion model using a few user-provided images so that it can generate the same object in various contexts. Specifically, the model is finetuned on a set of images labeled \"a photo of sks dog,\" and subsequently evaluated by generating images under the prompt \"a sks dog eating nachos.\" LoRA adapters with rank $r=4$ are attached to the U-Net component of Stable Diffusion, and experimental setups including hyperparameters are default to those in [1].  \nThe finetuning losses for LoRA, LoRA-Pro, LoRA-RITE, and RefLoRA are summarized in the table below, where average and standard deviation are calculated over 3 random runs. It is seen that RefLoRA achieves 14.0% , 13.1%, and 9.5% improvements over LoRA and LoRA-Pro, and  LoRA-RITE, respectively. In addition to quantitative gains in loss, we also observe noticeable improvements in image quality. The generations from RefLoRA-tuned models show markedly clearer details and better object fidelity, particularly in the mouth and tongue regions, where features are often distorted in outputs from other three baselines. \nAs external links are not permitted, the corresponding visual results will be included in the revised manuscript. We are also pleased to provide them during rebuttal if rule-compliant methods become available. \n Loss ($\\downarrow$) | LoRA | LoRA-Pro | LoRA-RITE | RefLoRA \n:---:|:---:|:---:|:---:|:---:\n Avg$\\pm$std | $0.100 \\pm 0.015$ | $0.099 \\pm 0.015$ | $0.095 \\pm 0.016$ | $0.086 \\pm 0.017$\n\n[1] N. Ruiz, Y. Li, V. Jampani, Y. Pritch, M. Rubinstein, K. Aberman, \"DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation,\" in *CVPR*, 2023. \n\n2. **Comparison with LoRA-Pro and LoRA-RITE**  \nWe encountered practical challenges when scaling LoRA-Pro to larger LLaMA models with 7B and 8B parameters. For instance, the estimated time for finetuning a LLaMA3-8B model with LoRA-Pro on two NVIDIA A100 GPUs required more than one day, which exceeded our available computational budget. Similar scalability concerns have been reported by other users in the community; see e.g., issue #6 on the LoRA-Pro GitHub repository. We hope to revisit these experiments once performance optimizations are available.  \nAs for LoRA-RITE, our initial experiments encountered some instability and sensitivity due to random seeds. Fortunately, the codebase has since been updated, and a stable version was released recently. Using the latest implementation, we obtained stable and reliable results. As shown in the table below, RefLoRA(-S) consistently outperforms LoRA-RITE across all three backbone models, showcasing its robustness and effectiveness. Additionally, RefLoRA(-S) also demonstrates better computational efficiency. For example, when finetuning LLaMA3-8B, the throughput of LoRA-RITE reaches merely 72.4% (63.6%) of RefLoRA(-S), yet consuming an additional 1.02 GB (1.16 GB) GPU memory. Please refer to the table in our response to Reviewer r7FQ for further details.  \n Model | Method | BoolQ | PIQA | SIQA | HS | WG | ARCe | ARCc | OBQA | Avg \n:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:\n LLaMA-7B | LoRA | 66.42 | 80.03 | 77.84 | 82.88 | 81.85 | 79.92 | 63.40 | 77.20 | 76.19 \n LLaMA-7B | LoRA-RITE | 69.82 | 82.75 | 78.55 | 84.72 | 81.69 | 82.15 | 66.23 | 81.40 | 78.54 \n LLaMA-7B | RefLoRA | 69.60 | 82.48 | 79.53 | 88.25 | 82.56 | 81.57 | 66.64 | 80.20 | **78.85** \n LLaMA-7B | RefLoRA-S | 70.18 | 82.48 | 78.15 | 87.41 | 82.08 | 81.52 | 65.36 | 81.60 | 78.60 \n LLaMA2-7B | LoRA | 69.8 | 79.9 | 79.5 | 83.6 | 82.6 | 79.8 | 64.7 | 81.0 | 77.6 \n LLaMA2-7B | LoRA-RITE | 71.04 | 82.43 | 79.79 | 89.12 | 84.53 | 83.88 | 68.77 | 81.20 | 80.10 \n LLaMA2-7B | RefLoRA | 72.54 | 83.79 | 80.04 | 86.94 | 84.85 | 86.36 | 71.50 | 80.20 | **80.78** \n LLaMA2-7B | RefLoRA-S | 73.36 | 83.84 | 80.76 | 90.02 | 82.48 | 84.55 | 67.92 | 82.60 | 80.69 \n LLaMA3-8B | LoRA | 70.8 | 85.2 | 79.9 | 91.7 | 84.3 | 84.2 | 71.2 | 79.0 | 80.8 \n LLaMA3-8B | LoRA-RITE | 74.19 | 89.44 | 81.52 | 95.44 | 86.74 | 90.45 | 80.12 | 86.60 | 85.56 \n LLaMA3-8B | RefLoRA | 75.35 | 88.74 | 80.91 | 95.71 | 86.66 | 90.49 | 80.20 | 87.40 | 85.68 \n LLaMA3-8B | RefLoRA-S | 75.50 | 89.72 | 81.11 | 95.59 | 87.29 | 90.99 | 79.78 | 86.00 | **85.75**"}, {"review_id": "oAFNL2TXQ8", "reviewer": "Reviewer_r7FQ", "summary": "The paper introduces RefLoRA, a novel method for efficient fine-tuning of large language models (LLMs) by addressing the limitations of Low-Rank Adaptation (LoRA). The authors propose dynamically selecting the optimal low-rank factorization per step to minimize an upper bound on the loss, leading to balanced updates and faster convergence. The method is evaluated on matrix factorization, natural language understanding, and commonsense reasoning tasks, demonstrating improved performance over existing LoRA variants. The theoretical analysis is rigorous, and the experimental results are comprehensive, covering multiple LLMs.", "questions": "1. The largest model tested is LLaMA3-8B. The authors mention scaling to 30B models as future work, but can RefLoRA’s overhead become prohibitive for even larger models (e.g., 100B+ parameters)?\n2. RefLoRA closes the gap to full fine-tuning, but the gap persists in some tasks. Is this due to inherent limitations of low-rank updates, or could further refinements help?", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n1. The paper provides a solid theoretical foundation for RefLoRA, including proofs for optimal factorization and balanced updates (Section 3). The derivation of the closed-form solution for the optimal S_t (Theorem 3) is particularly compelling.\n2. The experiments are extensive, covering diverse tasks and models. The results consistently show RefLoRA’s superiority over LoRA and other variants.\n3. The paper is well-written, with clear motivations, methods, and results. The visualization of loss landscapes (Figure 1) effectively supports the theoretical claims.\nWeaknesses:\n1. While RefLoRA-S is introduced as a simplified variant, its trade-offs (e.g., performance vs. complexity) are not thoroughly explored. For instance, how much performance is sacrificed compared to full RefLoRA, especially in low-rank settings (r=16)?\n2. The GLUE results for LoRA-Pro and LoRA-RITE are omitted due to runtime or convergence issues. This raises questions about the fairness of comparisons.", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the valuable feedback provided. The questions are addressed one-by-one below. \n\n1. **Performance-complexity trade-off of RefLoRA-S**  \nThere is indeed a tradeoff. The performance comparison of RefLoRA and RefLoRA-S were listed in Table 3 of the original submission. Specifically, RefLoRA achieves 0.09% higher average accuracy than RefLoRA-S across three models in the high-rank $r=32$ setting, while the gap increases to 0.23% in the low-rank $r=16$ setup. This trend is because a higher rank provides sufficient model expressiveness, allowing the loss to be minimized effectively even with scalar refactoring. In contrast, at lower rank $r=16$, RefLoRA-S exhibits slower convergence compared to RefLoRA. These observations suggest that full refactoring becomes more critical when the number of finetuning parameters is limited.  \nIn terms of runtime and memory, RefLoRA introduces additional computational overhead of $\\mathcal{O}((m+n+r)r^2)$ time and $\\mathcal{O}(r^2)$ space, which grows rapidly with $r$. For instance, when finetuning LLaMA3-8B with $r=32$, RefLoRA's throughput is 12.1% lower than RefLoRA-S, and consumes additional 0.14 GB GPU memory; see the table below for details. These considerations point to a practical trade-off: RefLoRA-S is particularly preferable for larger $r$, offering improved throughput and memory usage, while solely incurring a marginal drop in performance. \n\n2. **Comparison with LoRA-Pro and LoRA-RITE**  \nPlease refer to our \"Response 2\" to Reviewer VpnN, where additional numerical tests are conducted for the requested comparison. \n\n3. **Scaling to 100B+ models**  \nScalability does not confine the applicability of RefLoRA. In larger models, the major runtime bottleneck is the forward pass and gradient computation. In comparison, the additional overhead of RefLoRA becomes negligible.  \nLimited by computational resources, we are unable to perform experiments on models with 100B+ parameters. Instead, it is possible to scale up to Gemma3-27B-pt, where gradient checkpointing is turned on so that the model can fit within a single NVIDIA H100 96GB GPU. The computational comparison is summarized in the table below. Notably, the runtime and memory gap between LoRA and RefLoRA narrows as model size increases. These findings indicate that RefLoRA remains as practical and efficient as LoRA, especially for larger models.  \n Method | DeBERTaV3 |-base | LLaMA3 |-8B | Gemma3 |-27B |\n:---:|:---:|:---:|:---:|:---:|:---:|:---:\n | | Tp. (it/s) | Mem. (GB) | Tp. (it/s) | Mem. (GB) | Tp. (it/s) | Mem. (GB) \n LoRA | 1$\\times$ (2.26) | 8.73 | 1$\\times$ (1.53) | 36.21 | 1$\\times$ (0.58) | 64.28 \n LoRA-RITE | 0.73$\\times$ | 8.87 | 0.63$\\times$ | 37.37 | 0.95$\\times$ | 64.28 \n RefLoRA | 0.88$\\times$ | 8.86 | 0.87$\\times$ | 36.35 | 0.98$\\times$ | 64.28 \n RefLoRA-S | 0.99$\\times$ | 8.73 | 0.99$\\times$ | 36.21 | 0.99$\\times$ | 64.28 \n\n4. **Gap relative to full fine-tuning**  \nThe performance of RefLoRA is still limited by the low-rank premise of LoRA. However, the small gap observed is a reasonable tradeoff, given that the primary motivation behind LoRA is to fuel scenarios where full fine-tuning is infeasible.  \nTo further fill this gap, a promising direction is to integrate RefLoRA with recent advancements that support high-rank update, such as ReLoRA [1] and HiRA [2].  \n[1] V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky, \"ReLoRA: High-Rank Training Through Low-Rank Updates,\" in *ICLR*, 2024.  \n[2] Q. Huang, T. Ko, Z. Zhuang, L. Tang, and T. Zhang, \"HiRA: Parameter-Efficient Hadamard High-Rank Adaptation for Large Language Models,\" in *ICLR*, 2025."}, {"review_id": "xmt3KZLBa7", "reviewer": "Reviewer_USmE", "summary": "The paper “RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models” improves upon standard LoRA by dynamically re‐factorizing its low‐rank updates to minimize a provable upper bound on the loss, yielding more balanced and consistent weight adjustments. Empirical results show that RefLoRA and its lightweight variant converge faster and more stably on low‐rank matrix problems and standard PEFT benchmarks.", "questions": "Questions: \n1. Can the authors provide a convergence theorem, similar to [1] (Theorem 2)? I think the preconditioning of the gradients are somewhat related to the manifold optimization problem introduced in [1]. I.e. they combat the problem of high curvature of the manifold consisting of rank r matrices. Can the authors comment on this?\n2. What happens if the Gram matrices A^TA and B^TB are near singular? Computation of $\\tilde{S}$ can become hard? \n3. Have you evaluated performing the refactor step every k>1 iterations (instead of every step)? How does this affect convergence speed and runtime?\n\nComments:\n1. Assumption 1 can also be enforced by using the methods of [1,2]. \n2. In [1], the authors also discuss the shortcomings of the LoRA learning dynamics and how to circumvent them. It is relevant work that should be added to the manuscript. \n\n\n\n\n[1] https://arxiv.org/abs/2410.18720\n[2] https://arxiv.org/abs/2205.13571", "limitations": "- Ill conditioning of the Gram matrices needs to be addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n\n-  The method derivation is sold - I did not spot any logic flaws. \n- Pseudocode is well readable. \n- I appreciate the matrix completion benchmark - as it gives good insight in the method without the bloat of a full LLM architecture\n\nWeakness: \n\n- It is a bit unclear how the simplified method (Algorithm 2) is derived. \n- See questions below", "was_revised": true, "final_justification": "The authors have addressed my concerns and thus I have raised the quality score of my review. \nI think my current score (4) reflects the work well, since there are some theoretical shortcomings about convergence that other methods have figured out. \nThe empirical results of the paper and the presented analysis however, warrant acceptance.", "rebuttal": "The detailed comments and suggestions are greatly appreciated. Next, we deal with the concerns one-by-one below. \n\n1. **Derivation of the lightweight RefLoRA-S (Algorithm 2)**  \nThanks for pointing this out. We will elaborate further in the main context to improve clarity.  \nThe principle for deriving RefLoRA-S is the same as RefLoRA. But instead of finding an $r \\times r$ matrix $\\mathbf{S}_ t$, RefLoRA-S aims at a scalar $s_ t \\in \\mathbb{R}_ {++}$ for computational efficiency. In particular, we replace $\\mathbf{S}_ t = s_ t \\mathbf{I}_ r$ in Eq. (8), which results in $\n\\min_{s_t > 0} ( || \\mathbf{A}_ t ||_ \\mathrm{F}^2 s_ t + || \\mathbf{B}_ t ||_\\mathrm{F}^2 s_t^{-1} - \\frac{1}{L \\eta} )^2$; cf. Eq. (21) in Appendix B.4. More details for solving this quartic minimization problem can be also found in the Appendix. \n\n2. **Convergence theorem, and gradient preconditioning in relation to manifold optimization problem**  \nThanks for providing the nice intuition and relevant work [1]. We will make sure to touch upon it in the revised draft.  \nAgreed, RefLoRA is related to manifold optimization at high level. Yet, a similar derivation of [1, Theorem 2] is non-straightforward, as the manifold in RefLoRA is less structural. \nIn particular, GeoLoRA mainly works with two (generalized) Stiefel/Grassmanian manifolds, whereas the manifold of RefLoRA can be interpreted as $\\{(\\tilde{\\mathbf{A}}, \\tilde{\\mathbf{B}}) | {\\cal L}(\\mathbf{A}, \\mathbf{B})  = {\\cal L}(\\tilde{\\mathbf{A}}, \\tilde{\\mathbf{B}})\\}$ for given $\\mathbf{A}, \\mathbf{B}$. \nBecause $\\mathbf{A}$ and $\\mathbf{B}$ are updated over iterations, this \"manifold\" is time-varying and data-dependent. As a tradeoff, working with such an irregular manifold reduces the number of retractions compare to GeoLoRA.  \nWhile we are still working toward convergence analysis for the general case, the convergence of RefLoRA in the simpler matrix factorization case can be indeed established by adapting the proof of [3, Theorem 1.1]. Using the notation in [3], we redefine $A := (\\tilde{U} + \\tilde{V}) / 2$, and $B := (\\tilde{U} - \\tilde{V}) / 2$, where $\\tilde{U}$ and $\\tilde{V}$ are the correspondents of $U$ and $V$ after refactoring. A key step in the original proof involves bounding the asymmetry $|| B ||_F$, ensuring it remains sufficiently small after stage one and throughout stage two. This condition is inherently satisfied under our refactoring. As a result, it can be verified that the induction hypothesis in [3] remains valid under RefLoRA. \n\n3. **Convergence and runtime of refactoring per $k$ steps**  \nWe conducted additional ablation tests under the same configuration described in Section 4.4; and term the suggested variant as RefLoRA-I, for which the refactoring interval is set to $k = 10$ so that its throughput matches RefLoRA-S. \nThe following two tables respectively outline the finetuning loss over epochs, and the computational overheads comparison. The first table shows that the convergence of RefLoRA-I is slightly slower than RefLoRA, but comparable to RefLoRA-S. This is because the refactoring in RefLoRA not only balances the norms of $\\mathbf{A}_t$ and $\\mathbf{B}_t$, but also rotates and skews them to align their Gram matrices. While the norms of $\\mathbf{A}_t$ and $\\mathbf{B}_t$ evolve gradually due to the small learning rate, their Gram matrices can diverge significantly after gradient updates, necessitating more frequent adjustment to maintain stability. Furthermore, despite RefLoRA-I with $k=10$ matches RefLoRA-S in throughput, it consumes as much memory as RefLoRA. Thus, RefLoRA-S remains preferable in terms of both time and space efficiency. \n\n Epoch | LoRA | LoRA-Pro | LoRA-RITE | RefLoRA | RefLoRA-S | RefLoRA-I \n:---:|:---:|:---:|:---:|:---:|:---:|:---:\n 2 | 0.444 | 0.410 | 0.442 | 0.316 | 0.340 | 0.335 \n 4 | 0.248 | 0.238 | 0.160 | 0.082 | 0.091 | 0.124 \n 6 | 0.174 | 0.164 | 0.147 | 0.041 | 0.056 | 0.062 \n 8 | 0.151 | 0.115 | 0.065 | 0.006 | 0.040 | 0.040 \n 10 | 0.084 | 0.065 | 0.042 | 0.007 | 0.025 | 0.031\n\n Method | Throughput (it/s) | Memory (GB) \n:---:|:---:|:---:\n LoRA | 1$\\times$ (2.26) | 8.73 \n LoRA-RITE | 0.73$\\times$ | 8.87 \n RefLoRA | 0.88$\\times$ | 8.86 \n RefLoRA-S | 0.99$\\times$ | 8.73 \n RefLoRA-I | 0.99$\\times$ | 8.86 \n\n4. **Remedy for nearly singular Gram matrices**  \nTo ensure numerical stability, a small $\\epsilon$ has been added to the eigenvalues of $\\mathbf{A}^\\top \\mathbf{A}$ when calculating its inverse; see lines 66, 71 and 107 of codes `reflora.py`. This is a standard practice in optimization to prevent division by zero. The same technique is employed in optimizers including AdamW, LoRA-RITE, and LoRA-Pro.\n\n\n5. **Related works [1,2]**  \nThank you for bringing these two papers to our attention. They are closely related to our work, and the manifold perspective offers a complementary lens to interpret our results. We will incorporate a discussion of both studies in the revised submission.  \n\n[1] S. Schotthöfer, E. Zangrando, G. Ceruti, F. Tudisco, and J. Kusch, \"GeoLoRA: Geometric integration for parameter efficient fine-tuning,\" *arXiv preprint*, 2024.  \n[2] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco, \"Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations,\" *arXiv preprint*, 2022.  \n[3] T. Ye, and S. Du, \"Global convergence of gradient descent for asymmetric low-rank matrix\nfactorization,\" in *NeurIPS*, 2021."}]

Author rebuttal:

Meta-review: The paper received mixed ratings initially. There are a few minor concerns from the reviewers. The rebuttal has addressed most of them, and the reviewers raised their scores. The final decision is acceptance.
Decision: Accept (poster)