PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Authors: Kai Lion
Liang Zhang
Bingcong Li
Niao He
Venue: NeurIPS 2025 poster ()
Keywords: low-rank adaptation
architecture-optimizer co-design
large language models
lora
low-rank adapter
fine-tuning
Primary area: deep_learning
TL;DR: We propose PoLAR, a polar-decomposition-based parameterization, for efficient fine-tuning of LLMs. PoLAR mitigates the low stable rank seen in LoRA, provably accelerates convergence on a canonical LoRA problem, and lifts accuracy on real-world tasks.

Abstract: We show that low-rank adaptation of large-scale models suffers from a low stable rank that is well below the linear algebraic rank of the subspace, degrading fine-tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields an exponentially faster convergence rate on a canonical low-rank adaptation problem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, commonsense reasoning, and mathematical problem solving with base model sizes ranging from 350M to 27B.

PoLAR: Polar-Decomposed Low-Rank Adapter Representation
Kai Lion
Liang Zhang Bingcong Li Niao He
Department of Computer Science
ETH Zurich
Zurich, Switzerland
(kai. lion, liang. zhang, bingcong.li, li, niao. hei eeth.. ch
Abstract
We show that low-rank adaptation of large-scale models suffers from low stable rank that well below the linear algebraic rank of the subspace, degrading fine- tuning performance. To mitigate the underutilization of the allocated subspace, we propose PoLAR, a parameterization inspired by the polar decomposition that factorizes the low-rank update into two direction matrices constrained to Stiefel manifolds and an unconstrained scale matrix. Our theory shows that PoLAR yields exponentially faster convergence rate on canonical low-rank adaptation prob- lem. Pairing the parameterization with Riemannian optimization leads to consistent gains on three different benchmarks testing general language understanding, com- monsense reasoning, and mathematical problem solving with base model sizes ranging from 350M 27B.
Introduction
Large language models (LLMs) transformed the field of machine learning through their proficiency understanding and generating text, well as handling complex multimodal tasks. Typically, those models require number of parameters that is on the scale of billions, rendering fine-tuning of large- scale models infeasible hardware-constraing setups. Consequently, parameter-efficien fine-tuning methods have been developed to enable adaptation of LLMs with limited resources (Houlsby et al., 2019; Li and Liang, 2021; Lester et al., 2021; Hu et al., 2022). One notable work is that of Hu et al. (2022), which performs fine-tuning by learning an additive low-rank update parameterized as the product of two low-rank matrices. Such Low-Rank Adapters (LoRA) have been the subject of study in large body of work. Various improvements, ranging from initialization strategies (Meng et al., 2024; Li et al., 2025a) and custom learning rate settings for stable feature learning (Hayou et al., 2024b) alternative parameterizations (Liu et al., 2024a) have been proposed ever since.
Recent work attempts to overcome the low-rank constraint imposed by LoRA while preserving its parameter-effiiinnyy (Xia et al., 2024; Lialin et al., 2024; Zhao et al., 2024; Huang et al., 2025; Jiang et al., The underlying premise that the low-rank nature of the adapter limits its expressiveness. However, this premise is at odds with recent theoretical results that LoRA can approximate any target transformer model reasonably well under mild assumptions on the rank that depend on the relationship between the pre-trained and the target model (Zeng and Lee, 2024). Additionally, Kalajdzievski (2023) shows that raising the nominal rank does little to improve performance. Taken together, these findings suggest that the low-rank space offers sufficient expressiveness, but the classical low-rank adapter formulation struggles to fully utilize this potential.
'Equal supervision.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

{ov: Ph::ceR R
W2
300
W1
200
475
200
2864
W3
W4
300
1300
2000
2000
380
1010
000
620
485
19-0 1800
200 2000 nioo
(a) diversity collapse (DC)
(b) DC in LoRA
(c) PoLAR solves DC
(a) Illustration of directional diversity collapse (DC) of =w; where w, denotes the i-th row of low-rank update AW. (b) and (c) Diversity of update directions of LoRA and PoLAR for a Llama-2-7B down-proje nown-projection layer on dataset Social-IQA. respectively. Each pixel shows || - |2; rows and columns are rearranged reveal cluster patterns in both plots. Emergence of a cluster pattern is evidence for DC. The algebraic rank is 32 for both methods, yet the stable rank is 1.06 and 5.49 for LoRA and PoLAR, respectively. See also Section 2.1.
Figure 1:
Indeed, we observe comprehensive empirical evidence for this underutilization: when fine-tuning Llama-2-7B with LoRA, find that the stable rank, robust analogue of the matrix rank and measure of expressiveness, of the resulting update remains well below the linear algebraic rank. W (Rudelson and Vershynin (2006), see Appendix B), is as low as 1.06. This reveals that approximately rank 1 subspace is utilized even if the LoRA rank is chosen as 32. Similar behaviors of low stable rank are consistently observed across layers and datasets; see Figs. 2 and 6. Such deficiency results in diversity collapse the update directions among different neurons, where extreme cases the updates for all neurons strongly align along a single direction (up to a sign flip); see Fig. la for an illustration and Fig. lb for the directional diversity collapse when fine-tuning Llama-2-7B.
For some learned LoRA updates AW, the stable rank, defined as sr( sr(
To address this pathology. we put forth PoLAR, a co-design of architecture and optimizer that mitigates the directional diversity collapse, as shown in Fig. 1c On the architecture side, PoLAR facilitates effective exploitation of the linear algebraic rank by expressing the low-rank update as the r scale matrix. This parameterization
product of two column-orthogonaa direction matrices and can also be interpreted from the perspective of decoupling direction and magnitude, which arises naturally when the low-rank factors lifted via the polar decomposition. On the optimizer side, we apply methods from Riemannian optimization (Boumal, 2023). Theoretically, we demonstrate that our co-design enables exponentially faster convergence than vanilla LoRA on canonical problem. To further enable training at scale, we draw on recent advancee in infeasible methods for optimization over Stiefel manifolds (Ablin and Peyré, 2022). The resulting method is tailored GPU hardware, catalyzing a speedup of at least 3x over feasible methods. In nutshell, our contribution is four-fold:
Our empirical analysis demonstrates that the update matrices learned by LoRA have a stable rank far below their full linear algebraic rank, leading to a collapse in directional diversity and, in turn, preventing the adapters from fully realizing their expressiveness.
We introduce PoLAR, architecture-optimize co-design that ensures diverse update directions by factoring the low-rank updates into colunn-orthogonal direction matrices and an arbitrary scale matrix. Riemannian optimization is then adopted for our PoLAR parameterization.
On matrix factorization prototype problem, we prove that our PoLAR parameterization achieves an exponentially faster convergence rate than vanilla LoRA.
We evaluate PoLAR three benchmarks (language understanding, commonsense reasoning, and mathematical problem solving) using several backbones: DeBERTa-v3, Llama-2-78 Gemma-3. 12B, and Gemma-3-278B Consistent performance gains are observed across all settings.
Notation. Bold capital (lowercase) letters denote matrices (vectors); (.) and || ||r refer to transpose and Frobenius norm of a matrix; || is the Ea (spectral) norm of a vector (matrix); (.(-) and (x(-) denote the i-th largest singular value and eigenvalue, respectively. For a matrix X E grxr, , let Skew(X) { X be its skew-symmetric part. The set of matrices with orthonormal columns, i..., the Stiefel manifold, is denoted as St(m,,) = {X E Rmxr T I,). The set of r r positive semi-definit (PSD) matrices is denoted as S = { MERT grxr X 0}.

---

1.1 Related Work
Cemoryy-ffiiiien Fine- Tine-Tuning Low-Rank Adapters (Hu et al., 2022) are now a standard approach for fine-tuning large language models. There are various efforts to further enhance LoRA, e.g.. via adaptivity (Zhang et al., 2023), chaining (Lialin et al., 2024; Xia et al., 2024), low-bit training (Dettmers et al., 2023; Li et al., 2024), modifications for long sequences (Chen et al., 2024), weight decomposition (Liu et al., 2024a), or sparsity (Nikdan et al., 2024). For this work, LoRA-variants that change the parameterization of A W are particularly relevant. DoRA (Liu et al., 2024a) reparameter- izes the low-rank update according to weight normalization (Salimans and Kingma, 2016), learning direction and magnitude of the update separately. AdaLoRA (Zhang et al., 2023) allocates the rank of adapters adaptively according to layer importance, learning an SVD-type update with a diagonal matrix and orthogonal low-rank factors. A work concurrent to ours (Li al., 2025b) proposes reparameterization of LoRA that is similar to ours, using feasible methods from the literature of Rie- mannian optimization for training. VeRA and LoRA-XS propose novel parameterizations shrink the number of trainable parameters (Kopiczko et al., 2024; Balazy ct al.. 2024). Building on top of the LoRA-XS parameterization, Ponkshe al. (2024) develop custom initialization scheme which is informed by the SVD of the gradient at the pre-trained model. Another closely related line of research is dedicated to the development of subspace optimizers which maintain the memory efficiency of low-rank adapters, while enabling full rank parameter updates. Such subspace methods project gradients to a low-rank subspace within which optimizer states are maintained (Zhao et al., 2024; Liang et al., 2024; Hao et al., 2024; Robert et al., 2025; Chen et al., 2025).
further
Optimization on Manifolds. Riemannian optimization generalizes gradient methods smooth manifolds (Absil et al., 2008; Bonnabel, 2013; Boumal, 2023) and has proven useful in scenarios such as matrix completion (Vandereycken, 2013) and eigenvalue problems (Edelman al., 1998). Despite the theoretical appeal, such methods are rarely applied in large-scale settings, as expensive retraction operations are required ensure the feasibility of iterates. Ablin and Peyré (2022) propose an infeasible method for optimization on the orthogonal manifold that does not require retractions. Rather than ensuring the feasibility of all iterates, the specifically designed landing field guarantees convergence to the manifold in the limit. Gao et al. (2022) extend their analysis to the Stiefel manifold, and Schechtman al. (2023) generalize the landing method to more general cases.
2 PoLAR: A Co-Design of Architecture and Optimizer
We begin by empirically identifying inefficiencies in LoRA's utilization of its nominal rank, an insight that motivates PoLAR parameterization based on polar decomposition.
2.1 Overcoming Low Stable Rank with PoLAR
Given the pretrained weight (of linear layer) Wo Rmx=, LoRA Icarns an additive low-rank update AW E g with rank AW) < The adapted weight thus given by Wo + AW. In Hu et al. (2022), the parameterization AW- Z,Z, with
LoRA
Z, E mx and Zg E R2
x is used.
While LoRA significantly enhances parameter effi- ciency as (m + n)r & mn, it turns out that it strug- gles to fully utilize the expressiveness of its param- cterization. In particular, when fine-tuning Llama-2 2- 7B with LoRA, we find that the stable rank sr(A remains small on various datasets, oftentimes ap- proaching 1 for many layers even with a reasonably large choice of r = 32; see Fig. 2. Similar observa- tions also evident in official LoRA weights; see more in Fig. 6. Such a low stable rank translates to a loss in directional diversity of the neural updates.
Te
Figure 2: sr( of Llama-2-77 low-rank updates fine-tuned on commonsense tasks with rank 32.
The directional diversity of an update matrix is measured by the average pairwise Euclidean distance of neurons when projected to the unit sphere. Note that this distance within [0, 2] with the lower and upper bounds attained by pair of collinear neurons, pointing in the same or opposite directions,
3

---

respectively. Consequently, a distribution of pairwise distances with most mass at the ends of the interval can be interpreted as evidence for low directional diversity.
As observed in Fig. 1b, the LoRA update closely aligns along a single direction, with most neural updates being nearly collinear. Using the compact SVD AW UEVT, it is possible explain this observation and identify a low stable rank as a driver behind the lack of directional diversity, Here, U [u],..., u.] RExr and V [Vi,.... Vn] E Rexr have orthonormal columns and Σ = diag(a),..... .I....,
.r. E Rrxr contains the singular values on its diagonal. Also, let w, denote the i-th row of A W. With this notation, we can see that the direction of the LoRA update for the i-th neuron can be approximated by:
Wi w
Σ o U.yy
(a) sign(o) Ualvi e (1.2...., m}
(I)
E
of U by j=1
where (a) comes from sr( (W) 1. Equation (1) suggests that the weight update of all neurons tends to strongly align with the direction of the leading right singular vector up to a sign flip, causing the two-cluster pattern in Fig. lb. Moreover, Fig, demonstrates the wide applicability of this finding, as the LoRA updates across several datasets suffer from a very low stable rank.
We provide more thorough empirical evidence of this collapse in directional diversity in Appendix F.1.
This pathology suggests that LoRA does not fully utilize its rank capacity, and we conjecture that a parameterization addressing this pathology increases the performance of LoRA. These insights lead to the following desiderata: we wish to learn (r) roughly orthogonal directions whose contributions to the unit-norm neural update are roughly balanced, avoiding the collapse of directional diversity.
To this end, we advocate to incorporate orthogonality directly into the architecture. This can be done by applying the polar decomposition (Golub and Van Loan, 2013) to each of the low-rank factors. Definition 1 (Polar Decomposition). Let Z E TR with m and Zn Raxr with admits factorizations Z = x and Ze = YO,, where X E St(m,r) and Y E St(n, r) have orthonormal columns and 1 , 2 E Rr are positive semi-definite, respectively.
r. It
When r = 1, Definition reduces to the familiar magnitude-direction decomposition of vectors.
The polar decomposition thus generalizes magnitude-directtn decomposition to matrices, where the semi-orthogonal factor represents the directional component and the positive semi-definite factor captures the magnitude. Using this pngnitude-direction decomposition, the desirable orthogonality naturally enforced through the manifold structure of X and Y. Moreover, rather than relying on two individual scale matrices, it is more convenient to learn joint θ E R""* matrix for the overall update, which amounts to merging the product O O,O.
These considerations give rise to our Polar-decomposed Low-rank Adapter Representation (PoLAR):
AW T with X St(m,r), Y E St(n, r), and O R**r
(2)
As byproduct, PoLAR admits natural interpretation in terms of a direction tion. However, unlike alternative decompositions, such as the column-wist wise weight normalization used in DoRA (Salimans and Kingma, 2016; Liu et al., 2024a), PoLAR enforces orthogonality, substantially increasing the stable rank and generating low-rank updates with more competitive downstream task performance (see later in Section 4).
decomposi-
In the following subsection, we show that our PoLAR parameterization in (2) offers provable advantages over LoRA when paired with appropriate optimization algorithms.
2.2 Faster Optimization with PoLAR Parameterization
We now compare the convergence of PoLAR and LoRA on the problem of learning a low-rank adapter for a single layer with whitened data. As discussed in Arora et al. (2018); Zhang and Pilanci (2024); Li et al. (2025a) and Appendix C.3, applying LoRA in this case is equivalent to matrix factorization under the asymmetric Burer-Monteine (BM) parameterization (Burer and Monteiro, 2003)
1
min mm Z
IZ AlP.
(3)
2
'The Stiefel manifold is defined St(m,r) r) = RXRR X 1.).

---

In problem (3), the matrix to be factorized (or the target matrix of LoRA; see Appendix C.3) is denoted as A E mRx,, and Z1, Ze repre- sent the LoRA weights. In light of the low- assumed. Problem (3) has been widely adopted as a testbed for developing advanced optimiza- tion schemes for LoRA; see, e.g.. Zhang and Pilanci (2024) or Li et al. (2025a).
Algorithm 1 RGD for PoLAR parameterized (4)
Require: Learning rates 7,, " sample X, and Yo uniformly from St(m, r) and St(n, r), respectively. for t= 0...., ,
do
rank setting, r < min(m,n]
Find θ via (5a)
Obtain Riemannian gradients E, and F, Update X and Y(+1 with (5b) and (5c)
end for
Our goal in this subsection is to understand the optimization dynamics of our PoL.RR parameterization applied to the same one-layer setting, yielding the problem below
min X,Y 2 Xex - All,, s.t. X &ES,m,, Y E St(n, O E R"**"
Considering the sufficient expressiveness of LoRA (Zeng and Lee, 2024), we focus on the overpa- rameterized regime for both (3) and (4), where FΛ == rank(A) A). In this setting, zero loss is attainable. Let the compact SVD of A be UΣV with U E gMxrA, V E Inxrr and diagonal E E Rra Without loss of generality, we assume 01 [Σ) = and or (E) = 1/16 where k is the condition number. We also assume 2 n, one can transpose A if necessary.
Note that the semi-orthogonal low-rank factors live on Stiefel manifolds, requiring a treatment with Riemannian gradient descent (RGD). For technical simplicity, we consider a procedure that alternates between updating O, and (X,, Y.). At iteration I, starts by finding O with gradient descent (GD) using learning rate 7 > 0, i.c.
AY,.
(5a)
Setting = significantly simplifies our analysis. With this value and the updated matrix O the Riemannian gradient of X, can be obtained via E, -Im X,X, AY, O, Further involving polar retraction to remain on the manifold, the RGD update on X, is given by
X = (X; - -,,,,, + 1/ E E.) 1/2.
(5b)
Likewise, the Riemannian gradient of Y, is F. = (I, - Y.Y, A A X,O, leading to the update
Y (Y, FI) FF
(5c)
Alg. summarizes the resulting RGD procedure. Despite the main technical challenge of establishing global convergence in the presence of saddles, RGD for our PoLAR parameterization admits natural geometric interpretation in terms of principal angles between subspaces. In particular, define the alignment matrices $, := U X RTAXT and V,, == V Rr The singular values of 4; give the arc-cosine of the principal angles between the two subspaces spanned by bases X₁ and U, respectively. In other words, (a, ( + capture the alignment of the subspaces, yielding Tr( Appendix C.I. Analogous statements apply to complements the discussion of direction-ne -artiiiudem decomposition in the context of equation (2). Our first theoretical result demonstrates that the aligmment between X, and U is non -occeeccnn over iterations. This geometric observation bears resemblance to the descent lemma in standard GD.
E ($:) as a suitable metric measure total alignment; see more in This alignment perspective further
Lemma 1 (Increasing Alignment). Let Be == d (I, $ ) and 6, σn (I,, L V,DT) D.),, and suppose that the learning rates are chosen as 17 < 1 and 7 = 1. If the following conditions are met,
2(1 37 (1)
Tr( - -b, 1.
> n/3, TT(
R22 p Tr Tr((Ira V V.V.
2(1 - 211-726::0² 172
> no, Tr V.)
x2
Alg. 1 guarantees that Tr( Tr( 1
I 2 T and Tr V t+1 & IT) 2 (R
Next, we show that, with a proper n, Lemma leads to global convergence of Alg. 1.
5

---

77
1371
8V 65
1at0: 12800 2000 ola solo 1860
20.0 2010
50.8
2388
2008
2018
2008 60.00
1380 2000
2000
1200
1-n (E0 F,-33
33hard
122
150
(a) Asymmetric
(b) Symmetric
Figure 3: Overparameterized matrix factorization for different condition numbers and degrees of
overparameterization using the PoLAR parameterization with Riemannian Gradient Descent (PO + RGD) and the Burer-Monteiro parameterization with Gradient Descent (BM + GD). (a) Asymmetric scenario and (b) symmetric case. The experimental details can be found in Appendix G.4.
Theorem (Global Convergence). Suppose that FA S Let p: min (r-ra)²}
}. W.h.p. over the random initialization of Xu and yo, choosing the learning rates 1= O (r-ra)
rm m²
0 , (P)
and y =
Alg. ensures ,, Al? S cffor all T 2 0 Prare m m² log
(p((rra)) p(r-ra)
Theorem builds on a three-phase analysis to prove global convergence of Alg. 1. demonstrates that RGD with PoLAR parameterization avoids getting trapped by saddles of (4) without the need for artificial noise. Our rate compares favorably to previous results of GD in the overparameterized regime." In particular, Xiong et al. (2024) show that overparameterizztion slows down GD, leading an undesirable x dependence with O maxffxs x²) log (1/e)). Although this bound is originally obtained for matrix sensing, the numerical example in Fig. 3 confirms that GD barely progresses beyond certain iterations. Our rates Theorem exponentially improve the π dependence a quartic one. Compared with other works of overparameterized GD on BM such as Soltanolkotabi et al. (2023), our bound not only has more favorable dependence on K, but also ensures last-iteration convergence, removing the need for early stopping.
It is also worth noting that, unlike GD with BM, PoLAR improves with overparameterization. To see this, consider the slightly overparameterized regime with r z r', e.g., FA + 5. Our bound in Theorem 1 reads O 0m=7* + log( (o/)). However, once at the highly overparameterized regime r = Cr for some constant C, c.g., C = 5, we have bound with a drastically improved dependence on the dimension: O(m² n²4 + ="x log(1/e)) A graphical illustration can be found in ra
Fig, 3a, where the convergence of r 5 much faster than that of = = + 5.
Theorem 1 also suggests practical merits of PoLAR, notably its simplified initialization (via, for instance, Lemma 6). In contrast, initialization of BM can impose multiple requirements both theoretically and empirically (Ye and Du, 2021: Xiong et al., 2024; Hayou et al., 2024b). addition, our choice of an unconstrained O E R''* plays crucial role for convergence. Empirical evidence in (Mishra et al., 2013, Fig. 5) shows that substituting
by a diagonal matrix @d, as done in A (Zhang et al., 2023), can adversely affect convergence, potentially because of the presence of non-strict saddles in the loss landscape (Levin al., 2024). These spurious stationary points can be removed by a parameterization with positive-defiite @ (Levin et al., 2024, Prop. 20).). Our PoLAR parameterization poses no constraints on @ It thus avoids computational overheads associated with enforcing positive-definiteness (e.g., matrix exponentials), yet still ensures global convergence.
Lastly, we extend our analysis to the symmetric matrix factorization problem in Appendix E, where we prove last-iterate convergence with complexity O (k* log(1/c)). This represents an exponential improvement in the c-dependence over GD on BM-type problems, which is subject to the lower bound of Q(k²) /e) established in Xiong et al. (2024); see also Fig. 3b for an illustration.
'The works of (Ward and Kolda, 2023; Xu et al., 2024b) provide improved e dependence of Alternating GD and GD when the column/row space of A is known at initialization. They are excluded from comparison because knowing the column space of A amounts setting @0 ) r.A which is already optimal initialization.

---

3 Practical PoLAR for Scalable Fine-Tuning
Although PoLARR increases expressiveness and accelerates convergence, optimizing under the mani- fold constraint of (2) with standard feasible methods does not scale to tasks such as LLM fine-tuning. Every retraction back onto the Stiefel manifold requires a matrix inversion SVD (see (5b)), opera- tions whose sequential nature limits GPU parallelism and becomes a runtime bottleneck (see Table and Sun et al. (2024)). Retractions are also numerically brittle in the low-precision arithmetic, now standard in training. To sidestep these issues, we take inspiration from the recently proposed landing algorithm which completely eschews retractions, producing iterates that are not necessarily on the manifold, but provably land on it as training proceeds (Ablin and Peyré, 2022; Gao et al., 2022).
To fine-tune the PoL.RR parameterization of (2), initialize X, and Yo randomly the Stiefel manifold and set Og = 0. At iteration 1, we update X, and Y, with the landing field. Taking the update X, as example,
r(X;)= XX,, +
(XX
(6)
Algorithm 2 PoLAR Fine-tuning
Riemannian grad.
Infeusibility penalty
Require:
Parameterize via (2); Initialize Xo, Yo uniformly random from Stiefel manifolds, set
as a drop-in replacement for the Riemannian update described in Section 2.2. The first com- ponent in (6) is the standard Riemannian gradi- ent for matrices on the Stiefel manifold with (X) := Skew( V CXX, O, T)X"). The second component in (6) is given by the gra- dient of the infeasibility penalty, (XX)
O = O, and denote À regularization strength, Pe stateful gradient transformation (e.g., Adam)
for t = , do
T(X,) f- (X,) + AV
T(Y,) +- (Y.YY, AVN(Y.)
X t X, 1E:-7:P
||X X-I which attracts the iterate towards the Stiefel manifold, making retraction obsolete. The landing field (6) has several appealing prop- erties. First, the two components of the field are orthogonal to each other, decoupling loss mini- mization and convergence to the Stiefel manifold. Second, its computation involves only matrix multiplications, eliminating the need for sequential retraction routines that do not map well to GPU parallelism; see Section 4.4 for numerical comparisons.
Y(41 + (TTT),:(T(Ya))
O.+1 < -, - - ThP:: (V,, O Y.))
end for
The complete PoLAR fine-tuning procedure is summarized in Alg. 2. We treat À > in (6) tunable hyperparameter chosen such that iterates converge to the Stiefel manifold in the limit (see Fig. 8). Alg. 2 also omits the alternating update for & used in Alg. simplify training by avoiding second backward pass. Additional details relating to deployment appear in Appendix A.
Experiments
To demonstrate PoLAR's effectiveness scale, fine-tune backbones spanning 350M-27B param- eters on three benchmarks of varying difficulty. Our study includes both masked and autoregressive language models and evaluates their abilities in terms of general language understanding, common- sense reasoning, and multi-step mathematical problem solving. The code for our experiments is available at https://githi cwm/kcc- Dercc-lion com//ccc-ion/polla.. lion/polar/.
4.1 Commonsense Reasoning
Commonsense reasoning tasks aim to assess how well LLMs can mimic human-like understanding. A standard suite of eight different datasets is chosen, and LLMS are finetuned separately on each of the datasets. For evaluation, we employ the widely-adopted Eleuther-AI (Biderman et al., 2024). We report the accuracy based multiple-choiee likeliilddoodd evaluation to facilitate reproducibility. More details can be found in Appendix G.I.
reevalaation-hhaeeee framework from
We first isolate the contribution of our θ parameterization and the optimizer based on the landing field. To this end, we compare PoLAR with a modified procedure which sets E Diag(r) and (ii) replaces the Riemannian gradient with the Euclidean gradient. These changes bring our procedure closer to the SVD-type parameterization discussed A (Zhang et al., 2023). Gemma-2-2B is chosen as the base model (Gemma Team, 2024), and adapters with diverse ranks are tested. We report the results Table and remark that PoL.RR outperforms the diagonal parameterization across all ranks, validating the effectiveness of our co-design.
7

---

Table 1: Accuracy of Gemma-2-28 with PoLAR on commonsense reasoning tasks for different gradient types and parameterizations Rie. (Eucl.) refers to Riemannian (Euclidean) gradient.
Rank Param.
Grad.
BoolQ
PIQA
SIQA
HcSw
WiGr
ARC-e
ARC-c
OBQA
Avg.
Diag(r)
Eucl.
86.24
81.50
58.70
79.88
78.69
78.75
52.39
56.80
71.62
Rexr
Rie.
86.48
81.66
58.90
79.69
80.03
81.78
54.69
56.40
72.45
Diag(r)
Eucl.
86.06
82.05
59.52
80.38
77.19
79.50
$5.03
$7.20
72.12
8
Rexr
Rie.
86.94
81.77
58.80
80.49
79.64
81.94
56.14
55.80
72.69
Diag(r)
Eucl.
86.82
81.83
59.52
81.08
78.37
78.87
$4,35
54.80
71.96
16
Resr
Rie.
86.91
81.18
59.93
81.01
79.16
82.45
53.92
57.40
72.75
Diag(r)
Eucl.
87.03
81.39
60.08
81.73
77.51
79.21
55.29
56.60
72.35
32
Rexr
Rie. 87.28
81.61
59.72
81.40
77.74
81.78
54.86
57.80
72.77
Table 2: Performance on commonsense reasoning tasks with Llama-2-7B using PoLAR for different ranks in single-task setup. HeSw refers to HellaSwag and WiGr to WinoGrande.
Rank
Adapter
BoolQ
PIQA
SIQA
HeSw
WiGr
ARC-e
ARC-c
OBQA
Avg.
LoRA
87.16
81.01
58.85
82.36
74.35
81.90
57.68
56.80
72.51
4
DoRA
87.22
80.30
58.96
82.39
75.22
81.69
57.85
56.80
72.55
PoLAR
87.49
82.59
59.31
81.23
81.77
81.61
56.31
55.80
73.26
LoRA
87.34
81.07
58.80
82.68
74.66
82.24
58.11
55.80
72.59
8
DoRA
87.49
80.58
58.39
82.32
75.22
82.24
58.19
55.60
72.50
PoLAR
87.74
82.70
59.62
82.14
82.40
81.36
56.91
55.20
73.51
LoRA
86.94
81.18
58.96
82.57
76.01
82.58
57.85
56.00
72.76
16
DoRA
86.85
81.34
58.29
82.39
74.59
82.62
57.68
55.40
72.39
PoLAR
88.01
82.92
59.93
82.68
81.77
81.82
57.51
57.40
74.00
LoRA
87.89
81.56
59.06
82.51
72.61
82.37
56.83
54.60
72.18
32
DoRA
87.61
81.45
58.70
82.50
74.43
82.28
57.17
55.60
72.47
PoLAR
88.13
82.64
60.03
83.12
82.00
81.99
56.14
55.60
73.71
Scaling up to Llama-2-7B (Touvron et al., 2023), PoLAR again delivers the highest mean accuracy across tasks, outperforming both LoRA and DoRA (Table 2). Whereas the gains of DoRA and LoRA appear to be mostly flat as the adapter rank grows, PoLAR's accuracy increases with larger rank. This consistent with our conjecture that PoLAR counteracts directional-diversity collapse by exploiting the allocated rank more effectively. We next examine the stable rank of the learned update matrices (Fig. 2) at nominal rank 32. For datasets, except for ARC-e and ARC-c, PoLAR significantly increases the stable rank of the update. Correspondingly, these two tasks are also where PoLAR shows no accuracy advantage relative to LoRA and DoRA. These results support the intuition that a higher stable rank correlates with improved downstream performance. Figs. 4a and 4b (and more in Appendix F.1) extend the analysis over multiple nominal (linear algebraic) ranks: LoRA's stable rank rarely exceeds 2, while PoLAR's increases steadily with the nominal rank, indicating richer subspace and mirroring the observed accuracy gains. Figs. 4c and 4d further visualize the evolution of the stable rank the course of training. While the stable rank for LoRA remains relatively constant, PoLAR's i stable rank increases throughout training and appears to dynamically vary with the layer type: MLP layers (up and down projection) appear require larger stable rank, while that of the value projection layer often comes out lowest (see more examples in Fig. 7).
In Appendix F, provide further experimental results for the commonsense reasoning benchmark. comparing PoLAR to GaLore (Zhao et al., 2024). In addition, we further scale up the experiment and report detailed results for Gemma-3-122 3-2B (Gemma Team, 2025), where PoLAR is observed to improve upon LoRA by more than percentage point.
4.2 Mathematical Problem Solving
Our second set of experiments tests the ability of fine-tuned models to perform multi-step mathe- matical reasoning. To do so, we apply PoL.RR to adapt Gemma-3-27B on MetaMathQA (Yu et al., 2024) and evaluate on GSM8K (Cobbe et al., 2021), set of 8.5K grade school math word problems,

---

(a) Social-IQA
(b) HellaSwag
(c) Social-IQA
(d) HellaSwag
Figure 4: Left two: Stable rank distribution of Llama-2-7B adapter layers trained with PoLAR and LoRA and different r on the Social-IQA and HellaSwag datasets. See also Fig. 5. Right two: Stable rank dynamies of the fifth transformer block over the course of training by layer type. See also Fig. 7.
Table 3: Performance comparison on GSM8K and MATH.
Table 4: Runtime (18) of landing (Land.) and retraction (Retr.) on a H100 GPU with parameter size (4096, r)
1 results are taken from Yu et al. (2024).
Model
Method
Trainable (%)
GSM8K
MATH
GPT-4
92.00
42.50
32
64
256
Gemma-3-
LoRA
0.2816 0.2818
85.37
41.94
Retr.
541
1050
1916
9889
27B-PT
PoLAR
85.67
42.70
Land.
180
186
238
539
and MATH (Hendrycks et al., 2021), of 12.5K high school competition mathematics problems. As is done in Yu et al. (2024), zero-shot prompt is leveraged for evaluation. We report results in Table 3 and remark that PoLAR achieves superior performance compared to LoRA with the margin being larger on the more challenging MATH benchmark. In Appendix F.3, we show that approximate feasibility is maintained throughout training with the manifold constraints fulfilled convergence.
4.3 Natural Language Understanding
Lastly, we demonstrate the applicability of our method beyond auto-regressive language models. To achieve this, we use DeBERTa-V3 (He et al., 2021) as a representative of masked language models (Devlin et al., 2019) and fine-tune it on the GLUE benchmark (Wang et al., 2019). We repeat the experiment with four seeds and report the evaluation results Table with the standard deviation in the subscript. Comparing PoLAR with LoRA, note significant improvements across all tasks except for QQP and MRPC where results are comparable. These results highlight that PoLAR can handle both autoregressive as well as masked language model setups.
4.4 Practical Runtime Improvement over Retraction
For X E Rmxx, landing step requires O(m2r) FLOPs while retraction- based routine only requires O(mr²) operations. While landing offers no advantage in terms of FLOPs over retraction based methods, the General Matrix Multiplications (GEMMs) used in landing step achieve much higher FLOP utilization on GPU hardware than the SVD-based retraction routine. To quantify runtime differences, we benchmark the retraction and landing operations using the most frequently occurring attention and MLP weight dimensions in Llama-2-7B LoRA fine-tuning. As Table 4 shows, even for very small ranks of
4, which make retraction relatively cheaper, usage of the landing method provides roughly a speedup of 3x with the advantage increasing to s 18x for higher ranks. Appendix F.2 extends the study to broader range of layer shapes and additionally compares memory and runtime overhead to LoRA and DoRA.
Table 5: Performance of PoLAR on GLUE with DeBERTa-V3-base We report the matched accuracy for MNLI, Matthew's (Pearson) correlation for CoLA (STS-B), and accuracy for other tasks.
Name
MNLI
SST-2
MRPC
CoLA
QNLI
QQP
RTE
STS-B
Avg.
LoRA
90.01
18
89.71+1.44
68.17
94.06.0.18
92.47.0.08
84.84
20.99
88.18
PoLAR
90.67
95.96
59.52
69.89 69.8940.47
94.37
92. 92.028
87 18
91.83
88.94

---

5 Conclusion and Future Work
Low-rank adaptation has become the workhorse for efficient fine-tuning, yet our analysis revealed that the classical BM parameterization in LoRA often underutilizes its allocated subspace: the stable rank of the learned updates collapses, limiting expressive power. Building on this empirical finding, as well as theoretical insights from a canonical low-rank optimization problem, we introduced PoLAR, a reparameterization inspired by the polar decomposition that is coupled with a landing field optimization procedure. Across backbones from 350M to 27B parameters and tasks that span general language understanding, commonsense reasoning, and multi-step mathematics, PoLAR delivered consistent accuracy gains. The improvements often scale with adapter rank, validating that maintaining higher stable rank translates into richer, more task-aliss sask-alinndd updates. In practice, PoLAR's 's reliance on nothing more than matrix multiplications implies that it maps cleanly onto GPU hardware, offering a drop-in replacement for existing LoRA pipelines.
Given the low stable rank of the LoRA update, a closer examination of the spectral dynamics of low-rank fine-tuning seems to be a promising direction for future work. While several works already study the spectral evolution of weights during ordinary training of deep neural networks (Boix- -Bsera al., 2023; Martin and Mahoney, 2021; Mousavi-Hosseini et al., 2023), the impact of the explicit low-rank constraint on the spectrum is not yet well understood. An investigation into the dynamics of the spectral distributions of PoLAR could bring more insights into the benefits of the parameterization.
Acknowledgements
We are grateful to Christopher Criscitiello for insightful discussions and thank the anonymous reviewers for helpful feedback. Kai Lion is supported by Swiss National Science Foundation (SNSF) Sinergia Funding No. 216600. Liang Zhang gratefully acknowledges funding by the Max Planck ETH Center for Learning Systems (CLS). Bingcong Li is supported by SNSF Project Funding No. 200021-207 2007343. Niao He is supported by ETH research grant funded through ETH Zurich Foundations and SNSF Starting Grant. This work was supported as part of the Swiss AI Initiative by a grant from the Swiss National Supercomputing Centre (CSCS) under project ID a105 on Alps.
References
Pierre Ablin and Gabriel Peyré. Fast and Accurate Optimization on the Orthogonal Manifold without Retraction. In Proc. Int. Conf. on Artificial Intellignnce and Statistics (AISTATS), January 2022.
P.-. Absil, Mahony, and R. Sepulchre. Optimization Algorithmss on Matrix Manifolds. Princeton University Press, Princeton, NJ, 2008. ISBN 978-0-691-13298-3,
Sanjeev Arora, Nadav Cohen, Noah Golowich, and Wei Hu. A Convergence Analysis of Gradient Descent for Deep Linear Neural Networks. In Proc. Int. Conf. on Machine Learning (ICMI.), 2018.
Klaudia Batazy, Mohammadreza Banaei, Karl Aberer, and Jacek Tabor. LORA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters, October 2024. arXiv:2405.17604
Stella Biderman. Hailey Schoelkopf, Lintang Sutawika, Leo Gao, Jonathan Tow, Baber Abbasi, Alham Fikri Aji. Pawan Sasanka Ammanamanchi, Sidney Black, Jordan Clive, Anthony DiPofi, Julen Etxaniz, Benjamin Fattori, Jessica Zosa Forde, Charles Foster, Jeffrey Hsu, Mimansa Jaiswal, Wilson Y. Lee, Haonan Li, Charles Lovering, Niklas Muennighoff, Ellie Pavlick, Jason Phang, Aviya Skowron, Samson Tan, Xiangru Tang, Kevin A. Wang, Genta Indra Winata, François Yvon, and Andy Zou. Lessons from the Trenches on Reproducible Evaluation of Language Models, May 2024. arXiv:2405.14788
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proc. AAAI Conf. Artif. Intel., November 2019.
Ake Björck and Gene H Golub. Numerical Methods for Computing Angles between Linear Subspaces. Mathematics of Computation, 27(123):579-594 1973.
Enric Boix-Adsera, Etai L Littwin, Emmanuel Abbe, Samy Bengio, and Joshua Susskind. Transformers Learn through Gradual Rank Increase. Proc. Neural Information Processing Systems (NeurIPS), 2023.
10

---

Silvère Bonnabel. Stochastic Gradient Descent on Riemannian Manifolds. IEEE Trans. on Automatic Control, 5899)22217-2229, September 2013.
Nicolas Boumal. An Introduction to Optimization on Smooth Manifolds. Cambridge University Press, 2023. doi: 10. 17777880001611111 16.164.
Nicolas Boumal, Bamdev Mishra, P.-A. Absil, and Rodolphe Sepulchre. Manopt, Matlab Toolbox for Optimization on Manifolds. Journal of Machine Learning Research, 15(42):1455-1459 2014.
Samuel Burer and Renato DC Montciro. A Nonlinear Programming Algorithm for Solving Semidefi- nite Programs via Low-Rank Factorization. Mathematical Programming, 95(2):329-33 -57, 2003.
Daniel Cer, Mona Diab, Eneko Agiree, lñigo Lopez- and Lucia Specia. SemEval-2017 Task I: Semantic Textual Similarity Similarity-Multiinng -Multiiinuul and Cross Ctnss-Lingual Focused Evaluation. In Proc. Int.
Workshop Semant. Eval., pages 1-14. ACL, 2017.
Xi Chen, Kaituo Feng, Changsheng Li, Xunhao Lai, Xiangyu Yue, Ye Yuan, and Guoren Wang. Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? In Proc. Neural Information Processing Systems (NeurIPS), 2025.
Yukang Chen, Shengju Qian, Haotian Tang, Xin Lai, Zhijian Liu, Song Han, and Jiaya Jia. Long- LoRA: Efficient Fine-Tuning of Long-Context Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
Yasuko Chikuse. Statistics on Special Manifolds, volume 174. Springer Science Business Media, 2012.
François Chollet. On the Measure of Intelligence. arXiv:1911.01547, 2019.
Christopher Clark, Kenton Lee, Ming-Wos Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. Proc.
Conf. Assoc. Comput. Linguist. Meet., May 2019.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have Solved Question Answering? Try ARC, the A12 Reasoning Challenge, March 2018. A.G3::1003.05557
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training Verifiers to Solve Math Word Problems, November 2021. 1rXiv:2110.14166
Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLoRA: Efficient Finetuning of Quantized LLMs. In Proc. Neural Information Processing Systems (NeurIPS), volume 36, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proc. Conf. Assoc. Compul. Linguist. Meet., pages 4171-4186, June 2019.
Bill Dolan and Chris Brockett. Automatically Constructing a Corpus of Sentential Paraphrases, In Proc. Int. Workshop Paraphrasing, 2005.
Simon S Du, Wei Hu, and Jason D Lee. Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balanced, In Proc. Neural Information Processing Systems (NeurIPS), volume 31, 2018.
Alan Edelman, T. A. Arias, and Steven T. Smith. The Geometry of Algorithms with Orthogonality Constraints. SIAM journal on Matrix Analysis and Applications, pages 303-353, June 1998.
Bin Gao, Simon Vary, Pierre Ablin, and P.-A. Absil. Optimization Flows Landing on the Stiefel Manifold, July 2022. arXiv:2202.09008.
Gemma Team. Gemma 2: Improving Open Language Models Practical Size, October 2024. arXiv:2408.00118.
Gemma Team. Gemma 3 Technical Report, March 2025,

---

Bogdan Georgiev, Lukas Franken, Mayukh Mukherjee, and Georgios Arvanitidis. On the Impact of Stable Ranks Deep Nets, October 2021. arXiv:2110.02333.
Gene H Golub and Charles F Van Loan. Matrix Computations. JHU press, 2013.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge, Ying Shan, and Mike Zheng Shou. Mix-of-Show: Decentralized Low-Rank Adaptation for Multi-Concept Customization of Diffusion Models, In Proc. Neural Information Processing Systems (NeurIPS), 2023.
Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-Rank Adapters Are Secretly Gradient Compressors. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+:
Efficient Low Rank Adaptation of Large Models. In Proc. Int. Conf. on Machine Learning (ICML), 2024a.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. The Impact of Initialization on LoRA Finetuning Dynamics. In Proc. Neural Information Processing Systems (NeurIPS), June 2024b.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. DeBERTa: Decoding-Enhaneee BERT with Disentangled Attention. In Proc. Int. Conf. on Learning Representations (ICLR), October 2021.
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring Mathematical Problem Solving with the MATH Dataset. In Proc. Neural Information Processing Systems (NeurIPS), 2021.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-Efficient Transfer Learning for NLP. In Proc. Int. Conf. Machine Learning (ICML), June 2019.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), 2022.
Qiushi Huang, Tom Ko, Zhan Zhuang, Lilian Tang, and Yu Zhang. HiRA: Parameter-Efficien Hadamard High-Rank Adaptation for Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), 2025.
Ilse C. F. Ipsen and Arvind K. Saibaba. Stable Rank and Intrinsic Dimension of Real and Complex Matrices, December 2024. arXiv:2407.21594.
Ting Jiang, Shaohan Huang, Shengyue Luo, Zihan Zhang, Haizhen Huang, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Deqing Wang. and Fuzhen Zhuang. MoRA: High-Rank Updating for Parameter-Efficien Fine-Tuning, May 2024. Nww:2405.121300
Jikai Jin, Zhiyuan Li, Kaifeng Lyu, Simon Shaolei Du, and Jason D Lee. Understanding Incremental Learning of Gradient Descent: A Fine-Grained Analysis of Matrix Sensing. In Proc. Int. Conf. on Machine Learning (ICML), pages 15200-15238, 2023.
Damjan Kalajdzievski. A Rank Stabilization Scaling Factor for Fine-Tuning with LoRA, November 2023. arXiv:2312.03732.
Dawid Kopiczko, Tijmen Blankevoort, and Yuki M. Asano. VeRA: based Random Matrix Adaptation. In Proc. Inf. Conf. on Learning Representations (ICLR), January 2024.
Brian Lester, Rami AI-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proc. Conf. Assoc. Comput. Linguist. Meet., September 2021.
Eitan Levin, Joe Kileel, and Nicolas Boumal. The Effecc of Smooth Parametrizations on Nonconvex Optimization Landscapes. Mathematical Programming, March 2024.
Bingcong Li, Liang Zhang, Aryan Mokhtari, and Niao He. On the Crucial Role of Initialization for Matrix Factorization. In Proc. Int. Conf. on Learning Representations (ICLR), 2025a.
12

---

Xiang Lisa Li and Percy Liang, Prefix-Tume Tuning: Optimizing Continuous Prompts for Generation. In Proc. Conf. Assoc. Comput. Linguist. Meet., pages 4582-4597, August 2021.
Yixiao Li, Yifan Yu, Chen Liang, Pengcheng He. Nikos Karampatziakis, Weizhu Chen, and Tuo Zhao. LoftQ: LoRA-Fine- SiRRA--in-TTunig-AAAe Tuning-Aware Quantization for Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
Zhizhong Li, Sina Sajadmanesh, Jingtao Li, and Lingjuan Lyu. StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold. In Proc. Neural Information Processing Systems (NeurIPS), 2025b.
Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky, ReLoRA: High-Rank Training Through Low-Rank Updates. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
Kaizhao Liang, Bo Liu, Lizhang Chen, and Qiang Liu. Memory-Efficient LLM Training with Online Subspace Descent. In Proc. Neural Information Processing Systems (NeurIPS), 2024.
Rongmei Lin, Weiyang Liu, Zhen Liu, Chen Feng. Zhiding Yu, James M. Rehg, Li Xiong, and Le Song. Regularizing Neural Networks via Minimizing Hyperspherical Energy. In Proc. Conf. Computer Vision and Pattern Recognition (CVPR), April 2020.
Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-Shot Parameter-Efffiieen Fine-Tuning is Better and Cheaper than In-Context Learning. In Proc. Neural Information Processing Systems (NeurIPS), 2022.
Shih-Yane Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. DoRA: Weight-Deecopposed Low-Rank Adaptation. In Proc. Int. Conf. on Machine Learning (ICML), pages 32100-32121, 12,11,,,, July 2024a.
Weiyang Liu, Rongmei Lin, Zhen Liu, Lixin Liu, Zhiding Yu, Bo Dai, and Le Song. Learning towards Minimum Hyperspherical Energy. In Proc. Neural Information Processing Systems (NeurIPS), 2018.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo. Songyou Peng, Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter- Orthogonal Finetuning via Butterfly Factorization. In Proc. Int. Conf. on Learning Representations (ICLR), 2024b.
Charles H. Martin and Michael W. Mahoney. Implieit Self- Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learning. Journal of Machine Learning Research, 22(165):1-3,, 2021. ISSN 1533-7928.
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PiSSA: Principal Singular Values and Singular Vectors Adaptation Large Language Models. Proc. Neural Information Processing Systems (NeurIPS), 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), 2018.
B. Mishra, G. Meyer, S. Bonnabel, and R. Sepulchre. Fixed-Rank Matrix Factorizations and Riemannian Low-Rank Optimization. Computational Statistics, 29:591-621, April 2013.
Alireza Mossavi- Sejun Park, Manuela Girotti, Ioannis Mitliagkas, and Murat A. Erdogdu. Neural Networks Efficiently Learn Low-Dimensional Representations with SGD, In Proc. Int. Conf. on Learning Representations (ICLR), 2023.
Mahdi Nikdan, Soroush Tabesh, and Dan Alistarh. RoSA: Accurate Crameter-Efficiens Fine-Tuning via Robust Adaptation. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
13

---

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang. 7ach De Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang. Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, figh-PPeformance Deep Learning Library. In Proc. Neural Information P Processing Systems (NeurIPS), 2019.
Kaustubh Ponkshe, Raghav Singhal, Eduard Gorbunov, Alexey Tumanov, Samuel Horvath, and Praneeth Vepakomma. Initialization using update approximation is a silver bullet for extremely efficient low-rank fine-tuning, 2024.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng, Zhen Liu, Dan Zhang. Adrian Weller, and Bernhard Schölkopf. Controlling Text-to-Image Diffusion by Orthogonal Finetuning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Pranav Rajpurkar, Robin Jia, and Percy Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In Proc. Conf. Assoc. Comput. Linguist. Meet., pages 784-789, 2018.
Thomas Robert, Mher Safaryan, Ionut-Vlad Modoranu, and Dan Alistarh. LDAdam: Adaptive Optimization from oow-Dimessional Gradient Statistics. In Proc. Int. Conf. on Learning Repre- sentations (ICLR), 2025.
Mark Rudelson and Roman Vershynin. Sampling from Large Matrices: An Approach through Geometrie Functional Analysis. Journal of the ACM, 54(4), December 2006.
Mark Rudelson and Roman Vershynin. Smallest Singular Value of a Random Rectangular Matrix. Communications Pure and Applied Mathematics: A Journal Issued by the Courant Institute of Mathematical Sciences, (2(12))1707-1733 2009.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. Communications of the ACM, 64(9)999-106, 2019.
Tim Salimans and Durk Kingma. Weight Normalization: Simple Reparametcrization to Acceler- ate Training of Deep Neural Networks. In Proc. Neural Information Processing Systems (NeurIPS), volume 29, 2016.
Amartya Sanyal, Philip H. Torr, and Puneet K. Dokania. Stable Rank Normalization for Improved Generalization in Neural Networks and GANs. In Proc. Int. Conf. on Learning Representations (ICLR), 2020.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Common- sense Reasoning about Social Interactions. In Kentaro Inui, Jing Jiang. Vincent Ng, and Xiaojun Wan, editors, Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 4463 4473, November 2019.
Sholom Schechtman, Daniil Tiapkin, Michael Muchlebach, and Eric Moulines. Orthogonal Directions Constrained Gradient Method: from Non-Linear Equality Constraints to Stiefel Manifold. In Proc. Annual Conf. on Learning Theory (COLT), 2023.
Ying Sheng, Shiyi Cao, Dacheng Li, Coleman Hooper, Nicholas Lee, Shuo Yang, Christopher Chou, Banghua Zhu, Lianmin Zheng, Kurt Keutzer, Joseph E. Gonzalez, and Ion Stoica. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. In Proc. Conf. on Machine Learning and Systems (MLSys). 2024.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. Recursive Deep Models for Semantic Compositionality over a Sentiment Treebank. In Proc. Conf. on Empirical Methods in Natural Language Processing (EMNLP), pages 1631-1642, 2013.
Mahdi Soltanolkotabi, Dominik Stöger, and Changzhi Xie. Implicit Balancing and Regularization: Generalization and Convergence Guarantees for Overparameterized Asymmetric Matrix Sensing. In Proc. Annual Conf. on Learning Theory (COLT), 2023.
14

---

Dominik Stöger and Mahdi Soltanolkotabi. Small Random Initialization is Akin to Spectral Learning: Optimization and Generalization Guarantees for Overparameterized Low-Rank Matrix Reconstruc- tion. In Proc. Neural Information Processing Systems (NeurIPS), volume 34, pages 23831-23843, 2021.
Youbang Sun, Shixiang Chen, Alfredo Garcia, and Shahin Shahrampour. Retraction- Free Decentral- ized Non-convex Optimization with Orthogonal Constraints, December 2024. arXiv:2405. 1590.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang and Tatsunori B. Hashimoto. Stanford Alpaca: An Instruction-folloiing LLaMA model, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, and Others. Llama 2:
Open Foundation and Fine-Tuned Chat Models, July 2023, arXiv:2307.09288
Bart Vandereycken. Low-Rank Matrix Completion by Riemannian Optimization. SIAM Journal on Optimization, 121411323 January 2013.
Roman Vershynin. Introduction to the Non- Non-Asymptotic Analysis of Random Matrices. arXiv:1011.3027, 2010.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill. Omer Levy, and Samuel Bowman. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding, In Proc. Int. Conf. on Learning Representations (ICLR), November 2019.
Rachel Ward and Tamara Kolda. Convergence of Alternating Gradient Descent for Matrix Factoriza- tion. In Proc. Neural Information Processing Systems (NeurIPS), volume 36, pages 22369-22382, 2023.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural Network Acceptability Judgments. Trans. Assoc. Comput. Linguist., 7,625-641, 2019.
Adina Williams, Nikita Nangia, and Samuel R Bowman. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. In Proc. Conf. Assoc. Comput. Linguist. Meet., pages 1112-1122, 2018.
Wenhan Xia, Chengwei Qin, and Elad Hazan. Chain of LoRA: Efficient Fine-tuning of Language Models via Residual Learning. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
Nuoya Xiong, Lijun Ding, and Simon S Du. How Paremearamttiin Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initialization. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
Yuhui Xu, Lingxi Xie, Xiaotao Gu, Xin Chen, Heng Chang, Hengheng Zhang, Zhengsu Chen, Xiaopeng Zhang, and Qi Tian. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), 2024a.
Zhenghao Xu, Yuqing Wang, Tuo Zhao, Rachel Ward, and Molei Tao. Provable Acceleration of Nesterov's Accelerated Gradient for Rectangular Matrix Factorization and Linear Neural Networks. In Proc. Neural Information Processing Systems (NeurIPS), 2024b.
Tian Ye and Simon S. Du. Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorization. In Proc. Neural Information Processing Systems (NeurIPS), June 2021.
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for loRA optimization. In Proc. Int. Conf. Learning Representations (ICLR), 2025.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Models. In Proc. Int. Conf. on Learning Representations (ICLR), May 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a Machine Really Finish Your Sentence? In Proc. Conf. Assoc. Comput. Linguist. Meet., May 2019.
15

---

Yuchen Zeng and Kangwook Lee. The Expressive Power of Low-Rank Adaptation. In Proc. Int. Conf. on Learning Representations (ICLR), 2024.
Fangzhao Zhang and Mert Pilanci. Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models. In Proc. Int. Conf. on Machine Learning (ICML), June 2024.
Jialun Zhang, Salar Fattahi, and Richard Y Zhang. Preconditioned Gradient Descent for Over- Parameterized Nonconvex Matrix Factorization. In Proc. Neural Information Processing Systems (NeurlPS), 2021.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive Budget Allocation for Parameter-Efticient Fine-Tuning, In Proc. Int. Conf. on Learning Representations (ICLR), 2023.
Yilang Zhang, Bingcong Li, and Georgios Giannakis. RefL.RR:: Refactored low-rank adaptation for efficient fine-tuning of large models. (rxXiv2505./8877 2025.
Yuan Zhang, Jiang Hu, Jiaxi Cui, Lin Lin, Zaiwen Wen, and Quanzheng Li. Retraction- Optimization over the Stiefel Manifold with Application to the LoRA Fine-Tuning OpenReview, 2024.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
Jiacheng Zhu, Kristjan Greenewald, Kimia Nadjahi, Haitz Sáez Ocáriz Borde, Rickard Brüel Gabrielsson, Leshem Choshen, Marzyeh Ghassemi, Mikhail Yurochkin, and Justin Solomon. Asymmetry Low-Rank Adapters of Foundation Models. In Proc. Int. Conf. on Machine Learning (ICML), 2024.
16

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
All empirical results mentioned in the abstract are provided in the main text and/or appendix.
The theoretical results mentioned are proven in the appendix.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations of the applicability of empirical results are provided in Appendix A.
We discuss theoretical FLOP impact in Section 4.4; additionally, we provide an empiri- cal runtime comparison, illustrating the practical runtime impact in Appendix F.2.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approxinations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
17

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification:
Proofs for theoretical results are provided in Appendix D and E.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
We provide detailed pseudo-code outlining proposed procedure in Algorithm 2. We state detailed training hyperparameter configurations in Appendix G. Moreover, we provide details of our model evaluation strategy, employing widely accepted evaluation frameworks for reproducible evaluation.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
18

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
The code for our experiments and model checkpoints are available at httts:///ggthtt com/kcc--lonnpolaat
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: //n /nips. //wpps.cc///////e wttppcblcccgggiesssteessseeeeppe httpo////ggddees/eeeuuuiiiiiiis wwweeeuubmmsssooooooo.oo
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the general experimental setup in the main text and give the hyperparameters used Appendix G.
The supplemental material includes the code to reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
19

---

Justification:
We include error bars in terms of standard deviation over seeds where computational cost of multiple repetitions is within reasonable range (e.g., smaller scale experi- ments in Table 5).
In other cases, for fine-tuning models models with 2B+ parameters, we do not report error bars, as repetition with multiple seeds would incur significant computational overhead.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We outline the hardware used in Appendix G.
Peak memory usage and runtime impact are discussed in Appendix F.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics sttpp://neerrr httppppc.ccppplpppeep caubllc/Ethisccuddeee
Answer: [Yes]
Justification:
We have reviewed the NeurlPS Code of Ethics and confirm that our research conforms with it in every respect.
20

---

Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the broader impact in Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release assets that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
21

---

12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We provide the licenses in Appendix G.5,
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi wehoodewthhodeee com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide documentation for the released code in terms of README file and docstrings at appropriate locations.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not rely on crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
22

---

Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethies, workers involved in data collection, curation,
or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do rely on crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMS if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification:
We LLMs solely for purposes related to the methodology such as editing, writing, and formatting.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMss as any important, original, non-standard components.
Please refer to our LLM policy (https: /etarripp.ccc/ooffeeenneee////// for what should or should not be described.
23

---

Supplementary Document for PoLAR
Contents
Limitations, Broader Impact, and Practical Concerns
25
B Additional Related Work
25
Useful Facts
26
C.1 Angles Between Subspaces
26
C.2 Other Useful Lemmas
26
C,3 Equivalence of Matrix Factorization and LoRA
28
D Proofs for Asymmetric Problems
28
D.1 Riemannian Gradients of (4)
28
D.2 General Dynamics
29
D.3 Initialization.
30
D.4 Increasing Alignment
30
D.5 Non-Increasing Misalignment
32
D.6 Convergence @T) and Tr( v.)
33
D.7 Convergence of θ
36
D.8 Proof of Theorem
36
Theoretical Extensions to Symmetric Matrix Factorization
37
E.1 Detailed Setups and Main Results
37
E.2 Proof of Theorem
37
F
Additional Experiments
41
F.1
Additional Analysis on Stable Rank
41
F2
Runtime Comparison
42
E3
Approximate Feasibility
43
F4
Comparison with GaLore
44
F5
Gemma-3-122 Commonsense Reasoning
44
G
Experimental Setup
44
G.1
Commonsense Reasoning
45
G.2 Mathematical Reasoning
45
G.3 GLUE
45
G.4 Matrix Factorization
45
G.5 Licenses
45
24

---

A Limitations, Broader Impact, and Practical Concerns
Limitations. In our empirical evaluation of PoLAR we exclusively focused on natural language processing tasks using transformer architectures. We did not evaluate our method on other sequence modeling architectures (e.g., RNNs) or computer vision tasks. Also, we did not cover model sizes beyond 27B. Due to computational cost associated with fine-tuning decently-sized models with IB+ parameters, we repeat each dataset-model combination only for a single seed. For DeBERTa-v3-base, we repeat training with four different seeds.
Broader Impact. The theories and approaches developed are applicable in discriminative and generative use-cases. The proposed algorithmic tool improves fine-tuning performance on dis- criminative tasks such as sequence classification. Use-cases range from recommender systems to content-moderant pipelines where reliable classification is crucial. For generative downstream tasks, model output should be audited through automated gating or filtering mechanisms.
Practical Concerns. In the scenario of serving multiple adapters concurrently (Sheng et al., 2024) PoLAR and LoRA share identical storage and runtime costs at inference, as Or can be merged into either X Yr once fine-tuning is complete. We note that PoLAR incurs computational overhead during fine-tuning compared to LoRA, due to the landing field computation and the additional r X r matrix O As already mentioned, this runtime overhead compares favorably to a classical Riemannian treatment based on retractions (see Section 4.4). Moreover, we observe runtime benefits over other popular LoRA variants such as DoRA (see Table 6).
B Additional Related Work
Darameter-Efficien Fine-Tuning. The goal of PEFT is reduce the resource requirement for finetuning LLMs on downstream tasks. Some commonly employed methods include adapters (Houlsby et al., 2019) and prefix tuning (Li and Liang, 2021). The family of LoRA adapters (Hu et al., 2022) recently gained significant popularity. Beyond the works already mentioned in Section 1.1, there are several lines of work focused on improving or understanding different aspects of LoRA. Zhu et al. (2024); Hayou et al. (2024a) contribute to the understanding of the different roles of the low-rank factors. Gu et al. (2023) extend the framework to diffusion models. Xu et al. (2024a); Dettmers et al. (2023) improve memory- -emorr-efficiency through quantization strategies. Liu et al. (2022) further improve parameter efficiency. Zhang et al. (2024) apply landing methods to LoRA fine-tuning and present LoRA variants that constrain the projeejeetion to the Oblique and Stiefel manifold. Scale and rotation invariance is considered for efficiently optimizing vanilla LoRA parameterization (Yen et al., 2025; Zhang et al., 2025).
Optimization for Matrix Factorization. Existing works mostly handle matrix factorization problems under the BM parameterization, i.c., (3). Such problems are classical examples for understanding the optimization dynamics of LoRA. They involve a complex landscape characterized by nonconvexity. saddle points, and the absence of the PL condition.
Recent works have examined the convergence of several algorithms under the BM parameterization, such as GD, Alternating GD, and ScaledGD (Du et al., 2018; Ward and Kolda, 2023; Li et al., 2025a) in overparameterized settings. Another closely related problem is matrix sensing. The convergence behaviors of different algorithms under the BM parameterization can be found in e.g., (Stöger and Soltanolkotabi, 2021: Zhang et al., 2021; Jin et al., 2023; Xiong al., 2024).
Stable Rank. The stable rank (Rudelson and Vershynin, 2006) is a continuous and perturbation-roll analogue of the classical linear algebraic rank (Ipsen and Saibaba, 2024). It is given by the square of the ratio of the Frobenius and spectral norms. In the context of regularization for convolutional neural networks, constraining the stable rank has been found to curb expressivity, mitigate overfitting, and reduce memorization (Sanyal et al., 2020; Georgiev et al., 2021).
Other Related Work. Qiu et al. (2023); Liu et al. (2024b) use a multiplicative orthogonal matrix to fine-tune diffusion models, attempting to preserve the angular relationships between neurons of the pre-trained matrix. This is motivated through the preservation of hyperspherical energy of a weight matrix. Hyperspherical energy measures the distribution of neurons on the unit-norm hypersphere and attains its minimum when neurons are distributed uniformly across the sphere (Liu et al., 2018;
25

---

Lin et al., 2020). A low stable rank of the LoRA update can be understood in terms of a state of high hyperspherical energy of the low-rank update.
C
Useful Facts
C.1 Angles Between Subspaces
Angles between two subspaces are known as principal angles. Suppose that n 2 and n 2 q. and let U and V be two linear subspaces of dimension x and x q. Then the principle angles 0, E [0, 21. Vi & (pn g) is defined as
uu
81 = max
arccos
u
neu.vvy
(u
u Eu, U, vj, El,... 11.. #1. ||
0: = max
arccos
There is well-known relation between principal angles and SVD. Let U and V be basis of U and V respectively. It can be seen that all the singular values of U V belong to [0. 1]. Moreover, the principle angles defined above 1973). For convenience of this work, we refer to the singular values of as principal angles (instead of the arc-cosine of them). If the basis and are both from St(m,,), we sometimes use the term "alignment", where say U and V aligned all the singular values of U V are 1; or in other words, they share the same column space.
just the arc-cosine of these singular values (Björck and Golub,
The principal angles are also related to the geodesic distance on Grassmann manifolds. Oftentimes, people the term chordal distance refer to d(U, V) V sin 6,, where 0, are principal angles between two subspaces spanned by U and V. The square of the chordal distance coincides with our notation Tr(I - 1b. where 4 is defined in Section 2.2.
C.2 Other Useful Lemmas
Lemma 2. Given a PSD matrix A, we have that (I+A) I A.
Proof. Simply diagonalizing the LHS and RHS, and using 1/(1 + A) 2 1 X, AA 2 0 gives the result.
Lemma 3. Suppose that X E St(m,r), U St(m,ra) and F r. Let U1 E (m- be the orthogonal complement of U. Denote & = UTX and Ω UT X. It is guaranteed that of )b) + (S2) = holds for i E (1,2.....
Proof. We have that
I 1
X X UU
U X
(7)
T Φ n...
Equation (7) indicates that v TTu v and + T commute, i.e.,
+ + ( 22) = (Φ (I - ②) =
④b $ T
[I. + ( ②) (Ω ( TΦ).
The commutativity shows that the eigenspaces of T + and Ω coincide. As a result, we have again from (7) that (f( + of (2) = for E (1, The proof is thus completed.
D
Lemma 4. Suppose that P and Q are m x m diagonal matrices, and their diagonal entries are non-negative. Let S be PD matrix m with smallest eigenvalue Xmins then we have that
Tr(PSQ) > Amin Tr(PQ).
26

---

Proof. Let and be the (i, 2)-th entry of P and Q respectively. Then we have that
Tr(PSQ)= Σ piS Amin
Epig Amin Tr(PQ)
(8)
where the inequality above comes from the positive definiteness of i.c. St,s e, Se, 2 Amin Vi.
Lemma Let E Rmx" be a matrix with full column rank and B E R be a non-zero matrix. Let mminl.) be the smallest non-zero singular value. Then it holds that minn (AB) 2 Juin (A) miin
Proof. Using the min-max principle for singular values,
Mii (AB)
min
||Ax||
|x||=11,x SpooSpn B)
Bx
min
A
|Bx|
Collsen B)
Bx
| ye ECColpaanBB) Ayll - [x
min
min
Ayl|
min
Bx
|byl Iyll=lyyE
(
min
|Ayl
min
[x
|
min
(A) O (B)
where (a) is by changing of variables, i.e., y = Bx/ Bx|
Lemma 6 (Theorem 2.2.1 of (Chikuse, 2012)). If E gmxr has entries drawn iid from Gaussian distribution (0, 1), then X Z(Z 1)2 is random matrix uniformly distributed St(m, r).. Lemma 7 (Vershynin (2010)). / I E Rmxrr is a matrix whose entries are independently drawn from (0, 1). Then for every 2 0, with probability least (xp(-72/2), we have
(Z) mm + +
Lemma 8 (Rudelson and Vershynin (2009)). I E R" is matrix whose entries are indepen- dently drawn from (0, 1). Suppose that 2 r. Then for every T 2 0, we have for some universal constants C > 0 and C2 0 that
P o, (Z) S F( 1)) s (C17) exp(-C2r).
Lemma 9. ISU E rA) is a fixed matrix, X E St(m,r) is uniformly sumpled from St(m, r) using methods described in Lemma 6, and > TA then we have that with probability at least 1 wepppppmmmme (C17)"-rat exp((-2dd).
Tr (U 7(r-rA+1)
6/mr
Proof. Since X St(m,r) r) is uniformly sampled from St(m, r) using methods described in Lemma we can write X Z(Z" T ((ZZZ)-1/4) -1/2, where E Rmxr has entries iid sampled from A A 1). We thus have
or (U x σr Z(
Now consider U Z E Rr It is clear that entries of U Z also iid Gaussian random variables (0, 1). As consequence of Lemma 8, we have that w.p. at least "CGr)"-Tati exp(-C2p
or (UTZ) 2 T(V VTA ----1 1).
We also have from Lemma that with probability least 1 exp(-m/2)
9 ZZ 2) oZ - (2 vr)².
27

---

Taking union bound. we have with probability at least 1 1-cxp(mmm22) -
Tr (a)
ora (U Z) (2)
T(VF- Vra -1]
7(r (1r-ra+t]
T(r -A +
2/m + /r
3/m 2//
6/mr
where (a) comes from Lemma 5.
Lemma 10. Vf St(n,ra) AA) a fixed matrix, and Y St(n, r) is uniformly sampled from St(n, r)
using methods described in Lemma 6. Suppose TA. Then we have that with probability at least 1 exp(-n/2) (Cir) -At1 exp(-C2r),
TA (V 7(r r(r-rA+1)
6/
Proof. The proof is omitted since follows the same steps of Lemma 9.
C.3 Equivalence of Matrix Factorization and LoRA
Whitening data refers to transforming the data such that the empirical uncentered covariance matrix of the features is identity. Suppose D E R"xN holds N training examples in its columns with n features each, then whitened data refers to having DD In- We follow standard arguments laid out in Arora et al. (2018); Li et al. (2025a) to show that low-rank adaptation linear model with whitened data and squared loss can be written as a matrix factorization problem. Consider the minimization of
L(X, l(W, XY D D A
where A E mxN holds labels for each example, Wo mmxa is the pre-trained weight, and X E R mxr Y Rxr are the weights of LoRA. Rewriting L(X, Y) yields
L(X, ((Wo+XY A
=Tr([(Wo XY ()( XY))D-A)")
-T((Wo + =Tr((W, XY DDDT(WWW XY (
(DAT
-Tr(AD (Wo + +
( Tr( XY )WW + +XYT)) Tr((Wo + XY XYT)DAT)
-Tr(ADT (Wo x) Tr(AAT)
M Tr( XY A)((W + XY A) Tr(AAT) + Tr(AAT)
where (a) uses the fact that the data whitened and (b) defines A AD E Rmxn i.e. the matrix to be factorized. Thus, we can write
L(X, Y) = (I XY A1i +
IXY 1 +c
for A' Wo A and constant == Tr(AA Tr(AA T). Using the same arguments, one
can frame low-rank adaptation of linear model with the PoLAR parameterization as the matrix factorization problem given in (4).
D Proofs for Asymmetric Problems
D.1 Riemannian Gradients of (4)
One can start with the Euclidean gradient with respect X, as E, (X,,, A)Y,O)
(X,X, Im AY AY,OT. Note that for O = X. AY, (i.e., = 1) the Euclidean gradient is skew- symmetric such that it is already contained in the tangent space of St(m, r) at X, (i.c., X, E,
28

---

E, = 0). yielding equality between the Riemannian gradient E, and Euclidean gradient E, This equivalence holds regardless of whether the Euclidean or canonical metric is used. In other words, the Riemmanian gradient for X, at iteration is given by
E, - (Im X,X, XXY,)
Similarly, one can obtain the Riemannian gradient for Y via
F. = -In Y.Y, A X,O,
D.2 General Dynamies
Here we derive several equations that are useful throughout this section. Note that the choice of learning rate 7 = will be leveraged in some equations.
Dynamics on X:, E,, and p., From the updates in Alg. 1, it straightforward to arrive
R 3 U. = - (Im X,X, !YY,!!
- (Im X,X, UEV
(Ir, D.
Similarly, also have
R E, E, O,Y, A (Im X,X, X,X,) AY
O,Y, A [Im X,X, AY,0, 0, Σ(I,, D O
Applying 0i (V)) S. and 01 (A) = (2) 1 the equation above, and leveraging in the update of O (i.e., θ X. AY, Φ ΣT,), we have that
σ1 E, E.) S o (E) क. = (IrA $D
(10)
And the dynamics on the alignment .; E RR can be written as
$ 1+1
1/2 (a) L, 7!IrA 2 Σ V Σ + E,) 1/2
_ + nI 2 E + (I, + E E.)
where (a) uses O, X AY, o. Σv,, Hence, we have that
P.+1 Φ
(11)
2+1
[Ta + tI Φ EV, 4 Σ $ (I, + 1 क. [T, + +, E(,, bb )
Dynamics on Y,, F,, and I., From the updates Alg. and similar the derivation above, we anrive at
Braxr 3 V F. --V (I, - Y,Y,) A X,O,
= =-V (I, Y,Y, VEU X,O,
-II,, L. E
Moreover, we also have
grxr 9 F. F. = = O, A(I, Y.Y.) Tx0,,
= 0 A - Y.Y, A X,O,
θ, Φ E(I, V. O
Applying 0i (.)) < and (A) = to the equation above, we have that
(F, F,) < (E) - V. a (I,, V.
(12)
29

---

The alignment v, = V Y E R' can be tracked via
V P4+1 = | + (lrx - V. F -1/2
(h) Ixa n(rr V. トル ز v (I. + n7 FF F.)
1/2
where (b) uses O X, AY, . Σ Finally, have that
1 V 1+1
(13)
IL, + ,, I(T,, v. = [Tr + n V. ΣΦ, P Σ V (T, + F.) V. IT
With these preparations, ready prove our main results.
D.3 Initialization
Lemma 11. Suppose that Xu and Yo are uniformly sampledrrom St(m, r) and St(n, r), respectively, using methods described in Lemma 6. There exist universal constants and 02 such that whp the following holds
TT (8)) (U (U))))
Xx)
r -- TA +1
2
r - FA
Cimr
v /imm
Tra ( = Or YV)
r TA+1
TA
C2nT
C2n
Proof. The proofs, the constants Ci and (2, as well as the exact probability follow directly from Lemma 9 and Lemma 10.
D.4 Increasing Alignment
Lemma proved in this subsection, where the detailed proof is divided into two parts.
D.4.1 Increasing Alignment Between X, and U
Lemma 12. Consider Alg. 1 with 17 < 1 and 7 Let 3r == (I 2. If holds that
2(1 - 17 Be 80 (v.) Tr((Ir, (..0) Tr
x²
we have Tr( IT) TT)
Proof. From (11), we have that
P.+1 Φ
(14)
[r + 1/ - -. EV,
Σ 4. [I,
E, T [., + ME,, V. E(IrA bb
(a)
] Ir (I,,, $ ΣV
V Σ 1 (I,
E,
I,, E E(I, $.
(
(1 - 1/3,,)
I,, + +
F. 4.
2 (1 1281)
T + + Φ
1. ΣΦ
2 $ Σ V E(I, $
where (a) is by Lemma 2; and (b) is by [I. - " E,) ž (1 (1- 11-720 (I,, Φ I,, as result
of (10), and we write 3 == (I,, Φ for convenience. We also dropped the fourth term (Ira o. E. ΣΦ E. E( $ given its PSDness. Note that B, [0, 1].
30

---

Now let the EVD of Φ = Q,A,Q where both Q: and A, are PA matrices. Note that
A, 3 I Then we have that
Tr
(I, (I
2 D. E
(15)
Tr Qc Q. EV, V EQ.AQQ.)
Tr
(Ira
A. EV. v EQ.A:)
(4)
of
Tr (Ira - A1))
R²
o
Tr
(Ira $ )
x²
where (c) is by Lemma 4 and Lemma 5. More precisely, the PSDness of Q V. EQ, justifies the prerequisites for Lemma 4, and then we use or (Q) V EQ:) 2 o (D) (v.) of (k²)
Taking trace on both sides of (14) and plugging (15) in, we arrive 2n (1e)
Tr( f 2 (++1)
Tr( T.) +
Tr (Ira 0.0.)..000
(16)
R²
Simplifying this inequality gives the results.
D.4.2 Increasing Alignment Between Y, and V
We proceed by proving the analogue of Lemma 12 for the alignment between Y, and V by following the same steps.
Lemma 13. Consider Alg. 1 with 1j
< and 7 = Lei 6, : 01 V.V.). ). Ifil holds that 2(1 1-- 80) (D.)
Tr (IIrA L. v... 2 To)
n²
we have Tr( L T 2 Tr v.)
Proof. From (13), we have that
2z+1 t+1
(17)
[T^ + +T 1 V.
EΦ
-
V [I, + nE E(I,, DD) F.
I, + (IrA - V. E Σ
[L, D E(I,, [.
(1 n/6)) [,, + |II, - V.
1 [.
(1 7 V.V. + (IIA v 4 Σズ ΣΦ
ΣΦ, 4. E(I,, V v
where (a) is by Lemma 2; and (b) is by - F,) ž (1-
V. ) as result (Ira- L for convenience. Note that 6, [0,
of (12), and write
Now let the SVD of V.V. P.A.P.. where both P. and A, are A TA matrices. Note that 0 A, I,a. Then we have that
Tr (I, - 1 V 2 ΣV,V,
(18)
Tr Pi((I, - A.P. 0 EP A.P.
Tr (In A.)P. ΣP,A.
(0) 22
Tr
-.))A.)
x2
{
Tr
[Ira
V. 2 v V.V.
x
31

---

guaranteed to have where (c) is by Lemma 4 and Lemma 5. More precisely, we use the PSDness P; ΣΦ, v, EP, for applying Lemma 4, and then employ ora (P; DΦ, $ E), 2 o7 (E) (1,) of (4; /)/k²²
Taking trace both sides of (17) and plugging (18) in, arrive
Tr V 141 V 111 1- 1266 2 v +
2n
(
Tr (IrA - -V.R,
(19)
R²
Simplifying this inequality gives the results.
D.5
Non-Increasing Misalignment
Misalignment of X. refers to the principal angles between X; and the basis the orthogonal complement of U. Similarly, one define the misalignment of Y..
D.5.1
Non-Increasing Misalignment of X,
Lemma 14. Denote the orthogonal complement U as U E (mmx(m-ra), Define the (m-rA) xr matrix Qz == X characterize the alignment of X, and U_. Under the same setting of Lemma 12, we have that 1441 I 52,52 Moreover, if TA I m
o/ 4 (++1)= 2 o7 (t).
Proof. From update (5b), we have that
Ω UI(X,
n),,) 17 E E, -1/2
= (2: + 7
[Im - X AY 0, +1 E, Ex) 1/2
@ (S4 x, ( E,) 11/2
-2 - 1000 11, -1 E,) -1/2
where in (a) have used U A 0 and θ -X AY, With this,
that
2 (I, - 0 (I, + E E.) EI, -,,0,,,,
S
where the last inequality comes from the fact that the three matrices in between are all PSD and their largest eigenvalues are smaller than given our choices of This gives the proof for the first part of this Lemma.
To show o² (P++1) 2 of2 (P)), notice that given 2r S m, we have from Lemma that of (4)) 1 - o7 (52z) and o², (P++1) = 1 - o², (S2+1). The conclusion is straightforward.
D.5.2 Non-Increg Misalignment of Y:
Lemma 15. Denote the orthogonal complement of V V (n-rA)))), Define the (N-TA) X
matrix Ω VIY, characterize the alignment of Y, and V.. Under the same setting of Lemma 13, we have that 144151 S2,52. Moreover, if TA S n, it is guaranteed to have
o, (4+1) 2 o., (.)..
Proof. From update (5c), we have that
Ωu1 VT(Y, FI, + 1/ F. F,) -1/2
(Ô, + +/ - Y.Y,) A x,O,) (I, +I, F.)
-11-//
(1, - 0,) 11/
With this, we can see that
S 24112 = 02, [I, - (.,0)
F -1, (7.
3 Ω
32

---

where the last inequality comes from the fact that the three matrices in between are all PSD and their largest eigenvalues are smaller than given our choice of 17.
To show o (v (+11 2 of ()),, notice that given 2r S m. we have from Lemma 3 that of (1)) = 1 r (St) and of (++1) = 1 =1-o, of (St+1). The conclusion is straightforward.
D.6 Convergence of Tr( and TV(
D.6.1 Dynamics of Tr(
Lemma 16. Suppose that A S 2' and let p minf (r-ra)²} Choosing 17 01 0(p mi mer
and 1, Alg. 1 guarantees that after at most T 0 Tr(I,A - $.
('(r--aa) + m²r'n' p(r-ra) log !) steps
Proof. By rewriting (16), we arrive at
+r( o (+1 Tr(
(20)
2n (V.) (1 (1- Trr - (. - Tr
2 x2
Based on (20), we discuss the convergence in three different regimes.
Phase Tr(I,, 2, 2 2 TA- 0.5. This the initial phase, and the condition is equivalent
to Tr S 0.5. For notational convenience, let the SVD of T. Q,A,Q, Given
these conditions, it can be seen that σ, (I, T or Tr. (Ira A,) 2 0.5. Recalling that B. oI Φ < 1, we can simplify (20) as
2no7 (v))
Tr(2 1 T+1) Tr(
(1 - 10- - A.)A.) AA)A) ( TT(
x²
1 IV
10? (1))
11-1)))T(AA) Tr(
x²
n(r rA)?
(1 - 1/ Tr 2 2,) T2
R c2nr
where (a) uses or (I, A.) > 0.5; (b) uses Lemmas 15 and 11, which jointly imply that
of (v)) of (o)) 2 (r rA)² / for some universal constant 0o defined in Lemma 11.
Rearranging the terms, we arrive at
Tr( + TT1) 2 (1+ 1(-
(1- -1² (1=) - ) TT(
RZegnr
which is linearly increasing once the term in parentheses is greater than 1. This amounts to choosing a small enough 7), i.e., S 0 (1=ra)²)
R'
Phase II. 0.5 < Tr(I,, - 2 < <TA - 0.5. Suppose that Tr((I,, - A.)A.) 2 p. for some p > to be discussed shortly. Let 17 = 0
( mm
p(rr-ral)) and I 0.5, it is straightforward to have
2A)2 (1 17 Tr A.)A.) TPTA (21)
Tr(2 2 [42) Tr(2.
R222n
2n(r ra)2
(1- (1- A.J..) -1²r
k2
(r (r TA)* A1
Note that the O(-) notation ignores the dependence on constants including ei and c2. This means
that per step, Tr( Φ increases least by A1. Consequently, after at most (rA 1)/A, Or m (p2 2 (r () iterations, RGD leaves Phase II.
33

---

Next, we show that p 2 (min
(r-ra)" }). Notice that Tr Tr((T,, - A.)A.)
EA of
m mr
(++) (min (r-ra)e] }), where the last inequality comes from of 7 (+:)(1 o
m mer
the facts that i) for a E bJ with 0
0.5 < b < 1, the smallest value of x(1 x) is
min(a(1 - a), 6(1 - b)}: and, ii o (P:) belongs to interval (a, b] with = O (M-rs> and b= 24-05 = 1-
<
Lemmas 14 and 1l are adopted to calculate that is, o (.:) 2
2
O((r - (r)) m
Phase III. Tr 2. )= S 0.5. This is regime near the optimum. An implication of this phase
is that Tr( +b 2 T. 0.5. Given that the singular values of Φ belong [0, 1]. it can be
that rra 2 Fra (Ar) 2 0.5. Together with Bg S 0.5 in this scenario, we can simplify (20) as
Tr( 4 [+1 Tr(
2np (1,))
- 1/2
Tr Tr((Ira A.) Bc Tr(
R²
(c) 1/ T2)
1 112
Tr(IrA -A)) 1/33rA
62nrk?
2
= 1/rr - rA)2
1- 1
Tr(InA- - 2,00)) - - B
c2Rrk?
2
(4) (r - r2)
1 1/²
Tr(IrA 0 -
- -
02Rrk?
where (c) uses Tr (Ai) 2 0.5; and (d) comes from Bi S Tr(IrA -B,0,). This further implies that
Tr(I, P411 1
(+1)
Tr(I,. (.
S ( MiT-TAll T2)² (1 (1-2) 2
112
Tr(I. - x, + r - -.
C2nrR²
Reorganizing the terms, we arrive at
Tr(L. 1/+1 T D. (1
1 (1- + r²ra Tr(I.,, -D,
(22)
C2nrk?
This indicates a linear rate until we achieve optimality once 1j chosen sufficiently small.
Note that our choice of y ensures the conditions in Lemma 12 are satisfied, indicating that an increase of Tr(#, T..) per iteration is guaranteed. This means that Tr($b,0 +b) traverses Phase I, II, and III consecutively. Combining these three phases together gives the claimed complexity bound.
D.6.2 Dynamics of Tr( ( (
Lemma 17. Suppose that S 2. and let min 1 (r-ra)²} }. Choosing = O O(p(r-ra))
mer
and 7 = 1, Alg. guarantees that after at most T = O Mar (r(rr)))" + p(r- log 2) steps Tr(I.
=²,3,4
V, v.) S.
Proof. By rewriting (19), we arrive at
Tr( Tr( 1 T+1) Tr(
(23)
2n (.)) 1726,) Tr V V. Tr(3
R²
Based on (23), we discuss the convergence three different regimes.
Phase I. Tr(I, V.VT) 2
0.5. This is the initial phase, and the condition is equivalent to Tr( L V < 0.5. For notational convenience let the SVD of V. P.A.P... Given these
34

---

conditions, it can be seen that rra AI,
v
= Tra (IrA - Ad) 2 0.5. Together with 6G S 1
(recall that 6, = 01 (IrA - 4 41 )) we can simplify (23) as
210 (p.)
1+1 v [411 -
[1- (T- (Ira - A.)..) - TT
X²
(0) 10 ($t)
(1 -1-1"- Tr(a)) - -1)
x²
(9) 7 - ra)² (1 TT U -
where (a) uses Tra (1 Ad) 0.5; (b) uses Lemmas 14 and 11, which joindy imply that o² (+t) 2 of (oo) 2 (r- - TA) (c(2/(((rr) for some universal constant ej in defined in Lemma 11.
Rearranging the terms, we arrive
2 441 (1
n.eymr 11--11 172) (1
1r "(- (1-
- T((V,
-
which is linearly increasing once the term in parentheses is greater than 1. This amounts to choosing
a small enough 7). i.e., S 0 *mm (==)))
Phase IT. 0.5 < Tr(Irr V. <TA- 0.5. Suppose that Tr - A,)A,) 2 p. for some to ((r-TA)))
be discussed shortly. Choosing 17 S 0.5, and 17 O
p3 ) is straightforward to have
Tr( V(41)) (T) 2m(r rA)2
(1 (1-17))r((() (.)ā,) - T7²rA (24)
2n(r rA)2
(1 (1-17)Tr(((1a A,) - 12²r
x2emm
2 0 (r ( ra)* ;= A2-
mmmm²
Note that the (-) notation ignores the dependence on constants including and c2. This means that per step, Tr( x D at least increases by Az. Consequently, after at most (TA - 1)/A2 O(rAr3 2 Amm /p² (r rA)*) iterations, RGD leaves Phase II.
Next, we show that 2 ( n² (m-ral² }) 2 O(min (m-ra)²))
}). Notice that Tr((Ira
m
At)A.) 2 (Z (V.) 2 o² ( (1 o² ()))) 2 (minn (r-ra)²] }), where the last inequality comes from the facts that i) for x E [a, bj with 0 < a < 0.5 < b < 1, the smallest value of a<1 x) is min(a(1 - a), - b)); and, ii) o (V)) belongs to interval [a, b]
with a- -(( and b
r.-0.5 = 2F4 < I. Lemmas 15 and 11 adopted to ra
nr
calculate a, that is, a = of, (v)) 2 2 (Po) = (0((rra))))
- r2)2
Phase III. Tr - D I 0.5. This is a regime near the optimum. An implication of this phase 2 Fa- 0.5. Given that the singular values of L V belong to [0, 1]. it can be
that Tr L T v
seen that on (V, M = Tr (A.) 2 0.5. Together with d, S 0.5 in this scenario, we can simplify (20) as
Tr( ( V T 2noo (1))
1+1 (T
1 1
Tr Tr((I,, -A.)A.) - (.
R²
2
(6) nfr T2)2
1- 7
Tr(InA -A,) -17brA
umrrR
n(r - ?
1- 17?
Tr(IrA V. - -176.TA
quTrk?
2
(4) 1/r 2
1 1
Tr(I, V.D,) - -.
mrrk?
2
35

---

where (c) comes from Tra (A,) 2 0.5, as well as of ( of (1)) 2 (r (a)2/(cmmr)) and
(d) uses 6t s Tr(ha V. ). This further implies that
Tr(Lr, 441 v 14 Tr(Ira v. S
(1 (1-2) Tr(IrA V. + PfTTTTII - D.).)
17
cpmrk?
Reorganizing the terms, we arrive at
Tr(I,A L (14 (1- CIMTK²
n)(r 1 rA)
(1-²) + t Tr(l.,- v.
This indicates linear rate until achieve optimality once 1j is chosen sufficiently small.
Note that our choice of y ensures the conditions Lemma are satisfied. In other words, increasing Tr(V, ! across / is guaranteed. This means that Tr( (r, (e will traverse Phase I, II, and III consecutively. Combining these three phases together gives the claimed complexity bound.
D.7 Convergence of O
Lemma 18. Suppose that at iteration t, Alg. with = -1 satisfies Tr(I,, - S P1 and Tr(I,, W ) S pn. It is guaranteed to have [Xx,, Y,, O.) = Opp + p2).
Proof. Recall that = implies θ = X, AY,, we thus have that
| - A | AY,Y, AllF
|X,X, AY,Y, AY,Y, +AY,Y, All
[Xx,X I AYY, + IAYY,, I.IIFF
(2) 1XX,XX I. Y.Y, + | (Y,Y,
II)lIF
$ /| - X,X, UF + V YY-YYY,)II
where (a) uses the compact SVD of A UEV. Now we have that
[I X,X. Ju Tr Tr(U U (Im XX, X,X, )U)
= Tr(II, 2.0.) S11
Similarly, we also have
(I Y.YD) - (I, -.,,Y,,,,,,,
= =T([Ir - V.V.) S p2.
Combining these inequalities, we have that
(b) $ EI - + (I, Y,Y, = + + X,O,Y. A
where (b) uses (a + b)2 < 2a² + 2b2. This finishes the proof.
L
D.8 Proof of Theorem
Proof. The proof is straightforward. We apply Lemma 16 and Lemma 17 show that within m²,",²
O( mm3 Tras + p(r-ra) log 2) iterations, we have Tr(I., - T v < c and Tr(I,, - Φ < Then, Lemma 18 is adopted to reach the conclusion.
((((rrr))""
36

---

E Theoretical Extensions to Symmetric Matrix Factorization
This section extends theoretical results to symmetric matrix factorization problems. While slightly deviating from the LoRA ap- plication, we still include these results be- cause we believe the improvement of the and λ dependence is important in its own right.
Algorithm 3 RGD for porae--arrmettrrized (25)
Require: Learning rates y, sample Xo uniformly from St(m, r).
for t= 0 do
Find O via (26a)
Obtain Riemannian gradient G, using (26b)
Update X with (26c)
Let B E Rmxm be a symmetric positive semi-definite matrix with rank B) TB- Its compact SVD is given by B UΣU
end for
where U E gmxrr and E Rgrwxrn, Without loss of generality, we assume dI and Trn ()) 1/k with denoting the condition number.
E.1 Detailed Setups and Main Results
In the same spirit of the asymmetric problems (4), the standard BM factorization is reformulated as
min f X, )) 12 xexx - BIi, s.t. X St(m, r), O 1xxr.
(25)
X
We again consider the overparameterized setting where r > TB-
The algorithm solve (25) shares the same principles as Alg. 1. At iteration we first apply GD to update + using V (X,, .,-1). which gives
+ (1 7) 0 + 7x, BX,
(26a)
Next, the Riemannian gradient (of X,) calculated (X,, Q.) is given by
G; = - (In X,X, B BX,
(26b)
With G; and polar retraction, the update on X, is given by
X,+1 (X; GI, + G G.) The detailed algorithm summarized in Alg. 3.
(26c)
Main Results for the Overparameterized and Symmetric Problem. We show that the PoLAR parameterization optimized with Alg. can exponentially improve the dependence of compared with GD applied on BM.
(r-re)") Choosing 11= O (00(00((((0) p(r-ra)² ner
Theorem 2. Suppose that FB S 2, and let p == min
p2
and 7 = 1, Alg. 3 guarantees (XT, ①r) S é for all T 2 O
Mirrarn k'm mr3k*
log !).
(rrr-r p(
Note that lower bound of Q( ) applies to GD on the overparameterizee matrix factorization problem with BM parameterization (Xiong al., 2024). This shows that our PoLAR parameterization exponentially improves the dependence, transforming a sublinear rate into linear one. Moreover, can be seen that the convergence of Alg. improves with overparameterization, since cr
induces a tighter dependence on r compared to TB + e for proper e >> 0.
E.2
Proof of Theorem
E.2.1
Dynamics
The alignment matrix is defined b, :- U X, E Rraxx, Next we derive several equations and inequalities that help to establish the main theorem. Note that the choice of learning rate ? = will be leveraged in some equations. We start with
x 3 G. G. x BX, B X,X, BX,X BX, x. BX,X B - X,X, BX,X BX,
0. E(I, Φ ) 0
37

---

where we use R'xr θ = x, BX; + EΦ, Applying 01 ( < and 01 (B) = 1 to the equation above, have that
ai (G G.) S of(o)o - -.,,))
=0 or - ).
(27)
We also have that
1+11 = { + 7 - $ 0 G Gt) -1/2
[I + nTr -
Σ (T, G G₁) 1/2
This gives that
P++1 4
(28)
2+1
Irr + (Irr 22 E/ G [To + ,, E(1r x2
With these preparations, ready to prove our main results.
E.2.2 Initialization
Lemma 19. Suppose that X. is uniformly sampled from St(m,r) using methods described in Lemma 6. There exists universal constant 01 such that whp the following hods
Trg (Po) = Dre Xo)
Xo) 2
r TB +1
TB
Venteer
Crmr
Proof.
The proofs, well as the constants C1, and the exact probability, follow directly from Lemma
E.2.3 Increasing Alignment in Symmetric Problems
Lemma 20. Let Br = 01 (Ir
(b Assuming < 1, = 1, and
2(1 17 3,)0 (e))
Tr( - 2,, )., 2 m/. (1)
x2
holds, we have Tr 1 --1 I.) 2 TT)
Proof. From (28), we have that
P.41 $ 1+1
Irg + + -.
ΣΦ
$ [r + + E(I, +145))
(a)
I1g +1(( 2
EΦ, 2
4b (I, G G. 1 1, E E(I, 2
(6)
t (1 1 11) + (I, の
ΣΦ. Σ Φ [1 + E/
E(I, $.
where (a) is by Lemma 2; and (b) by [I, - Gf (1-
2 I as result
of (27), and we write 3, = a (I,g
Φ for convenience. Note that 3. E [0, 1].
Now let the thin SVD of & be Q.A.P. where Q2 E RreeT A R and P Rrrra This gives that
ト 1 +
(c)
Σ (1 -1-1) 17) Qr Ire + (1 A2) + 1SS,A,S, - -A7) Q
m (1 (1- ( A? + t A?)S,A,S,A, A - Q2
38

---

where in (c) we denote S, :- Q EQ, which is PD. Noticing that 1 - r2 2 0, and taking the trace on both sides, we have that
Tr( T $ (+1)
(29)
1 - 7
2
Tr((
A + A, -A)))
(4)
2njorn (A?)
2 Tr
+
Tr (Irg A3)A2)
k²
2n (A3)
- !
R² Tr Tr((Ira )
where (d) by Lemma 4 and Lemma 5. More precisely, we use org (S,A oig (S,)ora (A)) Trg (AT)/ Simplifying the inequality finishes the proof.
0
E,2.4 Non-Increasing Misalignment in Symmetric Problems
Lemma 21. Denote the orthogonal complement of be U1 E mm-xm--re) Define the (m-rB TB xr matrix V, = U to characterize the alignment of X, and U 1. Under the same setting of Lemma 20, we have that 1441 V t+1
V. Moreover, TB S 2' guaranteed to have
o 2 (++1) 2 o02 (R.).
Proof. From update (26c), we have that
V r+1 UI(X. G, + G.)
G,) = ( + n - X.X, BXX + G,)-1/2
(U)
-I G 1/2
With this, we can see that
1+ V T 2+1
v, (1
-.) 1)) T.
3 W.
where the last inequality comes from the fact that the three matrices between are all PSD) and their largest eigenvalues are all smaller than I given our choices of 1). This gives the proof for the first part of this lemma.
To show of ( (1+1) 2 of (..), notice that given 2r S m, we have from Lemma 3 that a (T.)
1 of2 (Iz) and of ( e++1 i) =1 - of q 1.).. The conclusion is straightforward.
E.2.5 Convergence
The proof of Theorem a direct result of the following Lemmas.
Lemma 22. Suppose that vB S 2. and let p p= min
(r-rg)? Choosing n = 0 0(((r-rp)) var
m
r"x²m
and 7 we have that after most O 00 (r-ra)) p(r-ra] log !) steps, Tr(Irg 1
(Earr mm mr?
Proof. By rewriting (29), we arrive at
Tr F P - TT)
(30)
2marg (A (1 3r - A?)A3) - -122,,TTT
R²
Based on (30), we discuss the convergence in three different regimes.
Phase I. T 2 ræ 0.5. This is the initial phase, and the condition is equivalent to Tr < 0.5. Given these conditions, it can be seen that miin (Irg 2 J Umin(Irg
39

---

A 2 0.5. Together with Br S (recall that 3t == :1:01[[1 01 - + J). we can simplify (30) as
21j (Ai)
Tr(1:+1 ((41) TT(
(1 T22TT(((rnn - A3)A3) ( TT
n²a
(0) 1rrB (A?)
(1 - - (,
(b)
me TB)²
R² cimr (1 Tr( -TTT((
where (a) uses mmin (I_g A 2 0.5; (b) uses Lemmas 21 and 19, which implies that drg (A?) 2
of (0o) 2 (r - rB) 3 (c) for some universal constant ej defined in Lemma 19, Rearranging the terms, we arrive at
Tr( & L1)
1+
7/ -(T-=B)² (1 - TT(
)
R2eimr
which is linearly increasing once the term in parentheses is greater than 1. This amounts to choosing
small enough 1), e.g.. 17 - O kmmr (r=r))" ).
Phase 11. 0.5 < Tr(Irg D < TB - 0.5. Suppose that Tr((I,, A))Aî) 2 for some p to
be discussed shortly. Choosing 17 S 0.5, and r O p(r-re)") pa is straightforward to have
21 - TB)² n² Rrc2mr
Tr($:+1 (111) TT(
(1 -()))3 - 12rB
(1 112) Tr -A])A) 12²r 20f(r TB)²
x2emm
p² (TB) A.
334mm2
This means that per step, Tr( (p,
increases at least by A. Consequently, after at most (rn iterations, RGD leaves Phase II.
1)/A O( m² 02 rn)
Next, we show that p 2 Omin
m (r-ra)² }). Notice that Tr( - - -
2 of
(TP.) 2 7g (1.) - o (P,)) 2 O (r-ra)² }), where the last inequality comes from the facts that i) for r E [a, b] with 0 < a < 0.5 b < 1, the smallest value of x(1 z) min(a(1 a), b(1 b)); and, ii) of (o.) belongs interval [a, b] with a O (r-r
mr 1 2rn < 1 1. Lemmas 21 and 19 are adopted to calculate a, that is,
and b re- 0.5
m rg 2 2 rg Po) = =(( - -rB)²/m/
is that Tr 4b p 2 < 0.5. This is regime near the optimum. An implication of this phase
Phase III. Tr(I,
2 rn - 0.5. Given that the singular values of $ T belong to [o, 1]. it can be
seen that rra ( = o.s (A?) 2 0.5. Together with Br S 0.5 in this scenario, we can simplify (30) as
Tr 1-
$ (41 TT(
2nJ
1 1 11²
Tr Tr((Irn A7)A7) - TTTT.
R2
(0)
11
1
Tr(Inn -A3) - 17² BrB
2n
2
1
1
1
Tr(Irp - -. - BrBB
2x
2
(4)
1/2
2
1
Tr(I,,, F. r2 Tr(I,, -.
2x²
2
40

---

where (c) applies rra (A3) 2 0.5 twice; and (d) follows from Br S Tr(Ira D, This further implies that
Tr(Iræ
P1+1 i (+11 Tr(Aro - +,)
7
1 11
Tr(lrg - -., + ) Tr(Ir - (.
2n
2
Reorganizing the terms, we arrive at
Tr(Ire 2++1 2 (+11) S
77 2x
1 2
+ 17 r
Tr(Irp -2
This indicates a linear rate until we achieve optimality once 1j chosen sufficiently small.
Note that our choice of ensures the conditions Lemma 20 are satisfied, guaranteeing an increase of Tr( ) in every iteration. This means that the Tr( o (,,)) will traverse Phase II, and III consecutively. Combining these three phases together gives the claimed complexity bound. 0
Lemma 23. Alg. 3 with = guarantees that [(X,, satisfied.
O(e) if Tr(I, &2, S
Proof. We ignore the subscript in X, and θ = X BX, for simplicity, We have that |Xx BXX - -BIlF
|xex
B
|XX BXx BXx + BXX B
|Xx Im BXX |B(X LIF)
|[xx T U XX | |UE [XX Im)
[m - Fl
Now since we have that
[1 - xx UU = Tr ( (Im - xx -xx
Tr(I 2 S
Combining above inequalities, we have that
|Xex - < - 4c.
This finishes the proof.
F Additional Experiments
F.1 Additional Analysis Stable Rank
In addition to low stable rank in commonsense reasoning tasks shown in Fig. 2, we analyze the official LoRA checkpoints from Hu et al. (2022) for DeBERTa-XXLL The results are shown in Fig. 6. demonstrates that for most tasks the median stable rank of LoRA is in [1.30, 1.40] even if r chosen 16 in LoRA. This implics that for half of the layers the stable rank is at the lower end, leading to low directional diversity as outlined in the main text.
We also provide more comprehensive results on the relationship between the stable and nominal rank for LoRA and PoLAR fine-tuned on commonsense reasoning tasks. Fig. 5 confirms the argument made in the main text that LoRA suffers from a low effective rank which addressed by our PoLAR parameterization. This applies to all commonsense reasoning datasets considered with the exception of ARC-e and ARC-c. Moreover, the stable rank of PoLAR increases with on all datasets, yet the trend is unclear for LoRA.
Available https: hatee:////thtee htt///tttuuuu/../ com.//.coooostt//..AA
41

---

(a) BoolQ
(b) HellaSwag
(c) WinoGrande
PLAA
flahk
Rlak
Stat:
(d) ARC-Challenge
(c) ARC-Easy
(f) OpenbookQA
Figure 5: Stable rank of A AW for Llama2-7B 7B fine-tuned on commonsense reasoning tasks.
122 1220,000000 3.37
mrp
0. 1.22, 0, m.39
371
D1 1.-1117.0--0 1,48
C0k
01 0.-1.16. 1.30
40p
4. 4,11.14
120
4. 120, 4p-
505b
1.177
1.12.
used LoRA
12
Stable Bank
Figure 6: Stable rank of the official DeBERTa-XXL LoRA checkpoints trained on GLUE tasks with r 16. 9i and d2 denote the first quartile and median, respectively.
In Fig. we show the evolution sr(A AW) over the course of training for PoLAR and LoRA at = 32. While the stable rank for LoRA does not deviate significantly from the stable rank early in training, PoLAR's stable rank appears to dynamically vary with the layer type: more often than not the MLP layers (up and down projection) appear to require a larger stable rank, while the stable rank of the value projection layer often comes out at the lowest.
F.2 Runtime Comparison
Benchmarking against Retraction. We provide the experimental details for the runtime compari- son of a landing step and retraction step based on polar retraction in Table 4. For the retraction step, we re-implement the polar retraction from Boumal et al. (2014) in PyTorch and compare it to our parameter update based run the benchmarking on a single NVIDIA H100 device. We sample measurements until the ratio of interquartile range to median is below 0.15 and report the median. We provide detailed results for
landing step. We use torch. compile to pre-compile each function and
42

---

Table Reserved memory, average step time, and total time of training different low-rank adapters for Llama-2-7B on PIQA with r = 32. A single NVIDIA H100 GPU was used.
Method
Mem. (GB)
Avg. Step Time (s) Total Time (min)
LoRA
52.37
2.72
28. 28.74
DoRA
67.05
5.71
60.38
PoLAR
52.68
3.13
33,13
HeSw
PIDA
BockQ
13.-
10-
Yank
10.
7.
5.0
5.-
ORQA
ARC-e
ARC-c
50-
Stadle
21x
20-
15-
100 Stop
3000
Stop
Stop
Figure 7: Stable rank evolution of the low-rank updates for the layers of the 5th transformer block of Llama-2-7B.
additional values of m in Table below, where the choice of m comes from the dimensions of linear layers of Llama models.
Benchmarking against LoRA/DoRA. We compare runtime and memory usage of PoLAR that of LoRA and DoRA by reporting average step time and reserved memory usage Table 6. We record memory usage with the PyTorch profiler and measure step time using the HuggingFace Trainer. We remark that PoLAR only slightly increases memory usage relative to LoRA. The step time overhead of PoLAR LoRA is at approximately 15%.
F.3 Approximate Feasibility
In Fig. 8, we display the Frobenius norm distance to the Stiefel manifold for X and Y of PoLAR well as Z₁ and Ze of LoRA. The displayed values refer to averages across all layers. The plot provides empirical evidence that PoLAR ensures (i) approximate fcasibility X and Y throughout training and (ii) convergence the Stiefel manifold.
Table 7: Retraction (Retr.) and landing (Land.) times (in ju) for different (m, r).
r
m
32
64
256
768
Retr.
507
1160
2178
8317
Land.
167
100
102
109
4096
Retr.
541
1050
1916
9889
Land.
180
186
238
539
11008
Retr.
606
1312
2237
11653
Land.
1785
1233
1632
3606
43

---

04--
Avg
Step
Step
(a)
|
I. and 2, 21 - 1.
(b) ||
I, and [ Zz 1.11
Figure 8: Average distance Stiefel manifold across all LoRA and PoLAR modules in Gemma-3- 27B on MetaMathQA.
Table 8: Performance on commonsense reasoning tasks with Llama-2-7B using GaLore and PoLAR for rank 16 in single-task setup. HeSw refers HellaSwag and WiGr to WinoGrande.
BoolQ
PIQA
SIQA
HeSw
WiGr
ARC-c
ARC-c
OBQA
Avg.
81.93
80.20
56.40
80.23
81.53
80.22
47.53
44.80
69. 10
GaLore 4 x 10-5
86.48
82.81
61.26
83.84
78.85
83.08
54.44
57.80
73.57
PoLAR
88.01
82.92
59.93
82.68
81.77
81.82
57.51
57.40
74.00
F,4
Comparison with GaLore
We compare PoLAR to GaLore (Zhao et al., 2024) Table We adopt our commonsense reasoning training setup from Table 2 with rank 16. For GaLore's hyperparameters, their fine-tuning experiments in Appendix D and set the scale to 2. We set the update projection gap the projections are updated once 15% increments of training are completed and tune the learning rate // E {1 x 10-5, x 10-5).
take inspiration from
F,5 Gemma-3-12B on Commonsense Reasoning
In Table 9, we evaluate PoLAR on Gemma-3 on the commonsense reasoning benchmark, replicating the setting described Appendix G.1 with the difference of setting 5 x 10-3 8, and using Gemma-3-12B.
G Experimental Setup
Experiments performed either of NVIDIA GH200 and NVIDIA H100 GPUs. We use PyTorch (Paszke et al., 2019) for all experiments.
Table 9: Performance on commonsense--rasooing tasks with Gemma-3- 3-B using LoRA and PoLAR rank 8 for different learning rates (1). HeSw refers to HellaSwag and WiGr WinoGrande.
BoolQ
PIQA
SIQA
HeSw
WiGr
ARC-e
ARC-c
OBQA
Avg.
90.03
84.33
59.77
86.25
79.40
88.55
70.90
63.60
77.86
LoRA
8x 10
90.43
82.43
57.68
83.70
77.35
87.79
69.28
63.00
76.46
4x 1
89.72
85.53
61.87
87.19
86.50
90.11
65.53
66.00
79.06
PoLAR
8x 10
89.54
85.58
61.87
86.58
86.50
90.61
66.81
65.80
79.16
44

---

G.1 Commonsense Reasoning
We consider the following tasks: BoolQ (Clark et al., 2019), PIQA (Bisk et al., 2019), SIQA (Sap et al., 2019), HellaSwag (Zellers et al., 2019). WinoGrande (Sakaguchi et al., 2019), ARC-e and ARC-c (Clark al., 2018), and OpenbookQA (Mihaylov al., 2018). To facilitate reproducibility. we use Fleuther-Al's in-evalluation-hurresss (Biderman et al., 2024) and report the accuracy based on multiple-choice log-likelihood evaluation, i.e., we select the answer choice with the highest conditional log-likelihooe as the predicted answer. For datasets with answer choices of varying length (PIQA, ARC-e, ARC-e, OpenbookQA, and Hellaswag), we perform byte-length normalization of the log-likelihood scores remove any bias due to the answer length.
For the results in Table 10, we train for 5 epochs on each task with batch size 128 and choose the learning rate within {4 x 10m4, 8 x 10-+ x 10-3), 3]. We tune À E {10-3, 3,5 x 10 }} for PoLAR and set 32. We choose the combination that performs best on average throughout all datasets (see Table 10) and report these.
For the comparison in Table 1, we train again each task for 5 epochs with 1) 4 10-3 and set = 0.1 for all datasets and settings.
G.2 Mathematical Reasoning
For the mathematical reasoning experiment, we tune the learning rate grid in x2x 10 x 10= 10 6 10-4 10-4). We train for epochs on MetaMathQA (Yu et al., 2024) using rank 16, batch size 128, and tune x E {10-3 5 x 10 10-3). Further following Yu et al. (2024), we do not use few-shot prompting, as the zero-shot setup reduces inference costs and tends to work better for fine-tuned models (see Appendix B in Yu et al. (2024)). We use the following Alpaca-based prompt (Taori et al., 2023) for training and evaluation:
"Below is an instruction that describes a task. Write a response that appropriately completes the request. Am\nnnAnnt
Instruction: infinstruction)W An# Response: An". We evaluate on GSM8K (Cobbe et al., 2021) and MATH (Hendrycks et al., 2021) using in--valuation-hamee and the above prompt.
G.3 GLUE
We set the rank 16 and batch size 32 for all methods. Additionally, we fix a and X For both methods, we do grid-search of the learning rate {4 x 10-2,8 x 10-1, x 10-3) and report the set of runs that yields best performance across all datasets. We report the average accuracy across four seeds.
G.4 Matrix Factorization
Here, provide the experimental details for Fig. 3. We compare Riemannian Gradient Descent (RGD) with the PoL parameterization (PO) to Gradient Descent (GD) with the Burer-Monteiro parameterization (BM). We set A E Rmxx with y 4 evenly spaced singular values in the interval [L. K| with K E {10, 100]. For λ = 10, we set 7 = 10-3 and y = 1 for PO + RGD and n 10-3 for BM GD. For 100, we use 17 = 10- for both methods. In the symmetric case, we use again 10 for 10 and 7) = 10 for K 100.
G.5 Licenses
Our evaluations are carried out on commonly-used datasets in the literature.
Language Understanding. GLUE (Wang et al., 2019) is designed to provide general-purpose evaluation of language understanding. Those adopted in our work include MNLI (inference, Williams al. (2018)), SST-2 (sentiment analysis, Socher al. (2013)). MRPC (paraphrase detection, Dolan and Brockett (2005)), CoLA (linguistic acceptability, Warstadt et al. (2019)), QNLI (inference Rajpurkar et al. (2018)), QQP (question-answerrng)) RTE (inference), and STS-B (textual similarity, Cer et al. (2017)). These datasets are released under different permissive licenses.
"ncope://quoradata. quora. com www.aa.co/o.iaatta//aaano Sowa---auurraaaaeeaeeeee Daw.-aat-ttt-eeeeeeeeee--------e Rewe-eeeeeeeeeeeeeeee-eeeeeeeeee Release-Question
"attpss//paporsswthhoooo wtt.o/dde.coo/ttttt////ttt
45

---

Table 10: Learning rate and regularization lambda values for different ranks and adapters.
Rank
Adapter
Learning Rate
a
DoRA
4 10-4
32
LoRA
10-4
PoLAR
4 x 10-3
5 10
DoRA
10 4
LoRA
4 x 10
32
PoL.R
4 x 10
5 10
32
16
DoRA
x 10
32
LoRA
4 10
32
PoLAR
10 3
5 10-3
32
DoRA
x 10
32
LoRA
10-4
PoLAR
4 x 10-3 5 10-3
32
Commonsense Reasoning. These datasets are collection tasks that require commonsense reasoning answer. The considered datasets include WinoGrande (Sakaguchi et al., 2019), PIQA (Bisk et al., 2019), Social Interaction QA (SIQA) (Sap al., 2019), HellaSwag (Zellers et al., 2019), ARC-easy, ARC-challenge (Chollet, 2019) and OpenbookQA (Mihaylov et al., 2018). These datasets are released under different permissive licenses.
Mathematical Reasoning, For mathematical problems, we consider GSM8K (Cobbe et al., 2021) dataset that consists of high quality linguistically diverse school math problems created by human problem writers. MATH (Hendryeks al., 2021) contains high-school competition mathematics problems from different areas of mathematics with varying degrees of difficulty. We also adopt MetaMathQA dataset (Yu al., 2024), which is constructed through bootstrapping mathematical questions by rewriting the question from multiple perspectives. All three datasets are under the MIT license.
Models. We summarize licenses and terms of use of the models employed in this work. All model checkpoints obtained from HuggingFace, if not stated otherwise. DeßBERaa-v3-ase and DeBERTa-v2-XX -2-XXL are under the MIT License. Llama-2 models are under the Llama 2 community license agreement." Gemma models are under the Gemma terms of use. The official LoRA checkpoints for DeBERTa-XX XXL available on Github are under the MIT license.
fectpa://ai.google..oe wwwggole..e/ggem///eeeeeeeeee
46


Reviews:
[{"review_id": "YS8hr1AKFZ", "reviewer": "Reviewer_35pC", "summary": "The authors start out with looking at PEFT -- parameter efficient fine-tuning. Their key enabling insight, which serves as strong motivation for this work, is that vanilla LoRA adapters demonstrate low stable rank, and severe underutilization of available algebraic rank and, thus, expressivity. Thus, while structurally, capacity is available, it is not utilized due to demonstrated directionality collapse (DC). What's interesting is that the effective stable rank over time converges to 1. Armed with this insight, the authors propose to factor low rank updates (ie \\delta{W} updates to W_0) into direction matrices and a separate scale matrix. This parameterization is then suported by a Riemannian optimization. As a result of this, the authors demonstrate exponentially faster convergence rate compared to vanilla LoRA! Findings generalize across tasks, models, and model sizes. Impressive work.", "questions": "* would you be able to demonstrate your proposed technique on the Silver Bullet, which reduces a set of FT-trainable parameters to R (rxr), instead of the typical A and B LoRA matrices. This reduction to R may lead to higher stable rank. I'm wondering whether Silver Bullet led to improving/mitigating directional collapse as well.\n* To what extent is PoLAR sensitive to initialization? Conversely, if we use the best possible initialization, does this help mitigate the stable rank or DC in LoRA? In other words, can initialization help with this problem, or is it fundamentally an issue with the optimization step and, therefore, truly requires proposed decomposition? \n\n[1] Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning, https://arxiv.org/abs/2411.19557. Submitted on 29 Nov 2024 (v1).", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths**\n\n* very good presentation. Well written, thoughtful, thorough disposition throughout the paper. I enjoyed reading it. Best in the pile.\n* the proposed technique is mathematically sound\n* strong key enabling insight, showing directional collapse in standard LoRA adapters as well as very low stable rank and, therefore, poor utilization of available expressivity. The fact that varying r typically doesn't help is strongly aligned with this insight.\n* findings generalize across tasks, models, and even model sizes (up to 27B)\n\n\n**Weaknesses**\n* some important LoRA initialization work hasn't been cited [1]. It would also be great to see if PoLAR can generalize to this type of LoRA parameterization (see questions). \n\n[1] Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning, https://arxiv.org/abs/2411.19557. Submitted on 29 Nov 2024 (v1).", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your insightful comments and positive feedback, which guided further improvements to our work. \n\n**Q1 would you be able to demonstrate your proposed technique on the Silver Bullet, which reduces a set of FT-trainable parameters to R (rxr), instead of the typical A and B LoRA matrices. This reduction to R may lead to higher stable rank. I'm wondering whether Silver Bullet led to improving/mitigating directional collapse as well.**\n\n**A1** We thank the reviewer for bringing this interesting related work to our attention. The parameterization does indeed share similarities, as it contains a trainable $r \\times r$ matrix and two semi-orthogonal factors. However, we remark that freezing the semi-orthogonal factors might cause a loss in expressivity compared to PoLAR, as the row and column spaces of the low-rank update remain fixed. As we discuss in **A2**, the increase in stable rank occurs during training, s.t. the question of whether a trainable $r \\times r$ matrix alone achieves similar benefits as PoLAR boils down to whether the parameterization alone already achieves those benefits or the trainability of the direction factors is required as well. Apart from that, we find it unlikely that a better initialization alone (without any constraints on the factors throughout training) will yield similar results; see the discussion in **A2**.   \n\n**Q2 To what extent is PoLAR sensitive to initialization? Conversely, if we use the best possible initialization, does this help mitigate the stable rank or DC in LoRA? In other words, can initialization help with this problem, or is it fundamentally an issue with the optimization step and, therefore, truly requires proposed decomposition?**\n\n**A2** We did not test alternative initialization schemes and how these would relate to the stable rank dynamics. Note though that Fig. 7 demonstrates that the increase in stable rank appears throughout training, rather than at initialization. Therefore, we are unsure whether a more advanced initialization alone mitigates the problem of low stable rank. Other than that, the initialization scheme proposed by LoRA-SB could further improve our PoLAR procedure, as the Silver Bullet initialization is informed by the SVD of the gradient evaluated at the pre-trained model. Going beyond our initialization that is randomly sampled from the Stiefel manifold, the information contained in the SB-basis might provide complementary benefits. A thorough investigation on whether these purported benefits materialize empirically is an interesting direction for follow-up work. We will include a discussion of LoRA-SB in a revised version.\n\n**W1 some important LoRA initialization work hasn't been cited [1]. It would also be great to see if PoLAR can generalize to this type of LoRA parameterization (see questions).**\n\n**R1** We refer to **A1**/**A2** for a discussion of the relationship to LoRA-SB. Given the potential for follow-up work, we will include a discussion of LoRA-SB in a revised draft."}, {"review_id": "UlNTjdDxjB", "reviewer": "Reviewer_DiUw", "summary": "This paper introduces PoLAR, a parameter-efficient fine-tuning (PEFT) method designed to enhance the directional diversity of weight updates ($\\Delta W$) in LoRA, thereby addressing the low stable rank issue. Experimental results demonstrate that PoLAR achieves consistent accuracy improvements across diverse datasets and base models.", "questions": "1. How does PoLAR compare to GaLore and other methods addressing directional diversity (*e.g.*, SVD-based approaches)? Please provide a discussion on similarities and distinctions.\n\n2. Is it possible to provide a convergence analysis for the alternating update algorithm used in PoLAR’s implementation, given that the theoretical analysis relies on Riemannian gradient descent?\n\n3. How does the proposed method performs compared to methods like GaLore, Fira, SLTrain, and full-rank fine-tuning?\n\n4. Does PoLAR offer any memory efficiency advantages over LoRA, or are there plans to address this limitation in future work?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. **Clear motivations.** The paper effectively identifies the directional diversity collapse in low-rank adapters, articulates its impact, and explains how PoLAR addresses this issue.\n2. **Theoretically sound analysis.** The study provides a comprehensive theoretical framework, including problem definition and convergence guarantees, strengthening the method’s foundation.\n3. **Practical implementation.** This paper introduces an alternating update method to solve the low-rank approximation problem with reduced computational requirement. \n\n**Weaknesses**\n\n1. **Limited discussions of related  work.** One major reason for the low directional diversity of LoRA may be attributed to the non-orthogonal projection. There are other related work that attempts to solve the same problem, such as SVD-based GaLore (Zhao et al., 2024) and other related works. The discussion on the similarity and distinctions between the proposed method and GaLore-like methods are necessary.\n2. **Misalignment between theory and practice.** While convergence is analyzed under Riemannian gradient descent, the practical implementation relies on an alternating update algorithm. An analysis of the convergence properties of this algorithm would bridge this gap.\n3. **Insufficient experiments.** The experiments compare PoLAR only with LoRA and DoRA. Including additional methods like GaLore, Fira (Chen et al., 2024), SLTrain (Han et al., 2024), and full-rank fine-tuning would provide a more comprehensive evaluation.\n4. **Limited memory efficiency.** Unlike recent PEFT methods that prioritize both parameter and memory efficiency, PoLAR does not demonstrate memory advantages over LoRA, potentially limiting its practical applicability.\n5. **The organization and presentations can be further improved.**\n   - Key content, such as Appendix C.1, is relegated to the appendices despite its importance and should be integrated into the main text. Frequent references to appendices disrupt the reading flow.\n   - [Minor] The core idea’s presentation could be more concise, avoiding excessive use of technical terms and complex concepts.\n\n**References**\n\n[R1] X. Chen, et al. (2024). Fira: Can We Achieve Full-rank Training of LLMs Under Low-rank Constraint? *arXiv preprint: 2410.01623*.\n\n[R2] A. Han, J. Li, W. Huang, M. Hong, & A. Takeda, (2024). SLTrain: A Sparse Plus Low-Rank Approach for Parameter and Memory Efficient Pretraining. *NeurIPS 2024*.", "was_revised": true, "final_justification": "The authors' rebuttal has effectively addressed my concerns regarding the related work, theoretical analysis, and experimental results. Accordingly, I have revised my overall score to Accept.", "rebuttal": "We are thankful for your suggestions and insightful comments, bringing up important points which help us in sharpening our contribution. \n\n**Q1 How does PoLAR compare to GaLore and other methods addressing directional diversity (e.g., SVD-based approaches)? Please provide a discussion on similarities and distinctions.**\n\n**A1** We thank the reviewer for bringing this up and agree that there are indeed some similarities to GaLore [1]. First and foremost, PoLAR as well as GaLore see low-rank structure during training as an opportunity to improve memory efficiency. In addition, both setups use semi-orthogonal matrices for compression and decompression operations. Nevertheless, there are also substantial differences; PoLAR imposes low-rank structure on the weight updates whereas GaLore restricts the gradients driving those updates to be low-rank. Moreover, the orthogonality of the subspace is enforced through different means and the frequency of subspace changes differs. GaLore updates the subspace only periodically via an SVD of the gradient, which introduces substantial overhead. PoLAR, on the other hand, updates the subspace at every iteration through the use of the landing field, which can be obtained using matrix multiplications only. Moreover, there are substantial differences in storing fine-tuned models; PoLAR natively stores a compressed representation requiring storage of $(m+n+r)r$ parameters. In contrast, GaLore requires storage of $mn$ parameters.\n\n**Q2 Is it possible to provide a convergence analysis for the alternating update algorithm used in PoLAR’s implementation, given that the theoretical analysis relies on Riemannian gradient descent?**\n\n**A2** We apologize for not making it clear in the main text. The theoretical analysis uses alternating updates (see Alg. 1 and Eq. 5a-5c). The updates in Alg. 2 (PoLAR), on the other hand, can be thought of happening simultaneously rather than alternatingly. The reason for this discrepancy is of theoretical nature, as the alternating update greatly facilitates the analysis. Nevertheless, we do believe that our analysis can be extended to cover the updates used in the practical variant.\n\n**Q3 How does the proposed method performs compared to methods like GaLore, Fira, SLTrain, and full-rank fine-tuning?**\n\n**A3** We provide results for GaLore+AdamW and full-rank fine-tuning below. We adopt our commonsense reasoning training setup from Table 2 with rank $16$. For GaLore's hyperparameters, we take inspiration from their fine-tuning experiments in Appendix D [1] and set the scale to $2$. We set the update projection gap s.t. the projections are updated once 15\\% increments of training are completed and tune the learning rate $\\eta \\in \\lbrace 1 \\times 10^{-5}, 4 \\times 10^{-5} \\rbrace$. We additionally show results for PoLAR at rank $16$ for ease of reference. \n\n |       | $\\eta$    | BoolQ | PIQA | SIQA | HeSw | WiGr | ARC‑e | ARC‑c | OBQA | Avg  |\n |-------|-------|-------|------|------|------|------|-------|-------|------|------|\n | GaLore | $1 \\times 10^{-5}$ | 81.93 | 80.2 | 56.4 | 80.23 | 81.53 | 80.22 | 47.53 | 44.8 | 69.1 |\n | GaLore | $4 \\times 10^{-5}$ | 86.48 | 82.81 | 61.26 | 83.84 | 78.85 | 83.08 | 54.44 | 57.8 | 73.57 |\n | Full-FT | $1 \\times 10^{-5}$ | 84.8  | 81.72 | 59.21 | 81.71 | 81.06 | 82.95 | 53.67 | 51.2 | 72.04 |\n | Full-FT | $4 \\times 10^{-5}$ | 87.68 | 80.09 | 55.63 | 80.93 | 76.56 | 81.36 | 58.02 | 54.8 | 71.88 |\n | PoLAR  |                     |88.01  | 82.92 | 59.93 | 82.68 | 81.77 | 81.82 | 57.51 | 57.40 | 74.00 |\n\nWe remark that the average performance of GaLore at $4 \\times 10^{-5}$ comes close to the performance of PoLAR at rank $16$ while full fine-tuning appears to lag behind. \n\n\n**Q4 Does PoLAR offer any memory efficiency advantages over LoRA, or are there plans to address this limitation in future work?**\n\n**A4** PoLAR's memory footprint is mostly comparable to that of LoRA, as it only adds $r^2$ additional trainable parameters to the procedure. We report peak memory usage for Llama2-7B on PIQA in Table 6 of Appendix F.2 and show the relevant values here for convenience.\n \n | Method    | Memory (GB)  |\n | ------    | ----------------: |\n | LoRA      | 52.37             |\n | PoLAR     | 52.68             |\n\nThe memory footprint of PoLAR could theoretically be further reduced through a reparameterization of the direction matrices living on the Stiefel manifold. While our current $m \\times r$ form exhibits an avoidable overhead of $r + {r \\choose 2}$ (see, e.g., Section 3.3.2 in [2]), we do not think that these rather small gains in memory warrant the extra computational cost and complexity associated with it. The bulk of the memory is still consumed by the model itself with memory complexity $O(mn)$, while the low-rank factors only consume $O((m+n)r)$. In the table above PoLAR has approximately 56M trainable low-rank parameters with each of them consuming 12 bytes (parameter and gradient with 2 bytes each and Adam states with 4 bytes each), totalling less than 1GB of memory. \n\n**W1 Limited discussions of related work. One major reason for the low directional diversity of LoRA may be attributed to the non-orthogonal projection. There are other related work that attempts to solve the same problem, such as SVD-based GaLore (Zhao et al., 2024) and other related works. The discussion on the similarity and distinctions between the proposed method and GaLore-like methods are necessary.**\n\n**R1** We agree that GaLore-like methods share some similarities, but also exhibit significant differences. We included a discussion of those in **A1** and will also integrate it in a revised version.\n\n**W2 Misalignment between theory and practice. While convergence is analyzed under Riemannian gradient descent, the practical implementation relies on an alternating update algorithm. An analysis of the convergence properties of this algorithm would bridge this gap.**\n\n**R2** As in our reply to **Q2**, we apologize for not making it clear that our practical variant does not use the alternating update procedure which was used in the theoretical analysis. Note that the use of the alternating RGD procedure in the practical variant would double the computational cost, as it requires two forward and backward passes per optimizer step. Moreover, Table 4 shows the runtime advantage of the practical variant over RGD. \n\n**W3 Insufficient experiments. The experiments compare PoLAR only with LoRA and DoRA. Including additional methods like GaLore, Fira (Chen et al., 2024), SLTrain (Han et al., 2024), and full-rank fine-tuning would provide a more comprehensive evaluation.**\n\n**R3** We provide results for GaLore and full fine-tuning in **A3**. \n\n**W4 Limited memory efficiency. Unlike recent PEFT methods that prioritize both parameter and memory efficiency, PoLAR does not demonstrate memory advantages over LoRA, potentially limiting its practical applicability.**\n\n**R4** As mentioned in **A4**, PoLAR maintains the memory footprint of LoRA, taking out the additional $O(r^2)$ memory incurred by the $\\Theta$ matrix.\n\n**W5 The organization and presentations can be further improved. Key content, such as Appendix C.1, is relegated to the appendices despite its importance and should be integrated into the main text. Frequent references to appendices disrupt the reading flow. [Minor] The core idea’s presentation could be more concise, avoiding excessive use of technical terms and complex concepts.**\n\n**R5** We thank the reviewer for this important feedback on the overall presentation of our work. We agree that the definition of the polar decomposition, a key concept in our work, should have a more prominent position in the main text, and we will change it accordingly. Also, we will reduce the number of references to the appendix.\n\n**References**\n\n[1] Zhao, Jiawei, et al. \"GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection.\" ICML. 2024.\n \n[2] Absil, Pierre-Antoine, et al. \"Optimization Algorithms on Matrix Manifolds.\" 2008.\n \n[3] Ponkshe, Kaustubh, et al. \"Initialization using Update Approximation is a Silver Bullet for Extremely Efficient Low-Rank Fine-Tuning.\" arXiv. 2025"}, {"review_id": "WANtKTWxll", "reviewer": "Reviewer_1G8C", "summary": "The work introduces a new method for fine-tuning large language models that addresses LoRA's underuse of the allocated rank, leading to low stable rank and collapsed update directions. PoLAR uses polar decomposition to factor low-rank updates into orthogonal direction matrices and a scale matrix, improving expressiveness. Combined with a Riemannian optimization strategy that avoids costly retractions, PoLAR achieves faster convergence and better GPU efficiency. PoLAR outperforms LoRA and DoRA  (the most recent reproducible SoTA) across models and tasks by maintaining higher stable rank.", "questions": "In Figure 5, the stable rank of LoRA seems to have a wider range compared to PoLAR different from other datasets. You mention this in L1060, but I couldn't find an explanation or hypothesis about it. Why do you think PoLAR doesn't provide any improvements here?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Comprehensive and detailed theoretical grounding.\n- The method scales well with model sizes.\n- The implementation of the landing method seems stable and significantly improves convergence time.\n\nWeaknesses:\n- Optimizing over Steifel manifolds still introduces overhead and significant complexity over DoRA. This is considering that the performance improvements are noticeable but not necessarily significant in the tested benchmarks.\n- It would also be interesting to see the difference in the jump in stable rank from this work compared to multi-lora applications. I say this because multi-lora methods bump up the rank by adding multiple matrices together, which could result in a rank that is higher than the sum of all ranks without changing anything about the typical adapter.\n- The state of the art being compared against is a bit limited. I understand that in most of the tested tasks, AdaLoRA barely performs better than DoRA, but both AdaLoRA and LoRA+ are missing.\n- Addressing the diversity collapse with this work leads to better stable rank within the adapter. Interpreting this from a LoRA’s perspective, the singular value spectrum is more uniform as opposed to being peaky. Other works like DoRA and LoRA+ have directly or indirectly addressed this issue, and their claims were mostly directed towards matching the training dynamics of full fine-tuning. Similar analyses would be interesting to see if the same behavior is seen from PoLAR.\n\nNitpick:\nThe references to the Appendix in the main text are too frequent, and it disturbs the flow of reading.", "was_revised": true, "final_justification": "Considering the clarifications and additional experimental results the authors have provided in their answer, I'll keep my score the same (5: Accept).\n\nI don't consider the weaknesses raised by the last reviewer to be actual weaknesses (performance and evaluation benchmarks), and I believe the authors' answer addresses these raised questions adequately. Same for weaknesses raised by other reviewers about GaLore, since GaLore is significantly different in terms of their goal and methodology.", "rebuttal": "We sincerely acknowledge your careful analysis of our work and thank you for recognizing its strengths.\n\n**W1 Optimizing over Stiefel manifolds still introduces overhead and significant complexity over DoRA. This is considering that the performance improvements are noticeable but not necessarily significant in the tested benchmarks.**\n\n**R1** Optimizing over the Stiefel manifold does add overhead. Nevertheless, the runtime of PoLAR compares favorably to that of DoRA (see the table below and Table 6 in the Appendix). To see why that is, consider a linear layer with weight $W\\in \\mathbb{R}^{m \\times n}$ and $m>n$. As outlined in Alg. 2 and discussed in Section 4.4, PoLAR requires $O(m^2r)$ additional operations to obtain the landing field on top of the layer's gradient. In contrast, the gradient projection in Eq. 6 of DoRA [1], requires $O(m^2n)$ computations. Similarly, in the forward pass, DoRA requires the explicit construction of the low-rank weight change to compute the columnwise norm of the updated layer. This raises the complexity of the forward pass from $O(mr)$ for LoRA/PoLAR to $O(mnr)$ for DoRA.\n\n| Method    | Total Time (min)  |\n| ------    | ----------------: |\n| LoRA      | 28.74             |\n| DoRA      | 60.38             |\n| PoLAR     | 33.13             |\n\n\n**W2 It would also be interesting to see the difference in the jump in stable rank from this work compared to multi-lora applications. I say this because multi-lora methods bump up the rank by adding multiple matrices together, which could result in a rank that is higher than the sum of all ranks without changing anything about the typical adapter.**\n\n**R2** It is indeed interesting to see whether Multi-LoRA approaches that sequentially fit multiple low-rank updates obtain similar improvements in stable rank as PoLAR does. We find it plausible that using sequential fitting of PoLAR adapters could further elevate the stable rank, since LoRA's update often exhibits low stable rank compared to PoLAR.  \n\n**W3 The state of the art being compared against is a bit limited. I understand that in most of the tested tasks, AdaLoRA barely performs better than DoRA, but both AdaLoRA and LoRA+ are missing.**\n\n**R3** Note that Table 1 includes a comparison to a variant of PoLAR that brings it closer to AdaLoRA by using the Euclidean gradient to update the parameters as well as a diagonal $\\Theta$. This variant can be interpreted as a memory-matched variant of AdaLoRA that does not redistribute ranks among matrices and does not rely on overbudgeting during training (see Section 4.1 in [2]). We consider this to be a fair baseline, as AdaLoRA itself matches the parameter count to LoRA based on a final rank-budget of all adapters, but starts training with $1.5\\times$ as many parameters. This overbudgeting increases memory load throughout training and makes a fair comparison hard. For this reason, we decided to benchmark PoLAR against the reduced variant described above.\n\n\n**W4 Addressing the diversity collapse with this work leads to better stable rank within the adapter. Interpreting this from a LoRA’s perspective, the singular value spectrum is more uniform as opposed to being peaky. Other works like DoRA and LoRA+ have directly or indirectly addressed this issue, and their claims were mostly directed towards matching the training dynamics of full fine-tuning. Similar analyses would be interesting to see if the same behavior is seen from PoLAR.**\n\n**R4** Thank you for raising this interesting point. While DoRA and LoRA+ indeed discuss concepts such as directional diversity or efficient feature learning, we do not see direct or indirect hints at how the spectrum of the low-rank update behaves.\n\nBased on your comment regarding the alignment with full fine-tuning, we ran further experiments to see whether PoLAR also exhibits a closer alignment with the training dynamics of full fine-tuning. To do so, we ran full fine-tuning on the commonsense reasoning benchmark and analyzed the stable rank distribution of $\\Delta W = W' - W$ where $W'$ and $W$ refer to the fine-tuned and pre-trained weights, respectively. We tune the learning rate in $\\lbrace 1 \\times 10^{-5}, 4 \\times 10^{-5} \\rbrace$ and report the three quartiles of the stable rank distribution across layers below.\n\n$\\eta = 1 \\times 10^{-5}$\n\n| Percentile | BoolQ | PIQA | SIQA | HeSw | WiGr | ARC‑e | ARC‑c | OBQA | Avg |\n|-----------:|------:|-----:|-----:|-----:|-----:|------:|------:|-----:|-----:|\n| 0.25 | 65.38 | 76.9 | 50.15 | 80.28 | 50.1 | 27.47 | 20.87 | 23.53 | 49.34 |\n| 0.5  | 104.88 | 104.84 | 74.82 | 114.24 | 69.47 | 43.64 | 33.87 | 33.43 | 72.40 |\n| 0.75 | 150.88 | 132.42 | 91.56 | 151.8 | 89.96 | 58.7 | 44.36 | 42.76 | 95.31 |\n\n\n$\\eta = 4 \\times 10^{-5}$\n \n| Percentile | BoolQ | PIQA | SIQA | HeSw | WiGr | ARC‑e | ARC‑c | OBQA | Avg |\n|-----------:|------:|-----:|-----:|-----:|-----:|------:|------:|-----:|-----:|\n| 0.25 | 78.90 | 59.28 | 53.16 | 83.53 | 54.27 | 19.60 | 12.77 | 26.84 | 48.54 |\n| 0.5  | 106.37 | 70.55 | 60.76 | 106.61 | 60.84 | 29.71 | 21.52 | 34.78 | 61.39 |\n| 0.75 | 126.42 | 82.59 | 76.32 | 134.89 | 73.77 | 34.55 | 26.44 | 39.42 | 74.30 |\n\n\nReferring to the results $\\eta=1\\times 10^{-5}$, the average of the median stable rank across datasets is $72.4$ and the average first quartile is $49.3$, implying that most weight updates exhibit a relatively high stable rank. Therefore, the higher stable rank obtained by PoLAR can be interpreted from the angle of closer alignment with full fine-tuning. \n\n**Q1 In Figure 5, the stable rank of LoRA seems to have a wider range compared to PoLAR different from other datasets. You mention this in L1060, but I couldn’t find an explanation or hypothesis about it. Why do you think PoLAR doesn’t provide any improvements here?**  \n\n**A1** While we can only provide a speculative answer, we hypothesize that this is related to the training set size and task format. ARC-c and ARC-e [3] are by some margin the smallest datasets in the benchmark with roughly 1k and 2k training examples, respectively. We suspect that this dataset size is not sufficiently large s.t. meaningful, orthogonal directions can be found. Moreover, the ARC datasets differ from the other datasets in that they do not provide additional per-choice context to answer the question. In contrast, the other datasets do incorporate some form of minimal context: HellaSwag has a context field; BoolQ provides a passage; Social-IQA gives a context for the question; WinoGrande provides the full sentence with a blank; OBQA provides related facts; and PIQA frames the question as a goal. \n\n**References**\n\n[1] Liu, Shih-Yang, et al. \"DoRA: Weight-Decomposed Low-Rank Adaptation.\" ICML. 2024.\n\n[2] Zhang, Qingru, et al. \"Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning.\" ICLR. 2023. \n\n[3] Clark, Peter, et al. \"Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge.\", arXiv. 2018."}, {"review_id": "ScMOZsDqm5", "reviewer": "Reviewer_guGQ", "summary": "This paper introduces PoLAR, a new method for PEFT of LLMs that leverages the polar-like decomposition of LoRA matrices into unitary and a low-dimensional components. The authors provide theoretical motivation with rigorous analysis and demonstrate empirical gains over strong LoRA-based baselines across various model sizes and tasks.", "questions": "1. A comment: \nThe authors reference Zeng and Lee (2024) to motivate the expressiveness of low-rank adaptation, noting that LoRA can approximate any target transformer under mild conditions. However, the cited result crucially depends on using relatively high ranks—specifically, LoRA rank being at least half the embedding dimension—which makes the theoretical result more about approximation capacity than practical efficiency. In contrast, this paper aims to improve fine-tuning efficiency by operating in a very low-rank regime, which is far from the high-rank setting assumed in Zeng and Lee. Therefore, the claim that “the low-rank space offers sufficient expressiveness” is not directly supported by that citation.\n\n2. In the theoretical analysis (e.g., lines 168–170), the convergence is derived under the assumption of zero training loss, which implies that the model operates in a high-rank regime—something the authors also acknowledge. However, this seems inconsistent with the practical low-rank settings  used in PEFT. Could the authors clarify how much insight the analysis in the full-rank or zero-loss regime provides for understanding the behavior of LoRA or PoLAR in low-rank regimes commonly used in practice?\n\n3. In Theorem 1, the expression uses T \\geq O(\\cdot), but the Big-O notation already implies an upper bound. Would it not be more precise to write T = O(\\cdot) or clarify the inequality with appropriate asymptotic notation?\nSimilarly, in the supplementary material (line 760, Condition C4), the equality marked as (a) is said to follow from the data being whitened. However, whitening ensures the identity covariance only in expectation, not deterministically. Shouldn’t this step be marked as an approximation, rather than a strict equality? While these may be minor points, they raise concerns about the precision and rigor of the theoretical claims in a paper that is otherwise theory-driven.\n\n\n4. In Algorithm 1, the matrix \\Theta is updated at the beginning of each iteration, whereas in Algorithm 2, it is updated at the very end. This inconsistency raises a question: what would happen if we swapped the update order in Algorithm 2 to match that of Algorithm 1? Is there a theoretical or empirical reason for this discrepancy? Some clarification would help understand whether the difference is deliberate or incidental.\n\n5. In Table 2, both LoRA and PoLAR show lower average performance at rank 32 compared to some smaller ranks. This seems counterintuitive since higher ranks should, in principle, offer more capacity. Can the authors comment on why this drop occurs? Additionally, did you train all configurations (across different ranks) with the same number of epochs? If so, could underfitting or overfitting at higher ranks be a factor?", "limitations": "The authors briefly mention some limitations of their approach, such as the orthogonal constraint’s potential optimization challenges, but they do not clearly discuss broader limitations or possible negative societal impacts. It would strengthen the paper if they included a more explicit reflection on the practical constraints of PoLAR (eg. the selection of \\lambda) and any potential risks in real-world deployment scenarios.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths\n1. The paper is well-organized and clearly written, with effective use of figures to illustrate key concepts and results.\n2.  The authors present a theoretical connection between PEFT and matrix factorization. They analyze convergence properties under this framework in a rigorous way.\n3. The proposed “landing trick” serves as a computationally efficient alternative to the more expensive Stiefel retraction, enabling the enforcement of orthonormality constraints with minimal overhead.\n4. The experimental results include both small- and large-scale language models, demonstrating the scalability of the approach across model sizes.\n\nWeaknesses:\n1. The theoretical analysis on matrix factorization does not clearly translate into deeper understanding of PEFT or address any of the known theoretical challenges in the field. The connection feels superficial and lacks strong justification for its practical relevance.\n2. The performance improvements shown in Table 2 and other benchmarks are modest. Given the competitiveness of the PEFT field, it is unclear whether polar decomposition offers a meaningful advantage over existing methods such as LoRA or QLoRA. More extensive ablation or comparisons would help clarify the benefits.", "was_revised": true, "final_justification": "I appreciate the authors’ response, so increased the score.", "rebuttal": "We appreciate your thorough evaluation and your constructive feedback, which greatly helps us to further refine the work. \n\n**W1 The theoretical analysis on matrix factorization does not clearly translate into deeper understanding of PEFT or address any of the known theoretical challenges in the field. The connection feels superficial and lacks strong justification for its practical relevance.**\n\n**R1** We agree that there is a gap between theory and practice in the field of parameter-efficient fine-tuning, and the relevance of theoretical results for practice might sometimes be limited. However, we would like to highlight that our theoretical analysis on the matrix factorization problem, which is equivalent to a linear LoRA adaptation problem under whitened data, can be interpreted as a basic threshold for efficient low-rank techniques. If one can obtain provable benefits on such a problem, it is an indication that the technique might provide meaningful benefits in a practically relevant setup. As such, our theoretical analysis provides value not in that has immediate implications for state-of-the-art fine-tuning pipelines, but in that it highlights that PoLAR provably overcomes a basic requirement of providing benefits in an analytically tractable scenario.\n\n**W2 The performance improvements shown in Table 2 and other benchmarks are modest. Given the competitiveness of the PEFT field, it is unclear whether polar decomposition offers a meaningful advantage over existing methods such as LoRA or QLoRA. More extensive ablation or comparisons would help clarify the benefits.**\n\n**R2** Performance gains in LLM evaluation are highly dependent on evaluation framework used and the\nmagnitude of differences can wildly differ, making evaluation an active area of research. We opted for\n_lm-eval-harness_ [1] with log-likelihood based evaluation which is a widely recognized and reproducible way to\nevaluate models. This evaluation methodology is in line with the\nmultiple-choice question answering format of the commonsense reasoning benchmark and results in more conservative estimates of improvements.\n\n\n**Q1 A comment: The authors reference Zeng and Lee (2024) to motivate the expressiveness of low-rank adaptation, noting that LoRA can approximate any target transformer under mild conditions. However, the cited result crucially depends on using relatively high ranks—specifically, LoRA rank being at least half the embedding dimension—which makes the theoretical result more about approximation capacity than practical efficiency. In contrast, this paper aims to improve fine-tuning efficiency by operating in a very low-rank regime, which is far from the high-rank setting assumed in Zeng and Lee. Therefore, the claim that “the low-rank space offers sufficient expressiveness” is not directly supported by that citation.**\n\n**A1** We thank the reviewer for this interesting commment and agree that the current phrasing of the rank requirement as being mild should be improved by providing a more detailed contextualization. The cited work [2] expresses the required LoRA rank in terms of what they refer to as _rank-based functionality gap_, which can be informally understood as measuring the closeness between the frozen model and the target model. The worst-case in which this measure is equal to the embedding dimension is only relevant in the scenario where the frozen and target network are completely unrelated (e.g., independent random initialization). In any practical fine-tuning scenario, we expect the frozen and the target model to be closer than this worst case, decreasing the rank-based functionality gap and thereby the required rank to achieve desired approximation capacity. The empirical applicability of these theoretical findings is validated in Appendix G.9 of [2]. We will revise the draft accordingly by providing further context for the supporting claim made. \n\n**Q2 In the theoretical analysis (e.g., lines 168–170), the convergence is derived under the assumption of zero training loss, which implies that the model operates in a high-rank regime—something the authors also acknowledge. However, this seems inconsistent with the practical low-rank settings used in PEFT. Could the authors clarify how much insight the analysis in the full-rank or zero-loss regime provides for understanding the behavior of LoRA or PoLAR in low-rank regimes commonly used in practice?**\n\n**A2** The significance of the high-rank analysis hinges on the relation between the rank of a hypothetical \"ground-truth\" update (think of $r_A$) and the chosen rank $r$ of the adapter. The premise of the entire low-rank adaptation field is that $r_A$ is sufficiently low (think $r_A < r$ or at least $r_A \\approx r$), making the case for the applicability of the high-rank, overparameterized regime in which $r > r_A$. An almost verbatim repetition of it is contained in the original LoRA paper, where it is hypothesized \"that the change in weights during model adaptation also has a low 'intrinsic rank'\" [3]. \nIn addition, there are some practical fine-tuning scenarios in which an approximately zero-loss, say below $10^{-5}$, is commonly encountered. This is for instance the case of fine-tuning 7B models with LoRA on some of the GLUE tasks.\n\n\n**Q3 In Theorem 1, the expression uses $T \\geq O(\\cdot)$, but the Big-O notation already implies an upper bound. Would it not be more precise to write $T = O(\\cdot)$ or clarify the inequality with appropriate asymptotic notation? Similarly, in the supplementary material (line 760, Condition C4), the equality marked as (a) is said to follow from the data being whitened. However, whitening ensures the identity covariance only in expectation, not deterministically. Shouldn’t this step be marked as an approximation, rather than a strict equality? While these may be minor points, they raise questions about the precision and rigor of the theoretical claims in a paper that is otherwise theory-driven.**\n\n**A3** We apologize for the confusion on these two matters and address them in order. (i) We will improve the notation from \"for all $T \\geq O(\\cdot)$\" to \"for all $T \\geq c f(\\cdot)$\" where $c$ is some constant and $f(\\cdot)$ the convergence rate. We remark though that writing $T = O(\\cdot)$ wouldn't suffice here, as Theorem 1 implies a strictly stronger result. It not only implies the existential bound $T=O(\\cdot)$, but also provides last-iterate convergence which is a result of PoLAR's monotone error. (ii) Whitening the data ensures that the empirical uncentered covariance matrix is exactly identity; this _holds deterministically_ and not just in expectation. The corresponding whitening operator can always be obtained from the eigendecomposition of the uncentered covariance matrix of the \"raw\" (unwhitened) data. We outline the procedure below. Following the notation in Appendix C.4, let $\\tilde{D} \\in \\mathbb{R}^{n \\times N}$ be the raw (unwhitened) data whose columns are training examples. Its empirical covariance matrix is given by $\\tilde{\\Sigma} = \\tilde{D} \\tilde{D}^\\top$. From the eigendecomposition of $\\tilde{\\Sigma} = Q \\Lambda Q^\\top$, we obtain the whitening operator $W := \\tilde{\\Sigma}^{-1/2} = Q \\Lambda^{-1/2}Q^T$. Multiplying the raw data with $W$ yields the whitened data $D := W\\tilde{D}$. This ensures $DD^{\\top} = W\\tilde{D}\\tilde{D}^\\top W^\\top = I_{n}$ exactly.\n\n**Q4 In Algorithm 1, the matrix $\\Theta$ is updated at the beginning of each iteration, whereas in Algorithm 2, it is updated at the very end. This inconsistency raises a question: what would happen if we swapped the update order in Algorithm 2 to match that of Algorithm 1? Is there a theoretical or empirical reason for this discrepancy? Some clarification would help understand whether the difference is deliberate or incidental.**  \n\n**A4** We remark that the order of the updates in Algorithm 2 does not matter, as the updates only depend on parameters from the previous iteration. In other words, the updates of Algorithm 2 can be thought of happening simultaneously rather than alternatingly. This is different from the update procedure in Algorithm 1 where the order matters due to the alternating procedure. The reason for this discrepancy is of theoretical nature, as the alternating update greatly facilitates the analysis.\n\n**Q5 In Table 2, both LoRA and PoLAR show lower average performance at rank 32 compared to some smaller ranks. This seems counterintuitive since higher ranks should, in principle, offer more capacity. Can the authors comment on why this drop occurs? Additionally, did you train all configurations (across different ranks) with the same number of epochs? If so, could underfitting or overfitting at higher ranks be a factor?**\n\n**A5** All configurations were trained with the same number of epochs. We agree that the lower average performance at rank 32 appears counterintuitive  and we hypothesize that the underlying reason for this phenomenon to occur is overfitting, as the rank of the optimal weight update could lie in-between 16 and 32.\n\n**References**\n\n[1] Biderman, Stella, et al. \"Lessons from the Trenches on Reproducible Evaluation of Language Models\". arXiv. 2024.\n \n[2] Zeng, Yuchen, et al. \"The Expressive Power of Low-Rank Adaptation\". ICLR. 2024.\n \n[3] Hu, Edward J., et al. \"LoRA: Low-Rank Adaptation of Large Language Models.\" ICLR. 2022."}]

Author rebuttal:

Meta-review: In this paper, the authors propose a novel parameter-efficient fine-tuning method to address the low stable rank issue in LoRA. The presentation is clear, the method is technically sound, and the authors have adequately addressed the reviewers' concerns during rebuttal. I recommend acceptance.
Decision: Accept (poster)