LoRA meets Riemannion: Muon Optimizer for Parametrization-independent Low-Rank Adapters
Authors: Vladimir Bogachev
Vladimir Aletov
Alexander Molozhavenko
Denis Bobkov
Vera Soboleva
Aibek Alanov
Maxim Rakhuba
Venue: ICLR 2026 Poster (2026)
Keywords: Low-rank Adaption
Fine-tuning
Smooth manifolds
Riemannian optimization
Fixed matrix rank manifold
LLM
Diffusion Models
Primary area: generative models

Abstract: This work presents a novel, fully Riemannian framework for Low-Rank Adaptation (LoRA) that geometrically treats low-rank adapters by optimizing them directly on the fixed-rank manifold. This formulation eliminates the parametrization ambiguity present in standard Euclidean optimizers. Our framework integrates three key components to achieve this: (1) we derive **Riemannion**, a new Riemannian optimizer on the fixed-rank matrix manifold that generalizes the recently proposed Muon optimizer; (2) we develop a Riemannian gradient-informed LoRA initialization, and (3) we provide an efficient implementation without prominent overhead that uses automatic differentiation to compute arising geometric operations while adhering to best practices in numerical linear algebra. Comprehensive experimental results on both LLM and diffusion model architectures demonstrate that our approach yields consistent and noticeable improvements in convergence speed and final task performance over both standard LoRA and its state-of-the-art modifications.

Published as a conference paper at ICLR 2026
LORA MEETS RIEMANNION: MUON OPTIMIZER FOR PARAMETRILAATION-NNDEEEENEEEDE LOW-RANK ADAPTERS
Vladimir Bogachev
Vladimir Aletov1.2
Alexander Molozhavenko'
Denis B,
Vera Soboleva 1,3
Aibek Alanoy 1.3
Maxim Rakhuba'
HSE University
Basic Research of Artificial Intelligence Laboratory (BRAIn Lab),
FusionBrain Lab, AXXX
ABSTRACT
This work presents novel. fully Riemannian framework for Low-Rank Adap- tation (LoRA) that geometrically treats low-rank adapters by optimizing them di- rectly the fixed-rank manifold. This formulation eliminates the parametrization ambiguity present in standard Euclidean optimizers. Our framework integrates achieve this: (1) we derive Riemannion, new Rie- mannian optimizer on the fixed-rank matrix manifold that generalizes the recently proposed Muon optimizer; (2) we develop a Riemannian cradient-informee LoRA initialization, and (3) we provide an efficient implementation without prominent overhead that uses automatic differentiation to compute arising geometrie opera- tions while adhering to best practices in numerical linear algebra. Comprehensive experimental results on both LLM and diffusion model architectures demonstrate that our approach yields consistent and noticeable improvements in convergence speed and final task performance over both standard LoRA and its state-of-the-art modifications.
three key components
INTRODUCTION
Large language models (LLMMs) have demonstrated remarkable capabilities across a wide range of natural language processing tasks Brown et al. (2020); [Tovrron et al.) (2023a However, the computational and storage costs associated with training and deploying such models at scale pose significant challenges. To reduce these costs, parameter-efficient fine-tuning techniques such as low-rank adaptation (LoRA) Hu et al. (022) have emerged as practical solution. LoRA enables efficient adaptation of pre-trained models by embedding learnable low-rank matrices into specific weight updates, allowing most of the original parameters to remain frozen. In particular, the main idea of LoRA is to fine-tune pretrained model using a rank-r correction matrix AW:
W + AW = W ABT,
E Rmxr
R
where W remains constant during training and A, B are optimized via gradient-based optimization methods.
Despite its efficiency, the dominant practice of optimizing the LoRA factors (A, B) with Euclidean optimizers such as SGD (Robbins Monro 195T). Adam (Kingma & Ba 2014). Adagrad (Duchi et al. 2011). RMSProp (Tieleman, 20122) etc. that misaligned with the geometry of the low-rank constraint. The same update AW can be represented by infinitely many factorizations: for any
"vabogachevv@hse.ru

---

Published as a conference paper at ICLR 2026
A R Rmxr, B Raxr and any invertible matrix S R"x", may write:
AW AB =AB where AS,
Ideally, training should be reparameterization (Iransformation) invariant: the update to AW must depend on which factorization (A, used (Yen et al. 2024). Empirically, this lack of in- variance manifests as unbalanced learning where one factor dominates and the other stalls, fragile hyperparameter sensitivity, and path-depenn auth-dependent solutions. These issues have prompted geometry- aware formulations. Riemannian treatments of low-rank models operate on the fixed-rank manifold rather than the ambient factor space, projecting gradients the tangent space and retracting back to the manifold. Such steps can be implemented efficiently when < min(m, n2 and avoid forming full-size matrices. Within the LoRA literature, existing Riemannian approaches either use standard SGD-type optimizers like LORO (Mo al. 2025 rely the Adam (Zhang & Pilanci 2024) optimizer for auxiliary matrices within the chosen parameterization, which deviate them from the Riemannian framework and introduce dependence parameterization.
In this work, we introduce a fully Riemannian framework for training LoRA that optimizes the adapter X = AW directly on the fixed-rank matrix manifold
M, {X E IR=x* rank(X)=
eliminating factorization ambiguity by construction. Central our approach is Riemannion, new Riemannian optimizer on M, that generalizes the recently proposed Muon optimizer (Jordan et al. 2024) the fixed-rank setting. In contrast to prior Riemannian LoRA variants that port Adam- like mechanics to the manifold with ad hoc choices, our design inherits Muon's 's aalieee normalization, yielding transformation invariance of the learned update. We further propose Rie- mannian gradient-informen initialization that places the initial adapter at good location on M,, and we provide a practical, low-overhead implementation that assembles projections, retractions, and vector transports via automatic differentiation, also following best practices from numerical linear algebra. Extensive experiments on LLM and diffusion architectures show consistent gains in convergence speed and final task performance over standard LoRA and recent state-of-the-an modifications.
Our contributions as follows:
Muon on the fixed-rank manifold. We derive Riemannion, the first opti- mizer that generalizes Muon to the manifold M, of fixed rank matrices.
Riemannion:
Riemannian gradient-informed initialization. We propose initialization strategy which yields best alignment between the initial Riemannian gradient and the Euclidean gradient. We also propose an efficient way for this strategy by using a randomized SVD algorithm with implicit matrix multiplication (Section Finally, we show the connection of this initialization LoRA-GA
Efficient implementation with automatic differentiation. We pay special attention to numerical implementation to make the method robust without any prominent overhead compared to vanilla LoRA at small ranks.
Comprehensive empirical validation. We showcase the performance of our framework for fine-tuning LLMS and in subject-driven generation using diffusion models. Among positive effects that we observe are: boost in target metrics, improved convergence, and reduction of variance.
RELATED WORK
The problem of an optimal initial guess selection for low-rank LLM adaptation has been addressed in sequence of works: Meng et al. (2024) (PiSSA) have suggested heuristic that involves using a low-rank truncated SVD of pretrained parameters as an initial point for LoRA and its orthogonal
complement as frozen layer's parameters, starting value of the loss function. A similar approach was implemented by Wang et al. (2025) (MiLoRA) with the main difference of optimizing the smallest singular components of unadapted parameter matrix. A context-aware initialization was considered in CorDA (Yang et al, 2024) and COALA (Parkina & Rakhuba 2025) proposes numerically robust inversion-free framework for
that the tuning process starts without changing the
2

---

Published as a conference paper at ICLR 2026
low-rank weighted approximations for this setting. Another idea is initialize LoRA with subset of left and right singular vectors of a doubled-ramd truncated SVD of the loss function gradient at the starting parameters, proposed by Wang al. (2024) (LoRA-GA). We show direct connection of this method to our Riemannian initialization strategy and propose how to additionally significantly accelerate the computation of SVD using our approach. Attempting overcome the asymmetry in the initialization of vanilla LoRA fine-tuning process, Hayou et al. (202)) (LoRA+) have introduced a scale-free step size selection for LoRA factors.
Riemannian optimization is widely used for algorithms on matrix manifolds and allows for exploit- ing task geometry or imposing additional constraints. For example, a Riemannian solution for the extreme eigenpairs search problem was described in Absil et al. (2009); Baker (2008). a matrix completion task, which common collaborative filtering for recommender systems, via opti- mization on the fixed-rank manifold (Vandereycken) (013), a Riemannian approach on the manifold of matrices with orthonormal columns (the Stiefel manifold) was used by /iisdom et al. (2016) for diminishing the problem of vanishing and exploding gradients in recurrent neural networks, etc. The book Trendafilov & Gallo (202T) also presents a comprehensive description of useful mani- folds for solutions of the data science problems. For the deeper understanding of applied differential geometry techniques see the books Absil et al. (2009) and Boumal| (2023).
The idea of using Riemannian optimization has recently started to emerge for the large language models. For example, the fine-tuning of L.MSs with the help of the Stiefel manifold was considered in the work Hu et al. (2024). The authors of (Zhang & Pilanci (022) introduced the Riemannian inspired modification of Adam. Mo et al.] (2025) (LORO) applied the Riemannian optimization techniques for pretraining LLMs on the fixed-rank manifold. Parametrization that is used in our work can potentially help in this setting as well, by additionally avoiding potential overheads and instabilities, arising due the explicit inversion of Gram matrices.
3 PRELIMINARIES
3.1 MUON OPTIMIZER
Muon is an optimizer designed specifically for matrix-valued parameters in a network's hidden lay- ers. Empirically, it accelerates training on language and vision workloads while leaving scalar/vec- tor parameters and the input/output layers level, Muon takes the step that stochastic gradient descent with momentum (SGDM) would make on weight matrix and orthogonalizes that update before applying it. Orthogonalization acts as per-layer, per-step preconditioner that equalizes singular values of the update, which mitigates the collapse of updates into few dominant directions (Jordan et al.) (024). More specifically, let W E gax be hidden-layer weight. With gradient G. VwL(W,) and momentum Me = 3M + Gr, Muon computes
conventional optimizer such as AdamW. At a high
M, = Ortho(M,) and W2+1 = W - r Mr
where Ortho(-) denotes the nearest semi-orthogonal matrix in Frobenius norm, i.e.,
Ortho(G) = arg max O (110 - Glle: OTO = or 00 = =I}.
(2)
Computing Ortho(G)) exactly amounts to taking the SVD G = USVT and returning UVT, which is too slow to do at every iteration. Muon instead applies a Newton-Schut (NS) iteration that-after normalizing G - implements composition of fixed low-degree polynomial in GGT acting on G and converges to UVT. Leveraging efficient matrix multiplication operations results in a highly performant iteration. We will write M,- NS(M) for short.
LMO interpretation. Muon's step admits clean linear minimization oracle (LMO) interpreta-
tion (Bernstein] 2025). Indeed, consider the operator-norm unit ball B₂ = {X ||X||2 < 1). The linear minimization oracle (LMO) over B, at matrix M, given by its SVD M, = UEVT T
UV E Arg max (M, S).
(3)
111
3

---

Published as a conference paper at ICLR 2026
Applying Muon to LoRA. In LoRA, frozen weight Wo R is adapted via a low-rank update = W6 + a BA with B E R A E Rxx" and small r. Each trainable factor (A and B) a 2D parameter, so Muon can be applied per factor:
M(4) < 3M + VAC, M (B) + BM + BBC B+41 t B, BBB NS (M, (D).
At11 A₁ - 7A NS (
Note that acting on the two factors separately makes Muon non aun--rarramteriiatioiinneeee per-factor orthogonalization depends on arbitrary scalings or rotations, skewing the weight-space step and often letting one factor dominate.
its
3.2 RIEMANNIAN OPTIMIZATION
Let M, = [X E T rank(X) = r] S RRxx be smooth manifold of fixed-rank matri- ces (Example 8. 14) Lee (203). Let every point X of M, be equipped with tangent plane Tx My. Thinking geometrically, the tangent plane plays the role of the best local, flat approximation to this curved set: if you "zoom in" at X, the manifold looks like a plane. We will now discuss how to numerically parametrize points on manifold and its tangent plane. Every rank-r matrix X E M, can be represented using matrices A E mRx, B, E gaxr with orthonormal columns and square matrix T E R as
X= ALB A (TBR) (ArT) BR
For example, one may think a thin SVD, in which case T becomes the diagonal matrix of singular values, but other representations are also possible and will be convenient for our purposes. Note that the last two representations in 4 coincide with the TT parametrization in two-dimesional case
(Oseledets. 2011). Combining this representation with the derivations from (Holtz et al. 2012). one can obtain the following form of the tangent vector & E Mx
= [A An] [n
BR B B)T
AT =0,
(5)
B). Since the factor matrices contain 2r columns, we immediately have that rank < 2r. Another remarkable fact is that the point X E M, itself lies in the MxM, with Å = 0, B = B. Given a matrix Z E R=xx, its orthogonal projection PT Z onto the tangent space Tx M, (with the parameterization given in 5)) can be computed as follows:
so that any tangent vector can be identified in terms of the tuple: (A. B).
PixM, A (1-A ZBBBR.
and, hence, can be represented in the form of 5 with Á -= (I-
and B Z Ar,
Let C: mm*= + R be a differentiable function with the Euclidean gradient VL E R=x", Within the Riemannian optimization framework, we solve the following task optimization problem:
min ((X). XEM,
When constructing the algorithms, we need to work with three key objects: Riemannian gradient, retraction and vector transport. The Fuclidean gradient is direction of the steepest local increase L. Therefore, it is common to use the Riemannian gradient - the direction of the steepest local increase of corresponding smooth function value along the manifold, which lies in the tangent space (Absil et al. 2009) chap. 3.6 ). Given the Euclidean gradient V C. one may endow the tangent space Mr with a natural scalar product and derive a formula for the direction of the local steepest ascent of L with respect the manifold. This unique direction is called the Riemannian gradient and can be computed as follows:
grad ((X) = PTx (VL(X)), X EEM..
simple and robust retraction that maps a tangent step & (for example, & is a negative Riemannian gradient) back to the manifold is the truncated SVD:
(x R(X + { SVD, +
(8)
4

---

Published as a conference paper at ICLR 2026
Le., the best rank-r approximation of X + 5 in Frobenius norm. Note that here we do not need to compute the full SVD and can utilize low-rank structure of X and ç. leading to ( + n)rr 2))r++ +r) operations (Absil & Oscledets 2013). Finally, because tangent spaces change from an optimization step to step, momentum (an escumulated tangent vector) must be moved between them via a vector transport. For embedded manifolds like My, standard choice is the projection transport
(E) = (E),
Tr.
which simply reprojects the same ambient matrix onto the new tangent space at X.
4 RIEMANNION
In this section, we focus on the setting of parameter-efficient fine-tuning and, hence, to the fixed-rank manifold. For fine-tuning of one layer the optimization problem becomes:
C(W - AW)
min AWEM,
where L is differentiable loss function. Note that optimizing M, removes the ambiguity of factorized parameterizations, because all computations are carried out in the intrinsic space of the product X rather than in any particular factorization. So the formulas we write below will
naturally be reparameterization invariant. Let Gr = (V be the Riemannian gradient.
A Riemannian heavy-ball (Polyak 1964) momentum step reads
M, = 3 Mr-1 + G1, Mr-1 = 1w. (Me-1).
(10)
AW = R(AW, -1 M)),
(11)
with My = O, momentum parameter 3 E [0, 1), and stepsize 1) >> 0.
Let us now discuss how to introduce Muon-like variant of this iteration, which we refer to as Riemannion. A direct projection onto the set of orthogonal matrices Ortho(M,) presents two chal- lenges. First, such step does not respect the underlying Riemannian geometry. To address this, we propose to find the best approximation of Ortho(M;) on the tangent plane MW, Such a so- lution is given via the projection onto the tangent plane PTw.M. (Ortho( (Ortho(M,)) However, a second issue arises: although this projection is low-rank, its computation remains inefficient because the input matrix of full rank. Let us notice that M, E Taw, M, and, hence, is of rank at most 2r (Section 3.2). At the same time, the LMO interpretation (Section 3.1) provides several admissible low-rank solutions, including one where Orthor() (-) replaces only the first 2r singular values with 1, while all others are set O. Consequently, we obtain the following update rule:
Mr Praw. (Ortho,
(12)
Note that (rtho,,() exactly preserves the column and row spaces of M, Mp. Although Praw, M. (Ortho,,,MM) does not yield singular values exactly equal to 1, in practice they remain in a close proximity. In particular, in our experiments they were always in the interval (0.9, 1.1). This behavior is reminiscent of the Newton-Schull. iteration, which likewise produces approximate singular values. To climinate this inexactness and obtain an accurate solution in the intersection of M, with the set of matrices whose first 2r singular values are exactly one could apply various strategies. For example, one may formally apply the alternating projection method as shown in Cheney & Goldstein 1959 Theorem 4, Section 2) and Boyd & Dattorro (2003):
M, M = Praw, M, (Ortho,(... Praw,,,((rthoo, (Ortho,
(13)
As an alternative, we could solve the LMO problem the tangent plane:
M Arg max SM,
(14)
SETaw,
||S||211
where we proposed to apply observations from (Cesista 2025) to the fixed-rank manifold. In Ap-
pendix ! we show that (12 is an approximate solution to the maximization problem (1). Apply- ing (13) or accurately solving (14) on the intersection of two convex sets using convex optimization methods is more computationally expensive, and our experiments indicate that it has little to no impact the overall convergence of the optimizer.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 OrthoLR (efficient computation of Ortho, (E) for & Tx.M).
Require: ç E Mr given by (Á, B) from (5) AL. BR such that A as in
Ensure: A Rmx2r, B E pmx2r; AB Ortho(E).
1: Q1. Tı, = qr([Ar, Al), Qn, T, = qr([13 Bn)
11 0 ((m + n)rr
2: UU,,, V, SVD (T R
110 (12)
3: A B QRVR.
((m + 22)
Algorithm 2 Pro jectLR (efficient computation of PTx M, (2) for a rank-r' matrix Z).
Require: A E Rmxx'' B RRar'' such that Z =ABT; Ar, BR such that X ArTBR as in 4
Ensure: { = MT (Z) given by (A, B) from 5
1: A = (A (R(A BR), B = B(A Ar.)
11 100 + m)r
Algorithm Retractionna (efficient computation of R(E) for tangent vector & Mx,
Require: AL, A Rmxr BR, B Raxr such that {= [A Ar] [BR
B]
Ensure: X A E Mr given by X = truncSVD (E, Qr. Tr. cr([Ar, Al), Qn. T, qr( Bnl )
11 O ((m + n)r²)
2: U,E, V = truncSVD (T2 rr).
11 (r2) |/ O ((m + n)r2)
3: Åz, B (QLU), (h)-
Let us finally show that M, from 12 can be computed efficiently using O((m + n)r² + r arith- metic operations. Indeed, first of all we need apply Ortho,--) to a tangent vector Mr. From (6) we know that tangent vector can be represented in a form of a rank-2r matrix. Such representa- tion can always be transformed into the compact SVD form with 2 QR decompositions and single full SVD of 2r x 2r matrix, see Algorithm I called OrthoLR. As next step, need to project the obtained result (decomposed matrix of rank 2r) onto the tangent plane. The operation can also be done efficiently via (6) and is summarized in Algorithm called ProjectLR.
5 LOCALLY OPTIMAL INITIALIZATION (LOI)
Once the theoretical framework for the Riemannian optimizer is established, it is natural consider an initialization scheme that accounts for the underlying Riemannian geometry. Given any AW e My, we may write
C(W) = -ww-wWWwwwwo +WW) = C (w" + AAW)
(15)
This raises the question: how should AW be chosen ensure the fastest loss decrease along the manifold? The solution is to consider the following optimization task:
AW (0) E Arg max P|PTaw.M VWWII
(16)
AWEM,
Since Praw Mr is an orthogonal projection matrix to the tangent plane, the task (66) essentially seeks for the point on the fixed-rank manifold, whose tangent space has most alignment with the Euclidean gradient. In other words, this means that the direction of the steepest local function decrease alongside the manifold aligned with the full model tuning direction. The solution to this task is presented in Theorem 5.1
Theorem 5.1. Let the SVD of V w C(W) be:
Σ1,
0
VWC(W) 1U
U
U.]
E,
0
[V.r
Vr.rr Va]
E1

---

Published as a conference paper at ICLR 2026
and let also 02r + 02 Then any optimal solution AW! to the problem
16
has the form:
AW
[U
Un,27
Sn1
[C21
Czal
3 E1.
Vir
S21
V1,2r
(17)
Sn1
S12
C11
C12
s
GL2,(R), 8-1
S21
S22
C21
C22
Proof. See Appendix A
In our experiments, we use S aI, 0 obtaining: AW!") =UU1, VT.2r E EM,, E R (0). I,
Interestingly, Theorem relates to the findings of Wang et al. (2024), although their analysis neither adopts Riemannian framework nor addresses parametrization-fee optimization. Our op- timizer further differs from Zhang Pilanci (2024) and LORO (Mo et al.) 2025) in that it avoids
inversion of the Gram matrix. As result, the method remains stable |AW O. Empiri-
cal results indicate that initializing AW:") with a small norm leads to improved performance. The
procedure for selecting the scaling parameter is described in Appendix
SINGLE BACKWARD- GRADIENT TRICK
This section introduces efficient method for computing gradient-times-matee matrix in matrix-fres way, at a computational cost equivalent to a single backward pass. We then apply this technique in both the LOI initialization procedure in Algorithm 4 and the Riemannion optimizer in Algorithm
The calculation of the full fine-tuning loss gradient W L (W) with pretrained parameters W E Rmx" is computationally expensive. At the time, for our framework we only need compute the products (V w C (W) M) or (Ww (W) .) for some M E Rmxr, N E g"xr, Using the trick from (Nvikovv et al. 2022), may calculate both quantities simultaneously using a single borward-bakwwatd pass with a doubled rank representation.
First, initialize differentiable parameters Z1 0 e m and Z₂ 0 Raxr and perform a simple forward pass L := C (W + N + .M,)). ). This step does not violate the pipelines of LoRA framework since it is equivalent to standard LoRA forward pass with special adapter:
N + MI = 1 [N
Z2) = ABT, Ā mx B E "x22
Then we invoke an autodiff algorithm for value L, which ensures us both:
Vz, L V v (W Z MZ2))
1z==0 VWL(W)N,
(18)
Vz, L = Vz,C (W ZT M2,))
1,,=0,7==0 = VwL(W)) M.
Note, that AW A, ABJ from W + AW, then one may take - BR. M AL, Z1 0, Z2 = B,,Y = W + AW and the exact same operations work.
Notably, (18) is crucial for computation of the Riemannian gradient in (6. Therefore the proposed approach is a key building block that allows us to avoid forming the full fine-tune gradient loss and effectively compute matrix-matrix multiplications. This idea is the core for both: LOI initialization (Algorithm 4 and Riemannion optimizer Baleiizerrlloooaa (Algorithm
Randomized SVD for effficintt initialization The computation of the 2r-truncated SVD of the full loss gradient, required by Theorem 5.1 has asymptotic complexity O(min (m, n mm)) which may be infeasible for large-scale models. In addition, explicitly forming this gradient itself com- putationally expensive, and thus need avoid it in practice. To overcome this problem, propose to use a randomized SVD with power iterations (see (Halko et al. (011)), which we also enhance with our one-step gradient trick is described in the Algorithm 4 In nutshell, need to compute (V w CWW) V. L(W)T)² Y, where Y = Vw C(W) and Ω is sampled from standard

---

Published as a conference paper at ICLR 2026
normal distribution. This iteration can be done in robust manner using QR decompositions, The steps 2, 4, 5, 6 correspond to trick from (18).
Overall, the LOI search procedure has asymptotic complexity (((m++)r²) plus 2(q++1 1] additional backward passes. Moreover, since LOI search is executed only once before fine-tuning (which typically involves thousands of backward passes), its runtime overhead is negligible, taking merely 0.25% of the total fine-tuning wall-clock time in our experiments. Section 77.1 about 2 thousand optimization steps were carried out and Algorithm accounted for merely 0.25% of the total fine- tuning wall-clock time.
Algorithm BackPropRsvT
Require: Weights W Rmxx rank N, oversampling parameter p, power-step parameter q. Ensure: Randomized r-truncated SVD (Ur. E,, Vy)) of C(W).
I: Choose k + p. Sample $2 E gn*k (0, 1).
1/ O (nr) // //1 backward pass +0 (mr²)
2: Y == Q L(W + AQ )l==)).
3: for := 1.... y do
4: Y := qr((Vn C(W + YYB [ ).QQ.
// backward pass +0 (nr22)
5: Y qr(V L(W + +AY A
// backward pass + + (mr?)
6: Y := [V C(W + YB B=0]
1 backward pass
7: U,E.VT truncSVD (Y,r).
// O (nr2) O (mr")
8: YU,
Efficient Riemannion implementation The Riemannion optimizer is presented in Algorithm 5 Similar to vanilla Muon, it relies on a chosen Ortho procedure. Since the LoRA approach repre- sents the fine-tur tne-ttnnng shift AW with low-rank matrices, we employ an SVD-based Ortho procedure. For computational efficiency, this procedure is adapted to the fixed-rank manifold, as described in Section 4 and implemented in the OrthoLR (Algorithm) [ In detail, in step we calculate the Riemannian gradient components via single backward call (18) Then, in step 2 the algorithm transports the Heavy-Baa tangent direction to the current point via Algorithm/2H) (aatthhmmhhhttt provides a sim- compute the final optimization direction
ple but effective implementation of (6). In line the tangent space of the given point with a Heavy-Ball momentum coefficient 3. In line 5 algorithm performs the retraction step.
The algorithm finalizes with saving the obtained minimization direction for momentum in line 6 and calculating of new point representation in step 7.
Algorithm 5 One step of Riemannion
Require: Weight matrix W" E Rm rank E N, initial point As,, BR. Heavy-ball momentum AHB, BHB, step size 77, momentum coefficient B, weight decay coefficient Ensure: Tuming parameters AW"= B E MM.
1: Á, = Vx, L (W' + 3 + A1.
1,,
Vz,6 (W' + B,
R + Ar 2,=0ZZ==b
// I backward pass
2: À Aprev: Bprev ProjectLR (( (AHI, Bun), Ar., BR).
// O ((m (mm+n)) n)r2) O ((n + +m)r? 11 O ((m + n))r + rr))
3: Á, B := BÁprev + (I ALA ) A,, BBreev +
4: Á, ;= ProjectLR(DrthoL.) AL, BR. A. B,T, A BR)
5: Ann, Bun == [Á Ar.), [Bn, B)
6: B² == RetractionLR( n Ar), [BR, (n + BR)l)
O ((m + n)r² + n3)
Overall, one iteration of the Riemannion loop has asymptotic complexity O((m + n)y² فهر and additionally the same number of backward passes as vanilla LoRA. For comparison, the Euclidean Muon optimizer for LoRA (Section 3.1] exhibits the same asymptotic complexity. The complete framework and its time performance are summarized in Algorithm 6 (see Appendix B and Ap- pendix H
8

---

Published as a conference paper at ICLR 2026
7 EXPERIMENTS
In this section, we also employ the term LOI (Locally Optimal Initialization), referring to the pro- posed initialization scheme described in Section This is done to distinguish between using Rie- mannL.RA A as an optimizer with zero initialization (as in basic LoRA) and RiemannL.RRA-Lo which is the proposed combination of initialization and optimization. We conduct series of LLM fine-tuning experiments for different tasks. All of the experiments were computed on NVIDIA V100-32Gb GPU and A100-80Gb GPU. We ran all the experiments within 2000 GPU hours.
7.1
COMMONSENSE REASONING FINE-TUNING
The results were obtained for the benchmark Clark al. 2019 Bisk al., 2020 Sap et al. 2019
Zellers et al. 2019 (Sakaguchi et al.] 2021] Clark et al. [0188] Mihaylov et al. (0018) common
reasoning. The structure of the dataset described Appendix In the following experiments we conduct fine-tuning procedures for multilayer perceptron (MLP) and attention layers of Llama 3 8b model (Dubey et al. (2024)).
The commonsense reasoning tasks comprise of 8 sub-tasks, each of them contains predefined training and a testing set. We follow the setting of Hu et al. (2023) and amalgamate the training datasets from all 8 tasks to create the final training dataset and conduct evaluations the individual testing dataset for each task.
The hyperparameter tuning protocol and the selected hyperparameters provided in E Table contains the accuracy of the trained model's responses on the test dataset. Within the LoRA frame- work, the proposed Riemannian fine-tuning method delivers clear performance gains. The Rie- mannion optimizer consistently outperforms LoRA, DoRA, and achieves superior metric results compared the standard Muon optimizer applied LoRA factors (see Section 3.1) Furthermore, relative to other Riemannian-geoomeeyy--nnn
approaches such as RPrecAdam (Zhang & Pilanci, 2024), our method also demonstrates better results. Finally, the variance of outcomes for the pro- posed method is the smallest among all compared approaches.
Table 1:
The average accuracy (in %) among tasks of fine-tuned Llama 3-8b using different ap- proaches, tested on Commonsense Reasoning benchmark. LoRA rank is set to 16.
BoolQ
PIQA
SIQA
hella-
wino- grande
ARC-E
ARC-C
OBQA
Initialization
Raw
65.0
76.6
73.0
66.1
61.3
92.5
82.3
79.6
74.5
Adam
74.821.9
89 89.810.a
82. 82.6.0.6
96.210.3
87
87.911.2
92. 92.4
84.910.
88.5 58.5
87. $7.110.4
74.820.8
89.420.5
82.420.7
95.9
87.810.4
90.721.2
83.8107
86.610.3 87.820.6
Muon
72.940.0
86.4+0.s
80.8-0.2
94.1
84.440.a
84.2=0.9
77.3+2.6
83.9+1.1
83.0+0.6
LORA-RITE
72.2+0.3
88.6 88.6+0.1
82 0..0...
95.1+a.1
85.640.2
87.7=1.
79.3+2.8
85.7
84.5
RPrecAdamW
75.820-4
89.540.4
82.4go.2
96.110.2
87.710.a
90.621.6
84.111.1
87.720.5
86.810.4
Riemannion
75.7
91.210.2
8
710.0
88.610.4
93.6go.a
88.410.4
89.3
88. 88.110.2
7.2 SURJECT-DRIVEN GENERATION
Subject-driven generation Ruuiz et al. 2023 Gal et al. 2023) is a task in which the user provides several reference photos of an object, called concept, and uses diffusion model generate this concept with certain conditions (e.g., textual prompt). One way to solve this task is to fine-tune pre-trained diffusion model using this small of reference images. However, this technique leads to degradation in understanding of the conditions and to a fast overfitting of the concept. Furthermore, it requires a high computational cost due to the large number of trainable parameters.
This is why previous works, such as (Qiu et al. 2023 Liu et al. 2024 Hu et al. 2022 Tewel et al.) 2023 Han et al., 2023) Gorbunov et al., 2024). propose training only a lightweight parameterization for the base model. In this section, we demonstrate the performance of our parameterization in this task.
In our experiments, we used Stable Diffusion 2 (Rombach et al. 2022) the base model. We choose LoRA Hu et al. (2022) as a baseline and train both models with ranks of 4, and 16.

---

Published as a conference paper at ICLR 2026
rank
rank
rank
Concept
LoRA
LoRA
V in Arelihter out!""
y
Figure 2: Visual results for Subject-driven generation on 600 training step.
We predict the parameterization of the q. v. and out . matrices in all attention layers. The Dreambooth dataset (Ruiz et al. 2023) was used in all our experiments.
0.65
0.72
LoRA was trained using the Adam optimizer. In Figure 2. present visual comparison of LoRA with different ranks. As can be seen, even for complex concepts such as "robot toy', method requires only 600 steps learn the concept while preserving appropriate text similarity. We found that for subject-driven generation tasks, the lower the rank, the faster our method converges. To eval- uate this, we also calculated metrics for different learning rates subset of the Dreambooth dataset. We use CLIP to measure text similarity and DINO to mcasure image similarity. Figure shows that, even with different learn- ing rates, our method achieves more accurate results in concept preservation. Further details and visual com- parison can be found in the Appendix
LaRk
LoRA In-
Oura In-5a-6
Ours 1rs1-5
0.35
Ours Ir=22 5
Oun Immda-3
0.50
0.320 0,325 0.330 0.315
T5
Figure 1: Comparison of text and image similarities for LoRA and our method with rank 4 at different learning rates on 400 step.
CONCLUSION
In this work, we propose a novel fully Riemannian framework that integrates a new muon- muon-based opti- mization method, locally-optimal initialization, and efficient implementation. This integrated ap- proach yields a reliable reparametrization-ine on fine-tuning large language models (LLMs) and exhibits additional favorable properties for low- rank approximations in diffusion models. Given the promising empirical results, a natural direction for future research is to investigate the theoretical properties of the proposed method.
method that outperforms competing approaches
REPRODUCIBILITY STATEMENT
The hyperparameter selection procedure is described in Appendix E The datasets used in the ex- periments and the corresponding preprocessing steps are detailed in Appendix [D] The proof of the stated assumptions is provided in Appendix A The hyperparameters for Subject-driven generation
task are provided the Appendix
ACKNOWLEDGMENTS
The work was supported by the grant for research centers in the field of Al provided by the Ministry of Economic Development of the Russian Federation in accordance with the agreement 000000C313925P4E50 and the agreement with HSE University Nt 139-15-2025-009. The cal- culations were performed in part through the computational resources HPC facilities at HSE University (Kostenetskiy al. 2021
10

---

Published as a conference paper at ICLR 2026
REFERENCES
P-A Absil and Ivan V Oseledets. Low-rank retractions: a survey and new results. Computational Optimization and Applications, 62(1):5--22 29, 2015.
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms on matrix manifolds.
In Optimization Algorithms on Matrix Manifolds. Princeton University Press, 2009.
Christopher G Baker. Riemannian manifold trust-region methods with applications to eigenprob- lems. The Florida State University, 2008.
Jeremy Bernstein. Deriving muon, 2025. URL
deriving-
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. PIQA: Reasoning about Physical Commonsense in Natural Language. In Proceedings of the AAAI conference artificial intelli- gence, number 05 in 34, pp. 7432-7439, 2020.
Nicolas Boumal. An introduction to optimization on smooth manifolds. Cambridge University Press, 2023.
Stephen Boyd and Jon Dattorro. Alternating projections. stanford. hat.foo...ca//class//332//////1. p.tpitt__ppopppp____e pdf
2003.
URL https: //web.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal,
Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Franz Louis Cesista. Muon and selective survey on Steepest Descent in Riemannian and non-Riemannian Manifolds, April 2025. URL http:h/tp///e... wew.eeloyknnggiihheee 10/ steepest descent-eeeeeeeee non riimanniannnnn/aeaama
Ward Cheney and Allen A Goldstein. Proximity maps for convex sets. Proceedings of the American Mathematical Society, 10(3):448-4555 1959.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings the the 2019 Conference the North Amer- ican Chapter of the Association for Computational Linguisties: Human Language Technologies, NAACL-HLI 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pp. 2924-2936. Association for Computational Linguistics, 2019. doi: 10. 10.18653/V1IN 1/N19-111000 URL
1300
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try ARC, the A12 reasoning chal- lenge. CoRR, abs/1803.05457 2018 URL http::/axxxi wwtp:/iiivrg//aaaa hbtpg//bb/1883 05457
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshom, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang, Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Ander- son, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Ko- revaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes. Jason Park, Jay Ma- hadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee, Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak,

---

Published as a conference paper at ICLR 2026
Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Al- wala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield, Kevin Stone, and et al. The Llama 3 herd of models. CoRR, abs/2407.21188. 2024. doi: 104850/ARXXIV2407.2113.. URL httpps://dd
or wtt.:oo.g......... 48550/550/ wtwp:///.w.w............ 21783
John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of machine learning research, 12(7), 2011.
Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychome- trika, 1(3):211-214 1936.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Haim Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. In The Fleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URLitttss//oppeeeeee net/forum?ich SitpummiddNAQOOOSTTEE NAQVFO8TCYG
Mikhail Gorbunov, Nikolay Yudin, Vera Soboleva, Aibek Alanov, Alexey Naumov, and Maxim Rakhuba. Group and shuffle: Efficient structured orthogonal parametrization. In Amir Glober- sons, Lester Mackey, Danielle Belgrave, Angela Fan, Ulrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Confer- ence on Neural Information Processing Systems 2024, NeurIPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024.
N. Halko, P. G. Martinsson, and J. A. Tropp. Finding structure with randomness: Probabilistic algorithms for constructing approximate matrix decompositions. SIAM Review, 53(2)2117-288 2011. doi: 18.11377997778866 URL http://://......o//0111111////00
Ligong Han, Yinxiao Li, Han Zhang, Peyman Milanfar, Dimitris Metaxas, and Feng Yang. Svdiff: Compact parameter space for diffusion fine-tuning. Proceedings of the IEEÉ/CVF Interna- lional Conference on Computer Vision, pp. 7323-7334, 2023.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient Low Rank Adaptation of Large Mod-
els. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2024. URL htttt:::///ppeeeeeee NEV8YqBROO
net/forum?id
Sebastian Holtz, Thorsten Rohwedder, and Reinhold Schneider. On manifolds of tensors of fixed tt-rank. Numerische Mathematik, 73044)
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, al. LoRA: Low-Rank Adaptation of Large Language Models. ICLR, 1(2):3 2022.
Jiang Hu, Jiaxi Cui, Lin Lin, Zaiwen Wen, Quanzheng Li, et al. Retraction-free optimization over the Stiefel manifold with application to the loRA fine-tuning, 2024.
Zhiqiang Hu, Lei Wang, Yihuai Lan, Wanyu Xu, Ee-Peng Lim, Lidong Bing, Xing Xu, Soujanya Poria, and Roy Ka- Lee. LLM-adapters: An adapter family for parameter-efficiae fine-tuning of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10. 2023, pp. 5254 5276. Association for Computational Linguistics, 2023. doi: 10.18653/ 10533////202333 B.N.:02....MMMMM-...3 v1/2023 emn m- 319
URL httpps:///ooii... http://......01.111111100
Keller Jordan, Yuchen Jin, Vlado Boza, Jiacheng You, Franz Cesista, Laker Newhouse, and Jeremy Bernstein. Muon: An optimizer for hidden layers in neural networks, 2024. URL https:
//kellerjordan wwtporaannggituuh. github. cacluu..ooppoo wt/posts/////ooocooooo http:/ossss/uucn
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations, 12 2014.
PS Kostenetskiy, RA Chulkevich, and VI Kozyrev. Hpe resources of the higher school of economics. In Journal of Physics: Conference Series, volume 1740, pp. 01 012050. IOP Publishing. 2021.
12

---

Published as a conference paper at ICLR 2026
John M Lee. Smooth manifolds. Springer, 2003.
Weiyang Liu, Zeju Qiu, Yao Feng, Yuliang Xiu, Yuxuan Xue, Longhui Yu, Haiwen Feng, Zhen Liu, Juyeon Heo, Songyou Peng. Yandong Wen, Michael J. Black, Adrian Weller, and Bernhard Schölkopf. Parameter-efficient orthogonal finetuning via butterfly factorization. In The Twelfth International Conference on Learning Representations, 2024. httttttts//oppeeeeeee.
net/forum?id= Niggndd
Iyy Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. wwwt/eeuum/ wee.idd--kkkigggge
OpenReview.net, 2019. BRRLhttps://ppeneiiiee
Fanxu Meng, Zhaohui Wang, and Muhan Zhang. PISSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models. Advances in Neural Information Processing Systems, 37:121038-121072. 2024.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can suit of armor conduct electricity? A new dataset for open book question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun'ie iici Tsujii (eds.), Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp. 2381 23911. Association for Computational Linguistics, 2018. doi: 10. 10.18653/V 1/D18- 1260. URL httth: //doi
Zhanfeng Mo. Long- Kai Huang. and Sinno Jialin Pan. Parameter and Memory Efficient Pretraining via Low-rank Riemannian Optimization. In The Thirteenth International Conference on Learning Representations, 2025.
Alexander Novikov, Maxim Rakhuba, and Ivan Oseledets. Automatic differentiation for Rieman- nian optimization on low-rank matrix and tensor-train manifolds. SIAM Journal on Scientific Computing, 44(2):A843-A869, 2022.
Ivan V Oseledets. Tensor-train decomposition. SIAM Journal on Scientific Computing, 33(5)22295 2317, 2011.
Uliana Parkina and Maxim Rakhuba. Coala: Numerically stable and framii for context- aware low-rank approximation, 2025.
07580
Boris T Polyak. Some methods of speeding up the convergence of iteration methods. Ussr compu- tational mathematics and mathematical physics, 4(5):1-17, 1964.
Zeju Qiu, Weiyang Liu, Haiwen Feng, Yuxuan Xue, Yao Feng. Zhen Liu, Dan Zhang, Adrian Weller, and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetun- ing. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL
https::/ //tptsee//eeeeeeeeww .et/ wowumoddd-mmwwweemmae
Herbert Robbins and Sutton Monro. A stochastic approximation method. The annals of mathemati- cal statistics, pp. 400-407, 1951.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent difflusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Keisuke Sakaguchi. Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. WinoGrande: An Ad- versarial Winograd Schema Challenge at Scale. Communications of the ACM, 64(9):99-106,
2021.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. SociallQa: Com- monsense Reasoning about Social Interactions. CoRR, abs/1904.09728, 2019. URL http:
13

---

Published as a conference paper at ICLR 2026
Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-
image personalization. In ACM SIGGRAPH 2023 Conference Proceedings, pp. 1-11, 2023.
Tijmen Tieleman. Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude. COURSERA: Neural networks for machine learning, 4(2):26, 2012.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Ar- mand Joulin, Edouard Grave, and Guillaume Lample. LLaMA: Open and efficient foundation language models. CoRR, abs/2302. 2023a, doi: 104455W/RXXI.200013377 URL https://doi. www.ooo.o..........o..oo 1..00.485500 48550/855500 wivi..iii... 2302 13971
Hugo Touvron, Louis Martin, Kevin Stone. Peter Albert, Amjad Almahairi, Yasmine Babaci, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee. Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang. Angela Fan. Melanie Kambadur, Sharan Narang. Aurélien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. Llama 2: Open foundation and fine- tuned chat models. CORR, abs/2307 2023b. doi: 10448550/ARXXV.230000999 https://dol. www.orgg..............ooo....... 48550/ 2307 09288
URL
Nickolay Trendafilov and Michele Gallo. Multivariate data analysis on matrix manifolds. Springer, 2021.
Bart Vandereycken. Low-rank matrix completion by Riemannian optimization. SIAM Journal on Optimization, 2312):12 2013.
Hanqing Wang, Yixia Li. Shuo Wang, Guanhua Chen, and Yun Chen. MiLoRA: Harnessing minor singular components for parameter-efffcient LLM finetuning. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL 2025 Volume 1: Long Papers, A Albuquerque, New Mexico, USA, April 29 May 4, 2025, pp. 4823-4836. Association for Computational Linguistics. 2025, doi: 10.18653/V WAW.:////I.22..AALLLLLNLN...CCC
URL / htt.ooiooooooo org/ 10. 1et..1666//v/222.2 http://222..aaacc....aaa
Shaowen Wang, Linxi Yu, and Jian Li. LoRA-GA: Low-rank adaptation with gradient approxima- tion. Advances in Neural Information Processing Systems, 37:54905-54931, 2024.
Scott Wisdom, Thomas Powers, John Hershey, Jonathan Le Roux. and Les Atlas. Full-Capacin C1-CCapcccty Unitary Recurrent Neural Networks. Advances in neural information processing systems, 29, 2016.
Yibo Yang. Xiaojie Li, Zhongzhu Zhou, Shuaiwen Song, Jianlong Wu, Liqiang Nie, and Bernard Ghanem. CorDA: Context-ori Context-oriented decomposition adaptation of large language models for task- aware parameter-efficient fine-tuning. Advances in Neural Information Processing Systems, 37: 71768-71791, 2024.
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. Lora done rite: Robust invariant transformation equilibration for lora optimization. arXiv preprint arXiv:2410.20625, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Anna Korhonen, David R. Traum, and Lluís Mårquez
(eds.), Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2. 2019. Volume I: Long Papers, pp. 4791- 4800. Association for Computational Linguistics, 2019. doi: 10.18653/V 1472. URL 18653/ wtt.//11111///1/////0////
14

---

Published as a conference paper at ICLR 2026
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned loRA for fine-tuning foundation models. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024.
15

---

Published as a conference paper at ICLR 2026
A INITAL POINT SEARCH
In this section we introduce the proof of Theorem 5.1
Proof. In order represent the optimization task (66 more simple way, we will use a slightly different formula for the projection of the full loss gradient onto the tangent space of the fixed-rank manifold. Let
AW = A, ABR E EM.
and the tangent space is parametrized as follows
MA {AB] + A
Rux AAA=
then one may derive an orthogonal projection formula for any matrix (see, e.g., eq. 7.53 in Boumal 2023
Praw M, Z Z (I AL4 BB E M,.
As the tangent space a linear space, the operation of orthogonal projection can be written as an optimization task
P.
arg min | - E Sr
(19)
Since
|| = || (VC - PTaw M, VL) + Praw M. VLI.
IVL- PTAW M, V + |PTAWM. M, VCII + +2V( - PTAW VL, PTa M, VL)
LV- PrawM, V + MPTAWM, VCIl + +2(Praw (VL Praw M, VL), VL)
V PIAWM, VLI2 M VCIl? 2(PTawM, VL PT VC, VL)
= |VC PTAWM, VC + [TAWWMM VCII
so the optimization task (16) is equivalent to
VL Praw M, VLI min.
which, in turn, using 19 is equivalent to the task
min min AW M,
VVL - El1
Due to the fact that every vector of the tangent space is element of set of all matrices with 27- bounded rank M one may use the Eckart- Lloug-MMrsky theorem (see (Eckart & Young 1936)) to obtain the following lower bound:
LV- PTAW-M, V Z VVC - truncSVD (V AW M,, E Mr. (20)
But it is possible to ensure AW! (0) E M, and &. E T M. which turn the inequality 20 into the equality. One may take
Aw (0) AA(aB)) aVU,r r.2r (aA)BR E R,
E. AB + A.B = EUr22 B + Ar. Vi,r) truneSVD (V2, 2r).
Note, that ATA, 0. So the initialization (W ensures that truncSVD (VL, 2r) lies in the tangent space "o Mr, which means that the first step of the RiemannLoRA Algorithm will get the Riemannian gradient equal to truncSVD (r)
16

---

Published as a conference paper at ICLR 2026
To generalize the result, should change the parametrization of the tangent space. Because of the fact that each of the tangent vectors & lies in Mg2r, we may use an unconstrained skeleton decomposition (without constraints for and Az):
B B.
B
&- = [A., A.]
(A,,A,,
T2r
B=
B B.
AA. SS-1
Se
2r
det S 0.
Representing S and its inverse as block matrices:
Sn S21
S12 S22
C11
C
s
s-
C21
C22
one arrives 17
&. [A,, A.]
S1
S12
Cn
C12
[.
B.]
S22
S22
C21
C22
AW!") |A,,
Sm
C22
Czal
B
B'=
S21
To derive the version that is utilized in practice, one may take to be block-diagonal with Sn aTr, S22 = Ir. a E R.
B RIEMANNION WITH LOI
The complete framework is summarized in Algorithm 6 which consists of an LOI initialization step followed by max_iters iterations of the Riemannion optimizer.
Specifically, the first 3 lines dedicated the computation of LOI via layer weights and oversampling parameter p. that increases the accuracy of the singular approx- imation. The 4-th step initializes Heavy-Ball momentum matrices. Therefore, the asymptotical complexity of this stage is determined by the complexity of Algorithm
for given
Algorithm Riemannion with LOI
Require: Weights W Rm rank step size n,, momentum coefficient 3, weight decay coefficient ?, oversampling parameter power-step parameter q. Ensure: Tuning parameters AW" E M,-
Function:
(2ckPrrppsVDD(2,p....444 C. 2: Ar, B = Ar(: ;: r]. r.:
1: AL BT
11 O ((m + n)r²)
3: W" :W - T
11 O (mn)
4: AнB, Buв 0.0.
5: for i O..... max_ max_iterr do
6: BR == qr (B) QQ.
(nr2)
7:
Á = V2, L (W' + B + +AZz) 12,=0,
1/ = Vw L(W) BR
8:
B Vz, L (W' Bi + ArZz) 12,=0,
1/ VWW( Ar
9:
Aprev: Bprev = ProjectLR ((AHn, BUR), Ar., BR).
11 110(m + (0m++)rr n))22 // 0 ((n + m)r?)
10:
Á, B. BA (I- A, _Bprev B
Á ProjectLR (OrthoLR (A BR, A. Ar,BR)
O ((m + n)r² + r²))
12:
AμB. BHE = Ad, [BR, B
Ar, B = RetractionLR ([-) Ar BRi -1 (B + (B+rBR))) 11 ((m + nnp + r2)
13:
14: return Az, B.
17

---

Published as a conference paper at ICLR 2026
C RIEMANN-SGD ALGORITHM
In Algorithm 7 we present the RiemannSGD version of the fine-tuning algorithm. The first steps dedicated the computation of the optimal initialization using BackP roprsvD 4). After initialization, the algorithm follows an Adam-like procedure adapted to the Riemannian setting: it maintains exponentially smoothed estimates of momentum and gradient norms (4th step). The overall cycle encapsulates the computation of the Riemannian gradient at the current point(steps and the retraction
7-8), the update of the adaptive momentum terms via
step that projects the updated tangent direction back onto the manifold(steps 15 15-17) 17). This approach has asymptotical complexity of ((m+n)r) + and (2(g+ max iters) amount of backward
calls.
Algorithm 7 RiemannSGD with LOI
Require: Weights W Rmxn, rank N, step size 7, momentum coefficient oversampling parameter P. power-step parameter q, simulate Adam, Adam momentum coefficient Ensure: Tuning parameters AW* E M,.
Function:
I: AL _ B.ck
1/ O ((m + m)r²)
2: Ar, B = Ar(: rl. rr:,r:].
3: W" -=
11 O (mn)
4: AнB: Внв, SA. O.
5: for i = O..... max eax_itersotee do
6:
BR qr (B).Q. .Q.
11 (nr²)
7:
Á = Vz, L (W + LB + A, 22=0,
8:
B Vz, L (W 7, B] + Ar 1z -0,=0,2=-B-
9:
Aprev: Bprer == ProjectLR ((AHB, BHB),AL, BR).
O ((m + +))r²)
10:
Á = BAprev (1 (1- B) A,A,)A, B. (1 B)
11 O (nr²)
11:
wiimulate Adam then
12:
SA. || + (1 S)a >|| p + (1 ~
I/ O ((m + n)r) I/ O ((m n)r) 11 O ((m + n)r² + r2)
À, A/SA, B/SB
U. RetractionLR (1 BR. nB + +B))
14:
AHB, BUB = [Á, Ar), [Bn, B)
AL, B = EV
O (nr²)
17: return B.
Table The average accuracy (in %) among 8 tasks of fine-tuned Llama 3.2- using different SGD variants, tested Commonsense Reasoning benchmark. The <R-> prefix stands for Riemannian, the postfix <-LOI> stands for locally optimal initialization. LoRA rank set to 16. In all experiments excpet for <L the A factor in AB has orthonormal columns and B is zero.
Task
BoolQ
PIQA
SIQA
hella- swag
wino- grande
ARC-E
ARC-C
OBQA
Initialization
50.0
LoRA
64.0+0.2
75.340.3
69.6+0.4
82.2+0.1
53.0+1.0
75.4+1.7
56.540.6
67.1+2.8
67.9+0.4
LoRA
64.9
76.8
71.2
84.1
56.7.0.6
77.0+4.2
61.123.6
68.9=3.3
70.1
RSLORA
64.5+0.2
77.240.3
68.140.8
86.340.1
58.2+0.5
76.541.6
58.642.9
70.0=1.9
70.040.8
RiomannLoRA
65.1
77.740.3
69.1
85.8
58 58.520.4
74.4+1.
56.9
69.3-2.b
69 09.64
RiemamLoRA-LOl
65.210.1
79.4
75.62..4
87.3
62.420.1
79.9
63.621.a
73.820.5
73.410.3
D DATASET
D.1 COMMONSENSE REASONING
Following the approach of Hu et al. 2003),, we combine the training datasets from all tasks to form the final training set and evaluate performance each task's individual test dataset. We adopt
18

---

Published as a conference paper at ICLR 2026
queries structure from Hu et al. (2023) to Llama 3.2 instruct template. The prompt structure is demonstrated in Table
E HYPERPARAMETERS
For each optimization method, we carefully preselected the learning rate (step size). For the Rie- mannion method, set the momentum 0.9, and for the Muon method, set the momentum to 0.95. In both cases, we additionally tuned the weight-decay hyperparameter (see Algorithm 5). The final choices of these hyperparameters summarized in Table
In the Commonsense reasoning benchmark, the hyperparameters are reported in Table for the SGD-like methods and in Table for the Adam-like methods. For the LOI method, selected the following hyperparameters: the backprop RSVD oversampling parameter was set to T& = 16, the backprop powerstep parameter was q = and the multiplier a was
=0,01 vr
The non-tuned hyperparameters used for experiments on the Commonsense Reasoning dataset are presented in 6
F
SUBJECT-DRIVEN GENERATION
Training details We used Stable Diffusion-2-ben model with batch size of 4 for all experiments. For both methods, we set the betas to 0.9 and 0.999 and the weight decay to 0.1. In all variations of approach,
g 15. rank and
a =
Vrank' We used learning rate of 2e-5 to train both our and the LoRA models, which were used to generate images shown in Figures and
Evaluation details We used the DreamBooth dataset, which contains 25 different prompts and 30 various concepts. Due computational costs, we only used half of the proposed concepts to evaluate metrics: can, candle, cat, cat2, ushie, dog8, bancy_boot, grey_s slt_s_hhhppppp hacpp_plsshieeeee similarity between the original concept and the generated images, used Image Similarity (IS): synthesized 30 images for the base prompt <a photo of a V*> and 10 images for each of 25 editing prompts (like <a V on top of dirt road>). We then measured the average pairwise cosine similarity with reference photos the concept using the DINO model. To check the correspondence between the generated images and the textual prompts, calculated Text Similarity (TS): evaluated each concept with each of 25 prompts, synthesizing 10 images per prompt, and calculating the average pairwise cosine similarity with the prompts using the CLIP ViTB/32 model.
dog2, dog3, dog5, dog dog 7, Larkk_sunllas asses, vase. To measure the
Additional results Figure 3 shows additional visual comparison of method and LoRA after 600 training steps. As can be seen, our method learns the concept much faster than the original LoRA, while preserving editing capabilities.
G ADDITIONAL EXPERIMENTAL RESULTS
G.1 ABLATION STUDY INITIALIZATION
The fine-tuning optimization was carried out by AdamW (Loshchilov Hutter (019)) First of all, for every approach tested we preselected suitable optimization step sizes. The list of all selected hyper-parameters for each method is specified in the Appendix D Table Table(ccome contains the accuracy of the trained model's responses
the test dataset. The notation <LLoR in the table means the utilization of the vanilla LoRA with non-zero A. the notation <<oRAAB>, means the same with non-zero B, the notations <Stiefel-A> and <Stiefel-Br indicate vanilla LoRA with orthonormal initialization (for matrix A and B respectively). The naming of <Stiefel-bothh involves taking the factors A, B with orthonormal columns and with initialization like in (15)
19

---

Published as a conference paper at ICLR 2026
Table 3: The structure of the queries for the Commonsense reasoning dataset
Task
Role
Fine-tuning Data Template
BoolQ
system
Please answer the following question with True Falsc.
Follow the answer format, full answer not needed.
user
Question: [QUESTION]
Answer format: True/False
assistant
The correct answer is [ANSWER]
PIQA
system
Please choose the correct solution to the question.
Follow the answer format, full answer not needed.
Question: [QUESTION]
Solution1:
[SOLUTION 1]
Solution2: [SOLUTION 2]
Answer format:
Solution I/Solution2
assistant
The correct answer [ANSWER]
SIQA
system
Please choose the correct answer the question
based on the context provided.
Follow the answer format, full answer not needed.
user
Context: [CONTEXT]
Question: [QUESTION]
A: [ANSWER A
B: [ANSWER B]
C: [ANSWER CJ
Answer format: A/B/C
assistant
The correct answer is [ANSWER]
hellaswag
system
Please choose the correct ending to complete the given sentence.
Follow the answer format, full not needed.
user
JACTIVITY JIBBE]] [CONTEXT]
Ending [ENDING_I]
Ending2: ENDING _]
Ending3: [ENDING
Ending4: [ENDING __
Answer format: Ending //ndiigg/Enninnn3/nnnnnnni
assistant
The correct answer is [ANSWER]
winogrande
system
Please choose the correct answer to fill
the blank to complete the given sentence.
Follow the answer format, full answer not needed.
Sentence: [SENTENCE]
Optionl: [OPTION 1]
Option2: [OPTION 2]
Answer format: Optionl/Option2
assistant
The correct answer [ANSWER]
ARC-e & ARC-c
system
Please choose the correct answer the question.
Follow the answer format, full answer not needed.
user
Question: [QUESTION]
Answerl: [ANSWER 1]
Answer2: [ANSWER 2]
Answer3: [ANSWER
Answer4: [ANSWER_4]
Answer format: Sawwer//Ansseer//nnwerr////n
assistant
The correct answer is [ANSWER]
OBQA
system
Please choose the correct answer to the question.
Follow the answer format, full answer not needed.
user
Question: [QUESTION]
Answerl: [ANSWER
Answer2: [ANSWER
Answer3: [ANSWER 3]
Answer4: [ANSWER
Answer format: Answer /wwee///wwwwr////ss//////wee
assistant
correct answer [ANSWER]
20

---

Published as a conference paper at ICLR 2026
Table 4: The parameters for different Adam variants for fine-tuning on the Commonsense reasoning
dataset
Optimizer
learning rate
weight decay
Adam
0.0002
0.01
DoRA
0.0003
0.01
Muon
0.0005
0.2
DoneRITE
0.0005
0.0001
RPrecA W
0.0005
0.5
Riemannion
0.0001
0,00316
Table 5: The parameters for different SGD variants (RiemannLoRA with simulate_Adoe Adam flag disabled) for fine-tuning on the Commonsense reasoning dataset
Optimizer
learning rate
LoRA
0.1
LoRA-LOI
0.06
RSLoRA
0.1
RiemannLoRA
0.1
RiemamLLoRA-LO1
0.07
Table Other non-tuned hyperparameter configurations for experiments on Commonsense reason-
ing dataset
Dataset
Commonsense reasoning
Hyperparameters
Rank r
16
Dropout
0.05
LR Scheduler
Linear
Batch size
64
Epochs
2
Warmup ratio
0.1
Table 7: The average accuracy among tasks of tne-ttned Llama 3.2-1b via Adam using different
LoRA initialization variants, tested on Commonsense reasoning benchmark. <LOI> stands for lo- cally optimal initialization. LoRA rank set to 16.
Task
BoolQ PIQA SIQA hella- swag
wino- grande
ARC-
ARC-
OBQA
All
Initialization
E
C
Raw
40.1
55.4
50.3
25.8
50.0
61.9
41.8
42.8
46.0
LoRA-A
66.2
80.2
76.0
87.0
65.1
78.4
63.6
76.3
74.1
LoRA-B
65.9
77.9
74.2
82.9
61.2
73.9
60.6
72.2
71.1
Pissa
66.4
80.1
75.6
87.6
63.1
77.6
64.3
75.0
73.7
Stiefel-A
65.4
79.5
74.9
87.2
62.4
79.3
62.3
75.5
73.3
Stiefel-B
66.4
80.6
76.0
87.5
64.3
78.5
64.9
74.6
74.1
Stihfel-both
66.3
79.7
75.8
86.4
64.0
78.3
62.4
73.9
73.4
LOI
65.6
81.3
75.7
87.7
65.7
77.9
65.0
75.2
74.3
LoRA-GA
65.4
77.1
74.9
84.5
58.3
75.4
61.9
72.2
71.2
21

---

Published as a conference paper at ICLR 2026
rank
rank 16
Concept
LoRA
V" with a city in the background
with mountein in background"
"a V with the Eiffel Tower in the background"
ta V in chef outffr
a V the jungie"
V" the beach"
Figure 3: Additional visual comparison of our method and LoRA, checkpoint 600
Table The average accuracy among 8 tasks of fine-tuned Llama 3.2-lb using different LoRA ini- tialization variants, tested on Commonsense reasoning benchmark. <LOI> stands for locally optimal initialization.
Optimizer
Initialization
rank
Accuracy
Raw
46.0
Riemannion
random
32
74.7
Riemannion
LOI
32
74.9+0.3
LoRA
22ro
32
73.441.1
Riemannion
random
64
74.6.n.3
Riemannion
LOI
75.1
LoRA
64
73.1
22

---

Figure shows the relative increase in per-step execution time of the proposed Riemannion method compared with Adam, computed as ((T Riemamnion Adaam) / AdAdam). Table 10 shows the average time cost for each epoch. Table [1] presents comparison both the theoretical time and memory Published as a conference paper at ICLR 2026
Table The average accuracy among tasks of fine-tuned Llama 3-8b using different LoRA initial- ization variants, tested on Commonsense reasoning benchmark. <LOb>, stands for locally optimal initialization.
Optimizer
Initialization
rank
Accuracy
Raw
74.5
Riemannion
randomn
32
86.9
Riemannion
LOI
LoRA(Adam)
zero
32
87.2
Riemanmion Riemannion
random
86.2
1.0l
87.9
LoRA(Adamm)
zero
H COMPUTATIONAL COST
To measure execution time, we used a virtual server instantiated on compute node equipped with an Intel Xeon Gold 6240R CPU and Nvidia Tesla V100 GPU. The virtual machine was provisioned with 8 CPU cores, 16 GB of RAM and single Tesla V100 GPU. Experiments were executed using the Hugging Face transformers library; timing corresponds to the execution time the trainer: train () call. Measurements were conducted according to the following procedure. For each combination of method, LoRA adapter rank, and batch size, four timing runs were performed and the minimum execution time was selected. The measured quantity was the time spent 16 optimizer steps. Time spent on initialization and batch size, execution times for each of the methods were measured sequentially. In the present work, rank refers the rank of the LoRA adapter, and method denotes one of optimization schemes: Adam (baseline) and Riemannion (proposed method).
excluded. For each combination of rank
complexity of several methods with our algorithm (more methods provided in Yen et al. (2024))
Relative ovethead on Liama 8b
35
30
27.5
25
22.5
Overaad
17.5
13
12.5
7.5
2.5
Batch size
Figure Relative Time Cost (calculated as (TRiemannion T TAdam) TAdam) of Riemannion vs. Adam during Ilama 3-8B fine-tuning, as function of LoRA rank and batch size.
I LMO BOUNDS
Let A E Mw be a vector from the tangent space. Consider the problem
(A, X) + max, X
(21)
s.t. X < 1,
whose (any) solution will be denoted by X, Ortho A). Denote f. (A, X.) the optimal value.
23

---

Published as a conference paper at ICLR 2026
Table 10: Time Cost of Riemannion vs. Adam during Llama 3-8B fine-tuning, a function of LoRA rank and batch size.
butch AdamW
27.32
35.48
8
50.41
61.60
8
32
97.46
107.96
190.14
8
128
373.89
389.50
8
31.59
46.41
16
57.82
73.33
132.26
16
64
217.11
237.50
128
431.31
450.79
32
8
35.08
64,81
32
32
114.21
148.63
219.34
257.81
128
433.32
476.11
Table 11: Time and space complexity comparison for LoRA optimization.
Method
Time
Memory
(Adam)
O((m + n)
O + n)r)
LORA-RITE
0( 2
0(mm r
Riemannion (our proposed)
O((m r)
O((m + (9)
Consider the restricted (LMO) problem
(A, X) Xx,
s.t. |||||2 S 1, X Tw.
(22)
Let X* denote an optimizer of the task derived in (2) and f = (A, its optimal value. Proposition I.1. Assume A E Tw Mr and let P the orthogonal projector onto Mp. If
|X, PX,
for some E 2 0, then
1+e S.
Proof.
Since E Mr, we have PA = and P PT_ By assumption [X, - PX,l1 S and, because ||X.|12 1, the reverse triangle inequality yields
|||x |PX ||a| Se
hence
1 1-85 PPX|12 < 1+
Define the normalized matrix
PX.
|PX,122 Mw, |||||=
By definition of f (the maximum of (A, over (X E Tw.M: |||1| 1}) have
(A, PX.) (PA.X.)
(A.X.) f.
8² (A,Y)
PX- PX.
|PX.111
|PX.112
Using the inequality PP 5 + we obtain the claimed lower bound
8 1+6
24

---

Published as a conference paper at ICLR 2026
The upper bound f* v is trivial because the feasible set of the task derived in (22 is subset of the feasible set of the problem described (2I). Combining the two inequalities gives
T+e S.SSS
which completes the proof.
Proposition I.2. (Upper bound on the projection residual.) With the notation above, the residual
::= || PX+112
satisfies S
Proof. Write the thin SVD of the base point W as W = U,E.V. where U, St(m, r) and V, E St(n, r) have r orthonormal columns. Let U. and VI be orthonormal complements so that [U, U.] E m2x22 and [V, Vi E Rnx2r are orthogonal matrices (here assume mi, 2 2r; if m or r is smaller the argument adapted in the obvious way).
Because A Mw. one write (using suitable 2r 2r block representation) (Vandereycken 2013
Á C
A= U,, U_
where A, B, é TR"x" (the (2, block vanishes for tangent-space elements of this form). The rank of A is therefore at most 2r.
Let X, Ortho(A) be maximizer of (1 represent X, in the same block basis,
A
B
V
X. = UU,
U1]
C
D
V
2
where the 2r x 2r block matrix is orthogonal,
A
B
E
((r)),
(23)
C
D
where ((2r) denotes the orthogonal group in 2rx22
By Lemma I.1
below have
A
B
V
PX. = UU
U1]
c
VT
Therefore
0
0
X. PX, UU,
U1]
DVI,
0
D
VI
and hence
|X. PX. = DD
Because the block matrix in 23 is orthogonal, its lower block
C D
E
2rxr
has orthonormal columns. Thus | l1 = 1, and in particular ||D||2 < 1. Consequently ||_ PX l| I 1, as claimed.
Lemma L1. Let
W = U,E,VV e and write an arbitrary matrix X E Rmxn in the block basis determined by [U. U and [V, Vi J.
e st(m,r), V, E St(n,r),
A
B
V
x U, U
C
D
V
Then the orthogonal projector P onto Tw. Mr acts
A
B
PX = UU, U1]
C
0
V
25

---

Published as a conference paper at ICLR 2026
Proof. One convenient expression for the orthogonal projector onto the tangent space at W U,E.V V
P() = 1)
Applying this to X written in the chosen block basis yields
PX = X (I - U,U,
V,V,T)
A
B
V
B
0
= UU U1
[0
U1]
D
VT
c
VT
AA B
0
2
0
= U U1]
C D
-
0
A K V
UU U1]
C
0 [1
which proves the lemma.
Remark I.1. In experiments observed that the residual is typically small (in reported runs it did not exceed 0.1).
J ADDITIONAL EXPERIMENTS ON DIFFERENT TASKS
Let M be a matrix with rank(M) = k. We aim to approximate with a matrix M E M, such that rank(M") k. The task be formulated
M Ml| min. M
(24)
Figure 5 shows the dependence of the loss the epoch number for three different optimizers: Rie- mannion, PrecAdam W, and LORA-RITE. This experiment shows that our optimizer demonstrates remarkable stability at singular points and achieves significantly lower loss than competing methods.
10º
Riemannion
hmm
RPrec-Adam
10
LoRA-RITE
10
Loss
10
10
10
10"
500
1000
1500
2000
2500
Iteration
Figure 5: Comparison of the performance of three optimizers task 24
K LLM USAGE
We used LLM only for minor language polishing to improve readability and grammar; it was not involved research ideation, methodology, analysis, or substantive writing, and all research ideas and arguments were developed entirely by the authors.
26


Reviews:
[{"review_id": "hdoJDubxke", "reviewer": "Reviewer_wX7P", "summary": "This paper identifies a key deficiency in standard Low-Rank Adaptation (LoRA) training: the lack of transformation invariance. The authors note that the update to a LoRA matrix $\\Delta W$ is dependent on its specific factorization ($A$, $B$), which can lead to unstable training and sub-optimal results.\n\nTo solve this, the authors propose a novel, fully Riemannian framework. Instead of optimizing the ambiguous factors ($A$, $B$) in Euclidean space, this work proposes to optimize the low-rank matrix $\\Delta W$ directly on the fixed-rank manifold $\\mathcal{M}_r = \\{X : rank(X) = r\\}$.", "questions": "Could you elaborate on the conceptual advantages of your geometric framework over the adaptive matrix preconditioning approach of LoRA-RITE? Why is optimizing on the manifold better than forcing invariance in Euclidean space?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Addresses a Fundamental Problem: The paper tackles a well-defined and important problem in parameter-efficient fine-tuning. The lack of transformation invariance is a genuine flaw in the standard LoRA optimization paradigm, and addressing it is a valuable contribution.\n2. Conceptually Elegant Solution: By re-framing the optimization problem on the fixed-rank manifold, the framework eliminates the root cause of the invariance problem by construction, as the ambiguous factors ($A$, $B$) are no longer part of the optimization. \n3. Novel Optimizer: The generalization of the Muon optimizer to the fixed-rank manifold (\"Riemannion\") is a novel algorithmic contribution.\n4. Thorough Handling of Efficiency: A primary concern for any manifold-based method is computational cost. The authors anticipate this and provide a convincing case for their method's efficiency. The design explicitly avoids forming full-size matrices, with a theoretical complexity of $\\mathcal{O}((m+n)r^{2}+r^{3})$.", "weaknesses": "1. Insufficient Comparison to SOTA (LoRA-RITE): The paper's primary weakness is its engagement with its most direct baseline, LoRA-RITE (Yen et al., 2024).\n\n2. Conceptual Comparison: While LoRA-RITE is cited for solving the invariance problem whilst conducting adaptive regularization, the paper misses a clear opportunity to discuss why its geometric approach should be theoretically superior to LoRA-RITE's preconditioning approach. The review a-priori is that a geometrically-native solution should be more stable or effective, but this is not argued explicitly. A wall-time comparison with LoRA-RITE would be great to have. \n\n3. Mismatch in Optimization Space: The paper defines its optimization space as the manifold $M_r = \\{X : rank(X) = r\\}$. However, the true set of all possible LoRA adapters is $M_{\\le r} = \\{X : rank(X) \\le r\\}$. The paper does not discuss the implications of this distinction. The set $M_{\\le r}$ has \"singularities\" at all points where $rank < r$, and it is unclear how the optimizer behaves if the true optimal solution lies on one of these singularities. The $SVD_r$ retraction (Alg. 4, line 5) explicitly forces the update to stay on the $\\mathcal{M}_r$ manifold, which could potentially prevent the model from finding a simpler, lower-rank solution.\n\n4. The authors can mention an article on a related topic, specifically the \"manifold Muon\" optimizer described in the Modular Manifolds [1] article from Thinking Machines Lab. This article discusses a similar recipe (a Muon-like optimizer on a manifold) but applies it to the Stiefel manifold. A discussion of this related work is essential for correctly positioning the paper's novelty.\n\nminor: typo in line 215\n\n[1] Jeremy Bernstein, \"Modular Manifolds\", Thinking Machines Lab: Connectionism, Sep 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EZr6zm1Nh3", "reviewer": "Reviewer_uGz9", "summary": "This paper proposes a Riemannian optimization framework for Low-Rank Adaptation (LoRA) that directly optimizes low-rank adapters on the fixed-rank manifold instead of in standard Euclidean space. The approach introduces the Riemannion optimizer(a variant of Muon optimizer), along with a Riemannian gradient-informed initialization. Experiments on both large language models and diffusion architectures demonstrate improvements in convergence speed and performance compared to relevant methods.", "questions": "See weakness", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well-written and provides solid mathematical formulation such as gradient project,retraction and vector transports via automatic differentiation.\n\nThis paper derives Riemannion, the first optimizer that generalizes Muon to manifold of fixed rank matrices, addressing a fundamenta issue in LoRA traiing that different factorizations A,B lead to different optimization trajectories.\n\nEmpirical experiments demonstrate the proposed method outperform baselines across LLM an diffuision benchmarks.", "weaknesses": "The main novelty(generalizing Muon) is somehow incremental.\n\nLLM results focus on Llama-3 8B using rank 16 and commonsense benchmarks only. No ablations for ranks, downstream tasks (e.g. summarization,instruction tuning), ViT models.\n\nThe computational overhead need to be clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YKOspnCQ2m", "reviewer": "Reviewer_52qu", "summary": "This paper extends the Muon optimizer to fixed-rank matrix manifolds, proposes a Riemannion algorithm, and introduces an initialization strategy based on Riemannian gradient information. Finally, it demonstrates the performance of Riemannion on LLM fine-tuning and diffusion models.", "questions": "as stated in weakness", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "This is the first work to apply the Muon optimizer to low-rank matrix manifolds, combining Muon with Riemannian manifolds.", "weaknesses": "1. Insufficient motivation in lines 63–69. \n\n$\\cdot$ The authors present the main motivation as extending Muon, but do not sufficiently explain the necessity of this extension. To strengthen the argument, the core advantages of Muon should be outlined to clarify the specific benefits of applying it to fixed-rank manifolds. Furthermore, a comparison with existing manifold optimizers is essential to emphasize the strengths of the proposed method.\n\n$\\cdot$ The statement, `our design inherits Muon’s geometry-aligned normalization, yielding transformation invariance of the learned update', lacks theoretical or empirical support in the paper. It should be supported by providing proofs or references, or by experiments validating.\n\n$\\cdot$ To demonstrate the advantages of Riemannian gradient–informed initialization, they should compare different initialization strategies for the same algorithm through experiments to showcase the benefits of their proposed initialization.\n\n2. In lines 160-161, the authors claim that `Note that acting on the two factors separately makes Muon non-reparameterization-invariant: its per-factor orthogonalization depends on arbitrary scalings or rotations, skewing the weight-space step and often letting one factor dominate.' Please provide the proof or detailed reasoning to support this claim.\n\n3. In lines 168–171, the symbol G is already used earlier in the paper to represent the gradient. Reusing G in this section creates confusion. \n\n4. Please provide detailed derivations or references to support (5).\n\n5. In line 188, the objective function is denoted as F, but is changed to L in Line 215. This inconsistency in notation should be resolved for clarity.\n\n6. The complexity and scalability of Algorithm 2 need clarification. If a full SVD is performed on an m×n matrix at each step to determine \n$A_L$ and $B_R$, the computational cost is typically 𝑂(𝑚𝑛 min{𝑚, 𝑛}). The algorithm requires computing a full SVD at each step, which becomes extremely expensive when \\(m\\) or \\(n\\) is very large, making the algorithm inefficient in these cases.\n\n7. In line 80,  `we show the connection of this initialization to LoRA-GA', but this paper does not explain the differences and connection between the proposed initialization and LoRA-GA.\n\n8. The experiments only report results for r=4,8,16. It should include results for larger ranks (r=32,64, or even higher) to evaluate the stability, performance, and computational overhead of the proposed algorithm. \n\n9. The experimental results in Figure 4 regarding runtime provide limited insights. It would be more informative to include a comparison of the per-step computation time and memory usage of the proposed algorithm versus state-of-the-art algorithms.\n\n10. The paper does not provide any theoretical guarantees for the convergence of the proposed algorithm. Including such guarantees, or at least a discussion on convergence properties, would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IL95nB3Q5Y", "reviewer": "Reviewer_h8Wt", "summary": "This paper proposed a novel Muon-style LoRA vairant through the lens of Riemannian optimization. Specifically, the authors proposed to apply the retraction-based Riemannian gradient update step (Eq. 8) to optimize LoRA factors, and augment the momentum term involved in the update step with the Muon-style orthorgonalization (Eq. 12). Empirically, the proposed method achieves decent performance on LoRA-style language model finetuning tasks (Table 1).", "questions": "See **Weaknesses**.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The authors propose a computationally efficient Muon-style extension of Riemannian optimization for LoRA parameterized LLMs (Algorithm 1).\n- The authors derive locally optimal initialization scheme to maximize the the LoRA factors can be optimized towards the fastest loss decrease direction on the low-rank manifold (Theorem 5.1).\n- The authors develops single backward-pass gradient trick, to compute gradient-times-matrix products efficiently (Algorithm 3).", "weaknesses": "This paper is well-written. The proposed method is theoretically grounded, and empirically validated on several downstream tasks. I will raise my score upon my concerns being addressed.\n\n\n--------------\n\n**Concern 1. Feasible Region of (Eq 14)**\n\nTo my understanding, the authors aim to initialize the LoRA factors such that it can be optimized towards the fastest loss decrease direction on the low-rank manifold. In this case, does that means we should constraint the norm (or magnitude) of the LoRA factors in Eq. 14? Otherwise, scaling the LoRA initializaiton will lead to larger Riemannian gradients. \n\n--------------\n\n**Concern 2. Ablation on LOI**\n\nFrom my experience, the intialization of LoRA factors is usually chosen to ensure the adapted weight is identical to the loaded pretrained model, ensuring the model does not deviate from the well-trained local optima significantly. However, it seems that in LOI, a non-zero modification is made to the adapted weight. Does this affect the performance of the model? I recommend the authors to add discussion on this issue, and provide ablation studies to clarify the impace of LOI on the general performance of the proposed method.\n\n--------------\n\n**Concern 3. Effectiveness of the proposed methods on classic low-rank matrix optimization.**\n\nTo my understanding, the derivation of the proposed methods does not relies on specific assumptions on the optimization task. I recommend the authors to provide additional experiments on classic Riemannian optimization tasks to validate the effectiveness of the proposed methods in solving classic low-rank matrix optimization problems [1] that are more general and natural than the LLM finetuning tasks.\n\n[1]. Bioli, I., Kressner, D., & Robol, L. (2025). Preconditioned low-rank Riemannian optimization for symmetric positive definite linear matrix equations. SIAM Journal on Scientific Computing, 47(2), A1091–A1116. https://doi.org/10.1137/24M1688540", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "92RDg8HKNt", "reviewer": "Reviewer_vCt2", "summary": "The paper proposes a fully Riemannian LoRA framework that optimizes the low-rank adapter directly on the fixed-rank manifold to remove factorization ambiguity. It proposes a Muon-style Riemannian optimizer (“Riemannion”) plus a locally optimal initialization. Experiments on commonsense reasoning and subject-driven diffusion report gains over LoRA variants.", "questions": "1. I think the idea of applying direct optimization on low rank manifold for LLM training is already considered in LORO [1]. Although LORO is for pre-training, it can be applied to fine-tuning very easily (with a fixed pre-trained model). I'm wondering how the proposed method perform conparing to this variant of LORO? Since LORO is the first to consider this proposed approach of low rank manifold optimization, I feel the discussion is not sufficient in the current manuscript to compare to it.\n2. Clarity around G in Eq. (4): Readers may be confused because $G$ appears in the manifold parameterization but seems to disappear in later sections. It remains unclear which quantities are actually treated as trainable parameters in Algorithm 4 and which are only auxiliary. Please state explicitly what is stored, what is recomputed during retraction, and what constitutes the model’s trainable state.\n3. No memory footprint comparison: The paper measures time overheads but omits GPU memory / parameter-state comparisons vs. LoRA, DoRA, etc. A table counting trainable params and optimizer states per layer (e.g., Adam’s m/v vs. Riemannion’s HB state) would strengthen claims of “no prominent overhead.” This is an important piece of information that is missing, since LoRA is designed to be a parameter or memory efficient optimizer.\n4. For Table 1, is it exactly one set of hyperparameters used for Riemannion to achieve all these superior performance **across all tasks**? If so, I think this would be quite astonishing. If not, would it be a bit unfair if the authors didn't search hyperparameters for other methods?\n5. A minor point: The geometry section is readable, but several practical choices (which Ortho operator was used in which layer types; retraction accuracy vs. rank; momentum transport details) are scattered. I suggest consolidating everything into an implementable algorithm, not just one step as in Algorithm 4.\n\nReferences:\n\n[1] Mo, Zhanfeng, Long-Kai Huang, and Sinno Jialin Pan. \"Parameter and memory efficient pretraining via low-rank riemannian optimization.\" The Thirteenth International Conference on Learning Representations. 2025.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Clear motivation for moving from factor-space updates to invariant steps on fixed rank manifold\n2. Concrete algorithms: a step-by-step Riemannian procedure with Ortho/Project components and explicit per-iteration complexity\n3. Experimental results suggest improved stability/accuracy vs. strong LoRA-style baselines (though see concerns below)", "weaknesses": "(Please reply to the Questions section directly, where I write the details of the weaknesses)\n1. Insufficient positioning and no empirical comparison to LORO\n2. Unclear and potentially confusing use of G in Eq. (4).\n2. Missing memory-consumption analysis (vs. LoRA/PEFT)\n4. Hyperparameter protocol is opaque near the main tables", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)