LoRA-S: An Efficient Low Rank Adaptation scheme via Sylvester equation
Authors: Jinyang ZHENG
Tong Wu
Venue: ICLR 2026 Poster (2026)
Keywords: optimization
LoRA
Primary area: optimization
TL;DR: Inspire the optimizer design of LoRA by proposing framework that guarantees efficient feature learning

Abstract: Numerous studies on low-rank adaptation (LoRA) emerged in recent years, with the aim of accelerating the convergence of the LoRA framework. In this paper, we leverage the horizontal lift theory from differential geometry to establish the general iteration scheme on the quotient manifold  \mathbb{R}\_\*^{m \times r} \times \mathbb{R}\_\*^{n \times r}/\sim. 
By endowing the LoRA framework with Riemannian quotient geometries, our theory not only guarantees efficient feature learning but also bridges the LoRA algorithms and the pre-training algorithms for large models. 
Furthermore, we theoretically analyze the role of the weight decay matrix $\epsilon_{decay}I$ in efficient feature learning and then replace it with the Sylvester matrix $K$, indicating that the theory helps remove an important hyperparameter while generating accurate and computationally efficient optimizers. 
Based on the general scheme, we propose two efficient LoRA optimizers with runtime analysis, Adam-Sylvester (AdamS) and LRACS, then conduct experiments on the transformer-based networks. The results demonstrate evident improvements over existing optimizers.

Published as a conference paper at ICLR 2026
LORA-S: AN EFFICIENT Low RANK ADAPTATION SCHEME VIA SYLVESTER EQUATION
Jinyang Zheng', Tong Wu!
1 "Department of Mathematics, The Hong Kong University of Science and Technology, Hong Kong, China jzzengbp, twubi porubiieconnnct...o ust.hk hk
ABSTRACT
Numerous studies on low-rank adaptation (LoRA) emerged recent years, with the aim of accelerating the convergence of the LoRA framework. In this paper, we leverage the horizontal lift theory from differential geometry to establish the general iteration scheme on the quotient manifold R /R ~ By endow- ing the LoRA framework with Riemannian quotient geometries, our theory not only guarantees efficient feature learning but also bridges the LoRA algorithms and the pre-training algorithms for large models. Furthermore, we theoretically analyze the role of the weight decay matrix 6decay T in efficient feature learning and then replace with the Sylvester matrix K, indicating that the theory helps remove an important hyperparameter while generating accurate and computation- ally efficient optimizers. Based on the general scheme, we propose two efficient LoRA optimizers with runtime analysis, Adam-Sylvester (AdamS) and LRACS, then conduct experiments on the transformer-base networks. The results demon- strate evident improvements over existing optimizers. The codes are available https///jtthbb.omm//eeeee///////
INTRODUCTION
Low-Rank Adaptation (LoRA) (Hu al., 2022) is widely used parameter efficient fine-tune (PEFT) method for training large language models. By freezing the pre-trained weights W E R and adding trainable low-rank matrix X MNN R"xm to each layer where W appears, LoRA can achieve: (1) reduced memory requirements for fine-tuning, and (2) avoidance of overfitting with relatively little training data. Let (WW denote for the original loss function and L(M, N) denote the factorial version of the loss function. In each layer, training LoRA is equivalent to minimize the I
loss function in the space maxr
min (M. x C(M,N) = fe(W + X), = MNT
variations and improvements in the LoRA algorithm (Zhang et al., 2023 Valipour et al. 2022 Hayou et al. 2024 Yen et al. 2025). Among them, efficient Teature learning (EFL) (Hayou et 2024) is particularly influential. Since is an case-to-verify mathematical property that ensures that both factors M, N are updated efficiently. Unfortunately, conventional methods cannot achieve efficient feature learning. To achieve this property, LoRA+ (Hayou et al. 2024) employs different learning rates for the two factors. which needs more fine- tuning, and LoRA-Rite (Yen al., 2025) designs a novel and sophisticate optimizer achieve the property. These methods either require multiple rounds of fine-tuning, or their implementations are overly complex, i.e. redesign the preconditioner, and hard to generalize.
Recent research explores
To address this challenge, we introduce an equivalence relation see (Definition T and propose our general scheme (Algorithm I by applying horizontal lift theory (Absil et al. 2014) on the quo- tient manifold pmxx x /R Through theoretical analysis (Theorem] optimizers conforming to our general scheme are guaranteed to achieve transformation invariance (Yen et al. 2025). thus ensuring efficient feature learning (EFL). The general scheme is highly flexible and scalable. It can endow almost all conventional optimizers with transformation invariance and capable of maintain- ing EFL even after any preconditioning is applied, which is not supported by former Riemannian

---

Published as a conference paper at ICLR 2026
optimization methods such as RGD (Bonnabel 2013) and ScaledAdam Zhang & Pilanci 2024). By introducing the state-of-the-an preconditioning to mitigate the condition number of the Hessian matrix in LLMs, new EFL approach has certain advantages
existing efficient optimizers, 2025) in all metrics and significantly exceeds the simple EFL methods, 2021) and Quotient GD (Mishra et al. 2014), used as the baseline.
i.e. LoRA-Rite (Yen et al.
L.e. Scaled GID (Tngg et al.
Fine-tuning hyperparameters is also crucial part of LoRA training. Most conventional optimizers require simultaneous fine-tuning of both the learning rate and the weight decay parameter, thus searching for the optimal weight-decay parameter X can be very time-consuning. However, we find that the weight-decay hyperparameter unnecessary in our framework. Its benefit EFL is far less significant than that of our proposed Sylvester equation. Both theoretical and empirical results show that the effect of efficient feature learning exceeds the effect reggrggllaiization Considering that the Sylvester decay matrix K2 can be computed directly without tuning hyperparameter, our deight-decay-free methods can practically save a substantial amount of time.
The contribution of this paper can be summarized below:
We propose LoRA-S, an efficient low rank adaption scheme via Sylvester equation which convert optimizers with any preconditionings to transformationinnen invartiinnnnnaiaant optimizers. By proposing a formal math definition of efficient feature learning which is more strict than transformation invariance, first provide perspective understand EFL methods with manifold optimization.
We also study the role of weight-decay hyperparameter in efficient feature learning. In our scheme, we find removing the weight decay parameter À and replacing it with the Sylvester decay matrix K2 can achieve more efficient learning, both theoretically and empirically.
We lift the two conventional optimizers, Adam (Kingma & Baj 2014) and RACS (Gong et al., 2025). proposing AdamS and LRACS. Empirically, our proposed AdamS method achieves an average CL.IP score of 32.64 on the Mix-of-Show model an improvement of 9% over the widely used Adam optimizer (29.86) and 3% over LoRA-Rin (31.90), the current ofate-of-thh-rr EFL method.
PRELIMINARIES
2.1
DIFFERENTIAL GEOMETRY
In this section, briefly introduce concepts from differential geometry and explain how Low- Rank Adaptation (LoRA) (Hu et al. 2022) framework fits in manifold optimization. Recall that X MNT belongs the manifold M(r,m m x n) which denotes the set of all rank-r matrices of size m x n,
M(r, n) {X E Rm*x,
rxn(x (rank(X)=r)
Let RTT Rmxr (X E RXxr rank(X) r} denotes the set of all full-rank m x r matrices. The product space Rmxr x Raxr forms manifold. Compared to the non-linear M(r, m x n), Brxx x Raxr is the linear space Raxxr x Raxr with nowhere dense set excerpted. Therefore we usually optimize LoRA in the ambient space Rmxr x gaxr in practice. Meanwhile, the tangent space of manifold RT xx x x RRxrr at N) and the tangent space manifold M(r, m x at M (Séguin
ct al. 2024) are:
T(M.NR BuXr x B = Raxr x Raxr
(1)
T T(MNTY m x n) = (T Ruxn, = MSNT + M MNT)
(2) For
where R Mp Rm and N₂ E Raxr satisfy M MT and N N convenience, we denote Ry x Rixx as T N(M,N), and T ((MN)) m x n)) as T ((MTT)- Notably, a map π can be defined to transport tangent vectors between these two tangent spaces via horizontal lift theory.
2.2 HORIZONTAL. LIFT THEORY
The horizontal lift theory of fixed-rank matrices was firstly studied in Absii et al., 2014). It is said that, there exists Riemannian submersion from the product space of two full-rank matrices
2

---

Published as a conference paper at ICLR 2026
R Rix to fixed-rank matrices manifold MMr, n):
IRXX x *M(r, n) (M,N) MN
The degree of freedom (d.f.) of M(r, m x n) is mr + nr - r while the degree of freedom of Rexx xr Raxx is nir + nr, The Riemannian submersion a is a projection. Furthermore, given point MN M(r x n) we can also write down its preimage:
π (MN+) ( (MRRNR E GL(r)).
where
GL(r) {R R" det(R) + +0}
denotes the general linear group of degree r. From (Absil et al. 2008 Proposition 3.3.3), we know that fibers π (M (T) are ri-dimensional submanifolds of R, x Raxr. The redundancy leads
most conventional methods to become transformation-vatte feature learning. To eliminate redundant dimensions, (Absil et al. relation ~ on Rexe x Ryxr:
Yen et al. 2025) and miss efficient
2014) defines the equivalence
Definition (Equivalence relation).
(Ma: Na) N6) if and only if TMM,, Na) T TMs, N6)
In this way, Ruxt R is quotient manifold diffeomorphic to MM m n). Let (M, N) Rmxr Raxi be the update pair, and also tangent vectors, (M, NN.. Then the derivative will give that:
DT (T(M, ((M.N) T(MNT) (M, NN M + MN
The horizontal lift theory (Absil et al.) 2014) builds the relation of the tangent vectors between T NMM.N) and T(MNTY. It is said that, given tangent vector X E TAMNT, there is one and only one lift vector:
((M.N) E H(M,N) such that (T(,,N [XILM,NN]] -MNT.
where Dm [X] denotes the differential of X applied to X, and ğ is specific metric Rmxr x Raxr, Equipped with the metric g. the tangent space T(M,N) can be decomposed as a direct sum the kernel of (x(M,N) and orthogonal complement, called the horizontal space H(M.N) S. T(M,N) Rmxr Rexr;
H(M = (M, RuXr Ruxr, 1(M.N) (IM, (MR, -NR" 0, YR B
Note that the 22 linear constraints the definition successfully remove the redundant dimensions, ie., dim (enkkeD))))
22 and as long the redundancy is removed, the transformation dependence will be removed. Since d.f. of (MNNT) is mr + nr 12 and d.f. of ((M,N) is mr + nr, the vector X*M,,N N)) is termed the horizontal lift of X XMNT at (M, And call T T(MNT the matrix tangent space and T(M,N) the lifting space.
EFFICIENT LORA ITERATION SCHEME
In this section, we first give formal mathematical definition of efficient feature learning (EFL) then show how the theory in Section 2 can help build general iteration scheme that guarantees efficient feature learning.
3.1 EFFICIENT FEATURE LEARNING
The concept of efficient feature learning (Hayou et al. 2024) describes the asymptotic train- ing behavior of LoRA the network width grows. Efficient feature learning requires that both and M,XTNN X NN, are of magnitude ((z²)) @(1) where is the width of network and w is the input embedding. The explosion of the gradient occurs when the magnitude reaches ((n") (a > 1) and the gradient vanishes when the magnitude is below O(z²) (a < 1). In other words, keeping both 6 | and 0? [T] same magnitude means that both M and N parameter updates significantly contribute to the change in L(M,N N). Based on these properties, propose formal mathematical definition of efficient feature learning with LoRA:
X X,M(M,,N.NNT

---

Published as a conference paper at ICLR 2026
Definition 2 (Efficient feature learning with LoRA). Suppose (M1 N2 (M2. N2) two pairs of LoRA factors that represent the same weight, that is, oW = M = MzN2 Then w.L.ogg where R E GL(r). We say update scheme satisfy assume M₂ R, N22 Ni R TT, R..R..
efficient feature learning the updates (M .Ñ1) (M1, and the updates (M2, N2) at (M2, N2) shares the same magnitude w.r.t. a specific metric g:
(N,) (A)),, (1)
= ((M))NN)) (112, (M2,, (2))
(4)
and the updates should satisfy:
M2 MRR, N2 -NN
(5)
The definition ( ensures that the magnitudes of updates, (Mi, Ni) and (M2,, N2). remain the same. From the perspective of manifold optimization, equation (4) is the necessary and sufficient condition for the existence of the induced metric 9MNT on tangent space T(NTT) In addition, note that equation (5) also indicates the transformation invariance property which means that our definition
has close relation with former studies:
Proposition 1.
The transformation invariance scheme in LoRA-Rite Yen et al. 2025 indicates
efficient feature learning (Definition with metric:
9(M.N) (M, N), (M, trace (ance(NTNT) (N MAMMM+MMMMTT
The proof of this proposition given in Appendix (A.4).
3.2 DEFINITION OF THE ITERATION SCHEMM
In this section, we choose the metric used in Grassmannian quotient manifold (Absil et al., 2007 Example 3.6.4) that meets efficient feature learning condition ( and apply horizontal lift theory to calculate the lifted vectors (X M(M..N)) XN)M.N)) as updates. Our metric induces update rules with richer structure and interpretability than the metric (13), which will be shown in AA.18.9. The metric 9 on RT
Rm xr x R.xT is:
DIM ((M, N), (M,N) = trace ((M*M) MTM + (NTN) NTN)
(7)
9MNT XMMNTXXNNY
(8)
where (M, N) and (M, N) are tangent vectors. Then let X MNT and let XMNT belong to T(MNTY. The horizontal lift X (XIM((M,))--XXCMM derived from 3 is:
X {1M(M.N) = (XMNTN - MNTNK) (NTN)-
XIN(M,M) -SMANTM-NN"MMM (MFM)
where K is the Sylvester matrix such that
T N M MTTMNTNE NK + KM
The equation is called the Sylvester equation, and in case there exists unique solution matrix K. We conclude the general iteration scheme in Algorithm (T). From the analysis in the previous section, we know that:
Theorem 1. Any optimizer following the general scheme (Algorithm achieves efficient feature learning (Definition 2 and will become transformation invariant.
The proof given in Appendix A.5 While the conventional optimizers being not transformation invariant, researchers transform any conventional optimizers transformation invariant opti- mizers now by applying the general scheme. See Appendix A. for a unified and comprehensive analysis of conventional optimizers under our general scheme.
Conventional
Transformation Invariant Optimizers
General Scheme
Optimizers
4

---

Published as a conference paper at ICLR 2026
Algorithm 1 General Scheme
1: for t = 1...T . do
2:
Assign the descent vector X M. Solve K, from the Sylvester equation
4: Compute the lift vector X NMM(M,.N))
M M.TNK.) (NTN)) (X -M.M. N.KF (M.
5: Compute the lift vector X (N(MG.NN)
6: Update model parameters for (M++1. N(41) + (Mr XMMMMMMN) Nr + XAN(MMNN)
7:
end for
3.3 REMOVING THE WEIGIIT DECAY PARAMETER
In machine learning. weight decay (Loshchllov & Hutter 2017) is a critical regularization tech- nique primarily used to prevent overtitting the model. typically implemented by adding penalty term proportional to the square of the model weights (L2 norm) to the loss function. The L regularizer edecny is often set to r x X, where r the learning rate and is the weight decay value.
Cdecay 11
(10)
To maintain training stability and consistent regularization effects, it is common adjust the weight decay coefficient À proportionally when modifying the learning rate 7,, which will cost lot time to find the overall best weight decay value X in practice. Practically, X* is non-zero in most EFL LoRA optimizers, ie. 0.01 in AdamW and ScaledAdam (Zhang Pilanci (002).. while being set to 0 in EFL LoRA optimizers, i.e. LoRA+ (Hayou et al., (024) and LoRA-Ritt (Yen al., 2025). In this section, we will theoretically explain this phenomenon and why this parameter can be removed in our scheme.
X XIM(M..M) X MONT No (NT
M,NNKK,, (NT
gradient term (GT)
decay (DT)
First, the lifting vectors X XpM(M,NN) and XoN(M,NN,) consist of two terms. The first term is called the "gradient term" since it directly contains the lifted gradient preconditioned gradient vector. The second term is called the "decay term" since it contains -M. The weight decay parameter in the LoRA framework can be viewed as approximation the "decay term". Scaling the lifted vector will scale K, and "decay term" at the same time, i.c.,
M (NNM,NT)) M,N, N.(KK,) + (nK,)M, (.K.)M,MMNTT.
In this sense, the magnitude of the Sylvester decay matrix K2,, is determined by two decoupled parameters, the hyperparameter learning rate 77 and the solution matrix Kt, just like the situation of AdamW and SGDW (Loshchilov & Hutter (017)) in (IO). Conventional methods typically do not solve the Sylvester equation; instead, they use an approximate matrix K, to estimate the Sylvester matrix K, taking M, for example,
-deoay Mj M,,N, NNT K Edecay
Of course, such estimation will result in computational enrors in the Sylvester equation emror Ex, can be bounded by the norm of <decay M.NT and X M.NT::
The
EK. Mf x N.. - MF M,N N.K, K. K.M. M.N. N MT((MM,N 2edecay M.NT)N.
|K
(relative error) Rk. I|&MINNT 2easey M,NTII
||MI]| X ||Ne]l
Therefore, fine-tuning the weight decay parameter is actually minimizing the computational errors of (9) in conventional methods. Less computational emrors means more efficient feature learning, while the efficient optimizers (Hayou et al. 2024) (Yen et al.] (025) may not need to apply weight decay since they already achieve efficient feature learning. Empirically, computed the norm of
5

---

Published as a conference paper at ICLR 2026
relative error in Adam optimizer and note that the enror decreases by iterations. And weight-decay param that yields lower relative error tends to achieve lower EMA loss in experiments. It gives us intuition to eliminate the relative error through Sylvester equation. See Fig. (3) A similar result about SGD is given in Appendix (A.6). In addition to these experiments, we also provide a practical intuition for the weight-decay matrix vs. sylvester matrix from the perspectives of the update norm and preconditioning in Appendix (.77)
Convergence Analysia Norm
0.85
brd 0.03
0.80
EMA Loss (meen))
0.75
Adam_ Adam_lambda, Jamboa_0. 0.0
0.70
Adam_lamber
Adam lambca
0.65
Lngrette
100
150
200
250
300
350
400
10:00
1400
3000
7500
Bucket index (each
100 EMA Loss
(a) Relative error of Adam for A 0, 0.01, 1
(b) EMA Loss Adam for À = 0, 0.01,
Figure I: Weight decay À that yields lower relative error tends to achieve lower EMA loss.
Effeet of regularization. Although, according to our experiment (3 the logarithm of the relative error Rx, is mainly between 1e and 1e-12, which means that adding the appropriate weight decay matrix can be viewed as good approximation to efficient feature learning. We still need to verify whether adding L2 regularization to the loss function can improve performance because adding regularization to K, contradicts efficient feature learning. We perform relevant experiments and find that the benefits of achieving accurate efficient feature learning outweigh the benefits of applying Le regularization. See Appendix (A.1I).
Cost of removing weight-decay. In fact, computing the Sylvester decay matrix is not cost-free. According the Bartels- algorithm (Bartels & Stewart, 1972). it requires O flops to compute the matrix K2 at each timestep, meaning that the computation of K, is fast only when r min(m, n). However, our method can avoid repeated experiments to search for X*, which saves the experimental time significantly. To validate our approach, we conducted experiments with our weight-decay-free methods, which turned out to achieve even better performance than conventional optimizers.
OUR PROPOSED METHODS
In this section, we gain intuition from optimizers proven effective in pre-training and present the corresponding lifted algorithms. These lifted algorithms far surpass conventional LoRA optimizers and achieve more efficient feature learning in transformer- based networks.
4.1 PRECONDITIONING ON THE MATRIX TANGENT SPACE
Our intuition comes from efficient optimizers for large language models (LLMs) with low-memory requirements and fast convergence. According the recent research (Gong al. 2025), state-of- the-art optimizers can be viewed as solutions to Fisher Information Matrix (FIM) approximation (under the Frobenius norm). Suppose that the approximate FIM F E (Diag, (v); o > 0} and F is
the empirical FIM. Then minimizing the objective min & || FI) yields the following analytic solution: F Diag (E [g²]), where g² indicates the element-wiss square of the gradient g. This diagonal approximation Fx gives the form of Adam:
vec( X M,,NT) (F*) -11/ X,,NN);

---

Published as a conference paper at ICLR 2026
where vee(-) means viewing the elements of matrix vector in column- order. Applying this kind Gauss-Newton methods on the matrix tangent space T(MNT) m x n) can guarantee efficient feature learning while most methods failed to achieve efficient feature learning by applying preconditioner to the lifting space II mxxx x TRxx.
To see this, suppose the horizontal lifts of XM,N at (M, N) and (MR,NR" are (M, and (M, .) where Rt E GL(r). Since they are vectors on the fiber π 1 (M,NT) the following invari- ance equation always holds:
M = MR, Ñ
(11)
This property can also be seen from the analysis in A.5 And leads to the transformation invariance condition (5). Since the dimension of the matrix tangent space, i.c. TM, is only mr + nr - -r2, the solution of the Sylvester equation 9 K only recovers the information of X M on TM,N7. Thus, mathematically, we need to project XMAN back to T M,NT to avoid computa- tional errors. We provide a detailed analysis in Appendix (A.8).
4.2 ADAMS LIFTING ADAM TO HORIZONTAL SPACE
Assigning the descent vector. The descent vector exists on the matrix tangent space and it is usually set to be the projected gradient, i.e.,
X MINT + p(e) NN (V M.NNJe(M.NN))..
accurate but costly way to obtain X N. to use hook grad. When the hook registered correctly and called during back propagation, it will return V MMNTTfC(M,,) (Yen al., 2025) applies QR decomposition (Gander 1980) to Mr, N₁ and then uses the gradients of the two factors, V V., and VN to recover M..NT Assuming M, = U RM, N V R
Another method
X N. + ((RR),)"NN V.M. VT U(R VVN.VV.
(12)
We adopt the efficient though may not accurate form in which works well in our experiments.
Incorporating moments. From the previous section we are about to complete the lifting version of Adam. We employ a simple accumulation of first and second moments as in Adam (Loshchilov & Hutter 2017). Suppose the 1s order decay parameter is 31 and the 2nd order decay
parameter is 32, we have:
m +- 31m4- + (1-3)) M.NT
2t +- 320 + - M M.NT
(13)
Besides, there are different ways to incorporate moments in related methods. LoRA-Rite (Yen al. 2025) adopts a varying basis by projecting the moments each step P. = U. m hr pt which introduces mass loss and requires additional calculations for the escape mass d,- Meanwhile, Riemannian adaptive methods (Becigneul & Ganea 2019) prefer to accumulate second moments under Riemannian metric (8)
mi-1
g That is, by replacing
13 with:
Vt 320,11 (1 82 M.NT
118 M,
where X M.NT
X2
is the element-wise product under g. Computing X M.NT can
||x NT N,
improve training, though it requires solving an extra Sylvester equation and incurs about 10-20% overhead. Therefore, we provide the Riemannian inner product as an optional alternative to the standard inner product (3). This completes the Adam lifting, termed AdamS (S Sylvester), with minimal changes. The full iteration scheme is given in Algorithm 2 (Appendix A.9)
4.3 LRACS - LIFTING RACS TO HORIZONTAL. SPACE
Although AdamS (Algorithm 2 is simple and efficient, it requires relatively high memory, i.c. the storage of m. and vi consumes 2mm memory. We propose skillful trick to halve the memory
7

---

Published as a conference paper at ICLR 2026
consumption of AdamS low-memory version in Appendix (A.17). However, easier way is to lift an efficient and fow-memory method from pre-training other than Adam. Gong et al. (2025) proposed RACS algorithm for LLM pre-training. They assume that FIM positive diagonal matrices. This assumption normalizes gradients by scaling both rows and columns, leading to the RACS algorithm. results in a good structural approximation of nearly block diagonal Hessian matrix in transformer-based networks with mean- square error (MSE) loss and cross-entropy (CE) loss; see (Dong et al. 2025). Then minimizing the objective min
{S @ Q) where
S R Q E mmxm
F
Fll will give the following equation:
Diag E G (CQQ))
Diag (E GSG")
F S Q
q q=
(14)
S
where s = Diag(S), (S), = Diag(Q), (Q). RACS (Gong et al. 2025 uses a one-sample estimate of E|| and performs 5-step power iteration to solve (14), starting from q 1. And they only accumulate the first moments of s and 9, reducing the memory consumption to m + n, that is:
m, 818 + (1 B1)8 mmg = -/ + (1 1-11))4 )1)..
Incorporating these moments with the same descent vector as AdamS (Algorithm 2 we propose LRACS, short for "Lifted RACS". The iteration scheme of LRACS is provided in Appendix (A.10). We provide a runtime analysis about the two algorithms section 5
EXPERIMENTAL RESULTS
We evaluated the proposed LoRA optimizers against other optimizers in transformer-base models with different loss functions. These include LoRA fine-tuning on the Mix-of-show (Gu et al. 2023
image generation model, evaluated using the happ--ittbase-ptteeee 1 model (Radford et al 202 and
fine-tuning a GPT-2 medium L.M (Radford et al.) 2019) on the E2E dataset (Novikova et al. 2017) for the natural language generation challenge. The results are shown Table and Table 2
RANK
CLIP 1
FID 4
RANK
CLIP 1
FID 4
4
23.90
77.66
4
25.17
72.75
SGD
24.52
74.00
Adam
26.83
67.09
16
24.76
70.78
16
29.86
57.35
4
25.74
70.99
24.34
75.48
Scaled GD
25.98
70.01
Quotient GD
24.46
74.91
16
29.07
59.87
16
28.04
63.20
27.86
64.38
31.43
52.67
AdamW
8
28.05
62.58
LRACS (ours)
8
31.46
52.12
16
30.78
62.57
32.09
49.52
30.96
59.04
32.20
54.39
LoRA-Rite
30.99
59.02
AdamS (ours)
8
32.38
51.87
16
31.90
55.68
16
32.64
46.32
Table 1: Experimental results of Mix-of-show generation model CLIP and FID
We compare the following optimizers:
SGD Ammai
One of the most widely used optimizer in machine learning. It is conventional and does not support EFL.
1993):
Adam (Kingma & Ba 2014) and AdamW (Loshchilov & Hutter 2017): The most widely used optimizers in vanilla LoRA. The vanilla Adam does not support weight-decay, while AdamW needs
finetuning about the weight-decay parameter. They are conventional
and do not support EFL.
Scaled GD (Tong et al. (02T) and Quotient GD Mishra et al. 2014h: Both are Riemannian optimization methods which support EFL and extra preconditioning is applied.
8

---

Published as a conference paper at ICLR 2026
Method
BLEU
NIST
MET
ROUGE-L
CIDEr
SGD,
66.6
8.54
44.2
68.2
2.32
Adam,
68.0
8.61
44.7
69.1
2.38
Scaled GD,,
68.5
8.72
45.5
69.4
2.40
Quotient GD 7=4
66.9
8.54
44.3
68.4
2.35
AdamW, r=4
68.6
8.69
46.5
71.3
2.51
LoRA-Rite,=a
69.3
8.75
46.5
71.7
2.53
AdamS (ours)r=4
69.4
8.75
46.5
71.7
2.53
LRACS (ours),=1
70.4
8.85
46.7
71.9
2.54
Table 2: Experimental results of GPT-2 medium LLM on E2E NLG challenge.
LoRA-Rite Yen et al.
2025):
The state-of-the-an optimizer supports EFL.
LRACS (Algorithm 3 Our proposed algorithm which is constructed by lifting RACS
(Gong et al. 2025) horizontal space. Following the general scheme, supports EFL.
AdamS (Algorithm 2): Our proposed algorithm which is constructed by lifting Adam to horizontal space. Following the general scheme, it supports EFL.
Optimizer setup. We perform grid search to select the best learning rate for each algorithm, with the optimal hyperparameters listed in Appendix (A.2). For AdamS, we implement the low-memory variant (Algorithm 4 and enable the Riemannian inner product option. For LRACS, following
GGng et al., 2025), we train the last LoRA layer with Adam (Kingma & Ba 2014) if is injected into the last layer of the network. All experiments robust training are provided in Appendix (A.16).
A100 GPU. Additional details
Results of Mix-of-show model. Mix-of-show is transformer-bee based model with Mcan-Squar Error (MSE) loss. Thus, admits block diagonal Hessian structure (Dong et al. 2025). used the same image dataset in the original Mix-of-Show (Gu et al. 2023) paper the Harry Potter image set-and trained for 3,500 steps. We then saved the generated images (see the Appendix) and computed their CLIP and FID (Parmar al. (022) scores. higher CLIP score indicates stronger relevance between the generated images and the text descriptions, while a lower FID signifies that the generated images possess more distinctive features of specific category. As demonstrated in Table methods that achieve efficient feature learning (EFL) generally outperform the correspond- ing conventional approaches that do not. At 4. compared the non-EFL baseline, our proposed method AdamS improves the CLIP score from 25.17 (Adam) to 32.20 and reduces the FID score from 72.75 (Adam) 54.39, At r = 16, compared the EFL baseline, AdamS outperforms Scaled GD, raising CLIP from 29.07 to 32.64 and lowering FID from 59.87 to 46.32. These results demon- strate substantial gains over the baselines. Moreover, our method demonstrates superior performance over state-of of-the-art approaches, i.e. LoRA-Rite, on key benchmark metrics. AdamS consistently achieves the highest CLIP scores (above 32) for r = 8,8, 16, indicating both superior quality and robustness.
Results of GPT-2. GPT-2 is a transformer-based model with Cross Entropy (CE) loss. It also admits block diagonal Hessian structure. For experiment in GPT-2 medium, we choose 4 for LoRA based on the previous E2E NLG challenge study (Zhang & Pilanci] 2024). The E2E dataset is dataset for training end-to-end, data-driven natural language generation systems in the restaurant domain. The dataset is split into training, validation and testing sets (in a 76.5 5-8.5- 5-15 ratio). Success on this task typically measured by achieving high BLEU (Papineni al. 2002), NIST, METEOR (Banerjee & Lavie 2003). Rouge-L (Lin) 2004). CIDEr. In the GPT-2 medium experiments, the total number of trainable parameters is 0.39M. We train on the dataset for 5 epochs, totaling 22,600 steps. The training hyperparameters are documented in the Appendix (..)). As demonstrated in Table 2, it can be observed that, the EFL version of Adam, AdamS generally outperform Adam on all evaluation metrics. Within the EFL methods, LRACS achieves the best performance demonstrating significant improvements over baseline methods as well as comprehensive superiority over other state-of- ofate-of-the-ar the-art approaches. We believe that this characteristic will greatly inspire future research. To enrich the diversity and scalability results, we also conducted experiments on WebNLG dataset and the GPT-2 small model, which is recorded in Appendix (A.14)

---

Published as a conference paper at ICLR 2026
Ablation Study. We tested 4, 16 the image generation model to ensure the robustness to rank of our proposed methods. It turned out that r = 16 is the optimal rank value determined experimentally in Mix-of-show model. Besides, in Section 3.3 and Appendix (A.6), ducted an ablation study replacing the solution of the Sylvester equation with a weight decay matrix. The results show that more accurate approximations to the Sylvester equation lead to better perfor- mance, indicating that solving the Sylvester equation is indispensable. In the Appendix (A.II). performed an ablation study on regularizing the Sylvester matrix K, and the results show that regularizing the Sylvester matrix does not improve performance. Finally, we also conduct an ab- lation study about Riemannian and Euclidean momentum accumulation on GPT-2 small model in Appendix A.13).
also
Sensitivity Analysis. We conducted a detailed sensitivity analysis of the hyperparameters (learn- ing rate & batch size) for AdamS and LRACS at = 4 on GPT-2 medium model. We first fix the learning rate of AdamS to 2e-4 and LRACS to 2 then vary it from 50% to 250%. A similar
result about varying batch size in Mix-of-show model is presented in the Appendix AA.15. The
results demonstrate that AdamS and LRACS are robust to these changes.
WebNLG
Method
learning rate
BLEU
chrf++
TER 4
U
S
A
S
A
U
S
A
le
43.40
61.72
53.62
62
74
49
.34
.41
AdamS
2e
42.95
62.73
53.92
62
74
69
49
.34
.41
5e
43.28
62.38
53.42
62
74
68
49
.34
.41
le
43.51
61.99
53.75
63
74
69
49
.34
.41
LRACS
2c-3
44.02
62.42
53.94
62
74
48
.33
.41
5e
43.29
61.46
53.39
62
74
69
49
.34
.41
Table 3: Results of optimizers on GPT-2 medium model in sensitivity analysis. The results show that our proposed methods are robust to hyperparameter changes
Time and Space Complexity In each iteration, the dominant computational costs for AdamS (Algorithm 2 are as follows: (1) matrix inverses for r-by-r matrices which takes (()²) time, (2) solving Sylvester equation (9) for r-by-r matrices which takes Orr3) time, (3) projections of n-by- m matrices which take O (nr- mrr + -mr) time and (4) matmuls with time complexity O(mr² 2 mr?). Additionally, the backpropagation step for the loss function is computationally intensive, requiring at least (2(mnr) flops to compute the chain rule: VM VWN, VN VWTM. Therefore, the overall per-step complexity for AdamS is ((mnr) O( 3 + mr² + nr?). Similarly, the per-step complexity for LRACS (Algorithm 3) is Q(mmr)) (r + +mr + nr? In comparison, standard AdamW W has per-step complexity Q(mmr) Omr +nr²). As for the memory cost due to moments storage, AdamS takes (m)) while LRACS takes O(m + + 1). Meanwhile, Adam W needs Omnr + nr) for moments storage. For memory consumption and training time per step in real experiments, see the table (7) in Appendix )2.12..
6 CONCLUSIONS
Current LoRA optimization methods lack emphasis on efficient feature learning (EFL). The few existing EFL methods are hard to generalize. This paper proposes LoRA-S, which is a framework applying horizontal lift theory to bridge pre- pre-training optimizers (Adam and RACS) and lifting LoRA methods (AdamS and LRACS) that admit efficient feature learning. The new algorithms are weight- decay-free, indicating less hyperparameter tuning. In empirical tasks, our algorithms consistently achieve better results than existing optimizers diverse datasets and transformer-be cause our contribution of this paper remain valid even in complex engineering scenarios such as distributed computing and larger-scale LLMs with reasonable additional computational overhead. We hope that the framework and theoretical analysis presented in this work will help in designing and iterating EFL algorithms more efficiently and rationally.
models. Be- mathematical, theory-driven method for optimizer design, the conclusions
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS AND FUNDING DISCLOSURE
We thank the anonymous reviewers for their valuable suggestions on theory and experiments. This work is partially supported by Hong Kong Research Grant Council GRFs 16307023 and 16306124.
REFERENCES
P.-.. Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms Matrix Manifolds. Princeton University Press, USA, 2007. ISBN 0691132984.
P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms matrix manifolds. Princeton University Press, 2008.
P-A Absil, Luca Amodei, and Gilles Meyer. Two newton methods on the manifold of fixed-rank matrices endowed with riemannian quotient geometries. Computational Statistics, 29(3):569- 590, 2014.
Shun-ichi Amari Backpropagation and stochastic gradient descent method. Neurocomputingg (4-5):85-196, 1993.
Satanjeev Banerjee and Alon Lavie. METEOR:
An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on In- trinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65-72, Ann Arbor, Michigan, June 2005. Association for Computational Linguistics. URL https:/ www. www.wwwe.bb. htt.:e/.rrrggacoooogo///.00....0
Richard H Bartels and George W Stewart. Algorithm 432 [c2]: Solution of the matrix equation ax+ xb= [f4]. Communications of the ACM, 15(9):220-82 1972.
Gary Becigneul and Octavian Sctavian-Eugen Ganea Riemannian adaptive optimization methods. In In- ternational Conference Learning Representations, 2019. URL /ittss://ppereviee:
Silvere Bonnabel. Stochastic gradient descent riemannian manifolds. Automatic Control, 58(9):2117-22229, 2013.
IEEE Transactions
Zhaorui Dong, Yushun Zhang, Zhi-Quan Luo, Jianfeng Yao, and Ruoyu Sun. Towards quantifying the hessian structure of neural networks. arXiv preprint arXiv:2505.02889 2025.
Walter Gander. Algorithms for the qr decomposition. Res. Rep, 80(02):2511-1200 1980.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachini The WebNLG challenge: Generating text from RDF data. In Jose M. Alonso, Alberto Bugarín, and Ehud Reiter (eds.), Proceedings of the 10th International Conference on Natural Language Generation. pp- 133, Santiago de Compostela, Spain, September 2017. Association for Computational Lin- guistics. doi: 10. 10.18653/v1/W117-3511
URL //ac anthologyy...... www.aaoooo..ggg////-1--00011
Wenbo Gong, Meyer Scetbon, Chao Ma, and Edward Meeds. Towards efficient optimizer de- sign for IIm via structured fisher approximation with a low-rank extension. arXiv preprint arXiv.2502.07752. 2025.
Yuchao Gu, Xintao Wang, Jay Zhangjie Wu, Yujun Shi, Chen Yunpeng, Zihan Fan, Wayou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, Yixiao Ge. Shan Ying, and Mike Zheng Shou. Mix-of- show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305.18200 2023.
Vineet Gupta, Tomer Koren, and Yoram Singer. Shampoo: Preconditioned stochastic tensor opti- mization. In International Conference on Machine Learning, pp. 1842-1850. PMLR, 2018.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. LoRA+: Efficient low rank adaptation of large
models. In Forty-first International Conference on Machine Learning. 2024. URL https: //openrevi ew. net. wwt.:/ntttoom////o forum? http:/oorrmm..ddaevvySeee d=NE

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):, 2022.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 2014.
Chin- Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computational Linguis- tics. URL.ttpss www. ac web. antho ogy 1013
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.0510,, 2017.
Bamdev Mishra, Gilles Meyer, Silvere Bonnabel, and Rodolphe Sepulchre. Fixed-rank matrix
factorizations and riemannian low-rank optimization. Computational Statistics, 2913):911-61,, 2014.
Jekaterina Novikova, Ondiej Dušek, and Verena Rieser. The e2e dataset: New challenges for end- to-end generation. arXiv preprint arXiv:1706.0920 092544 2017.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. Bleu: a method for automatic evaluation of machine translation. pp. 311-318, 2002.
Gaurav Parmar, Richard Zhang, and Jun- Yan Zhu. On aliased resizing and surprising subtleties in gan evaluation. In CVPR, 2022.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PmLR, 2021.
Axel Séguin, Gianluca Ceruti, and Daniel Kressner. From low-rank retractions to dynamical low- rank approximation and back. BIT Numerical Mathematics, 64(3):25, 2024.
Tian Tong, Cong Ma, and Yuejie Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150):1-63. 2021. 2021.
Phuong Thi Tran et al. On the convergence proof of amsgrad and a new version. IEEE Access, 7: 61706-61716, 2019.
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter effi- cient tuning of pre-trained models using dynamic search-free low-rank adaptation. arXiv preprint arXiv:2210.07558 2022.
Rachel Ward, Xiaoxia Wu, and Leon Bottou. Adagrad stepsizes: Sharp convergence over nonconvex landscapes. Journal of Machine Learning Research, 21(219):1-30, 2020.
Jui-Nan Yen, Si Si, Zhao Meng, Felix Yu, Sai Surya Duvvuri, Inderjit S Dhillon, Cho-Jui Hsieh, and Sanjiv Kumar. LoRA done RITE: Robust invariant transformation equilibration for loRA optimization. In The Thirteenth International Conference on Learning Representations, 2025. URL http::::///ppnrevieee net www.rrrmmmmi/ppiiiii...
Fangzhao Zhang and Mert Pilanci. Riemannian preconditioned lora for fine-tuning foundation mod- els, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter- efficient fine-tuning. arXiv preprint 2303 2023.
Yuanhe Zhang, Fanghui Liu, and Yudong Chen. LoRA-one: One-step full gradient could suffice for fine-tuning large language models, provably and efficiently. In Forty-second International Conference on Machine Learning, 2025. URL htwps::tt//ccc.. htttps://ppenreview net wwrummtuummmmmm idd
Kw vmI.D.m
12

---

Published as a conference paper at ICLR 2026
A
APPENDIX
A.1 LARGE LANGUAGE MODELS (LLMs) IN PAPER WRITING
L.M is used in this paper to improve grammar and wording.
A.2 HYPERPARAMETERS
Method
AdamW
LoRA-Rite Rite
AdamS
LRACS
Rank
4
16
16
8
16
8
16
Training
Weight decay
0.01
0.0
Batch Size
LR Scheduler
Linear
LR (tuned, x x10-3)
le-2
le-2
2e-2
1
0.2
0.2
0.3
1
3
st Momentum decay 3
0.9
0.9
0.9
0.9
2nd Momentum decay B3
0.999
0.999
0.999
Inference
Inference Steps
50
Guidance Scale
7.5
Fusion parameter a
0.7
Table Hyperparameters for Mix-of-show model fine-tuning.
Method
AdamW
LoRA-Rite
AdamS
LRACS
Rank
4
4
4
Training
Weight decay
0.01
0.0
Train batch size
LR Scheduler
Linear
Warmup step
500
Label smooth
0.1
LoRA a
32
LoRA dropout
0.1
LR (tuned)
2e-4
le-3
5e-5
le-3
1 Momentum decay /1
0.9
0.9
0.9
0.9
2nd Momentum decay 32
0.999
0.999
0.999
Inference
Beam Size
10
Length Penalty
0.8
No Repeat Ngram Size
Table Hyperparameters for GPT-2 model fine-tuning.
13

---

Published as a conference paper at ICLR 2026
Train PPL with model size: 233M
17.0
Adams
16.0
15.5
Train
14.5
600
1030
1200
1400
1600
1800
2000
Update Step
Figure 2: 335M GPT-2 medium model LoRA training PPL curve begins from warmup steps.
A.3
TRAINING Loss CURVE VISUALIZATION
A.4
INDUCED METRIC FOR TRANSFORMATION INVARIANCE SCHEME (PROPOSITION
Recall the LoRA-Rite update scheme
Yen et al.
2025):
8M1 = VM (VM, VM, 3 (RT,)*
8M2 VM2 M2 V22
(RTT))'
Then the first part of our inner product ((M,N) becomes
trace(NN] N16M MM)) (race(R2 RN, R (VM, VM) AM (VMTVM))
(1
(race(I,) = trace [ N28M, 8M2) 5M,
(15)
Similarly, we have symmetric results for 8N, and 8N2 (the second part):
trace(M Mf &N1) trace(I,) trace(M3 M28N, 8N2)
(16)
Equations 15 and (16) lead to the induced metric:
((M,NN) ((6M1, 8N1), (6M, &N1)) = trace( N N18M, 5M, + M M,N -trace (2 225 5M2 + M. N2 8N2) 9(M2.Na)
Since (8M, 8N;), = 2,2 preconditioned with AdaGrad (Ward et al. 2020 LoRA-Rite can be viewed as lifting AdaGrad under this specific metric.
A.5 PROOF OF TRANSFORMATION INVARIANCE (THEOREM 1)
Yen al. 2025) proposed the definition of transformation invariance and all optimizers following the general iteration scheme (Algorithm I matrix in the equation
are transformation invariance. Let Kg be the Sylvester
M X M MMMNNKK KNT
And K, be the solution of the equation
(MR)T X A (
(17)
in which R E GL(r). Note that the equation can be reduced to
MT X M MFMNTNR
K, R R M
14

---

Published as a conference paper at ICLR 2026
Thus gain the relation between Kg and K
Ko = R K
(18)
holds for 1, T. Now we look at the iteration scheme for particular time t, and LoRA pair (M,, N.), (Algorithm [ gives the following:
M+11 = M, MM(M,N and No+1 N₂ + TN
And the iteration scheme for another LoRA pair N,R,NNRR... is
M M,R + MMM. and N 1+1 N,R n (N(MRNA-)
To prove the transformation invariance property, just observe that
X.M.MR....R-.)
N.R M,N N.R T T (NT
= (N M.N NARNTNRTTANE T (NNT N
From equation 18) we can substitute K!") with K!!, thus
X XMM(MRNATT) = X
We can derive similar result for XNN(MRN.
XNNGRNN T) = XNN(M.N..
These relations of the lift vectors lead to the following
XTM(M.N)N X.MMMRRM T NT
M.X (N(M..N)) = M,RXT AMMAN-Y
X X (N(M.N.) X-MMMMRA.TT X
NOM.RN T)
which leads the transformation invariance condition
M1 = M4
(19)
A.6 RELATIVE ERROR OF SGD OPTIMIZER
lambdo
lambba 0.01
Lagartthi the Average (IL 91
lambba
EMA Looso (mean)
0.70
0.65
0.60
-10
0.5
SGD Jamadaa
-12
SGD lambda 0.01
Jambda
10
25
30
35
40
terations
Bucket index (each 100 EMA Loss records)
(a) Relative error of SGD for À 0, 0.01,
(b) EMA Loss of SGD for 0.01,
Figure 3: Weight decay À that yields lower relative error tends to achieve lower E.M loss.
A.7 A PRACTICAL INTUITION FOR WEIGHTTDEECY MATRIX VS SYLVESTER MATRIX
matrix that is closer to K achieves a lower loss in Fig 3 and Fig (3.3 Now we provide two complementary perspectives for intuition.
We have already made experiment showing that
Perspective: Update Norm
15

---

Published as a conference paper at ICLR 2026
In fact, another interpretation of efficient feature learning studies the update norm divided by the weight norm, i... ||6M]// || and ||6N11/||]]] see LoRA-Rite's Yenn et al. 2025) (A.8). Recall the general iteration scheme 6M GTM + DTM, ON GIN DIN, the gradient terms GTM XN (N GGT = X (M"M) When applying weight-decay matrix, the decay terms are DTar(e) - (T) = -NN. When applying sylvester matrix, they are
DTM = (NTN) DTN -N -N (MTM) KT (MTM)-.
One can find that the magnitudes of both ||GTM |/|| and |
[[] while the
magnitude of DD M]] is O(e) which results in a magnitude gap as is a hyperparameter. For example, in weight-decay matrix case, when >>> ||GTa/l/||| I. | O(e). The preconditioning information in GTM cannot be properly reflected in the updates, which leads to less efficient learning. Similar results also appear in | |
Perspective: Preconditioning
Our preconditioning vector X mitigates the condition number of the Hessian V /2))(W efficiently during large-model training, which is also important reason for the success. When applying the Sylvester matrix, the overall update is 5MNT + MMNT X. However, applying weight-decay matrix will undermine the preconditioning effect. As the overall update becomes:
8MN M8NT XNN+ MM 2CMNT
In this case, the matrices .NT. NN+ and MM+ likely be ill-conditioned, making the preconditioning less effective. Thus, the sylvester-based update tends to be more efficient which indicates the weight-decay parameter our methods can be removed.
A.8 COMPUTATION OF PROJECTION
Mathematically, we need to project XMNT back to T M.NT using the following:
ple) T
MINT (XMNT) = UUTXMN + VV - -UUT - VVT.
However, the projection be omitted since the update rule for (M, N2) (lines and in Algo- rithm will eliminate the components of XMT orthogonal to T M.NT. Consider X M N and X M M in "Gradient Terms", we have:
XMNT p(e) TMANT (I- (UUUUU
NMNTN- =P(c) P
p(() (I-
=P(O) PPT,, (XMNT)
(20)
XT M (ple) (P T
VV -
(Pl)) NN (XMNT))"M,
(21)
where and V are the orthonormal bases from the QR decompositions of M and N, i.c., M U URM, N R.. The tricks in (0 21 significcay reduce the computational cost of projection.
16

---

Published as a conference paper at ICLR 2026
A.9 ITERATION SCHEME OF ADAMS
Algorithm 2 AdamS
1: Input: learning rate A, first momentum decay 81, second momentum decay B2. optimization
steps T. Init with mo = 0; 2/200=1 =0
2: for / = 1 ....Td.. do
3: Obtain the descent vector X M.NT from 12
4: Calculate first momentum my +- 31 1 (1 31 M.NT
5:
Calculate second momentum 24 + 3204 (1 B2 32) MANT
Calculate preconditioned vector X M,NT me/vie
6:
7:
Solve Ke from the Sylvester equation
to lift X.,,TT
8:
Compute the lift vector XMM(M,.N.
NT N; M,TTNKK) (NTN.)
9:
Compute the lift vector (NC(M,,N.) + (NN. - M -N,MF (MF
10: Update model parameters for (M++1. Ne41) + (Mc +XP((MMM))) Nr + X9N(MM.N))
11:
end for
A.10 ITERATION SCHEME LRACS
Algorithm LRACS
Input: learning rate X, first momentum decay 31. limiter threshold y, optimization steps T.
2: Init: mgg = 0; gu = 0; do = 0
3: for =1 1,.... T do
4:
Obtain the projected gradient G from
12
5: Obtain 8 and q: from (14)
6: Calculate momentums: m. + 3 mp + (1 B1)8s, Ma, +- B. + (1 Bi)
7:
Calculate preconditioned vector: X NT wF Diag, (m... /1/2 )
8:
Solve Kg from the Sylvester equation to lift XM.NT
Compute the lift vector XTM(M,,NN) + N - M.NTN.K.) (NTN.) -1
9:
Compute the lift vector XIN(MENN) + Mc MN...NMTAMM MKT) (M) Incorporate the norm-growth limiter:
10:
12: 7 7/ max ||
if 1 else 1, 1, = 1.00= =
Update model parameters for (e441,N441 (Mr XN- No- Xn (N((L..N)
14:
end for
A.11 EFFECT OF REGULARIZATION
As explained in Section 3.3 we can apply weight decay together with the Sylvester matrix. How- ever, this regularization will introduce computational error for Sylvester equation (9) then further influence the effect of efficient feature learning. Regularize K, with the weight decay parameter X and the learning rate ry:
K2 + Kg + AnI
(22)
Then we conduct experiments on Mixofshow model with AdamS (Algorithm 2 optimizer. We test the influence of À on an empirical task, the CLIP score, since Lg regularization mainly addresses the overfitting problem. The results show that the method with the least regularization achieves the highest CLIP score. Thus, empirically, we have the confidence to replace the weight decay matrix with the Sylvester decay matrix K, directly.
17

---

Published as a conference paper at ICLR 2026
À = 0.0
À = 0.01
À = 0.1
X == 1.0
CLIP
32.64
32.28
31.97
32.29
Table 6: Regularization Sylvester matrix K will not improve performance.
AdamW
LoRA-Rite
LRACS
AdamS
Memory (MiB)
10490
10530
10522
10546
Time (ms/batch)
569
767
734
653
Table 7: Memory consumption and training time per step in Mix-of-show model.
A.12
TIME AND SPACE COMPLEXITY
A.13
ABLATION STUDY RIEMANNIAN AND EUCLIDEAN MOMENTUM
A.14
DIVERSITY AND SCALABILITY RESULTS
Besides GPT-2 medium model with the E2E dataset, to enrich the scalability of our experimental results, conducted experiments on the GPT-2 small model with E2E dataset. To enrich the diver- sity of our experiments, we conduct experiments on the GPT-2 medium model, WebNLG (Gardent al. 2017) dataset. WebNLG popular dataset for data-to-text evaluation which includes 22K examples from 14 distinct categories. The validation set is spitted into U (unseen), S (seen) and A (all). The results show that our proposed method outperforms current methods across all metrics and we have recorded the results in Table (8).
E2E
Method
BLEU
NIST
MET
ROUGE-L
CIDEr
Adam,=4
68.8
8.71
46.0
70.3
2.45
LoRA-Rite,=4
69.1
8.75
46.1
70.5
2.47
AdamS,=4
69.4
8.75
46.5
71.3
2.51
LRACS 69.7
8.79
46.4
71.5
2.50
Table 8: Results of optimizers on GPT-2 small model in E2E NLG challenge. The results show that our proposed method outperforms current methods across all metrics.
A.
SANSITIVITY ANALYSIS
A.16 ROBUST TRAINING
In this section, we enumerate the challenges encountered during training and the corresponding solutions. As variant of Riemannian optimization algorithm, LoRA-S faces similar training failure cases. For example, the computation of (MT M) inaccurate under the case of zero initialization. The same issue also arises in LoRA-Rite Yen al., 20025.. ScaledGD, and ScaledAdam ZZhang & Pilanci] 2024). Here, we list approaches from other works as well as our proposed methods to ensure robust training of LoRA-S. Among them, we implement the zero initialization (zero-init) in our experiments.
Zero Initialization. Zero initialization is used most common in LoRA training. In this start, the first LoRA factor is initialized with normal distribution with mean 0 and variance a², that is:
[Moly N' a?) [Nols 0, 0. (zero-init)
In this circumstance. the computation of (NT No) explodes and the loss becomes nan, which makes the whole training inefficient. In response this situation, other methods take the following
18

---

Published as a conference paper at ICLR 2026
Performance Iteration
AdamS_ Eu
2.80
AdamS Rie
LRACS Eu
LRACS
Metric
2.70
2.60
2000
4000
6000
8000
10000
12000
14000
16000
18000
Iteration
Figure 4: Results of train loss of our optimizers on GPT2 small model E2E NLG challenge with
different momentums. The results show that Riemannian momentum accumlation provides slight improvement for AdamS, but not for LRACS.
WebNLG
Method
BLEU f
chrf++ f
TER
U
S
A
U
S
A
U
S
A
Adam,=4
41.2
60.6
53.0
60
72
68
50
.35
.41
LoRA-Rite,=4
43.1
61.4
53.2
62
73
68
49
.35
.41
AdamS,=4
43.0
62.7
53.9
62
74
69
.34
.41
LRACS,=11 44.0
62.5
54.0
62
74
69
.33
.40
Table 9: Results of optimizers on GPT-2 medium model in WebNLG challenge. The results show that our proposed method outperforms current methods across BLEU, chrf++ and TER metrics.
approaches: ScaledAdam uses the identity matrix in place of a matrix inverse, while LoRA-Rite applies the Moore-Penrose inverse and uses nan_to_num to handle extremely small values that arise during computation.
Biased Initialization. As variant of zero initialization, is possible way which applies nonzero initializations both to Mo and No. That is,
Moly ~ N (0) [Noly ~ (0, a, 3 > 0. (biased-init)
In this setting. need subtract the initial weight matrix oW MoN from the pre-trained weights W to ensure the starting point remains at O. Although this method works well in our experiments, it requires modifications the pipeline to support the initialization. Thus we give it up.
Besides the above two starts, we propose using robust start during training and introducing robust. sleps as hyperparameter. Before the training step reaches robust sleps, we replace our EFL method with AdamW; after reaching robust_steps, we switch back our EFL method. The advantage is that we avoid convergence issues near zero without changing the initialization scheme. In our experiments, robust_steps is set to 100.
Spectral Initializaiton. Denoting one-step gradient of full fine-tuning G*, Zhang et al. 2025 compute the singular value decomposition (SVD) of G S VT. Then they apply nonzero
19

---

Published as a conference paper at ICLR 2026
Method
learning rate
CLIP
Method
learning rate
CLIP t
1e 4
32.11
le 3
31.19
AdamS
2e
4
32.20
LRACS
2e
3
31.42
5e
31.95
5e 3
31.08
batch size
CLIP
batch size
CLIP t
1
32.13
1
30.86
AdamS
2
32.20
LRACS
2
31.42
32.30
4
32.16
Table 10: Results of optimizers on Mix-of-show model in sensitivity analysis. The results show that our proposed methods are robust to hyperparameter changes
WebNLG
Method
learning rate
BLEU t
chrf++
TER +
U
S
A
U
S
A
U
S
A
le-4
43.40
61.72
53.62
62
74
68
49
.34
.41
AdamS
2
42.95
62.73
53.92
62
74
69
49
.34
.41
5e-4
43.28
62.38
53.42
49
.34
.41
le
43.51
61.99
53.75
63
74
69
49
.34
.41
LRACS
2e
44.02
62.42
53.94
62
74
69
48
.33
.41
5e
43.29
61.46
53.39
62
74
69
49
.34
.41
Table 11: Results of optimizers on GPT-2 medium model in sensitivity analysis. The results show that our proposed methods are robust to hyperparameter changes
initialization strategy, termed spectral initialization.
Mo = V7 Üa:
22/2 Gt
[>]]
(Spectral-init)
No = vス /
VG
[1rr)
[.1r]]
This initialization be quite beneficial for LoRA-S because the starting point is no longer zero. However, it requires access to G, which means registering hooks during backpropagation and ensur- ing they called successfully. Such pipeline modifications overly complex, most LoRA methods, and cannot be achieved solely by changing the optimizer. For ease use, we eventually did adopt this approach. However, this initialization very favorable for LoRA-S, as it eliminates the need for the robust_steps hyperparameter.
adopted by
A.17 LOW-MEMORY VARIANTS OF ADAMS AND LRACS
A low-memory variant of AdamS, inspired by the structure of the tangent space of the popular manifold M(r, m n). Although the descent vector XMNT occupies m space store, when viewed in manifold coordinates it only has d = mr + nr r dimensions. From Equation (2), we can see that the projected gradient can be decomposed into three mutually orthogonal components S, Mp and N,
Md + VMRY - TUTRRR NOT VL RV VNVVT
(23)
S U-
(24)
By applying the first momentum to these components, we can then recover the overall first moment together with the current coordinates U, V solved from QR decomposition M U UR,, N = V Rx.
20

---

Published as a conference paper at ICLR 2026
which greatly reduced the storage required for the first-order momentum. That is,
mom (Mp) < m1 M2) + (1 Mp
mom( mom(Np) (N₂) < 3 31mom(N) (NT) +1 (1-81)NN
mom S) <- 31 (S) (15
M4 f- U Ummm(S)VT mom(M,) Ummom(N7))
(25)
The memory used by mom m (mm(m)))) (X))) and mom(S) is only mr + nr + + which is greatly reduced comparing with n m. However, this technique cannot solve the storage problem of the second moment, because there is no mutually orthogonal coordinate structure within the second moment. Therefore, it can only halve the memory usage, and the overall stroge remains O(mm). Finally, we present the complete low-memory variant algorithm:
Algorithm 4 AdamS (lm)
1: Input: learning rate A. first momentum decay $1, second momentum decay B2, optimization
steps 7. Init with my = 0; up =
2: for 1 1...T do
3:
Obtain the descent vector X NT from
4: Obtain the decomposed components M p, N
S from 23 24
5: Calculate first momentum my from (25)
6:
Calculate second momentum et 32 + (1 X2) M.NT
7:
Calculate preconditioned vector X MENT m4/ Vee
8:
Solve K, from the Sylvester equation
to lift X N,,
9:
Compute the lift vector XTM(M,,N. +
M - M.NTN.K.) (NTN.)
10: Compute the lift vector X N +
XT M.NT M, M -NM,,KKK (MFM))
11: Update model parameters for MMMe+1,N4440 + (Mr + MXMMM.. + XIN(M..N)))
12:
end for
In our experiments, the low-memory version of the algorithm shows no noticeable change in perfor- mance or training speed, but significantly reduces memory usage.
For LRACS in Algorithm 3 the projected gradient G 12 by default stored an m matrix, but this state is transient and can be released after use, so it does not significantly increase runtime memory in our experiment. Memory usage can be further reduced by storing G in its decomposed form G = M,VT UNT + USV*, which requires only O(nr nr) space. The computation of 8, and 4: from 14p can then be performed directly from the decomposed form in O(mr² + nr²) time.
A.18 RELATED OPTIMIZERS
In this section, list several first-order algorithms commonly used as baselines in both pre-training and LoRA training. We first start with pre-training optimizers and show how to fit them into our general scheme step by step.
A.18.1 SGD
Stochastic gradient descent (SGD) is the simplest first-order method leveraging gradient informa- tion, which wildly used in almost all neural networks. and replace Sylvester matrix K with weight decay matrix I. Thus it does not admit efficient feature learning. However, lifting SGD is sim-
ple. Firstly, assign the descent vector as the gradient ((() L(M, N₂) then calculate Sylvester matrix K. That is,
&..NT L(Me, N,)
Because this method does not use preconditioning, its training performance still not as good as other EFL methods.
21

---

Published as a conference paper at ICLR 2026
A.18.2 ADAMW
AdamW was first proposed in (Loshchilov & Hutter 2017). By applying diagonal preconditioning. it is now one of thet popular and efficient optimizers in neural networks. It is also the most common baseline the LoRA framework. As discussed in (Yen al. 2025). AdamW did not achieve efficient feature learning. We propose lifting version of Adam. Adam-Sylvester (AdamS) 2 to
ensure that. The details are already explained in section
A.18.3 SHAMMOO
Shampoo is proposed by (Gupta et al. 2018) for stochastic tensor optimization. applies Kronecker product approximation to FIM through power iteration algorithm. Preconditioner of Shampoo can be derived by minimizing an upper bound of min | Fli with structural assumption:
H = R & Li;; Rn E R I'm RR
where Rn and Lm are symmetric positive definite (SPD) matrices. The problem of applying sham- poo to transformer-
- networks is that the computation cost O(m* + n²) and memory cost mn + m² + n² are too high. Here provide lifting version of Shampoo, however, its feasibility still needs to be determined and details need to be refined.
X MINT + 6X = - 1/- VXJN(MNN 1/4
(Lift Shampoo)
A.18.4 RACS
Row and Column Scaled SGD (RACS) is proposed by (Gong et al. 2025) as LLM specialized efficient optimizer. By applying diagonal Kronecker product structure preconditioning, it performs strongly on pre-training the IB LLaMA. RACS is highly memory efficient since it only needs the storage of two diagonal matrices and Q and a scalar for the limiter, consuming O(m + n + 1) memory. In this paper, we lifted RACS to the LoRA framework as LRACS, see section (4.3) and observed very good performance on the GPT-2 LoRA fine-tuning. We hope this phenomenon can inspire the strategy for further optimizers design.
Now we begin to analyze optimizers degined specific to LoRA framework.
A.18.5 SCALED GD (RGD)
Scaled GD proposed by (Tong et al. 202I). Since it applies Riemannian metric ((rr,v)on top of SGD:
9(M.N) ((M, M N), (M, N)) = trace(N" M + MTMNTNN Ñ)
The metric different the metric in this paper (7) and this method also guaranties EFL. However, because it does not use preconditioning, its training performance is fairly poor. From the perspective of our scheme, Scaled GD is equivalent the following setting:
XM.NT ( C(Me, N,)
K2 Edecay I
We use this method as baseline for EFL methods.
1.18.6 QUOTIENT GD
Quotient space GD on fixed-rank manifold was proposed by (Mishra et al. 2014). To solve low-rank matrix completion problem, they proposed two quotient space gradient descent methods, GHT gra- dient descent and MMMF gradient descent with respect to the metric in this paper 7 and Euclidean metric. These Riemannian methods guarantee transformation invariance. However, since the usage of Euclidean metric will lead to non-Riemannian quotient manifold which conflicts to invariance MMMF failed to meet our requirement for efficient feature learning. Thus conduct experiments with G gradient descent quotient GD method. Similar with Scaled GD, quotient GD does not use preconditioning, its training performance is fairly poor. We this method baseline for EFL methods.
condition (4) in Definition
22

---

Published as a conference paper at ICLR 2026
Quotient manifold
Embedded manifold
(M1,N1)
(M2,N2)
project
X
lift
IRarr x IRA
M, (n
Figure 5: Quotient manifold VS embedded manifold. When (M1,N11 and NM2,N2) represent same point X E (M, m), the update vectors (in yellow) at (M,N1) and (M2 N2) should keep same magnitude guarantee efficient feature learning.
A.18.7 SCALED ADAMW
Scaled Adam W is proposed by iZhang & Pilanci 2024). It combines the same Riemannian metric with Scaled GD (8.15.5) and diagonal preconditioning from Adam (Kingma & Bal (014) in its design. Since they ignore the horizontal lift theory and apply preconditioning to the lifting space, according to (Yen et al.) 2025), they also failed achieve transformation invariance. As stated in AdamS (4.2) the combination of diagonal preconditioning and Riemannian metric should obey our general scheme to guarantee efficient feature learning.
1..8 RIEMANNIAN ADAPTIVE METHODS
Since the LoRA framework is naturally related with Riemannian manifold (n, m)) there is pos- sibility to apply Riemannian adaptive methods (Becigneul & Ganea 2019) to LoRA. These meth- ods mainly carry the optimizers from Euclidean space, i.e. AMSgrad (Tran et al. 2019). Adagrad (Ward et al. 2020) and Adam, to tangent space assigned with specific Riemannian metric, i.e. RAMSgrad, RAdagrad and RAdam. These methods not diseuss about efficient feature learn- ing since they only design algorithms on the matrix tangent space TMNT. However, efficient fea- ture learning require methods to care more about behaviors update factors on the quotient space Rmxr Raxt/ ~. The differences can be seen in Fig 5 All of them work in the right space, but not in the left one.
Besides, scheme does not Riemannian gradient and aiming to remove unstable parts of Riemannian adaptive methods which may lead to divergence, i.c. SVD on manifold, and only fo- cuses on memory and time efficient updates transformer-based networks, which leads to better performance than any previous Riemannian methods.
A.18.9 LORA-RITE
LoRA-Rite Yenn et al. invariant method with preconditioning. Our work is inspired lot by LoRA-Rite; in this paper, we aim to extend the favorable properties of LoRA-Rite
2025) a method that guarantees EFL., and is also the first transformation-
broader family of methods. Under the horizontal lift theory (section 2.2), LoRA-Rite can be viewed
lift version of Adagrad under a Riemannian metric (6) that is different from ours (7), thereby guaranteeing EFL and inheriting Adagrad-like preconditioning effects. In their metric (:r))s three mutually orthogonal components
Mc. UG and Ve in equation (26) of the descent vector G X NT applied with same inner
23

---

Published as a conference paper at ICLR 2026
products, shown in 27
G= G-UMGVT + UGV +UVE
(26)
(G, G)o = Lu(NNNNN + M MNTMN
-
+ VCCa)
(27)
where a constant. comparison, our scheme (G, G) satisfies:
(G, G)
tr((MM M + (NTN)-INTN)
Then we dig into (M²M)- -M and lifting scheme
substitute M and Ñ:
M) MTM
+ -
(28)
(29)
(30)
Equation (8 indicates the same inner product towards Mc. Ua and Vo. However, 29 and 30
do not. To this, need apply Sylvester equation
MTGN
RNTTTUMMG UGVT VUVT)RR MoRN MMTNNKK NK KKM
which means a one-to-one relationship between K and Mg independent of UG and Va. Thus, (29) and (30) apply inner product Me only, which leads to different scaled inner products three mutually orthogonal components Mc. UG and VG overall. Similar phenomenon also appears in (N" NNT NN N. Different Riemannian metrics also lead to different performances in experiments, and the effects of these Riemannian metries still need further study.
A.19 GENERATION RESULTS OF MIX-OF-SHOW MODEL
Methods
Fig.1
Fig.2
Fig.3
Fig.4
Fig.5
AdamW (CLIP 33.19)
LoRA-Rite (CLIP= 33.68)
LRACS
(CLIP = 34.03)
AdamS (CLIP= 34.84)
Figure 6: batch of generation results visualization Mix-of-show model with average CLIP score. The prompt is "a < V Vpotter > in blue shirt".
24


Reviews:
[{"review_id": "le96Oa6wAy", "reviewer": "Reviewer_D7kg", "summary": "This paper proposes a general iteration scheme for LoRA that works on a quotient manifold of pairs $(M, N)$ modulo the equivalence which preserves products $MN^\\top$ and updates them via \"horizontal lifts\" of a descent direction in the product space. The horizontal lift gives the unique update with no component along the equivalence directions. Computing this update requires a small Sylvester equation solve which can be done in $O(r^3)$ time where $r$ is the LoRA rank. This scheme wraps conventional optimizers to produce updates that by construction enforce desirable properties such as transformation invariance and efficient feature learning (EFL). The authors also observe that the Sylvester solution introduces a decay term and remark that this decay may obviate the need for explicit weight decay. The performance of this scheme applied to Adam and RACS are demonstrated across several experiments.", "questions": "What sufficient conditions ensure the Sylvester equation solution exists and is unique?\n\nAre there any numerical stability issues concerning computation of the inverses or the Sylvester solution?\n\nAre there metrics other than the Grassmannian quotient metric considered which can also be used?\n\nAre there any ablations on using the Riemannian moment accumulation (Eq. 36) vs Euclidean moment accumulation (Eq. 35)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "The framework introduced by this paper is very interesting and is a novel application of quotient spaces used to \"convert\" optimizers to a principled LoRA update. The generality of the approach and the connection to prior ideas such as transformation invariance and EFL is valuable. Discussions of practical concerns are thorough and informative.", "weaknesses": "The diversity and scale of empirical validations is lacking. \n\nWhen arguing about removing the weight decay parameter, it doesn't appear that the baseline weight-decay in Adam is tuned.\n\nSome of the technical exposition is hard to parse, especially to readers not familiar with the mathematical notions. More high-level exposition and background would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eTZ0luiC9e", "reviewer": "Reviewer_LhJ9", "summary": "This paper presents LORA-S, a novel LoRA optimization framework that leverages horizontal lift theory and Sylvester equations, achieving efficient feature learning (EFL) and transformation invariance. The authors propose two optimizers, AdamS and LRACS, both free of weight decay hyperparameters and shown to outperform existing methods in transformer-based models.", "questions": "Please see weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "**Theoretical Innovation and Novelty:**\n\nThis paper correctly establishes a general iterative scheme that generalizes conventional optimizers. The use of horizontal lift theory and quotient manifolds is both mathematically sound and novel in the context of LoRA. Besides, the removal of weight decay and the introduction of Sylvester-based updates reduce the need for hyperparameter tuning, offering a deeper understanding of how regularization interacts with low-rank parameterization.\n\n**Clarity and Presentation:**\n\nThe paper demonstrates a clear and logical structure. The authors properly define equivalence relations, tangent spaces, and induced metrics, and the derivations from the quotient manifold formulation to the horizontal lift operator are clearly and coherently presented.", "weaknesses": "- Although replacing the weight decay parameter with a sylvester matrix is mathematically sound, can you provide a practical intuition behind why it promotes efficient feature learning?\n- How do other hyperparameters affect the proposed framework? For instance, can a large learning rate lead to instability in the results, or can gradient noise from a small batch size undermine the underlying assumptions? \n- Does the computational overhead become a bottleneck for larger models, and does the scheme remain efficient in large-scale distributed training environments, where synchronization is a critical factor?\n- Sections 3 and 4 can be simplified, and unnecessary parts moved to the appendix, leaving more analysis of results in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ClSOpTzucj", "reviewer": "Reviewer_TWk9", "summary": "The reviewed work is concerned with low rank adaptation. An important feature of low-rank based parametrizations is that many pairs of left and right factors can give rise to the same weight matrix. Likewise, many pairs of infinitesimal changes of these factors give rise to the same tangent vector in weight space.\n\nThe \"horizontal lift theory\" (Avsil et al., 2014) proposes to select the representation that is orthogonal to the kernel of the map from pairs of factors to weight matrices. The reviewed work suggests using this mechanism to select updates in low rank training algorithms. Computing these updates requires solving a Sylvester equation. The authors provide a high-level argument for why the Sylvester-based update already accounts for the need of weight-decay, thus allowing to remove the need for explicit regularization and in particular hyperparameter optimization. \n\nThe numerical examples show small improvements over baselines.", "questions": "1. How is it possible for $\\bar{g}$ to be a metric (as opposed to a pseudometric). After all, the map $(\\dot{M}, \\dot{N}) \\mapsto \\dot{X}$ has a kernel by dimensional considerations.\n\n2. I have a hard time understanding why the proposed scheme is transformation invariant. After all, the definition of the horizontal complement relies on the \"orthogonal complement\" of the kernel of $D\\pi(M, N)$.\nWhat this orthogonal complement is depends on the choice of inner product on $T_(M, N)$. If the Euclidean inner product is used, then doesn't this again result in transformation dependence? If an inner product is prescribed, then doesn't this play the same role as describing a coordinate transformation? \n\n3. Could you help me appreciate the empirical contribution? As it stands, it seems that there is very little difference to the reference.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The use of Riemannian geometric approaches for improving low rank learning is a natural approach.", "weaknesses": "I found this paper difficult to read. This may be partly due to technical nature of its contribution, but unnecessarily aggravated by the way in which the heavy notation is introduced (or not). For instance in line 122 the paper talks about \"the metric $\\bar{g}$\", but this metric is only introduced in the next section.\n\nThe paper also claims benefits \"Our metric induces update rules with richer structure and interpretability than the metric (13)\" without (in the main text) substnatiating these claims. In contrast, the numerical improvements are very minor. I will confess that I was unable to follow the argument why the weight decay parameter can be removed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EPcTDWxO0p", "reviewer": "Reviewer_hpZ2", "summary": "In this work, the authors propose and analyze a Riemannian-based optimizer to fine-tune pretrained models using Low-rank adapters.\n\nIn particular, the framework is the one of [Absil et al., 2014], in which the manifold \n$$\n\\mathcal M_r =\\{ W \\in \\mathbb R^{n \\times m} \\,| \\, \\mathrm{rank}(W) = r \\}\n$$\nis seen as a quotient of $\\mathbb R_*^{n \\times r} \\times \\mathbb R_*^{n \\times r}$ through the projection $\\pi(A,B) = AB^\\top$. The framework proposed is essentially a general Riemannian optimization algorithm over the fibers of the projection, with the metric on the product defined through the metric of a chosen metric on $\\mathcal M_r$. In practice, then optimization is performed on liftings in the horizontal space. After this, the authors discuss the effect of $L^2$ regularization in their framework.\n\nFinally, the framework proposed allows us to easily lift preconditioners from $\\mathcal M_r$ to the quotient space, allowing us to easily precondition the dynamics and therefore to reproduce classical optimizers such as Adam.", "questions": "I would appreciate it if the authors could discuss the points I raised in the \"weaknesses\" section, as I believe the line of research per-se is interesting and valuable, but in my opinion, the work needs to be better presented.\n\nMinor comments and typos:\n1. As a general rule, I suggest that the authors not number equations if they are not referenced in the main text, as this creates unnecessary confusion.\n2. I would suggest including an initial paragraph summarizing the contributions of the work\n3. All initial quotes are inverted, use `` ... \" instead of \"\"... \"", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "The approach the authors propose is fairly general, and the theory is well-founded. The notation is pretty precise, and the authors often give good intuitions behind the formalism.\nThe proposed framework allows for easy extension for quasi-Newton methods, as the preconditioning in $\\mathcal M_r$ can be lifted to horizontal spaces.", "weaknesses": "1. A lot of the theory described in the majority of the work (including the horizontal lifting) was already presented in [1]. It is not clear what the actual contributions of the work are, as even the \"Efficient feature learning\" proposed in [Hayou et al., 2024] seems in this case a simple consequence of performing optimization in the quotient space, and this is well known in the Riemannian optimization literature (see [1]). \n2. Section 4.1 is not clear at all, as it seems the discussion is all to point out again that preconditioning is \"well-defined\" again because of the quotient, therefore one can use Gauss-Newton-like methods similar to Adam, and the preconditioner to use won't depend on the fiber of $\\pi$ we are working on. I guess also the title \"ANY PRECONDITIONING WORKS WELL ON THE MATRIX TANGENT SPACE\" is not clear, as it does not seem to reflect the content of the section.\n3. The claims of the theory are correct, but they don't seem to be reflected in the experimental section. In particular, numerical improvement with respect to much simpler methods is minor; therefore, one can ask if all this machinery has a use case in practice. I believe the authors should focus the experimental section more on showing the advantages of the proposed framework with respect to vanilla SGD or Adam on LoRA instead of just benchmark performances.\n\n[1] B. Mishra et al., \"Fixed-rank matrix factorizations and Riemannian low-rank optimization\", Computational Statistics, Volume 29, pages 591–621, (2014).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)