Finding Low-Rank Matrix Weights in DNNs via Riemannian Optimization: RAdaGrad and RAdamW
Authors: Fengmiao Bian
Jinyang ZHENG
Ziyun Liu
Jianzhou Luo
Jian-Feng Cai
Venue: NeurIPS 2025 poster ()
Keywords: Riemannian Gradient Descent
Low-rank matrix weights
Fine-tuning
AdaGrad
AdamW
Primary area: optimization

Abstract: Finding low-rank matrix weights is a key technique for addressing the high memory usage and computational demands of large models. Most existing algorithms rely on the factorization of the low-rank matrix weights, which is non-unique and redundant. Their convergence is slow especially when the target low-rank matrices are ill-conditioned, because the convergence rate depends on the condition number of the Jacobian operator for the factorization and the Hessian of the loss function with respect to the weight matrix. To address this challenge, we adopt the Riemannian gradient descent (RGD) algorithm on the Riemannian manifold of fixed-rank matrices to update the entire low-rank weight matrix. This algorithm completely avoids the factorization, thereby eliminating the negative impact of the Jacobian condition number. Furthermore, by leveraging the geometric structure of the Riemannian manifold and selecting an appropriate metric, it mitigates the negative impact of the Hessian condition number. Ultimately, this results in our two plug-and-play optimizers: RAdaGrad and RAdamW, which are RGD with metrics adapted from AdaGrad and AdamW and restricted to the manifold. Our algorithms can be seamlessly integrated with various deep neural network architectures without any modifications. We evaluate the effectiveness of our algorithms through fine-tuning experiments on large language models and diffusion models. Experimental results consistently demonstrate that our algorithms provide superior performance compared to state-of-the-art methods. Additionally, our algorithm is not only effective for fine-tuning large models but is also applicable to deep neural network (DNN) compression.

Finding Low-Rank Matrix Weights in DNNs via Riemannian Optimization: RAdaGrad and RAdam
Fengmiao Bian',
Jinyang Zheng', Ziyun Liu', Jianzhou Luo', Jian-Feng Cai!
Department of Mathematics
The Hong Kong University of Science and Technology, Hong Kong, CHINA
(ma fmabminn,, jfcai) @ust.hk, (izhengbp, z zliveq, jluobn (uoobn)@oonneeeeeeeeeeeeeeeeeeee @connect. ust. hk
Abstract
Finding low-rank matrix weights is key technique for addressing the high memory usage and computational demands of large models. Most existing algorithms rely on the factorization of the low-rank matrix weights, which is non-unique and redundant. Their convergence is slow especially when the target low-rank matrices are ill ill-conditioned, because the convergence rate depends on the condition number of the Jacobian operator for the factorization and the Hessian of the loss function with respect to the weight matrix. To address this challenge, we adopt the Riemannian gradient descent (RGD) algorithm on the Riemannian manifold of fixed-rank matrices to update the entire low-rank weight matrix. This algorithm completely avoids the factorization, thereby climinating the negative impact of the Jacobian condition number. Furthermore, by leveraging the geometric structure the Riemannian manifold and selecting an appropriate metric, it mitigates the negative impact of the Hessian condition number. Ultimately, this results in plug-and and-play optimizers: RAdaGrad and RAdam W, which are RGD with metrics adapted from AdaGrad and AdamW W and restricted to the manifold. Our algorithms can be seamlessly integrated with various deep neural network architectures without any modifications. We evaluate the effectiveness of algorithms through fine- tuning experiments on large language models and diffusion models. Experimental results consistently demonstrate that our algorithms provide superior performance compared to state-of-the- methods. Additionally, our algorithm is not only effective for fine-tuning large models but is also applicable to deep neural network (DNN) compression.
Introduction
Deep Neural Networks (DNNs) have achieved remarkable success in tasks such as image classification [19] object detection 200 and semantic segmentation [8). However, their high memory and computational demands pose significant challenges for deployment on resource-constrained devices. For example, ResNet-50 requires up to 4G FLOPs for single image classification task (6), making unsuitable for embedded systems. To address these limitations, techniques such as pruning 47 and quantization 228 44 have been developed. While effective, these methods often rely specialized hardware for acceleration, limiting their general applicability.
Low-rank matrix approximation has emerged as promising alternative, widely used in DNN compression 30 43), fine-tuning of large models (LMs) [18 21 47], and prompt engineering [1] 24). By representing weight matrices as the product of two low-rank factors, this approach
*Correspondence to: Jian-Feng Cairjfcai @ust.hhkiu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

reduces memory usage while preserving input and output dimensions, without requiring specialized hardware. In LMs, methods like Low-Rank Adaptation (LoRA) [8] 21 train additional low-rank matrix weights, significantly reducing computational costs while maintaining or even improving performance compared to full full-parameter fine-tuning. These advantages make low-rank matrix methods indispensable for efficient adaptation of large models to downstream tasks.
Numerous methods have been proposed to obtain low-rank matrix weights in neural networks. Some approaches [TD 22 [2 [50] employ singular value decomposition (SVD) to directly compute low-rank matrices weight, but the frequent large-scale SVD operations are computationally expensive. More factorize a rank-r matrix weight W E R mm x as W PQ" where P E R x X and Q Raxr, While this approach reduces memory usage by storing P and Q, it has several inherent drawbacks. First, the factorization W = =PQ is not unique, introducing redundancy that can lead to unbalanced factors, slow convergence, or even failure to converge. Second, as we will demonstrate later, the convergence rate of these algorithms depends on the condition number of the Jacobian operator (P, Q) of the generator G(P, Q) = PQT For ill- conditioned target matrices W, the condition number of Jo Q) becomes very large, significantly slowing convergence. Finally, efficient optimizers such as AdaGrad [12] and Adam W [26] 22), which are designed to optimize the full weight matrix W, cannot be directly applied to the low-rank factors P and Q in a way that fully exploits their efficiency.
commonly, factorization based methods [88 21
38 47 49
51
To overcome these limitations, we propose a novel optimization framework based on Riemannian optimization. Specifically, we employ the Riemannian Gradient Descent (RGD) algorithm on the manifold of fixed-rank matrices to directly update the entire low-rank weight matrix W, rather than its low-rank factors P and Q. While our method has the same order of computational complexity and memory usage as factorization-bant based methods, it offers significant advantages. First, it fully exploits the geometric structure of the Riemannian manifold and the intrinsic representation of low-rank matrices, avoiding the non-uniqueness and redundancy introduced by factorization. Second, by directly optimizing W, our method only involves V&(W) and is independent the Jacobian operator (P, Q), making the convergence rate independent of the condition number of the target matrix W. Finally, the RGD algorithm allows us to define different metrics on the manifold, enabling the extension of classical adaptive learning rate and momentum algorithms, such AdaGrad [12] and AdamW 26 22]] onto the Riemannian manifold. Building on this framework, we introduce two novel algorithms: RAdaGrrad and R R.damW. These algorithms retain the adaptability of AdaGrad and AdamW W while being better suited to the unique structure of low-rank matrix optimization problems.
Moreover, RAdaGrad and RAdamW function as plug-and-play optimizers that can seamlessly integrate into a wide range of deep neural network architectures without requiring any changes to the network structure. To evaluate the effectiveness of our approach, we conduct extensive experiments, including fine-tuning of large language models and diffusion models. The results consistently show that RAdaGrad and RAdamW significantly outperform state-of-the the-art methods. Furthermore, approach is not only highly effective for fine-tuning large models but also demonstrates great potential in deep neural network compression tasks, highlighting its versatility and practicality.
Finding Low-Rank Weights in DNNs
In this section, we examine existing algorithms, discuss their limitations, and propose a novel framework designed to overcome these challenges. To find the low-rank matrix weights of a deep neural network, we solve the following optimization problem
min E(W), subject to rank(W:(()) = Tk: he W
K,
where W = (W( W (()) E ==
k= Rd represents the weight matrices with (AA) E R ((W) is the training loss function, and r4 represents the rank of ((A), We use (((W) to denote its gradient, or (batch) stochastic gradient, without ambiguity.
K
2.1 cnctorization-Basee Algorithms
Current mainstream low-rank adaptation methods 188 38 47 49
51 for finding low-rank weights
primarily rely on factorization- based algorithms, where each weight matrix W(() is parameterized as w(( p(k))(()))) (Q())) T with p(k) e - Q(+) e xrk, Here, T represents matrix transpose.

---

Let P = (P(), p()) ((2),
and Q = (Q Q Q(K)) be the factors, and define the .(K)) .l.).....PP
generator
g Q) (Q (Q))) ,.... (" (Q(O))
as the mapping from these factors to the weight matrices. Factorization-base algorithms obtain low-rank weights by solving
min (P,Q), where (PPQ))=((((P.O)))
(3)
P.Q
The gradient of I(,, Q) is given by the chain rule
Vf(P. Q) Jz. ve(W) with W = G(P,Q)
(4)
where Tg is the Jacobian operator of the operator G and * denotes the adjoint of the operator. The Hessian given by
V²)(P,Q) = (q..Q) ((V) + the terms related to the changes in Ic)P.Q). (5)
Jp(P.Q)
The original LoRA algorithm (21] directly applies gradient descent solve (T. and its convergence rate is influenced by the condition number of the Hessian f Q) 4 54 From (5). is evident
that even when only the first term is considered, the convergence rate of the algorithm is significantly affected by the condition numbers of both the Jacobian operator To (P, Q) and the Hessian V2C(W) Since the condition number of (P, Q) depends on the condition number of W 13 the convergence rate deteriorates significantly when the target matrix W is ill-conditioned. Moreover, imbalances in the factorization of W and poor initialization can also lead slower convergence.
To develop efficient algorithms, therefore crucial to improve the condition of (P, Q) and
V
Efforts to improve the condition of Jp( Q): several recent attempts have been made address this issue:
LoRA+ [18] employs different learning rates for the two low-rank factors to improve conver- gence. However, it does not fully eliminate dependence the condition number of [P, Q). Riemannian preconditioned LoRA 47 applies simple preconditioners to each factor, partially mitigating but not entirely eliminating this dependence.
LORA-RITE 46 LoRA-RITE employs preconditioners based on transformation invariance, alleviating the impact of the condition number of [P, Q).
LoRA-PRO 49 removes dependence on the condition number of () by projecting
factor-based gradients back to a subspace of the matrix space.
DLRT (38) eliminates this dependence by updating low-rank factors on the quotient manifold. Imbalance- Imbalance-Regularie Regularized LoRA [5T] reduces the impact through regularization terms applied to the low-rank factors.
Addressing the condition of (2((W):: none of the aforementioned works directly address the condition number of V²((W).
When the low-rank constraint is absent, there are well-established methods to address this issue. Classical optimizers such as AdaGrad and AdamW adaptively assign different step sizes to each component of V((W). This essentially imposes an adaptive metric in the matrix space, under which the condition number of VZ((W) is smaller compared to the standard metric. However, extending this idea to low-rank factors presents significant challenges. Although is possible to directly use AdaGrad or AdamW to solve (3) by assigning adaptive step sizes to each component of V [[P, (), the redundancy the parameterization of the low-rank factorization leads to poor algorithm performance, demonstrated in experimental section.
2.2 Riemannian Optimization Framework
To develop efficient algorithms, we propose using Riemannian optimization solve 11. which addresses all the issues caused by the factorization- methods discussed in Section 2.1
It is well known that all rank-rı matrices form smooth Riemannian manifold 55 42 denoted as
Mrg embedded the matrix space 1 Consequently, the low-rank weight matrices W
belong to M, = I Myg, and M, is itself smooth Riemannian manifold embedded in E. Therefore, (1) can be reformulated as
min E(W). WEM,
(6)
3

---

We solve this problem using the Riemannian Gradient Descent (RGD) algorithm
W++ = R(W, 7 VM,, ((W:))
where W, M, represents the weight matrix at the t-th iteration, 7/- R., denotes the step size, ((W) VM,,(W, E Tw, is the Riemannian gradient that lies in the tangent space Tw, of the manifold M, at W,, and Tw; - M, retraction operator that maps matrices from Tw, back to M,. At iteration I, the low-rank weight matrices W, are updated on the tangent space using the Riemannian gradient, yielding
Z = W. T/ V M, E Tw.
where Z, is then retracted back onto the manifold M, using the retraction operator R. resulting in the new low-rank weight matrices 1++ chosen to define the inner product on the tangent spaces of the Riemannian manifold, These metrics affect both the direction and magnitude of the Riemannian gradient, thereby giving rise different algorithms. This provides general framework where various optimization algorithms can be derived by selecting appropriate Riemannian metrics.
worth noting that different Riemannian metrics can be
Tw
Z. W. V.VM
W
R
W
M,
By caleulation, the Riemannian gradient of ((W) M, is given by
(w, Prw VE(W),
where PIw is the orthogonal projection from onto the tangent space Tw, and V((W) is the gradient of ((W) with respect to W in E. The Riemannian Hessian of ((W) on M, is given by:
VM,e(W) PTw- V2E(W). PTw + terms related to the changes in Prw:
(10)
where V2((W) the Hessian of ((W) in Here, all Pow. VE(W), and V2((W) are computed under the metric on Tw extended to E.
We will demonstrate in the next section that, with suitable choices of Riemannian metrics, the computational and memory complexities of RGD are the same order as those of factorization-basee algorithms. Furthermore, RGD overcomes all the drawbacks of factorization-base algorithms mentioned in Section 2.T and offers several advantages:
Avoiding Redundancy. RGD operates on the manifold of fixed-rank matrices, directly updating the entire low-rank weight matrix W rather than its factorized components P and Q. By leveraging the geometric structure of the Riemannian manifold and the intrinsic representation of the tangent space, RGD avoids the non-uniquee non-uniqueness and redundancy introduced by factorization. Specifically, as demonstrated in 42 45), it is easy to find an orthonormal basis for Tw,- This allows us to efficiently compute updates for Z, in 29 under the orthonormal basis of T w,, maintaining the computational complexity and memory usage at the same order as factorization based algorithms while eliminating redundancy.
Convergence Rate Depends Only on the Euclidean Hessian. The convergence rate of RGD is primarily influenced by the condition number of V2((W) only. It is well known that the convergence rate of RGD is affected by the condition number of (M, M. From I00 if the second term is negligible, then (() Pr vP((W)) Prw which is the restriction of V²((W) Tw. Hence, the condition number of VM, ((W) is approximately bounded by the condition number (=((W). Therefore, the convergence rate of RGD is mainly influenced by the condition number of V²((W) only. In contrast, as shown in (), the convergence rate of factorization-bar based algorithms depends not only on the condition number of V²((W) but also on that of (P, Q), which is very large if the condition number of W is very large.

---

Designing Riemannian Metrics for Improved Convergence. The geometric structure of the Rie- mannian manifold allows us to flexibly design a Riemannian metric improve the convergence of RGD. As previously discussed, the condition number of VM, <(W) is approximately bounded by the condition number of (')( Therefore, it suffices to construct a metric on Tw such that, when extended to E, the condition number of V'E(W) is reduced. Many classical optimization al- gorithms, such as AdaGrad and AdamW, inherently define metric E that reduces the condition number (W) by applying different learning rates to different components of the gradient. Naturally, restrict such metric on Tw obtain Riemannian metric that reduces the condition number of V ((W), resulting in accelerated RGD algorithms. This approach leads two new RGD algorithms: RAdamW and RAdaGrad, which the main contributions of this paper.
3 The Proposed Algorithms
For simplicity, we present our algorithms with K 1. In this case, W E E = R and M, = MM,, For convenience, we set r = Ti. Additionally, we omit the superscript (k) throughout simplify the notation. Extending the case K
general K is straightforward.
Recall that the Riemannian Gradient Descent (RGD) algorithm can be written as:
W++1 R(W, M, ((W)),
(1I)
where V V., (WW) is the Riemannian gradient, which depends on the Riemannian metric of My, and R is retraction operator.
To develop practical and efficient RGD algorithms, we must select both a Riemannian metric and retraction operator RR. in (IT). For all algorithms, we choose the retraction operator RR to be the r-truncated singular value decomposition (SVD), i.e., R Hr. where H4 E M, defined as given the SVD of Z Σ: aqu.o.
H. (Z) - Ei 0,u;o
For the Riemannian metric, we either adopt the standard metric, resulting in the Plain RGD algorithm presented in Section 3.1 or use modified metric inspired by AdaGrad and AdamW, leading to the RAdaGrad and RAdam algorithms described Sections 3.2 and 3.3 respectively.
3.1 Plain RGD Riemannian Gradient Descent with the Standard Metric
We choose R = Mr and equip M, with the standard metric from the Euclidean space E. Under this setting, the Riemannian gradient is given by (,W) ((e) Vel(W). where p(e) denotes the standard orthogonal projection onto the tangent space Tw, and V. (WV( represents the standard gradient of in E.
With this setup, the Riemannian Gradient Descent (RGD) from becomes:
W H., - YP V. V.((W)))
(12)
where, for simplicity, T, = Tw,, We refer to the algorithm in [2) as the plain RGD. This algorithm serves as baseline for the RGD method, and we it demonstrate the RGD method in the simplest setting.
Throughout the computation, We is represented by its SVD W, U.E.V.T where D, grxr, U, E Rr and V, E xr are matrices containing the singular values and singular vectors, respectively. Below, we list the computations involved in the plain RGD algorithm (12)
Since G. = V.((W,) is the standard gradient in E, it be directly obtained using standard backpropagation.
With the help of U, and V,, the tangent space T2 is expressed as:
T TU,T + YV, E X YeixK,,YER Y
(13)
An orthogonal basis for Tr can be formed, under which the projection Pf. (G.) is computed as:
(G) (G.) = U,U,C G. + G.V.. U.u. G.V.V.
(14)
5

---

W Since Zz := Wt - P/o (G) E Tr is expressed in the form of 13 we only need to compute the co- efficient matrices U/ G., G.V, and U,! G.V, in the computation of P(.)) (G.). These computations can be efficiently performed using ((r) matrix-vector products with Gt.
Since Ze E T₁ and every matrix in T₃ has rank at most 2r, the operator (Z,) finds the best rank-r approximation of rank- 2r matrix. This computation involves two QR decompositions of size x 2r and n1 x 2r, one matrix matrix-matrix product of size 2r x 2r, and one SVD of size 2r x 2r. The total computational cost for these operations is ((mo + m1)r² + r+).
In summary, the computation in 12). in addition evaluating Ve((W)), involves O(r) matrix- vector products and an additional ( + n2 nrrr² + r²) operations. The memory complexity is ((1o + )r)r.. These complexities are of the same order those of mecooiizttion-basee algorithms. More details are provided in Appendix
3.2 RAdaGrad - Riemannian Gradient Descent with an Adaptive Data-Driven Metric
RGD can be accelerated by choosing an appropriate adaptive metric on the Riemannian manifold M,. Specifically, we can define new weighted inner products (X, Y)g, = AXY) for any X,Y E Te on each tangent space Т, of the Riemannian manifold, where A T, Te linear operator serving as weights for the inner product.
As we have previously argued, the new metric should be able to reduce the condition number of V2, ((W,,, which ultimately translates to reduction the condition number of V²&(W,, under the metric T, extended to E. With the low-rank constraint, PRGD 3 also demonstrates the effectiveness of defining a new metric on Te, When the low-rank constraint is absent, many classical optimization algorithms, such as AdaGrad [2), Adam [26] 29] and Shampoo [16], inherently define a metric on E that reduces the condition number of V ) by applying different learning rates to different components of the gradient. We modify and restrict such a metric E to Tę to obtain
Riemannian metric that reduces the condition number of VM, (W,).
At step f, any metric in E is in the form of
(X, f..)... g: (X,AY) AX,Y
(15)
where A, : E + E is linear operator serving as weights for the inner product. Ideally, the best choice for A, would be multiple of VVr(W, the Euclidean Hessian of é under the standard metric in E. However, computing ver(W,, is very expensive. A common approach [9]. [31 Chapter 10, 11, and 12] to the accumulation of the outer products of G, V (WW,). i.e., choose Ar = (E, (Ge, +G,) Nevertheless, computing the gradient under this metric involves the inverse of Ae, which computationally expensive and may even be infeasible. To address this issue, further approximate At using several efficient methods:
AdaGrad 12
and AdamW 26,
29],
we use a diagonal operator to approximate A,, yielding
A, = OY,
- = [GI))) for i 1,.... no, j 1,...,,y n1- (16)
Here, o denotes entry-wise multiplication, and E small positive number for numerical stability. The inverse of A, is simply the entry-wise multiplication by the inverse of each entry of A,
In Shampoo [6]. we exploit the matrix structure of the variables by using the Kronecker product approximate A,, which gives
A. R. where T.=eT+ +,
(17)
Here again, small positive number for numerical stability. The inverse of Ae is simply A y L, R
-
We aim to utilize both the matrix structure of the variable W and the easy invertibility of diagonal approximations. Thus, we combine both approximations and use the following A,, which both diagonal and in Kronecker product form:
A. Y R², where L, =e
el + E, diag(G, = + E, diag G. Gt). (18)

---

With this new metric on E, the condition number of the Hessian (2((W; is reduced. Then, we restrict this metric onto T, to obtain our Riemannian metric, under which the condition number of Vi,,((W)) ((W,) is small since it is approximately bounded by that of (2)(W,) More explicitly, the Riemannian metric we used here is: For any X,Y E TT, we define their inner product through (15)
with A, from 17
Under this new Riemannian metric, the Riemannian gradient
(g) M. e(w,) = ( T. (Vge(W.)) = P (A, = P (
(19)
where p( denotes the orthogonal projection onto the tangent space T, under the weighted inner product in [5 and 18). We choose R Hr and apply the general RGD framework in [IT) This gives
G₁ vel(W,);
La B1 (1 31 ddjagg (G.G))
(RAdaGrad)
R2 32R, + +11 B2) (G. Gt),
(20)
W4+1 = Hr W. - -PPi (Li R, 1)
Here, we introduce the parameters 31 and B2 to use a weighted accumulation instead of direct accumulation (17) and AdaGrad, providing more flexibility. The proposed algorithm is referred to
as RAdaGrad.
Similar to the plain RGD, throughout the computation of RAdaGrad, W, is represented as its SVD W. = U,E,V Below, we list the computation in RAdaGrad (20):
Ge computed using standard backpropagation, the same as plain RGD.
L: and R2 are diagonal matrices whose diagonals are actually the squared row norms and column norms of Gr, respectively. Thus, it requires only computational complexity of (1i)))) or even lower if G, carries some structure.
Since the metric weighting operator A, is Kronecker product form, the column vector space ned and the row vector space "K are weighted separately by L and R2 respectively. To find an orthogonal basis of T,, only need to orthogonalize U, under T.द wwighhtd inner product obtain U, L and Vú under -eeihhttt inner product to obtain V. V.(V," R v)- Similar to [4]. the projection p(a) is given by:
- +
+ L. Ü,Ü, G.V.V,
(21)
Again, we only need to calculate the coefficient matrices U.G.R, G and U. G,V, in the computation of (190) (L, G. +). These computations can be efficiently performed using (rr) matrix- products with Gt.
The computation of (Z,) is the same as in plain RGD, with a total computational cost of O(no n1)r² + r
In total, the computation of RAdaGrad is, in addition to evaluating G., O(r) matrix-vector products with G, and extra O(nn, + nr)r² r²) operation. The memory usage O((ng + n))r). Both complexities are in the same order as factorization-bass algorithms. The detailed calculations can be found in the Appendix B
3.3 RAdam W - RGD with an Adaptive Momentum and Weight Decay
Momentum commonly used acceleration technique for gradient-based algorithms. Adam has demonstrated certain advantages on convex problems and the Stiefel manifold [2 36 41 To further
enhance the performance RAdaGrad, we introduce the momentum from Adam 26 29) into the RAdaGrad algorithm.

---

In fact, RAdamW can be regarded as variant of the Riemannian Conjugate Gradient (RCG)
algorithm. The resulting RAdamW algorithm is as follows
Gt = Vel(Wi),
La B1 + (1 -1 (:)) ).
(RAdamW)
Rs = B2R -1 1122) Gt),
(22)
Mr = 83 + (1 B -01--80
We+1 - Hr (11 (1- -1-00) -),, TPP (LC (,RR
where Mr represents the update direction at step, which is a linear combination of the original gradient G2 and M,-1. Its detailed computation is almost same with RAdaGrad. Here we omit the detail.
Experiments
To comprehensively evaluate the performance of the RAdaGradd and RAdamW algorithms, we conduct experiments across range of tasks, including fine-tuning large language models (GPT-2
33 ). fine-tuning diffusion models (Mix-of-Show [14] and Stable Diffusion V1.5 (55). and deep
neural network (DNN) compression tasks.
For the large model fine-tuning, we compare two kinds of optimization algorithms: SGD-based algorithms and Adam W-based algorithms. The SGD-based algorithms include LoRA using SGD (denoted as SGD) [21]. ScaledGD 440 47), plain RGD [45. and RAdaGrad (Algorithm 20 This comparison allows for an evaluation
the improvements brought by RAdaGrad over conventional SGD approaches. On the other hand, the AdamW-basee W-based algorithms include AdamW (vanilla LoRA)
21. ScaledAdam 40 47 and RAdam (Algorithm 3I). By comparing these AdamW-based algorithms, we assess the performance advantages of RAdam
We validate the effectiveness of the proposed algorithms in DNNs compression the MNIST and CIFARIC 10 datasets using fully connected and convolutional networks. The experimental results (detailed in Appendix ED demonstrate that RAdaGrad and RAdamW, as plug-and-play optimizers, can be seamlessly integrated into various network architectures. The large model fine-tuning experiments are conducted on the NVIDIA A100 GPUs, while the DNN compression experiments are performed on a system equipped with an AMD Ryzen 7800X3D 8-core CPU and an Nvidia RTX 4090 GPU, The code for all our experiments can be found in the supplementary materials.
4.1 Large Models Fine- Fine-Tuning
In this section, we evaluate RAdaGrad and RAdam algorithms on large model fine-tuning, We present the experimental results for these algorithms on the fine-tuning of diffusion models (Mix-of- Show, see sections 4.1. and C.3) and Stable Diffusion,sec Appendix C.2) and large language models
(GPT-2, see sections 4.1.2 and
4.1.1 Mix-of-Show
This section preforms our experiment details on Mix-of-show model 14 We use OpenAl's clip- vit- hase-patch16 16 model to calculate the CLIP score, measuring the consistency between generated images and text prompts. Frèchet Inception Distance (FID) is commonly used metric for evaluating class-conditioned generative models. The specific parameter settings for each algorithm
provided in Table g in Appendix C.3 During the experiments, we employ a linear learning rate schedule and set the rank evident that RAdaGrad and RAdamW significantly outperform other algorithms, whether they are baDeb or Adam W- basdd... Specifically, when = 1 or 4 ,RAdamW W significantly outperforms Sealed Adam W, achieving nearly 1 point higher CLIP score and about 10 points lower FID score, demonstrating the effectiveness of our proposed Riemannian algorithms.
1, 4. The CLIP and FID scores are reported in Table From Table it is
4.1.2 GPT-2
We evaluate the performance of RAdaGrad and RAdamW optimizers on GPT-2. To ensure fairness, all hyperparameters except for the learning rate and weight decay are kept consistent, and the optimal
8

---

Table I: Experiments for EDLoRA on Clip and FID score.
Metrics
Metries
Methods
Rank
Rank
CLIPt
FID
CLIPP
FID
SGD Scaled GD
22.93
68.43
4
27.88
66.57
32.19
53.75
4
32.14
48.65
plain RGD
32.56
69.39
4
32.94
52.07
RAdaGrad(ours)
32.84
50.45
33.95
47.01
AdamW
32.84
53.11
33.75
52.16
Scaled Adam
31.94
66.77
4
33.61
60.71
RAdam Wours)
33.02
50.84
4
34.69
51.14
combinations are determined via grid search (complete implementation details and hyperparameter settings can be found in Appendix C.I. 1. On the E2E natural language generation benchmark [23), fine-tune GPT-2 small with ranks of 4, 8, and 16. The results are summarized in Table
The experiments demonstrate that RAdaGrad and RAdam consistently outperform other methods across all evaluation metrics, validating the efficiency and accuracy of our proposed framework in LLMs fine-tuning. Moreover, all RGD-base based algorithms generally outperform all other algorithms, further demonstrating the advantages of the RGD framework effectively leveraging the smooth manifold structure of low-rank matrices and adaptive metrics. Notably, RAdaGrra achieves perfor- mance improvements with reduced memory compared to Adam W-based methods, as it avoids storing historical G matrices. Meanwhile, RAdamW surpassed both AdamW and scaled AdamW across all evaluation metrics. This performance gain can be attributed to its combination of Riemannian geometry-aware updates and adaptive learning rates based first- and second-moment estimation.
Table 2: Scores for LoRA fine-tuning (r Generation challenge.
4, 8 and 16) of GPT-2 on E2E Natural Language
E2E
rank
Method
BLEU
NIST
MET
ROUGE-L
CIDEr
SGD
66.6
8.54
44.2
68.2
2.32
Scaled GD
68.5
8.72
45.5
69.4
2.40
plain RGD
69.2
8.77
45.7
70.1
2.45
4
RAdaGrad (ours)
69.8
8.80
46.5
71.1
2.49
Adam W
69.1
8.75
46.0
70.5
2.47
Scaled AdamW
69.5
8.80
46.2
70.9
2.48
RAdam W (ours)
69.8
8.81
46.5
71.1
2.51
SGD Scaled GD
65.8
8.46
43.5
68.7
2.33
68.8
8.75
45.3
69.4
2.43
plain RGD
69.1
8.73
46.0
70.7
2.45
8
RAdaGrad (ours)
70.1
8.80
46.8
71.7
2.51
Adam W Scaled AdamW
69.4
8.77
46.2
71.0
2.46
69.7
8.80
46.5
71.3
2.52
RAdamW (ours)
70.3
8.82
46.5
71.8
2.53
SGD Scaled GD
65.4
8.07
40.7
67.0
2.07
68.8
8.75
45.0
69.2
2.39
plain RGD
68.7
8.67
46.1
70.8
2.44
16
RAdaGrad (ours)
70.3
8.84
46.6
71.8
2.52
Adam
69.5
8.77
46.4
71.2
2.48
Scaled AdamW
69.8
8.79
46.5
71.7
2.51
RAdam W (ours)
70.1
8.85
46.6
71.6
2.52
Conclusion
Most existing algorithms for finding low-rank weights in large models are based matrix factor- ization and are constrained by the condition numbers the Jacobian and Hessian operators. To address these limitations, propose general RGD framework for efficiently finding low-rank

---

matrix weights in DNNs. This framework avoids the redundant representation introduced by matrix factorization, avoids the influence of the Jacobian condition number, and mitigates the impact of the Hessian condition number by selecting appropriate metrics. Inspired by AdaGraa and Adam W we extend their metrics to Riemannian manifolds and propose two new algorithms: RAdaGrad and RAdamW. These algorithms completely avoid the influence of the Jacobian condition number and significantly alleviate the effect of the Hessian condition number.
Through experiments on large model fine-tuning and DNN compression tasks, our plug-and-play optimizers are shown to outperform state-of offee-of-uae -fe-------hhheeee the-hharr art methods and are applicable various network architectures. Our work not only introduces RAdaGrad and RAdam W but also establishes a unified framework that provides novel method to finding the low-rank weights and offers fresh perspective for designing efficient algorithms. In the future, we plan to extend this framework to address the problem of finding low-rank tensor weights DNNs.
Limitations.
The rank-r Riemannian manifold constraint in lank adaptation methods inherently limits their expressiveness compared to full palammeeeet fine- tuning.
Broader Impact. The proposed RAdaGrad and RAdamW optimizers demonstrate broad compati- bility across diverse DNN architectures, which is useful to fields like healthcare, farming, and schools, especially in places with limited resources.
Acknowledgments and Disclosure of Funding
The authors would like to thank the anonymous referees very much for their careful reading and valuable comments, which greatly improved the quality of this manuscript. Jian-Feng Cai is partially supported by Hong Kong Research Grant Council GRF 16307023, GRF 16306124, and GRF 16307325.
References
[1] Absil, R. Mahony, and R. Sepulchre. Optimization Algorithms on Matrix Manifolds. Princeton University Press, Princeton, NJ, USA, 2008. ISBN 17780691132953 doi: 10.1515/9781400330222 400830244.
[2] G. Becigneul and O.-E. Ganea. Riemannian adaptive optimization methods. In International Conference Learning Representations (ICLR), 2019.
[3] F. Bian, J.-F. Cai, and Zhang. A preconditioned riemannian gradient descent algorithm low-rank matrix recovery. SIAM Journal on Matrix Analysis and Applications, 45(4):2075-2103, 2024.
[4] S. Boyd and L. Vandenberghe, Convex optimization. Cambridge university press, 2004.
[5] L. Cambier and P. Absil. Robust low-rank matrix completion by riemannian optimization. SIAM Journal Scientific Computing, 38:440-460, 2016.
[6] A. Canziani, A. Paszke, E. Culurciello. analysis of deep neural network models practical applications. arXiv preprint wxXiv:1605.06611 2016.
[7] E. Cervenka. Naruto blip captions.. 2022.
[8] L.-C. Chen, G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 4044)833-848, 2017.
[9] F. Curtis, J. Nie, and J. Reyes. A matrix-free approach for approximating the hcssian in optimization problems. SIAM Journal on Optimization, 2761):66-89 2017.
[10] L. Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6): 141-142, 2012.
[11] E. Denton, W. Zaremba, Bruna, Y. LeCun, and R. Fergus. Exploiting linear structure within convolutional networks for efficient evaluation. In Advances in Neural Information Processing Systems, (NeurIPS), pages 2014.
10

---

[12] J. Duchi, E. Hazan, and Y. Singer. Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Rexearch, 12(3):121121559. 2011.
[13] G. Golub and C. Loan. Matrix Computations. Johns Hopkins University Press, 2013.
[14] Y. Gu, X. Wang, J. Z. Wu, Y. Shi, Y. Chen, Z. Fan, W. Xiao, R. Zhao, S. Chang, W. Wu, Y. Ge, Y. Shan, and M. Z. Shou. Mix-of-show: show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. In Advances in Neural Information Processing Systems, (NeurlPS), 2023.
[15] S. Guo, S. Damani, and K. Chang. Lopt: Low-rank prompt tuning for parameter efficient language models. arXiv preprint ar arXiv:2466. 199469966..6 2024.
[16] V. Gupta. T. Koren, and Y. Singer. Shampoo: Preconditioned stochastic tensor optimization. In Proceedings of the 35st International Conference on Machine Learning (ICML), pages 1842-1857 2018.
[17] S. Han, J. Pool, J. Tran, and W. Dally, Learning both weights and connections for efficient neural network. In Advances in Newral Information Processing Systems, (NeurlPS), pages 1135- 1143, 2015.
[18] S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation large models. In Proceedings of the 41st International Conference on Machine Learning (ICML), pages 17783 - 17806, 2024.
[19] K. He, Zhang, Ren, and Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 1(5):770-778, 2016.
[20] K. He, G. Gkioxari, Dollar, and Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pages 9961-2969, 2017.
[21] E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022.
[22] M. Jaderberg, A. Vedaldi, and A. Zisserman. Speeding up convolutional neural networks with low rank expansions. In Proceedings of the British Machine Vision Conference (BMVC), pages 1-13, 2014.
[23] N. Jekaterina, D. Ondřej, and R. Verena. The ele dataset: New challenges for end-to-end generation. arXiv preprint arXiv:1706.092: 2017.
[24] C. Jin, Y. Li, M. Zhao, S. Zhao, Z. Wang, X. He, L. Han, Che, and D. Metaxas. Lor-VP: Low-rank visual prompting for efficient vision model adaptation? In International Conference on Learning Representations (ICLR), 2025.
[25] E. Kieri, C. Lubich, and H. Walach. Low-rank matrix completion by riemannian optimization. SIAM Journal on Numerical Analysis, 54:1020-1038, 2016.
[26] D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference on Learning Reprexentations (ICLR), 2015.
[27] V. Lebedev. Y. Ganin, M. Rakhuba, I. Oseledets. and V. Lempitsky. Speeding-up convolutional neural networks using fine-tuned op-dccomposition. In International Conference on Learning Representations (ICLR), 2015.
[28] 7. Liu, B. Wu, W. Luo, X. Yang, W. Liu, and K.-T. Cheng. Bi-real net: Enhaneing the performance of I-bit cnns with improved representational capability and advanced training algorithm. In Proceedings of the European Conference on Computer Vision (ECCV), pages 722-737, 2018.
[29] Loshchilov and F. Hutter. Decoupled weight decay regularization. International Conference on Learning Representations (ICLR), 2019.
[30] Z. Mo, L. Huang, and J. Pan. Parameter and memory efficient pretraining via low-rank riemannian optimization. International Conference on Learning Representations (ICLR), 2025.
[31] J. Nocedal and S. J. Wright. Numerical Optimization. Springer Science & Business Media, 2nd edition, 2006.
[32] J. Pennington, S. Schoenholz, and Ganguli. The emergence of spectral universality in deep networks. In Proceedings of the 41st International Conference on Machine Learning (PMLR), 2018.
[33] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei. and I. Sutskever. Language models unsupervised mul- titask learners. 2019. URL httt:sttt://app.. semant www.entcsccoo.....ooo.. wtw.:llaa...rgoooouuu...... 160025533
[34] R.-T. Rockafellar. Convex analysis, volume 28. Princeton university press, 1997.
11

---

[35] R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer. High-resolution image synthesis with latent diffusiin models. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pages 10684-10695, 2022.
[36] S. Kumar Roy, Z. Mhammedi, and M. Harandi. Averaging stochastic gradient descent riemannian manifolds. Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR). pages 4460-4469. 2018. 2018.
[37] R. Saha, Srivastava, and M. Pilanci. Matrix compression via randomized low rank and low precision factorization. In Advances in Neural Information Processing Systems (NewrPPS), 2023.
[38] S. Schotthöfer, E. Zangrando, Kusch. G. Ceruti, and F. Tudisco, Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In Advances in Neurul Information Processing Systems, (NeurlPS), pages
2022.
[39] W. Thomas, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz, and J. Brew. Hugging face model hub. https://ugginggueee.e wwt.r//focscccooooofforrrrr 2020.
[40] T. Tong, C. Ma, and Y. Chi. Accelerating ill-conditioned low-rank matrix estimation via scaled gradient descent. Journal of Machine Learning Research, 22(150):1-66 2021.
[41] N. Tripuraneni, N. Flammarion, F. Bach, and M. L. Jordan. Averaging stochastic gradient descent on riemannian manifolds. In Proceedings of the 41xt International Conference on Machine Learning (ICML). pages 650-687, 2018.
[42] B. Vandereycken. Low rank matrix completion by riemannian optimization. SIAM Journal on Optimization, 23:1214- 1236, 2013.
[43] T. Vogels, S. P. Karimireddy, and M. Jaggi. Powersgd: Practical low-rank gradient compression for distributed optimization. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[44] K. Wang, Z. Liu, Y. Lin, J. Lin, and S. Han. Haq: Hardware- automated quantization with mixed precision. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 8612-8620, 2019.
[45] K. Wei, J.-F. Cai, T. Chen, and S. Leung. Guarantees of riemannian optimization for low rank matrix completion. Inverse Problems and Imaging, 14(2):233 265. 2020.
[46] J.-N. Yen, S. Si, Z. Meng. F. Yu, S. Duvvuri, I. S. Dhillon, C.-J. Hsieh, and S. Kumar. LoRA done RITE: Robust invariant transformation equilibration loRA optimization. In The Thirteenth International Conference on Learning Representations (ICLR), 2025.
[47] F. Zhang and Pilanci. Riemannian preconditioned lora fine-tuning foundation models. Proceedings of the 41st International Conference on Machine Learning (ICML). 2024.
[48] T. Zhang, S. Ye, K. Zhang. Tang, W. Wen, M. Fardad, and Y. Wang. systematic dan weight pruning framework using alternating direction method of multipliers. In Proceedings of the European Conference on Computer Vision (ECCV), pages 184-19% 2018
[49] W. Zhang, J. Liang, R. He, Z. Wang, and T. Tan. Lora-pro: Are low-rank adapters properly optimized? In International Conference on Learning Representations (ICLR), 2025.
[50] X. Zhang, J. Zou, K. He, and J. Sun. Accelerating very deep convolutional networks for classification and detection. IEEE transactions on pattern analysis and machine intelligence, 38(10):1943 1990-1959999,,, 2015.
[51] Z. Zhu, Y. Wu, Q. Gu, and V. Cevher. Imbalance-regulaiizedd lora: A plug-and-play method for improving fine-tuning of foundation models. The 38th Conference on Neural Information Processing Systems (NeurIPS), 2024.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
This paper's contributions and scope are reflected.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Provided in Section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
13

---

Justification:
The theoretical analysis is provided in Section B and supplementary materials.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
The implementation details can be found in Appendix C and
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
14

---

Answer:
[Yes]
Justification:
Code is provided in the attached files.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm cc/
public calic//uiieee CodeeSuie ss onPolicy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https?
//nips. w/npssccppubbll wttpccccpppppppcteeececccccccccp cesii//uuiie///eeeeeeeoe.cseseee ies CodeeSume ss cioooooliio ooooicyyy icy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
The implementation details can be found in Appendix and
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: Due to constrained computational resources, the experiment conducted only once or twice.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
15

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The computer resources are provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
This paper conforms with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Provided in Section 5
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
16

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not have such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The assets cited and the license and terms respected.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, papers thcode com dataaetteett has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
17

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: This paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This paper does not involve crowdsourcing nor search with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor search with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
18

---

A Computational Details in the Plain RGD
In this section, we provide the computational details for plain RGD (see Eq. (12) in Section 3.1 of the main text). Recall that plain RGD generates the iterates via
W++1 = Hr (wo- v.l(w...
(23)
where p is the orthogonal projection from I onto the tangent space T,, and ve((W,) denotes the standard gradient of the loss function l in I..
Throughout the computation, the rank-r weight matrix W, is represented by its singular value decomposition (SVD) Wt V, where Σ, x"* is diagonal matrix whose diagonal elements are the singular values of W,, U2 E R"ox" contains the left singular vectors, and V2 E R" x contains the right singular vectors.
Computation of G2 Ve,(W,) Since G, is the standard Euclidean gradient of the loss function in E, can be computed efficiently via standard backpropagation.
Computation of Z W, yP(.. (G,). We first establish representations for the matrices involved in computing Z..
Representation of P(() (G,) in terms of U,, Vi, and G., Let V R(* and U, (m1-r) be the orthogonal complements of Vi and Us, respectively. The tangent space Tr admits the decomposition:
T. = [U,XT + YV,
Xr Y [R""XT)
= (UKK,, K E Rx @ KVV, K2 E (a(00-))
(UrKs(V;-) K3 E 1=x((1--r).
Using this decomposition, the projection
(G.) evaluates to:
p (G.) = U,U, G.V.V + +U,U, G.V, + + =U,U, G.VVV' + +UU -U-
(24)
U.U
Expression of Ze in terms of UC, Vi, E,, and G.. Using the projection, can
reformulate Z, as
2, = U.E.VV -U.T G.V.V - -UU,U Ge(T VVV,,) - 7(1 -UU--U,,,))NN
= U, ( UU G.V.V. -UU,U G. -7 -7 (T- U.U.
Y
V2
U,YVV UUYT TY2
(25)
Therefore, to compute Ze we only need to compute the coefficient matrices Yo E grxr, Y1 Rr and Y2 R The detailed procedure is follows:
Compute G.V., GJ U., and U,T (G.V).
Compute Yo F,, - (U, G.V.).
Compute Y1 (GTU,, - ViUU, G.V)T and Y2 = (G.V) - U, G.V..
This procedure requires (nonir) operations and O + n) )r) memory.
Computation of (Z.). The rank 2, is at most 2r due to its structured representation:
7, = U Y2
Yo
7
1
ne 2r
2r
2rX01
Since U. Y2 = O. the QR decomposition Yg Q2R2 yields an orthognal matrix [U, Q2] E
IR 2r Similarly, the QR decomposition Y1 Ri gives orthognal matrix (V/ Q1
19

---

R11x2r This allows us to rewrite Ze as:
Zz = [ Ue Q2
Yo -2
R 0
VT Q1
(26)
2rx2r
where both the left and right factors are orthogonal matrices. Therefore, computing the SVD of Ze reduces to finding the SVD of the central 2r 2r matrix. By retaining only the leading r singular values and corresponding singular vectors, we obtain the rank-r approximation W = (Z,) in its SVD form. The complete computational procedure consists of the
following steps:
Compute QR decompositions Yi = Q1 R1 and Y2 Q2R2.
- Compute the SVD
Yo
R
CAD
R7
0
Compute
U4+1 UU Q2) C(:11:) Σ++1 :r)))), V4+1 = [V. Qi) D(:, Irr)
The whole procedure requires two QR decompositions, one SVD of size 2r 2r, and two matrix-matrix products. The computational complexity is O( + nı )r)r",,, and the memory used is O(no + n1)r).
B Computational Details in RAdaGrad and RAdamW
In this section, we provide the computational details of the Riemannian Gradient Descent with an Adaptive Data-Driven Metric (RAdaGrad, Equation (20) in Section 3.2 of the main text). Recall that the standard RAdaGrad generates updates using the following formula:
G. Vee(W,),
Le = 3141-1 + -1- +11-B1)diag (GG )
(RAdaGrad)
RG = 32 P2 1 + (1 diag (G, Gt), W4+1 = Hr (Wr (P (Li + G,R,
(27)
where p(9) denotes the orthogonal projection from E to the tangent space T, under the weighted inner product (X, (X.Y),. = (X, L R for X, Y e E, and Ve((W,) represents the standard gradient of the loss function in the Euclidean space IE.
We now present the computational details the RAdaGrad algorithm. The matrix We is expressed using its singular value decomposition (SVD) as W, U,E,V. where E, Rxx is diagonal matrix whose diagonal elements are the singular values of W., and U, E Ro and V, E R Xr are the left and right singular vector matrices, respectively. The computations involved in RAdaGrad outlined below.
Computation of G. V.l(W.). In RAdaGrad and RAdamW, G, is also the standard Euclidean gradient of the loss function in E. Therefore, similar to the plain RGD, it can be directly obtained through backpropagation.
Computation of La and RG- According to (7). L, and R2 are diagonal matrices defined by the squared column 2-norms and squared row 2-norms of GG respectively. The computation involves the following steps:
Calculate the squared 2-norm of each column and each row of G
Update the diagonal entries of L, and RG using the squared column norms and row norms, respectively.
The total computational cost for these operations is Onon)))
Computation of Z, == W. - W (L, G, This is similar to the computation of Pf. (G.) plain RGD. We first establish representations for the matrices involved in computing Z,-
20

---

Expression P!() (I, T, This very similar to Ple) (G.) in plain RGD. The only difference is that we need to find an orthogonal basis for T2 under the weighted inner products. Notice that the weighting operator the weighted inner product (., a) is in Kronecker product form and is therefore separable in the column vector and row
vector spaces. Specifically, we orthogonalize U using the L -inner product to obtain U, = U,(U," U, à, and orthogonalize V, using the R2 -inner product to obtain V. = VVVV. R V As a result, we have:
Pl (4, G.R, Ü,Ü, G,R, + + LLiiG. -,U.
(28)
Representation of Ze terms of Ul, V1, Et, and G.. Using the projection, we can reformulate Z4 as:
( (1 + Z = W. -PI, (L,
U,EV ,ū,U, C, - +-ū,Ü, G.V.V.
Each term can be expressed in terms of U, Ve, and Ex:
Ü,Ü G,R, U, (U, L;U,)- G.R,+ U,Ki T G.R,
Ki
* L G..G.VVV = L V.VV R) Vj = = G.KVV.
V.V.
K2
* 0,Ü, G.V.V = U.Ki T.
Substituting these into the expression for Ze, obtain:
Z: = W, - (L, & ²)
- -U "G -T, G.LV -UUKTTG.K
Us (2: (E:--1Ki GrR, VV VV, L5'G,Ke + rKT V,
(29)
Yo
-UU K1 T.R. (I V.V. -7 (I- U,U D. )
Y == U.YOV - -UUYY -YVT
Therefore, to compute Zr, it is sufficient to calculate the coefficient matrices Yo E R Xr and Y2 E Rnoxr, The detailed procedure is as follows:
Compute K, = U,, L U,) and K₂ = VVVV, -1
Compute G. K1, K2 and KT (G.K2).
Compute L, (G) and R, (G Kj) by rescaling each row.
Compute Yo Σ, (KK G.R, *)V, + U, (L, G,K2) [K] GrK2)). Compute Y1 (Ki G.R ((Ki T G. VT and Y2 = (L, GG,Ke)
Since L, and RG are diagonal matrices, the multiplications involved in L, and R rescale the rows and columns, making them computationally cheap. The entire procedure requires O(nonr)) operations and O( + ni)r) memory.
Computation of (Z,). From 29 it follows that Ze can be rewritten in the following form:
Yo
1
Z,
Ur
Y2
(30)
X2r
2r
2rxR1
21

---

with Y2 and V. Y1 0. This form is exactly the same as 26 in plain RGD, except
for different yo. Y, and Y2 Therefore, the computation of W (2,),) ( is identical to that of plain RGD. As such, we omit the detailed computational steps here. Similar to plain RGD, the computational complexity is (( m))r)) and the memory usage O(mo + m))r).
Regarding the computational details of the RAdam W algorithm (see Equation (22) in Section 3.3 of the main text), the RAdamW algorithm is as follows:
G. Vel(W,),
L = 81L4-1 + -1 (i)gg
(RAdamW)
R& = B + (1 D2)diag(G)) Gt),
(31)
Mr B- +1 -B
Wi+1 Hrr (1L (1--8W- MP (L, MR
Note that compared to the RAdaGrad algorithm, the only difference is that G, in RAdaGrad is replaced with M, in the last line. As a result, the computational procedure remains almost the same. For this reason, we omit the computational details of the RAdam W algorithm here.
C
Supplementary Experiments for Large Models Fine-Tuning
C.1
GPT-2
C.1.1
Sensitivity Analysis of ayperparameters
To evaluate the stability of the proposed optimizer, we test the sensitivity of RAdaGrad and RAdam the parameters on GPT-2. Specifically, we validate their sensitivity weight decay, learning rate, and smoothing parameters, with the results shown in Table Table 4 and Table respectively. The results indicate that all metrics exhibit slight fluctuations within the error range.
In detail, Table presents the training loss and evaluation metrics under different weight decay settings. For these experiments, RAdaGrad's learning rate was fixed at 5e-3, RAdamW's learning rate was fixed at 8e-3, smoothing parameters were set to 31 = 32 0.98, and the rank was set to Table shows the training loss and evaluation metrics under varying learning rates, with weight decay fixed at le-2, smoothing parameters set to 33 0.9, 31 = B₂ 0.98, and the rank set to 4. Table provides the training loss and evaluation metrics for diffeernnt smoothing parameters 31 82. with the learning rate fixed at 8e-3, smoothing parameter B3- = 0.9, weight decay set to le-2, and the rank set 4.
Table Training loss and evaluation scores for the GPT-2 fine-tuned model (r 4) under varying weight decay schedules.
E2E
Methods
Weight Deacy
Training Loss
BLEU
NIST
MET
ROUGE-L
CIDEr
le-2
2.56
69.8
8.80
46.5
71.1
2.49
RAdaGrad
le-3
2.56
70.0
8.82
46.5
71.2
2.51
le-4
2.56
69.7
8.79
46.5
70.9
2.50
le-2
2.56
69.8
8.81
46.5
71.1
2.51
RAdamW
le-3
2.56
68.8
8.76
45.5
70.1
2.42
le-4
2.56
70.0
8.83
46.3
71.2
2.48
C.1.2 Training Loss Curve
To further demonstrate the advantages of RAdaGrad and RAdamW, we compare the loss within the same runtime, and the results are shown in Figure These results highlight the effectiveness of the proposed optimizers.
22

---

Table 4: Training loss and evaluation scores for the GPT-2 fine-tuned model (r = 4) under varying learning rates.
E2E
Methods
Learning Rate
Training Loss
BLEU
NIST
MET
ROUGE-L
CIDEr
le-2
2.55
69.5
8.76
46.9
71.4
2.52
RAdaGrad
5e-3
2.56
69.8
8.80
46.5
71.1
2.49
le-3
2.59
69.3
8.79
45.9
70.5
2.44
le-2
2.57
69.2
8.76
46.0
70.7
2.46
RAdamW
5e-3
2.56
69.1
8.75
46.1
70.7
2.47
le-3
2.59
69.2
8.76
45.8
70.2
2.44
Table Training loss and evaluation scores for the GPT-2 fine-tuned model (r = 4) under varying smooth parameters.
E2E
Methods
Parameter 32
Training Loss
BLEU
NIST
MET
ROUGE-L
CIDEr
0.96
2.56
69.3
8.81
45.9
70.5
2.45
RAdam W
0.98
2.56
69.8
8.81
46.5
71.1
2.51
0.99
2.56
69.2
8.78
46.0
70.8
2.47
C.1.3 Parameter Setting
Our training and inference configurations for GPT-2 fine-tuning (Table maintain full consistency with the experimental setup in [47. The model is trained using a linear learning rate schedule over epochs. Optimizer hyperparameters are systematically outlined in Table with key adaptations based on empirical observations:() plain RGD Stabilization: To address the comparatively large learning rate of plain Riemannian Gradient Descent (RGD) and mitigate potential training instability caused by gradient updates, we reduced the weight decay parameters. (2) RAdam W Configuration: The first-order moment parameter Bg is fixed at 0.9, while grid search performed for the second- order moment parameter 31 B2 within the set 0.96, 0.98, 0.99, 0.999.
C.2 Stable Difftusion V1.5
Diffusion models are widely used in image generation tasks, and LoRA has become a common technique for fine-tuning diffusion models. We take the commonly used Stable Diffusion V1.5 model as an example to demonstrate the effectiveness of RAdaGrad in LoRA fine-tuning for object generation. The experiments conducted on three datasets from Huggingface open-source models [39] naruto-blip blipto--bii--aptions (7]. flowers- flower-blip-caption blip-captions [39, and simpsons-blij blip-captions, with corre- sponding prompts "a hellokitty with naruto style", A woman in long hair in <simpsons>> and "a
Training Loss Curves for SGD Based Optimizers
Training Loss Curves for AdamW Based Optimizers
3.2
Tuising Loss (ong)
3.0
Training Loss (Ong)
Scaled GD
Sciled Adamik RAdann
3.0
RAdaCtra
2.8
Loss
2.8
2.6
2.6
0
1000
2000
3000
4000
5000
2000
4000
6000
8000
10000
12000
Training Time (8)
Training Steps
(a) Training loss
training time when fine- tuning with rank 8 using SGD-based methods.
(b) Training loss training step when fine- tuning with rank using AdamW-based methods.
Figure 1:
Training loss of GPT-2 small model fine-tuned using different optimizers.
23

---

Table 6: Training and Inference Contiguration for GPT-2 fine-tuning.
Parameter
Value
Parameter
Value
Training
Inference
Dropout Prob
0.1
Batch Size
# Epoch
Beam Size
10
Warmup Steps
500
Length Penalty
0.8
LR Scheduler
Linear
No Repeat Ngram Size
Label Smooth
0.1
LoRA ar
Table Core Optimizer Parameters for GPT-2 fine-tuning,
Learning Rate 10 -3) rank 4
Methods
Weight Decay
B1
$2
rank rank 16
SGD
0.01
90
90
200
/
scaled GD
0.01
20
30
40
/
plain RGD
0.0001
80
100
200
/
RAdaGrad
0.01
5
8
40
0.98
Adam
0.01
0.2
0.2
0.2
0.9
0.999
scaled Adam
0.01
0.8
2
2
0.7
0.8
RAdamW
0.01
8
10
0.9
0.98
yellow flower". Tigure[ddssplas the generated results from the four algorithms, with all experiments training LoRA for 4000 epochs.
<naruto>
<simpsons>
<fowers>>
Scaled GD
Scaled Adam
plain RGD
RAdaGrad
Figure Generation results for LoRA <naruto>, <flowers>> and <simpsons>>> with different optimizers. Default optimizer is AdamW with default learning rate 1e for U-Net tuning and le 5 for text-encoder tuning.
The Simpsons-blip-captoon dataset contains 786 (image. description) pairs, and Figure 2 shows the final generated results from four algorithms. The image quality produced by all four optimizers is comparable, with each successfully capturing the distinctive style of the Simpsons animation. However, the intermediate results of the generation process highlight the advantages of RAdaGrad. As shown in Figure 3 at the 500th epoch, while the other algorithms had not yet produced clear images, the images generated by RAdaGrad already exhibited the strong stylistic features of the Simpsons, indicating faster convergence and greater efficiency.
Hiinsson--bli--appiee
C.3 Mix-Or-Shor
We begin by training the model using images of Potter provided by the original project repository, replacing the character names in the image captions with < VPotter >, The provided tokens include

---

results at 500th epoch
results at 2000th epoch
results at 4000th epoch
Scaled GD
Scaled
A
plain RGD
RAdaGrad
Figure 3: Generation results for LoRA <simpsons>> with prompt "A woman with long hair with <simpsons>>>> style." using different optimizers. It shows the changes in objects generated by different algorithms over training time.
Table 8: Hyperparameters for Mix-of-Show fine-tuning
plain RGD
RAdaGrad
RAdamW
4
4
Training
Mixed precision
fpl6
Weight Decay
0.0
le-2
le.4
Batch Size
# Steps
3500
Unet Kv Drop Rate
LR Scheduler
Linear
LR (tuned, x x10-1)
1e3
1e3
0.7
1.0 /
0.7 1.0
AdamW Bg
/
0.9
AdamW 31 ,2
/
0.999
0.999
LoRA
1.0
Inference
Mixed precision
fp16
Num Samples Per Prompt
10
Batch Size
4 [0, 0.7. 1.0] 50
Alpha List
Num Inference Steps
Guidance Scale
7.5
"blurry background", "upper body", "looking at the viewer", "holding wand", "standing", and others. Figure[]shows the generation results for the prompt "A Vrotter wearing a blue shirt" and Figure 5 shows the generation results for the prompt "A Votttr in front of eiffel tower" The images generated by RAdaGrad and R Adam W more closely resemble Potter. Most images generated by ScaledAdam W resemble Potter, but the eyes in the images influenced by the word "blue" and appear blue eyes, differing from the real eyes of Potter.
Further, fine-tune the model using 15 images of Hermione provided by the original repository, replacing the character names in the image captions with < VHermiioee >. The provided tokens include "blurry background", "upper body", "looking the viewer". "head tilt". "arms at sides", "brown background", "holding a wand", "standing". and others. Figure the results for the prompt "A < VHermiioee wearing red hat" and Figure 7 shows the generation results for the prompt "A photo < Vitermione
The results in Figure 6 indicate that RAdaGrad and RAdamW W significantly outperform the other algorithms, especially capturing facial features. The Hermione wearing a red hat generated by these two algorithms more closely resembles Hermione's 's real facial characteristies. However, the
25

---

facial features in the images generated by Scaled SGD and SGD appear highly unnatural and even fail to resemble Hermione herself. In Figure 7 the images of Hermione generated by RAdaGrad and RAdam W exhibit more distinctive characteristics of Hermione compared to other algorithms. In particular, the images generated by RAdaGrad display richer facial details, such as freckles on Hermione's s face, capturing her facial features more precisely. Similarly, RAdamW W also captures some details more finely. These all results highlight the powerful image-generation capabilities of RAdaGrad and RAdamW.
SGD
Scaled SGD
plain RGD
(AdaGrad(ours)
Adam
Scaled Adam
RAdam Wours)
Figure 4: Generation results for LoRA <Potter> with prompt "A VPoter wearing blue shirt." " using different optimizers.
26

---

SGD
Scaled SGD
plain RGD
RAdaGrad(ours)
Adam W
Scaled Adam W
RAdam W(urr))
Figure 5: Generation results for LoRA <Potter>> with prompt "A Vatter in front of eiffel tower." using different optimizers.
27

---

SGD
Scaled SGD
plain RGD
(AdaGrad(ours)
Adam
Scaled Adam W
RAdam W(ours)
e Staa
Figure 6: Generation results for LoRA <Hermione>> with prompt "A <Hermione>> wearing red hat." using different optimizers.
28

---

SGD
Scaled SGD
plain RGD
RAdaGrad(ours)
Adam
Scaled Adam W
RAdam W(ours)
Figure 7: Generation results for LoRA <Hermione>> with prompt "A photo of <Hermione>. " "using different optimizers.
29

---

D Code Implementation Details in Our Paper
The corresponding code for Algorithms 2 and can be found in Codes/ eeaae/eee /NLG/ srcc/
opt zer wwt.eerrrsutoeeeeeeeeeeeo py. The classes corresponding to these algorithms are listed below:
Algorithm - class plain RGD (Optimizen
Algorithm 2 class RAdaGrad (Opt imezz))
Algorithm 3 class RAdamW (Optimizer)
Algorithm 1 Pseudocode of plain RGD in Pytorch
Create lora_u, lora_s, lora_Vh in loralib/layer.p
lora_U nn-parameter (self (swf.. zeross zeros (out_features, r))
1ora_$ nn an.parameter (self. wewgh.....ttttt.eeeeee.ee (r))
1ora_ nn.parameter (self. zeros fn_tuurr)))
Initialize lora_U, lora_5, 1ora_Wh in loralib/layer.py
nn, nn. nn, on.init.oness ww.intt.oees__sef..rrra... nn. un.init.i kaiii..aimingguniea wnwuuuniiiuueeeeeeeeeeee (wwlf.see..lle. Vh, s-mat..sgrttia........ (5))
Use hook to catch the gradient of # in loralib/layer.pp M. wergsiiter_hookt (cok(())
Group trainable parameters into LoRa pairs optimizer, py.
for lora_U, lcra_$, lora Vh in pairwise iettrannbblepprrneeeee param_groups [rram_ggoupssappedd(("" (ppend(""parmmms" [lora_U, lora_,, lora_Vhl, "Ir": learning_rate))
Update ruies in optimizer.py for group in paran_groups:
U, S, G_W w_grad J gradient of compute the weight update component pro_grad = -group('lr'] * G_K
group "pparams"]
compute some matrices for later use Ut_T_inv = torch. inverse (U.I (U. @ U)
W_T_inv torch. inverse (vh @ vh.T)
compute matrix
Y1 uU_T_inv U.T 0 pre_grad (1 vh.r 0 vh)
Q1, к1 torch. 1inasg......g..1........
compute matrix KO
KO tirch.diag($) JJ_T_inv U.T 6 pre_grad @ Vh. T \
+ (U.I UU_T_inv U.T) @ pre_grad 8 Wh.T @ W_T_inv
compute matrix K2
Y2 (torch, eye (U. shapeele e U.I) 8 pre_grad Vh.I VV_
Q2, K2 torch. 1inalg.qr
SVD of a matrix size x 2r
U_m, $_m, Vh_m forch.linalg.swv [KKO, k1.T], [K2, 011)
Update parameters
U [U, Q2] @ ULm, = S_m, Vh = Vh_m 8 [[vh], [Q1.I]]
U C(:, :self.
S[ :self. rank], Vh Vh[:self rank, :]
30

---

Algorithm 2 Pseudocode of RAdaGrad in Pytorch
Create Tora_U, 1ora_s, Tora_v, Vh in toralib/layer.r
lora D an. paraneter (self weight zeros (out features, r))
lora_$ ww..parmmterrrrr wee. (r))
lora Vh am.parameterrsell. www.h..wwwgg......... zeros in features))
Initialize 1ora_,, 1ora_s, Vo in loralib/layer.py
teatt.terrssseeff.._.. nn, ontt..onss_ssll..orr.
nn wtt.:kiiiinnnniorommeecoc.. lera__, a-math. (5))
# Use book to catch the gradient of # in loralib/Layer.pt
W. reggisiter_hhokk)
Group trainable parameters into LoRA pairs in optimizer.pi py.
for lora_U, lora_s, lora_Vh in pairwise wtwp:eeetrainbbleparrraae (rrra_ggropsspppend(( "ir": learning_rate))
[lora_U, lora_S, vora_Vh],
Update ruies in optimizer.py for group in paran_groups:
v, S, Vn = "roupl"params"
G_w W_grad # gradient of
It bttal
R beta2=RL(t-1)- (1-betal) storch. einsum ('ij, G_W)
(1-beta2) storch.einsum (11j,1j->3', G_M, G_W)
compute the weight update component
pro_grad = -group(''rr)-----ppppp
diag (-0.25)) 0 G_W
0 torch. diag (R_t+* (-O.22))) compute some matrices for later use
UnU inv torch. inverse (U.T @ tcrch.diag (1_tx+00.0
U)
VRV inv torch. inverse (Vh @ terch. aarch.diag(fe (R_tv+0.255 @ Vh.T)
ULZ u.t torch. [l_t++0, 25) pre_grad
2RV pre_
torch.dia tR_ 8 Vh.T
# compute matrix K1
Y1 ULU_ime 0 UL2 8 (torch. (Vh. shape [0]) Vh.T 0 Vh)
@1, K1 = torch. wiabhliingggrrt
compute matrix Kо
K0 torch.
JLU inv ULZ è Vh.I (U.T ULu inv J.r terch. .25)) XRV VRV_inv
compute matrix K2
Y2 (torch. (U. shape((())) - U E U.T) 8 2RV W_T_inv
Q2, X2 = torch. 1inalg..i..ggggg.... (Y2)
SVD of a matrix of size 2r x 2r
U_m, S_m, Vh_m torch.linalg. avd [KO, K1.T], [32, 01])
Update parameters
[U, Q2] @ U_m, s_m, Vh Wh_m [[vh], [Q1_T]] Ul:, :self, rself.rank), S1 tself. :self.rank), Vh Val :self rank, :
E
Supplementary Experiments for DNNs Compression
E. DNNs Compression MNIST Dataset.
To evaluate the performance of the algorithms, randomly divide the MNIST dataset [10 into training dataset with 50,000 samples and a test dataset with 10,000 samples. The image dataset is normalized wixel-wise, without any other data augmentation regularization techniques applied.
Five- Layer Fully Connected Network. first train a five-layer fully connected neural network on the MNIST dataset. The number of neurons for each layer is (5120, 5120, 5120. 5120, 10). To achieve DNNs compression, we employ a dynamic rank adjustment strategy (25) from DLRT for the weight matrix of each layer. This strategy adaptively truncates the singular values in = diag(o.) based on a parameter т and selects the smallest r x r submatrix that satisfies a specific condition (2)) < r. The initial rank for weight matrix of each layer is set to (500, 500, 500, 500, 10). We set the optimal dynamic rank adjustment thresholds 7 for plain RGD, RAdaGrad, and DLRT to be 0.5, 0.5, and 031 respectively. Notably, the 7 value for DLRT is smaller because, during experimentation, we observe that larger or smaller values led a decrease in test accuracy for DLRT. 0.13 is identified as the optimal threshold for maintaining high test accuracy.
31

---

Algorithm 3 Pseudocode of RAdamW in Pytorch
Create Tora_U, Tora_s, Tora_v, Vh in coralib/layer.g
lora D an. paraneter (self weight zeros (out features, r))
lora_$ nn wwr.arraeterrr w (r))
lora Vh am.parameterrsel1
zeros in features))
Initialize 1ora_,, 1ora_s, Vo in loralib/layer.py
nn. nn. teatt...rcs_sse...1.1 nn, ontt..nnssssell...rr. nn waiitkiggnnnngnoormm////// lera__, a-math. (5))
# Use book to catch the gradient of # in loralib/Layer.pt W. regissiter_hoakk)
Group trainable parameters into LoRA pairs in optimizer.pi py.
for lora_U, lora_s, lora_Vh in (trassrtiliabbepaarreeee (rrra_ggropsspppeed((( "ir": learning_rate)) Update ruies in optimizer.py for group in paran_groups:
[lora_U, lora_S, Lora_Vhl,
U, 9, Vh groupi"params" G_w w_grad & gradient of
Wmm Bettal@___itt11--____ [t-] w & update first moment in step www.eeeninnnnnneneeeee..
l_t betal+L_(t-1) (1-betal)
G_M, G_W) (1-beta2) storch. storch.einsum ('insummi/ G_M, G_W) f compute the woight update component
R_t beta22R_(t-1]
step_size = group ['lr'] sqrt (1-beta2.*t) / (1-betalirt) pre_grad = -step_size-
@ W_n_t
@ torsh.diag (-O.+++0)))))) # compute some matrices for later
ULU_inv inverse (U.T terch.diag (al
@ U)
VRV inv torch. (Vh torch. (R @ Vh.T)
ULZ U.T @ torch. torch.diag (I_t=+0.25) @ pre_grad
2RV pre_grad
(Ra
8 Vh.T
f compute matrix K1
Y1 = LLU_inv @ UL2 -I- vh.T Vh)
Q1, k1 torch. iinal.grrYY.1
compute matrix K0, K2 similarly
8 Update parameters
$ SVD of matrix of size 2r x 2r
U.m, S_m, Vh_m Norch.linalg, svd [KO, K1.T], [32, 0]1)
= [C, Q2] U_m, $ = s_m, Vh Vh_m [[Vh], [Q1.T]]
Uli, :self.rank), s S[ :self. Wh Val:self.rank :]
Furthermore, we also select the optimal training parameters for the three algorithms. The batch size of plain RGD and RAdaGrad is 128, and the learning rates are 0.07 and 0.06, respectively. The batch size of DLRT is 256 and the learning rate 0.01.
Table 9: The best test accuracy and the final rank of different methods for fully-connected network.
Algorithms
Rank
Test Accuracy
DLRT
(30.29.28.3.55)
96.77%
plain RGD
(37,27,36.35..
98.38%
RAdaGrad (16,6,9,24,5)
98.65%
Under these experimental settings, train fully connected neural networks using these three algorithms for a total of 75 epochs, and record the test accuracy and loss function, as shown in Figures 8 a) and (b). To more clearly illustrate the effect of DNN compression, we also display the evolution of the rank of weight matrices for each layer during the training process, shown in Figures aa), (c), and (e). The final test accuracy results are shown in Table 9 The experimental results indicate that the RAdaGrad algorithm achieves the highest test accuracy while achieving the maximum DNNs compression rate. The final ranks of the weight matrices obtained by the RAdaGrad algorithm are (16, 6, 9, 24, 5), which are the lowest among the three algorithms, and its test accuracy
32

---

reached 98.8%, also the highest among the three algorithms. This convincingly demonstrates that compared to the factorized- algorithm (baseline DLRT), the RAdaGrad algorithm can eliminate the dependence on the condition numbers of two low-rank factors. Furthermore, compared to plain RGD, the AdaGGra algorithm achieves the same level of test accuracy while providing superior DNN compression.
coconeccedd network
Pulp-conrmcledd rehwork
Teat ACCUBOC
Text hoze
BURT
Epech
Spach
(a) Test Accuracy FC
(b) Test Loss FC
100
Lakeet
Lahies
107
Test coursty
107
RGD
DURT
RAdiind
8
100
60
Epoch
Epoch
(c) Test Accuracy LeNet5
(d) Test Loss LeNet5
Figure 8: Test accuracy and loss function for the fully connected network and LeNet5 network.
Convolutional Network LeNet5. To further verify the effectiveness of the RAdaGrad algorithm, we apply it to compress the LeNet5 network on the MNIST dataset. For the convolutional layers in LeNet5, we reshaped their convolutional kernels into matrices, with dimensions calculated as: (number of output channels) x m (number of input channels * kernel size). Consistent with the fully connected networks compression, we still adopt the dynamic rank adjustment strategy from DLRT. The initial rank for weight matrix of each convolutional layer is set to r3 min(n;, my)) / 2. Through experimentation, we determine the optimal thresholds 7 for the plain RGD, RAdaGrad, and DLKT algorithms to be 0.6, 0.9, and 0. 1, respectively. Additionally, we set the optimal training parameters for the three algorithms: both plain RGD and RAdaGrad had a batch size with learning rate 0.11; DLRT had batch size 128 with learning rate 0.02.
Table 10: The best test accuracy and the final rank of different methods for LeNet5.
Algorithms
Rank
Test Accuracy
DLRT
(7.12,19.5)
97.231% 1% 99.2337% 99.2536%
plain RGD
(10.25.14.5)
RAdaGrad
(10,25,9.5)
We train LeNet5 using these three algorithms for 70 epochs. The test accuracy and test loss are shown in Figures c) and (d), respectively. The evolution of the rank for each layer during the training process, as depicted in Figures b). (d), and (f). The final test accuracy and ranks are listed in Table 10. The results indicate that the RAdaGrad algorithm not only achieved the highest compression rate but also attained the best test accuracy, consistent with the findings from the experiments on fully connected networks. Specifically, the final ranks of the weight matrices achieved by the RAdaGrad algorithm are (10, 25, 9, 5), the lowest among the three algorithms, and its test accuracy reached 99.25%, the highest among all algorithms. This further demonstrates that the convergence rate of the RAdaGrad algorithm is independent of the condition numbers of two low-rank factors.
33

---

-tver
The mor-- of eath
Jayer
The
Spece
(a) Rank of DLRT FC
Rank DLRT LeNet5
(c) Rank of plain RGD FC
(d) Rank of plain RGD LeNet5
RAdadired Puly-cornectedd nebwork
PAdagad Lotess
The of 01.h Tayee
Esooh
(c) Rank of RAdaGrad FC
Rank of RAdaGrad LeNet5
Figure 9: The evolution of ranks for each layer in the fully connected network and LeNet5 network.
E.2 DNNs Compression on CIFAR10 Dataset.
We further test the three algorithms on the CIFAR-10 dataset. The CIFAR-10 dataset consists of 60000 color images, each with a resolution of 32 x 32 pixels, divided into 10 classes, with each class containing 6000 images. Among these, 50000 images images are used as the test dataset. In this experiment, we train the VGGII and VGG16 networks with low-rank weight matrices. All algorithms are trained for 400 epochs.
used the training dataset, and 10000
VGG11 and VGG16. VGGI1 and VGG16 are two widely used deep convolutional neural network models developed by the Visual Geometry Group (VGG) at the University of Oxford. The models are renowned for their excellent performance in image recognition and classification, particularly in the 2014 ImageNet challenge. These models are characterized by the use of small 3 3 convo- lutional kernels and multiple convolutional layers. VGG11 includes 11 weight layers, including convolutional layers and fully connected layers, while VGG16 includes 16 weight layers, with 13 convolutional layers and fully connected layers.
Table 11: The best test accuracy and the final rank of different methods for VGG11.
Algorithms DLRT plain RGD RAdaGrad
Rank
Test Accuracy
(001111414413444.22441222113
66. 66.7623% 90.3877% 90.1701 %
(222222222222222222...
(2.2222.222222222222....
We conduct compression experiments on the VGG1 1 and VGG16 DNNs using three optimizers: RAdaGrad, plain RGD, and DLRT, with the final results presented in TT and 12 From the results in these two tables, can be observed that the compression performance of plain RGD and
34

---

Table 12: The best test accuracy and the final rank of different methods for VGG16.
Algorithms
Rank
Test Accuracy
DLRT plain RGD RAdaGrad
77. 2449% 92.0886% 92.1477%
2.22222222222::2222255
(2222222222222222222222
RAdaGrad is comparable, and both significantly outperform the DLRT algorithm. Specifically, plain RGD and RAdaGrad achieve extremely low ranks while maintaining high test accuracy. In contrast, the DLRT algorithm results in higher ranks and relatively lower test accuracy. These results strongly support our core idea: optmization algorithms based on RGDframework (such as plain RGD and RAdaGrad) have proven to be efficient method for DNNs compression by finding low-rank weight matrices on Riemannian manifolds. This kind of method not only significantly reduces the memory usage of DNNs but also largely preserves their performance, achieving excellent results in practical applications. This fully validates our previous statement: optimization algorithms based on the RGD framework not only eliminate the redundancy introduced by matrix factorization methods and completely avoid the negative impact of the Jacobian's 's condition number, but also effectively reduce the condition number of the Hessian by selecting appropriate metrics on the Riemannian manifold.
35


Reviews:
[{"review_id": "aXGt6oajhl", "reviewer": "Reviewer_eKS4", "summary": "This paper proposes a Riemannian optimization algorithm for the problem of low rank training of deep networks. The authors claim that existing methods are vulnerable to the condition number of the jacobian operator for the factor and their proposed method avoids this trap. The authors derive riemannian equivalents of adamW and adagrad by proposing alternative riemannian metrics. The paper empirically demonstrates improved performance in LM finetuning and pre-training tasks.", "questions": "See weaknesses", "limitations": "None", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths: The paper studies an important problem of LoRA finetuning. The empirical results show that their training algorithms improve performance.\n\nWeaknesses: This paper fails to adequately demonstrate the existence of a convergence problem in LoRA finetuning, and further demonstrate that their method adequately addresses the problem. \n\n1. The authors do not include any examples of failed convergence in training low rank factorization models, and only argue that the condition number of W is a problem through a derivation of the hessian. They do not compare convergence rates that the condition number of the jacobian plays a factor in the poor convergence of SGD in the factored parameterization.\n\n2. The proposed method is Riemannian gradient descent/Adam using the manifold of low rank matrices. This does not improve on the time complexity of the factorized parameterization, and might involve more computation since an SVD is required at each step.\n\n3. The empirical evaluation only establishes improved performance in terms of BLEU, etc. The authors do not show that their convergence times are better.\n\n4. Prior work has already developed riemannian adaptive gradient methods, which the authors do not cite. https://arxiv.org/abs/1810.00760", "was_revised": false, "final_justification": "", "rebuttal": "**W1/Q1.**  **A1.**  Thank you for your valuable feedback. We greatly appreciate your comments and believe they will help improve our work. \n\n**[1]**  Although the current version does not include a complete theoretical analysis due to time constraints, this omission is not due to any fundamental difficulty in the derivations. In fact, the theoretical analysis framework for our algorithm has already been preliminarily established, and its derivation process is more straightforward compared to factorization-based algorithms. This is because our algorithm updates the weight matrix $W$ as a whole, involving only $\\nabla \\ell (W)$, without needing to handle the complexities introduced by the factorization form, such as $\\mathcal{J}_{\\mathcal{G}}^*(P, Q)$. Thus, by applying standard smoothness assumptions, such as Lipschitz continuity, to $\\nabla \\ell (W)$, it is relatively easy to establish the convergence analysis and derive the convergence rate of the algorithm. \n\n**[2]** In the supplementary material, we presented experiments on fine-tuning a mix-of-show language model. Figures 3 and 4 illustrate the final results and intermediate results from different training epochs, respectively. Figure 4 shows the model's generated outputs at  500th and 2000th. The results clearly demonstrate that at 500th epoch, scaledGD and scaledAdamW are unable to generate any meaningful information, whereas our algorithm, RAdaGrad, has already generated images highly relevant to the prompt ``A woman with long hair in Simpsons style\". By 2000th epoch, our algorithm has produced images that matched the prompt very well, while the images generated by scaledGD and scaledAdamW still contained significant flaws and were far from usable.\n\nThese experimental results clearly indicate that our algorithm can generate usable images with far fewer training epochs. In our responses to W2/Q2, we also provided the loss. The table demonstrates that while the training time for our algorithm is comparable to that of other algorithms, the final image quality produced by our algorithm is the highest, with the best alignment to the input prompts, which is critical for generative networks. Given the same training time, the superior image quality generated by our algorithm is strong evidence of its higher efficiency.\n\n**[3]** During the experiments, we observe that using standard decomposition methods can lead to pathological optimization when small singular values are present. This is a characteristic of the low-rank matrix manifold itself, where the curvature of the Hessian matrix is inversely proportional to the smallest singular value of the weight matrix. To evaluate the efficiency of our algorithm under such conditions, we chose a larger  r=16 to ensure that the Hessian matrix has a high condition number. The results show that our proposed RAdaGrad significantly outperforms ScaledGD on all metrics. For example, in terms of the BLEU score, our algorithm improves from 65.4 (SGD) to 70.0, and from 68.8 (ScaledGD) to 70.0. Similarly, for the ROUGE-L score, our algorithm improves from 67.0 (SGD) to 70.0, and from 69.2 (ScaledGD) to 71.9.\n\n| Methods/Metric | BLEU | NIST | MET  | ROUGE-L | CIDEr |\n| -------------- | ---- | ---- | ---- | ------- | ----- |\n| SGD            | 65.4 | 8.07 | 40.7 | 67.0    | 2.07  |\n| Scaled GD      | 68.8 | 8.75 | 45.0 | 69.2    | 2.39  |\n| RieAdaGrad     | 70.0 | 8.81 | 46.8 | 71.9    | 2.52  |\n\n**W2/Q2.**  **A2.**  Thank you for your insightful question. We greatly value your feedback and would like to clarify the computational aspects of our proposed method.\n\n**[1]** Compared to factorization-based algorithms, our method does indeed perform an SVD computation at each step. However, as analyzed in Sections 2 and 3 of the supplementary materials, the computational complexity of performing SVD on $\\mathcal{H}_r(Z_t)$ at each step is $\\mathcal{O}(r^3)$, depending only on the rank $r$. LoRA fine-tuning typically selects a very small rank $r$, which makes the computational cost of SVD extremely low. In practice, this computational cost is negligible and has an almost imperceptible impact on the overall efficiency.\n\n**[2]** To further validate the efficiency of our algorithm, we compare the training loss of all algorithms under the same number of iterations (i.e., the same number of epochs), as shown in the table below. The experimental results demonstrate that our algorithm achieves the fastest loss reduction during training. For example, the loss trend table shows that at the 2000th iteration, our algorithm reduces the loss to 2.73, while the losses of other algorithms remain above 2.80. By the 18000th iteration, our algorithm further reduces the loss to 2.59, whereas the losses of other algorithms remain above 2.60. This indicates that, given the same training time, our algorithm achieves significantly lower loss values, resulting in superior overall performance. Furthermore, as demonstrated in Figures 3 and 4 of the supplementary materials, the output quality generated by our algorithm is also superior. These results highlight the effectiveness of our method, especially for network training based on limited data, where efficient and effective learning is crucial.\n\nTraining Average Loss:\n\n| Iteration  | 2000 | 4000 | 6000 | 8000 | 10000 | 12000 | 14000 | 16000 | 18000 | 20000 | 22000 | \n| -----------------  | ---- | ---- | ---- | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | \n| SGD                 | 2.89 | 2.89 | 2.81 | 2.86 | 2.83  | 2.81  | 2.82  | 2.80  | 2.76  | 2.75  | 2.69  | \n| scaled GD           | 2.80 | 2.79 | 2.71 | 2.75 | 2.72  | 2.70  | 2.72  | 2.70  | 2.67  | 2.66  | 2.60  | \n| plain RGD          | 2.80 | 2.78 | 2.69 | 2.73 | 2.70  | 2.67  | 2.70  | 2.67  | 2.64  | 2.64  | 2.59  | \n| RAdaGrad (ours)     | 2.73 | 2.72 | 2.63 | 2.68 | 2.65  | 2.63  | 2.64  | 2.62  | 2.59  | 2.59  | 2.53  | \n\n**W3/Q3.**   **A3.**  Thank you for highlighting this important evaluation metric, which will improve the presentation of our work. Our experimental evaluation focuses on improved performance metrics, which directly demonstrate better results within the same number of iterations and thus, greater algorithm efficiency. To further validate the effectiveness, we analyze the loss reduction trends (see the table in W2/Q2). The results show that our method achieves faster loss reduction compared to other algorithms. For example, at the 2000th iteration, the loss value of our method is lower than that of the baselines. This demonstrates that our algorithm requires fewer iterations to achieve good results, which is particularly crucial for low-data or few-shot learning. Additionally, Figures 3 and 4 in the supplementary materials further support that, within the same training time, our method can generate outputs that align better with the input prompts. All these experimental results clearly show that our algorithm converges more quickly and effectively. \n\n**W4/Q4.**  **A4.**  Thank you very much for bringing this important reference to our attention. This paper is closely related to our research, and we will revise our manuscript to cite and discuss [1]. In this work, the authors extend adaptive optimization methods to Riemannian manifolds, particularly for product manifolds. While both our algorithm and [1] introduce adaptive methods, there are several key differences\n\n**[1]**  [1] generalizes adaptive methods to Riemannian manifolds, whereas our algorithm specifically introduces an adaptive method tailored for low-rank matrix manifolds. This allows our algorithm to exploit the compact and smooth structure of low-rank matrix manifolds, further enhancing its efficiency. For example, our algorithm consists of three steps\n\n**1.**  $M_t = W_t - \\gamma_t L_t^{-1/4} G_t R_t^{-1/4}.$\n\n**2.** $Z_t= \\mathcal{P}_{\\mathbb{T}_t}^{g} (W_t - \\gamma_t L_t^{-1/4} G_t R_t^{-1/4}). $\n\n**3.** $W_{t+1} =\\mathcal{H}_r (Z_t) =\\mathcal{H}_r(W_t -  \\gamma_t\\mathcal{P} (L_t^{-1/4} G_t R_t^{-1/4})).$\n\nWe parameterize the tangent space of the low-rank matrix manifold at a point, then steps 2 and 3 can be combined for computation (see the details in Sections 2 and 3 of the supplementary materials). This allows us to perform SVD on only a $2r\\times 2r$ matrix, reducing the computational complexity of SVD to $\\mathcal{O}(r^3)$. In contrast, the algorithm in [1] does not include a tangent space projection (Step 2), resulting in an SVD computational complexity of $\\mathcal{O}(rn^2)$. When $r$ is small, the computational efficiency of our algorithm is significantly higher than that of the algorithm in [1].\n\n**[2]** More importantly, we propose a unified Riemannian gradient framework $W_{t+1} = \\mathcal{R}(W_t - \\gamma_t \\nabla_{\\mathcal{M}_r} \\ell(W_t)$, which provides new insights for designing efficient algorithms. As shown in 3.1, when the standard metric is chosen, our method reduces to the plain RGD [2]. In 3.2, we select a metric that combines AdaGrad and Shampoo, ensuring isotropy of gradients and mitigating the effects of the Hessian condition number, resulting in efficient RAdaGrad. This demonstrates that selecting more effective retraction and metrics can lead to more efficient algorithms. This idea is critical for designing efficient algorithms and is particularly inspiring for developing efficient fine-tuning methods for large language models. We hope this heuristic idea can stimulate the design of even more efficient algorithms.\n\nWe sincerely appreciate the reviewer's constructive suggestions and believe that the additional experiments, analysis, and explanations significantly improve the quality of our paper. We hope that this provides sufficient reasons to raise the score.\n\n[1] G. Becigneul and O. Ganea. Riemannian adaptive optimization methods. ICLR, 2019.\n\n[2] K. Wei, J.-F. Cai, T. Chen and S. Leung. Guarantees of Riemannian optimization for low rank matrix recovery.\nSIAM J. Matrix Anal. Appl., 37(3):1198–1222, 2016."}, {"review_id": "CTs4T2jyEj", "reviewer": "Reviewer_S2n3", "summary": "The authors study memory efficient tuning of large models through low rank optimization. They propose a framework based on the optimization of the Riemannian manifold. This approach is able to eliminate the negative impact of the Jacobian condition number. Specifically, they propose RAdaGrad and RAdamW. The empirical result shows the effectiveness of the proposed method.", "questions": "My main question is the same as the one mentioned in the weakness section. What is the difference between the proposed method, ScaledAdam, and LoRA-RITE, which all seems like attempt to avoid the condition number problem and adopts adaptive preconditioning?", "limitations": "The limitation section feels a bit short.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe strength of the paper is that it seems to propose a framework that generalizes the existing methods. The empirical result also supports the effectiveness of the proposed framework.\n\nWeakness:\n\nHowever, the relation with the existing works is not discussed in detail. Specifically, the difference between the proposed method and ScaledGD and ScaledAdamW is not clearly stated, which are also methods based on Riemannian optimization. Additionally, there is LoRA-RITE[1], which also seems to avoid the Jacobian condition number, while adopts adaptive preconditioning similar to AdamW and Shampoo.\n\n[1] Yen, J. N., Si, S., Meng, Z., Yu, F., Duvvuri, S. S., Dhillon, I. S., ... & Kumar, S. (2024). LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. The Thirteenth International Conference on Learning Representations (ICLR 2025)", "was_revised": true, "final_justification": "I appreciate the authors providing the details I ask for in the rebuttal. I believe incorporating these into the paper will make it more self-contained and make the contributions clearer to the readers. I retain my original score as it is already positive.", "rebuttal": "**W1/Q1.**  However, the relation with the existing works is not discussed in detail. Specifically, the difference between the proposed method and ScaledGD and ScaledAdamW is not clearly stated, which are also methods based on Riemannian optimization. Additionally, there is LoRA-RITE[1], which also seems to avoid the Jacobian condition number, while adopts adaptive preconditioning similar to AdamW and Shampoo.\n\n**A1.** Thank you for pointing out that our discussion of existing work is not sufficiently thorough. We greatly value your feedback and will include a more detailed comparison and discussion in the revised manuscript. Below, we provide a specific explanation of the connections and differences between our algorithm and related works:\n\n**[1] Differences from ScaledGD and ScaledAdamW.**\n\nAlthough ScaledGD and ScaledAdamW are also based on preconditioning methods for Riemannian optimization, there are several key differences between their algorithms and ours:\n\n**1.**  **Optimization framework for weight updates.** According to [1, Section 3], the weight matrix update in ScaledGD can be reformulated as\n\n$$\n\\begin{aligned}\nX_{t+1} &\\approx  X_t - \\eta (\\nabla_{X_t} L) A_t^{T} (A_tA_t^{T})^{-1}A_t - \\eta B_t(B_t^TB_t)^{-1}B_t^T(\\nabla_{X_t}L) =X_t - \\eta Proj_{row(A_t)} (\\nabla_{X_t} L) - \\eta Proj_{col(B_t)}(\\nabla_{X_t} L)^T.\n\\end{aligned}\n$$\n\nThe update in our algorithm for the whole weight matrix on the tangent space $\\mathbb{T}_t$ can be expressed as\n\n$$\nW_t - \\gamma_t \\mathcal{P}_{\\mathbb{T}_t}^{(g)}(Z_t) = W_t - \\tilde{U}_t\\tilde{U}_t^{T}L_t^{-1/4}Z - Z R_t^{-1/4}\\tilde{V}_t \\tilde{V}_t^{T} + \\tilde{U}_t\\tilde{T}_t^{T}L_t^{-1/4}ZR_t^{-1/4}\\tilde{V}_t \\tilde{V}_t^{T}\n$$\n\n$$\n\\qquad \\qquad \\qquad \\quad =W_t - \\tilde{U}_t\\tilde{U}_t^{T}G_tR_t^{-1/4} - L_t^{-1/4} G_t \\tilde{V}_t \\tilde{V}_t^{T} + \\tilde{U}_t\\tilde{T}_t^{T} G_t \\tilde{V}_t \\tilde{V}_t^{T}\n$$\n\n$$\n\\qquad \\qquad \\qquad \\quad =W_t  - Proj_{row} (G_t) - Proj_{col}(G_t) + Proj_{row}Proj_{col}(G_t),\n$$\nwhere $Z_t = L_t^{-1/4}G_tR_t^{-1/4}$. From the update rules of the two algorithms, it is evident that ScaledGD applies projections of the gradient onto both the row and column spaces twice for overlapping gradient components $Proj_{row}Proj_{col}(G_t)$. In contrast, our algorithm applies these projections only once. As a result, ScaledGD contains redundant projected gradient information, which may lead to excessive updates, deviations in the optimization path, and ultimately inferior training performance for the network. This also adversely affects the generative results of language models, as demonstrated by our experimental results.\n\n**2. Different methods for mitigating the Jacobian condition number.**  ScaledGD, based on a decomposition formulation, essentially solves the optimization problem on the quotient manifold M/ ∼, where each element [x]={y∈M:y∼x} represents an equivalence class. ScaledGD defines a new metric on the quotient manifold [x]=(A,B) as follows\n\n$$\ng [x] (\\eta [x],\\xi [x] ) = \\langle \\eta_A, \\xi_A B^TB\\rangle + \\langle \\eta B, \\xi_{B} A^TA \\rangle. \n$$\n\nUnder this metric, the update formulas for $A$ and $B$ can be derived as\n\n$$\n(\\frac{\\partial}{\\partial A}) \\to (\\frac{\\partial}{\\partial A}) (B^TB)^{-1}, \\quad (\\frac{\\partial}{\\partial B}) \\to (\\frac{\\partial}{\\partial B}) (A^TA)^{-1}.\n$$\n\nFrom these formulas, we observe that the new gradients in ScaledGD for the low-rank factors $A$ and $B$ are preconditioned by $(B^TB)^{-1}$ and $(A^TA)^{-1}$, respectively, which partially alleviate the effects of the Jacobi condition number. However, as noted earlier, ScaledGD does not fully eliminate the influence of the $\\mathcal{J}_{\\mathcal{G}}^*$ condition number. More importantly, it does not address the condition number of the Hessian $\\nabla^2 \\ell(W)$.\n\nAs you pointed out, our main contribution lies in proposing a unified framework for Riemannian gradient descent algorithms for the entire weight matrix update. By selecting appropriate metrics and retraction operators, we achieve a more efficient algorithm. Specifically, our approach targets the $W$ update, thereby completely avoiding the influence of the $\\mathcal{J}_{\\mathcal{G}}^*$ condition number. Furthermore, we introduce the following metric, which combines AdaGrad and Shampoo\n\n$$\n\\langle X, Y \\rangle_{g_t} = \\langle X, A_t Y \\rangle = \\langle X, L_t^{1/4} Y R_t^{1/4} \\rangle,\n$$\n\nwhere $L_t = \\epsilon I + \\sum_{t} diag(G_t G_t^T)$ and $R_t = \\epsilon I + \\sum_{t} diag(G_t^T G_t)$.  Under this new metric, we obtain a newly preconditioned Riemannian gradient $\\tilde{\\mathcal{P}}_{\\mathbb{T}}(L_t^{-1/4}G_t R_t^{-1/4})$, which exhibits isotropic properties, effectively mitigating the influence of the Hessian condition number. In summary, the metric we select is more effective for updating low-rank matrix weights. All of our experimental results strongly support this conclusion.\n\n**3. Different convergence rates.** Further referencing the convergence analysis in [1], the convergence of ScaledGD relies on the following assumptions\n\n$$\n(1−δ_r)∥M∥_F^2 ≤∥AM∥_F^2 ≤(1+δ_r)∥M∥_F^2.\n$$\nBased on these assumptions, the convergence rate of ScaledGD depends on $\\kappa= 2 \\frac{1+δ_r}{1−δ_r}$. When $δ_r$ is very small,  $\\kappa \\approx 2$. We analyze the convergence of our algorithm under the same assumptions. Preliminary calculations show that the convergence rate of our algorithm depends on the condition number $\\kappa= \\frac{1+δ_r}{1−δ_r}$, when $δ_r$  is very small, $\\kappa \\approx 1$. Clearly, under the same assumptions, our algorithm demonstrates better convergence than ScaledGD. For ScaledAdamW, similar conclusions apply as with ScaledGD, since the primary difference lies in the optimizer used.\n\n[1] F. Zhang and M. Pilanci. Riemannian preconditioned LORA for fine-tuning foundation models. In Proceedings of the 41st International Conference on Machine Learning (ICML), 2024.\n\n**[2] Differences from LoRA-RITE.**    We will cite LoRA-RITE [1] in the revised version of our paper. Below, we elaborate on the key differences between our algorithm and LoRA-RITE, along with their potential implications, while highlighting the advantages of our approach.\n\nAlthough LoRA-RITE achieves transformation invariance by introducing preconditioners, its core framework remains based on factorization for updating low-rank factors. Specifically, LoRA-RITE designs adaptive preconditioners and incorporates first- and second-order momentum using gradients of the low-rank factors. However, its gradient updates rely on the pseudo-inverse of the low-rank factor gradients to approximate the inverse of the Hessian. Due to the over-parameterized nature of low-rank factorization, these gradient matrices are often not invertible. As a result, the use of pseudo-inverses leads to the loss of gradient information in directions orthogonal to the pseudo-inverse, introducing errors that can affect the optimization path and final performance.\n\nThis highlights the fundamental difference between our algorithm and factorization-based methods. Our method directly utilizes the gradients of the entire weight matrix $W$ to construct the preconditioner, first-order momentum, and second-order momentum. Since we do not rely on low-rank factorization, our gradient matrices are invertible, allowing us to directly and more effectively approximate the inverse of the Hessian matrix. This avoids the information loss and potential errors introduced by pseudo-inverses in LoRA-RITE. By leveraging the full weight matrix gradients, our approach demonstrates potential advantages in optimization efficiency and final performance.\n\nWe sincerely appreciate the reviewer's constructive suggestions and believe that the additional experiments, analysis, and explanations significantly improve the quality of our paper. We hope that this provides sufficient reasons to raise the score.\n\n[1] Yen, J. N., Si, S., Meng, Z., Yu, F., Duvvuri, S. S., Dhillon, I. S., ... & Kumar, S. LoRA Done RITE: Robust Invariant Transformation Equilibration for LoRA Optimization. The Thirteenth International Conference on Learning Representations (ICLR 2025)"}, {"review_id": "wXtB9mxCve", "reviewer": "Reviewer_oub2", "summary": "The paper proposes learning of the low-rank weights using Reimanian gradient descent method. To this end the paper adapts Adam and RMSprop accelerated gradient descent methods to be compatible with Reimanian gradient descent. The authors provide in depth mathematical formulation for the approach and back the method with solid experimental results.", "questions": "Please address the weaknesses above^", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper’s strength lies in its principled foundation in optimization theory and its connection to a rich body of prior work. However, this also seems to be its main weakness.\n\nThe Riemanninan manifold optimizaton has been applied to the neural networks in many different ways and produced plethora of works. I think the paper would strongly benefit from thorough comparison to the previous works and having clear statement on what makes this paper different from preivous one. I especially would like to see comparisons to the ICLR-2019 paper of \"Riemannian Adaptive Optimization Methods\" that propose modifications of Adam, Adagrad, and Amsgrad with respect to Riemannian optimization. There are many more relevant works(if you follow the references), but I would list couple more here:\n- Geometry Aware Constrained Optimization Techniques for Deep Learning, CVPR2018\n- Averaging Stochastic Gradient Descent on Riemannian Manifolds, COLT 2018\nThis is by no means a full list of relevant papers, there is probably dozens of papers proposing some accelerationg scheme + Rimeannian manifold; \n\nGiven the amount of research happened in the field, I think what would put this paper apart would be the actual practical implementation (most of the Riemanian mehtods require SVD or other full decomposition on every gradient step), however, the paper is very scarce on the practical implementation. There is the part of the pseudocode in supplementary materials and the complexity analysis of the steps (lines 270-274), but it is hard to understand what is the actual runtime implications. In practice, many of the efficient factorizations (like QR, or computing inverse) take considerably longer times (have higher constants) wrt to similarly complex operations, and as such, the time cost of every update might be considerably longer. Previous works typically circumvent this by doing proper projections ever K steps. If authors can share the wallclock of every projection and compare it to vanilla SGD updates that would better idea about pracitcal usability of the method. Also, it seems this method favors smaller ranks (due to cost of updates), however, when training  LoRA adpaters you rarely see rank-1 and rank-4 apdaters, typically I would say it is rank 10 at the very least. \n\nAdditionally, no experiments are presented for pure compression (i.e., training a smaller model directly), which would broaden the applicability of the method beyond adapters.\n\nThere are also issues with citation accuracy. For example:\n-  on lines 31-32, when referring to low-rank  being used in dnn compression paper cites 14, 24, 38, but none of these papers in fact use low-rank compression.\n- on line 232 the reference 26 is great (I love this book), but can you pinpoint exact chapter/equation you are citing?\n\nSome claims are imprecise:\n- on line 30, papers says that \"these methods often rely on specialized hardware\" where \"these\" referes to \"quantization\" and \"pruning\", however, quantization is more-or-less hardware agnostic these days (as long as your hardware supports integer multiplications & has specialized kernels/code) and for instance structured pruning is definitely hardware agnostic. Therefore, the statement is imprecise and should be corrected.  \n- on line 81 \"Current mainstream methods\" supposed to refer to \"general low-rank optimization methods\", however only cite low-rank adapter research (i.e., learing w+uv with frozen w), which by no means would be \"mainstream\". Probably, you meant to write \"current mainstream low-rank adaptation methods\"?\n\nOverall, this is a promising paper, but it would benefit greatly from a deeper discussion of prior work, clarity on practical implications, and more careful citation and phrasing.", "was_revised": true, "final_justification": "Authors made strong rebuttal that clarified my initial reservations.", "rebuttal": "**W1.** **A1.** Thank you for highlighting these important references and other related works. These papers are indeed relevant to our research, and we will revise the manuscript to include and discuss these related references. Below, we emphasize the distinctions between our work and these prior studies, as well as our innovations.\n\n**[1]** The references [1, 2, 3] that you mentioned extend adaptive optimization methods or stochastic gradient descent methods to Riemannian manifolds. However, all these works consider general manifold structures. In contrast, our algorithm is specifically designed for the Riemannian gradient descent on the low-rank matrix manifold. Focusing on optimization problems on the specific low-rank matrix manifold enables us to leverage the unique properties of the manifold's structure—such as the parameterization of the tangent space of low-rank matrices—to accelerate the convergence of the algorithm. Specifically, our algorithm performs the following three steps \n\n**1.** $M_t = W_t - \\gamma_t(L_t^{-1/4} G_t R_t^{-1/4}).$\n\n**2.** $Z_t= \\mathcal{P}_{\\mathbb{T}} (M_t). $\n\n**3.** $W_{t+1} =\\mathcal{H}_r (Z_t) =\\mathcal{H}_r(W_t - \\gamma_t\\mathcal{P}(L_t^{-1/4} G_t R_t^{-1/4})).$\n\nBy parameterizing the tangent space of low-rank matrices at a certain point, we combine steps 2 and 3 into a single computation (as detailed in Sections 2 and 3 of the supplementary material). This allows us to perform an SVD on only a 2r×2r matrix, reducing the computational complexity of SVD to $\\mathcal{O}(r^3)$. This technique has been validated in applications such as low-rank matrix recovery[4], robust PCA[5], and tensor train format completion[6], demonstrating its efficiency. Building on this foundation, we further consider the stochastic gradient case and incorporate both first- and second-order momentum for additional acceleration. While our per-step computation time is indeed higher than that of SGD, the significantly reduced number of iterations required by our method results in a shorter total time to reach the same target. Moreover, the per-step computation time can be further reduced through optimizations in the underlying code.\n\nBased on your suggestion, we also realize that updating frequency can be reduced to optimize the cost of updates. For instance, as you mentioned, performing projections every K steps instead of every step could further reduce runtime. Our method can adopt a similar strategy, reducing update frequency while maintaining model performance. We will include these optimizations in the revised manuscript and report their impact on runtime and model performance.\n\n**[2]** As you pointed out, the implementation of our method in fine-tuning large language models is a significant highlight. Current optimization algorithms for fine-tuning large language models often rely on factorization, which suffers from dependencies on Jacobian condition numbers. To circumvent this issue, we propose updating the entire low-rank weight matrix and mitigating the effects of the Hessian condition number by preprocessing the gradient and incorporating first- and second-order momentum.\n\n**[3]** More importantly, we propose a unified Riemannian gradient framework $W_{t+1} = \\mathcal{R}(W_t - \\gamma_t \\nabla_{\\mathcal{M}_r} \\ell(W_t)$ which provides new insights for designing efficient algorithms. This framework is not only computationally efficient but also extensible. By selecting different retraction mappings 𝑅 and metrics, various efficient algorithms can be designed. As shown in Section 3.1, when the standard metric is chosen, our method reduces to the conventional Riemannian gradient descent (RGD) as proposed in [4]. In Section 3.2, we select a data-driven metric combining AdaGrad and Shampoo to ensure isotropic gradients and alleviate Hessian condition number issues, resulting in the efficient RAdaGrad method. This demonstrates that choosing more effective retraction mappings and metrics can yield more efficient algorithms. This idea is crucial for designing efficient algorithms, particularly for developing efficient fine-tuning methods for large language models. We hope this heuristic approach will inspire the design of more efficient algorithms in the future.\n\n**W2** **A2** Thank you for the valuable comments! Our experiments and methods are also applicable to higher-rank cases. In training LoRA adapters, current experiments primarily focus on lower ranks (e.g., 1, 4, or 8). This choice is mainly made to ensure that the parameter storage space remains very small. To further validate the generality of our algorithm, we also tested results for r=16, which are shown in the table below.\n| Methods/Metric | BLEU | NIST | MET  | ROUGE-L | CIDEr |\n| -------------- | ---- | ---- | ---- | ------- | ----- |\n| SGD            | 65.4 | 8.07 | 40.7 | 67.0    | 2.07  |\n| Scaled GD      | 68.8 | 8.75 | 45.0 | 69.2    | 2.39  |\n| RieAdaGrad     | 70.0 | 8.81 | 46.8 | 71.9    | 2.52  |\n\n| Methods/Metric | BLEU | NIST | MET  | ROUGE-L | CIDEr |\n| -------------- | ---- | ---- | ---- | ------- | ----- |\n| AdamW          | 69.5 | 8.77 | 46.4 | 71.2    | 2.48  |\n| Scaled AdamW   | 69.8 | 8.79 | 46.5 | 71.7    | 2.51  |\n| RieAdamW       | 70.1 | 8.85 | 46.6 | 71.6    | 2.52  |\n\nThe results show that our proposed RAdaGrad significantly outperforms ScaledGD on all metrics. For example, in terms of the BLEU score, our algorithm improves from 65.4 (SGD) to 70.0, and from 68.8 (ScaledGD) to 70.0. Similarly, for the ROUGE-L score, our algorithm improves from 67.0 (SGD) to 70.0, and from 69.2 (ScaledGD) to 71.9.\n\n**Comments**   **A1**  We sincerely thank you for the detailed comments on the accuracy of references and wording!\nWe have carefully reviewed the issues you raised and will make comprehensive revisions in the updated manuscript to ensure the accuracy of references and the precision of our statements.\n\n**1.**  We will replace the references with more relevant ones, such as (not listed exhaustively):\n\nM. Zhang, L.-K. Tong, S. Pan, Parameter and memory efficient pretraining via low-rank Riemannian optimization, ICLR, (2025).\n\n**2.** We greatly appreciate your positive feedback on the cited book! Your suggestion to specify the exact chapters or formulas is very reasonable. We will revise the reference accordingly, citing [26, Chapter 10, 11 and 12]. \n\n**3** Thank you very much for your comment on the dependency of pruning and quantization on hardware. The actual acceleration of pruning indeed depends on hardware support for sparse matrix operations. Pruned models usually result in sparse weight matrices, and sparse matrix multiplication requires hardware support for efficient sparse storage formats (e.g., CSR or CSC) and corresponding computational kernels. For example, GPUs based on NVIDIA's Ampere architecture support sparse tensor core operations, enabling performance improvements under 2:4 sparsity. However, other hardware architectures may not be able to exploit such sparsity. Similarly, Google's TPU hardware is specifically designed to support sparse matrix operations, but this requires customized compression formats and sparse acceleration kernels.\n\nLikewise, the actual acceleration and storage savings of quantization depend on hardware support and optimization for low-precision data types (e.g., INT8, BF16). Quantized models require hardware capable of efficiently performing computations with low-precision integers (e.g., INT8) or floating-point numbers (e.g., BF16). For example, NVIDIA Tensor Cores support INT8 and BF16 computations, significantly boosting the training and inference speed of quantized models. Google's TPU hardware has been optimized for BF16, enabling faster model inference without significantly reducing precision, whereas other hardware may lack similar support.\n\n**4** Regarding low-rank adaptive methods, we greatly appreciate your suggestion. We will revise our description to use the more accurate term “current mainstream low-rank adaptive methods.”\n\nWe sincerely appreciate the reviewer's constructive suggestions and believe that the additional experiments, analysis, and explanations significantly improve the quality of our paper. We hope that this provides sufficient reasons to raise the score.\n\n[1] G. Becigneul and O. Ganea. Riemannian adaptive optimization methods. ICLR, 2019.\n\n[2] Geometry Aware Constrained Optimization Techniques for Deep Learning, CVPR,2018.\n\n[3] Averaging Stochastic Gradient Descent on Riemannian Manifolds, COLT 2018\n\n[4] K. Wei, J.-F. Cai, T. Chen and S. Leung. Guarantees of Riemannian optimization for low rank matrix recovery. SIMX, 2016. \n\n[5] H. Cai, J.-F. Cai, K. Wei. Accelerated alternating projections for robust principal component analysis. JMLR, 2019.\n\n[6] J.-F. Cai, J. Li, D. Xia. Provable tensor-train format tensor completion by riemannian optimization. JMLR, 2022."}, {"review_id": "x2nTyOuNOx", "reviewer": "Reviewer_VP56", "summary": "This paper studies training models with low-rank weight matrices. These may arise in a number of contexts like LoRA. Prior approaches (like maintaining the entries of a low-rank factorization of the matrix) lead to ambiguities (the same matrix has many representations) and dependences on the condition numbers of the Jacobian and Hessian in the convergence. \n\nThis work suggests a new approach using Riemannian Optimization. Roughly, one does gradient descent on the manifold of low-rank matrices. There are two variants presented, building on AdaGrad and AdamW.\n\nThe paper first describes the mathematical details of the approach, including its computational complexity. It then describes experiments, showing that this approach leads to better correctness in various settings, including LoRA for GPT-2.", "questions": "- In the experiments, were the Jacobian and Hessian condition numbers large? Is the main source of improvement that the algorithm can not make use of matrices with large condition numbers? I was expecting the experiments to show faster convergence, rather than actually better quality scores, so I'm wondering if you could explain this more.\n\n- Could you elaborate on \"it is easy to find an orthonormal basis...\" on line 152? I was expecting calculations in the manifold to be slower than in prior approaches, but it seems to be roughly just as fast?\n\n- Could you compare more precisely the computational complexity with prior work? What are the running times of prior algorithms?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Natural idea for getting around the issues with prior approaches for low rank weights\n- Strong theoretical analysis, and very clear presentation\n- Experiments show real improvements by just making this change to LoRA and in other settings.\n- The approach is general and can be \"plugged in\" to many different models and frameworks.\n\nWeaknesses\n- I have a couple questions about the experiments and running time; see below.", "was_revised": true, "final_justification": "This is a nice theoretical contribution to low-rank optimization which, I think, will lead to empirical work down the road. My questions were properly addressed.", "rebuttal": "**W1/Q1.** In the experiments, were the Jacobian and Hessian condition numbers large? Is the main source of improvement that the algorithm can not make use of matrices with large condition numbers? I was expecting the experiments to show faster convergence, rather than actually better quality scores, so I'm wondering if you could explain this more.\n\n**A1.** We sincerely thank you for your valuable feedback, which is crucial for improving the quality of our paper. We will carefully consider your suggestions and include richer experimental comparisons in the revised manuscript to provide a more comprehensive and in-depth demonstration of the advantages and effectiveness of our method. Below, we provide detailed responses to the questions you have raised:\n\n**1. Efficiency and Loss Reduction.** Although the current experimental results primarily focus on performance improvements, they do demonstrate that our algorithm achieves better outcomes within the same number of iterations, reflecting higher efficiency. To further validate the effectiveness of our algorithm, we provide the loss reduction trends in the table below. The results show that, compared to other algorithms, our algorithm requires fewer iterations (or less training data) to achieve the same loss value.\n\n| Iteration  | 2000 | 4000 | 6000 | 8000 | 10000 | 12000 | 14000 | 16000 | 18000 | 20000 | 22000 | \n| -----------------  | ---- | ---- | ---- | ---- | ----- | ----- | ----- | ----- | ----- | ----- | ----- | \n| SGD                 | 2.89 | 2.89 | 2.81 | 2.86 | 2.83  | 2.81  | 2.82  | 2.80  | 2.76  | 2.75  | 2.69  | \n| scaled GD           | 2.80 | 2.79 | 2.71 | 2.75 | 2.72  | 2.70  | 2.72  | 2.70  | 2.67  | 2.66  | 2.60  | \n| plain RGD          | 2.80 | 2.78 | 2.69 | 2.73 | 2.70  | 2.67  | 2.70  | 2.67  | 2.64  | 2.64  | 2.59  | \n| RAdaGrad (ours)     | 2.73 | 2.72 | 2.63 | 2.68 | 2.65  | 2.63  | 2.64  | 2.62  | 2.59  | 2.59  | 2.53  | \n\nAt the 2,000th iteration, our algorithm reduces the loss to 2.73, while the losses of other algorithms remain above 2.80. By the 18,000th iteration, our algorithm further reduces the loss to 2.59, while the losses of other algorithms are still above 2.60. Moreover, the per-step computational time of our algorithm is comparable to that of other algorithms, resulting in overall less or at least equivalent total computation time. This demonstrates that, within the same training time, our algorithm achieves significantly lower loss values, leading to superior overall performance.\n\nAdditionally, in the supplementary material, we present fine-tuning experiments on a multimodal language model. Figures 3 and 4 illustrate the final results and intermediate outcomes at different training epochs. Specifically, Figure 4 shows the generated outputs at the 500th epoch and the 2,000th epoch. The results clearly indicate that, at the 500th epoch, ScaledGD and ScaledAdamW fail to generate any meaningful outputs, while our algorithm RAdaGrad already produces images highly relevant to the given prompt, such as ``a long-haired woman in Simpson style.\" By the 2,000th epoch, the images generated by our algorithm closely match the prompts, whereas the outputs of ScaledGD and ScaledAdamW still exhibit significant defects, rendering them unusable.\n\nThese experimental results clearly demonstrate that our algorithm can generate usable outputs with fewer training epochs, which is crucial for generative networks. Within the same training time, our algorithm produces higher-quality outputs, providing strong evidence of its superior efficiency.\n\n**2. Jacobian and Hessian Condition Numbers.** In our experiments, we did observe that using standard decomposition methods can lead to pathological optimization when small singular values are present. This is because the optimization problem for low-rank matrices is inherently influenced by the distribution of singular values. This property is also intrinsic to the low-rank matrix manifold: when the weight matrix contains small singular values (which often occurs when a larger rank is selected), the curvature of the Hessian matrix becomes inversely proportional to those singular values, resulting in a high condition number. Such high condition numbers can significantly affect the optimization trajectory and often lead to pathological optimization issues.\n\nTo further validate this, we conducted an additional experiment where we selected a larger rank r=16 to ensure that the Hessian matrix has a high condition number. The results are shown in the table below\n\n| Methods/Metric | BLEU | NIST | MET  | ROUGE-L | CIDEr |\n| -------------- | ---- | ---- | ---- | ------- | ----- |\n| SGD            | 65.4 | 8.07 | 40.7 | 67.0    | 2.07  |\n| Scaled GD      | 68.8 | 8.75 | 45.0 | 69.2    | 2.39  |\n| RieAdaGrad     | 70.0 | 8.81 | 46.8 | 71.9    | 2.52  |\n\nThe results illustrate that our proposed RAdaGrad algorithm significantly outperforms ScaledGD across all metrics. For example, in terms of the BLEU score, our algorithm improves from 65.4 (SGD) to 70.0, and from 68.8 (ScaledGD) to 70.0.\nSimilarly, for the ROUGE-L score, our algorithm improves from 67.0 (SGD) to 70.0, and from 69.2 (ScaledGD) to 71.9.\n\n**W2/Q2.** Could you elaborate on \"it is easy to find an orthonormal basis...\" on line 152? I was expecting calculations in the manifold to be slower than in prior approaches, but it seems to be roughly just as fast?\n\n**A2.** Thank you very much for your valuable question!\n\n**[1]. Regarding the statement in line 152.**  On the low-rank matrix manifold, our algorithm requires finding a set of orthogonal bases in a specific subspace to ensure that the update directions are consistent with the geometric structure of the manifold. The detailed process for computing the orthogonal bases is provided in Sections 2 and 3 of the supplementary material.\n\nSpecifically, under the newly constructed metric $\\langle X, Y \\rangle_{g_t} = \\langle X, L_t^{1/4} Y R_t^{1/4}\\rangle$, the orthogonal basis for the row space $\\tilde{U}_t = U_t (U_t^T L_t^{1/4} U_t)^{-1/2}$ and the orthogonal basis for the column space $\\tilde{V}_t = V_t (V_t^T R_t^{1/4} V_t)^{-1/2}$. These two orthogonal bases can be derived via the singular value decomposition (SVD). As analyzed in the supplementary material, our algorithm only requires performing an SVD on a 2r×2r matrix, with a computational complexity of $\\mathcal{O}(r^3)$. When r is small, this cost is significantly lower than performing an SVD on the entire matrix. Therefore, computing the orthogonal bases is highly efficient and does not become a bottleneck in the optimization process.\n\n**[2]. Regarding efficiency and speed.** Although in theory, manifold-based computation may introduce additional steps, two key factors ensure that our method is faster—or at least comparable in speed—to prior methods in practice.\n\n1. Efficient SVD computation. As shown in the supplementary material, we combine the gradient descent step in the tangent space with an r-truncated SVD, requiring only an SVD on a 2r×2r matrix. This significantly reduces the computational cost of the SVD operation.\n\n2. Integration of first- and second-order momentum. By incorporating first- and second-order momentum into the algorithm, we preprocess the gradient to make it isotropic, thereby accelerating the convergence of the algorithm. As shown in the experimental results discussed in Q1, although the per-step computation time of our algorithm is comparable to that of other methods, it requires fewer iterations (or less training data) to achieve the same loss value. As a result, the overall computation time is shorter—or at least equivalent—when compared to other factorization-based methods.\n\n**W3/Q3.** Could you compare more precisely the computational complexity with prior work? What are the running times of prior algorithms?\n\n**A3.** Thank you for raising this valuable question. We will provide a detailed explanation of the computational complexity of our algorithm.\n\nFirst, the rank of each matrix in $\\mathbb{T}_t$ is at most $2r$, and the operator $\\mathcal{H}_r(Z_t)$ is used to find the best rank-$r$ approximation of a rank-$2r$ matrix. This process involves the following steps:  \n\n**1.** Two QR decompositions of sizes $n \\times 2r$ and $m \\times 2r$,  \n\n**2.** A matrix multiplication of size $2r \\times 2r$,  \n\n**3.** A singular value decomposition (SVD) of size $2r \\times 2r$.  \n\nThe total computational cost of these operations is $\\mathcal{O}((n+ m)r^2 + r^3)$. In addition to computing $G_t$, RAdaGrad also involves $O(r)$ matrix-vector multiplications with $G_t$. The memory usage is $\\mathcal{O}((n+ m)r)$. Both the time complexity and space complexity are of the same order as those of matrix factorization-based algorithms. A more detailed explanation of the computational process is provided in the supplementary material.\n\nWe sincerely appreciate the reviewer's constructive suggestions and believe that the additional experiments, analysis, and explanations significantly improve the quality of our paper. We hope that this provides sufficient reasons to raise the score."}, {"review_id": "HwsfJ64tH3", "reviewer": "Reviewer_9j4S", "summary": "The paper provides a riemannian perspective on low-rank training. And introduces preconditioners for the weight gradients to improve convergence of low-rank equivalents of ADAM and AdaGrad. \n\nThe method is evaluated on LLM finetuning.", "questions": "Question\n- Is G_t the gradient w.r.t W (i.e. a nxn matrix) or somehow split into gradients w.r.t USigmaV^T? \n- Similarly, is H_r an SVD on \\sigma or on W? \n- How sensitive are RAdaGrad and RAdamW to choices of step‑size, decay rates, and smoothing constants? [Actionable item: provide a brief hyperparameter search (can be a smaller model, not gpt2, if compute is a bottleneck). \n\nComments:\n\n- Line 42: Citation [31] does not factorize W=PQ^\\top, but instead factorizes W=USV^\\top. Furthermore, it has been shown [a], that the convergence of the method (DLRT) is not dependent on the condition of W. More generally, the underlying dynamical low-rank approximation method [b] is robust w.r.t the curvature of the manifold, and ill conditioning of the matrix W. This should be rectified. \nIn particular, the issue of ill conditioning of the gradient would only apply to \\nabla_U L and \\nabla_V L for DLRT, but the special treatment in the basis augmentation step prevents the issue.  \n\n[a] https://arxiv.org/abs/2305.19059\n[b] DISCRETIZED DYNAMICAL LOW-RANK APPROXIMATION IN THE PRESENCE OF SMALL SINGULAR VALUES: EMIL KIERI, CHRISTIAN LUBICH, HANNA WALACH", "limitations": "I think scope and limitations are well addressed.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "pro: \n- The paper formulates low‑rank matrix training as optimization on the Riemannian manifold of fixed‑rank matrices, providing clear definitions of the geometry, providing an intuitive but lightweight introduction into the topic. \n- The paper is well structured and easy to read\ncontra:\n- The paper somewhat reinvents the wheel with robust low-rank optimization. There are prior work (see questions below) on this.", "was_revised": true, "final_justification": "I raise my score to 4. \nReason is the discussion with the authors, where they clarified my concerns.", "rebuttal": "**Q1.**  Is G_t the gradient w.r.t W (i.e. a nxn matrix) or somehow split into gradients w.r.t USigmaV^T?\n\n**A1.**  Thank you for raising this critical question. In our paper, $G_t = \\nabla_{e} \\ell (W)$ refers to the gradient of the loss function $\\ell (W)$ concerning the weight matrix $W$ during the $t$-th iteration. Its computation follows standard gradient calculation techniques, specifically using the backpropagation algorithm to compute the partial derivatives of the loss function with respect to $W$. In particular, when calculating $G_t$, we treat $W$ as a complete matrix, rather than decomposing $G_t$ into the gradients of $U$, $S$, and $V$.\n\nWhile it is theoretically possible to project the gradient onto the components of the singular value decomposition (SVD) of $W$, i.e., UΣVᵀ, this is neither necessary nor optimal in our algorithm. The reasons are as follows:\n\n**[1]** **Core Idea of the Algorithm.**  Our algorithm is designed to directly optimize the weight matrix $W$. Calculating the gradient of $W$ and updating it as a whole aligns with this core idea, making our algorithms both simpler and more efficient. If $W$ were decomposed into low-rank factors $U$, $S$, and $V$, and updated separately, this would still introduce dependencies on the Jacobian matrix $J$ and the condition number of the Hessian matrix, potentially slowing convergence. By avoiding these dependencies, our method achieves a more efficient optimization process.\n\n**[2]** **Computational Efficiency.** Decomposing the gradient into $U$, $S$, and $V$ incurs additional computational overhead, such as calculating the inverses of $U$ and $V$, among other operations. This overhead becomes particularly significant when the dimensions of $W$ are large. Directly computing the gradient of W avoids these extra computations, making the algorithm more efficient.\n\n**[3]** **Numerical Stability.** Computing gradients with respect to the singular vectors $U$ and $V$ can lead to numerical instability, especially when $W$ approaches singularity. On the other hand, directly calculating the gradient with respect to $W$ avoids this issue, thereby improving the numerical stability of the algorithm.\n\nIn summary, to balance efficiency, numerical stability, and simplicity, we choose to directly compute the gradient with respect to $W$ and perform optimization in the space of $W$.\n\n**Q2.** Similarly, is H_r an SVD on \\sigma or on W?\n\n**A2.** Thank you very much for your question. Here, $H_r$ represents the $r$-truncated SVD. Specifically, when it is applied to a matrix $W \\in \\mathbb{R}^{m \\times n}$, the computation process is as follows:\n\n**[1]. SVD.** First, we perform SVD of the matrix $W$, obtaining $W =U \\Sigma V^T$, where $U$ and $V$ are the left and right singular vector matrices, respectively, and $\\Sigma$ is the diagonal matrix containing the singular values.\n\n**[2]** **Truncation of Singular Values.** We retain the top $r$ largest singular values in the diagonal matrix $\\Sigma$, and set all other singular values to 0, resulting in the truncated singular value matrix $\\Sigma_r$. The core purpose of this step is to reduce the rank of the matrix and extract the most significant feature information.\n\n**[3]** **Reconstruction of the Low-Rank Matrix.** Using $U$, $\\Sigma_r$, and $V^T$, we reconstruct the matrix as $H_r(W) = U \\Sigma_r V^T$. The reconstructed matrix $H_r(W)$ has a rank of at most $r$, and it is the closest rank-$r$ approximation to the original matrix $W$.\n\n**Q3.**  How sensitive are RAdaGrad and RAdamW ... if compute is a bottleneck.\n\n**A3.**  Thank you for your valuable question, which helps us further improve our experiments. We have actually conducted a detailed sensitivity analysis of the hyperparameters (learning rate, decay rate, and smoothing constant) for RAdaGrad and RAdamW, although this information was not included in the original manuscript. We now present it here. The experimental results show that both algorithms exhibit low sensitivity to the choice of these hyperparameters, meaning that their performance remains stable even when these values are varied within a relatively wide range.\n\nTo illustrate this more clearly, we conducted a hyperparameter search and summarized the results in the tables below. Specifically, we performed a grid search for the learning rate, decay rate, and smoothing constant to examine their impact on the target performance. The results demonstrate that RAdaGrad and RAdamW are robust to changes in these hyperparameters.\n\n\n**RAdaGrad**\n\nThis table shows the training loss and evaluation metric with different weight decay when fixing the learning rate (step-size) as 5e-3, smooth parameter $\\beta_1=\\beta_2=0.98$, rank as 4. The metrics vary slightly within the margin of error.\n\n| Metrics/weight decay | 1e-2  | 1e-3  | 1e-4  |\n| -------------------- | ----- | ----- | ----- |\n| Training loss        | 2.56  | 2.56  | 2.56  |\n| BLEU                 | 69.8  | 70.0  | 69.7  |\n| NIST                 | 8.80  | 8.82  | 8.79  |\n| MET                  | 46.5  | 46.5  | 46.5  |\n| ROUGE-L              | 71.1  | 71.2  | 70.9  |\n| CIDEr                | 2.49  | 2.51  | 2.50  |\n\nThis table shows the training loss and evaluation metric with different learning rate when fix the weight decay as 1e-2, smooth parameter $\\beta_1=\\beta_2=0.98$, rank as 4.\n\n| Metrics/learning rate | 1e-2  | 5e-3  | 1e-3  |\n| --------------------- | ----- | ----- | ----- |\n| Training loss         | 2.55  | 2.56  | 2.59  |\n| BLEU                  | 69.5  | 69.8  | 69.3  |\n| NIST                  | 8.76  | 8.80  | 8.79  |\n| MET                   | 46.9  | 46.5  | 45.9  |\n| ROUGE-L               | 71.4  | 71.1  | 70.5  |\n| CIDEr                 | 2.52  | 2.49  | 2.44  |\n\n**RAdamW**\n\nThis table shows the training loss and evaluation metric with different weight decay when fix the learning rate as 8e-3, smooth parameter $\\beta_3=0.9, \\beta_1=\\beta_2=0.98$, rank as 4. The metrics vary slightly within the margin of error.\n\n| Metrics/weight decay | 1e-2  | 1e-3  | 1e-4  |\n| -------------------- | ----- | ----- | ----- |\n| Training loss        | 2.56  | 2.56  | 2.56  |\n| BLEU                 | 69.8  | 68.8  | 70.0  |\n| NIST                 | 8.81  | 8.76  | 8.83  |\n| MET                  | 46.5  | 45.5  | 46.3  |\n| ROUGE-L              | 71.1  | 70.1  | 71.2  |\n| CIDEr                | 2.51  | 2.42  | 2.48  |\n\nThis table shows the training loss and evaluation metric with different learning rate when fix the weight decay as 1e-2, smooth parameter $\\beta_3=0.9, \\beta_1=\\beta_2=0.98$, rank as 4.\n\n| Metrics/learning rate | 1e-2  | 8e-3  | 5e-3  | 1e-3  |\n| --------------------- | ----- | ----- | ----- | ----- |\n| Training loss         | 2.57  | 2.56  | 2.56  | 2.59  |\n| BLEU                  | 69.2  | 69.8  | 69.1  | 69.2  |\n| NIST                  | 8.76  | 8.81  | 8.75  | 8.76  |\n| MET                   | 46.0  | 46.5  | 46.1  | 45.8  |\n| ROUGE-L               | 70.7  | 71.1  | 70.7  | 70.2  |\n| CIDEr                 | 2.46  | 2.51  | 2.47  | 2.44  |\n\nIn all experiments, we use the same training dataset and evaluation metrics. The results indicate that the performance of RAdaGrad and RAdamW varies only slightly across a broad range of hyperparameter values, confirming that both algorithms are not sensitive to hyperparameter selection. Based on these results, we conclude that RAdaGrad and RAdamW have high robustness to the choice of learning rate, decay rate, and smoothing constant, allowing good performance to be achieved without extensive hyperparameter tuning.\n\n**Comments**  Line 42: Citation [31] does not factorize W=PQ^\\top $\\cdots$ step prevents the issue.\n\n**A1.**  Thank you very much for your thorough review. Your feedback has greatly improved the clarity and presentation of this paper. Regarding reference [31], our original intention was to align its description with other references [15, 18, 37, 39, 41] in the context of factorization methods. We will revise this citation in the next version and include references [a, b].\n\nRegarding the dependency on the condition number, we will modify our explanation. In fact, we explicitly stated in line 107 that DLRT does not depend on the condition number of the weight matrix $W$. However, the primary distinction between our algorithm and DLRT lies in the use of a novel data-driven metric in our algorithm, which yields a preconditioned Riemannian gradient and mitigates the effect of the Hessian matrix's condition number.\n\nMore importantly, we aim to reveal a unified framework for Riemannian gradient descent algorithms:\n\n$$\nW_{t+1} = \\mathcal{R}(W_t - \\gamma_t \\nabla_{\\mathcal{M}_r} \\ell (W_t)),\n$$\n\nwhere $\\nabla_{\\mathcal{M}_r} \\ell (W_t)$ is the Riemannian gradient and $\\mathcal{R}$ is a retraction operator. Within this unified framework, more efficient algorithms can be constructed by selecting different metrics and retraction operators. As described in Section 3.1, we select the standard metric and $r$-truncated SVD as the retraction operator, resulting in the plain RGD. In Section 3.2, we similarly chose the $r$-truncated SVD as the retraction operator but adopt a new metric that combines AdaGrad and Shampoo. This new metric ensures that the gradient exhibits isotropic properties, thereby mitigating the effects of the condition number of Hessian $\\nabla^2 \\ell (W)$ and ultimately leading to the fast and efficient RAdaGrad. **This is the key distinction between our method and DLRT.**\n\nWe sincerely appreciate the reviewer's constructive suggestions and believe that the additional experiments, analysis, and explanations significantly improve the quality of our paper. We hope that this provides sufficient reasons to raise the score."}]

Author rebuttal:

Meta-review: Following an active discussion between authors and reviewers, all reviewers recommended acceptance except for one, who did not reply to the authors or participated in the discussion. I think the authors provided a satisfactory reply to this reviewer. In summary, the paper was deemed an interesting contribution to the timely area of estimating or adapting neural nets with low-rank matrices.

A further comment from the AC: in line 33 and 40, an important problem in low-rank DNN compression is to learn the rank at each layer. A relevant citation: "Low-rank compression of neural nets: Learning the rank of each layer", CVPR 2020. This work uses gradients on the full matrix but penalizes it towards a low-rank matrix which is the result of a SVD at each step, but whose factors are not updated. How does the discussion in lines 39-65 apply here? Although the authors' method is different, there are similarities in the goals to directly update the entire low-rank weight matrix rather than its low-rank factors, and to avoid the non-uniqueness and redundancy introduced by factorization.
Decision: Accept (poster)