A geometric framework for momentum-based optimizers for low-rank training
Authors: Steffen Schotthöfer
Timon Klein
Jonas Kusch
Venue: NeurIPS 2025 poster ()
Keywords: Low-Rank
Compression
Finetuning
Optimization
Manifold
Primary area: deep_learning

Abstract: Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used to train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape. To address this, we introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure. Our approach leverages and combines tools from dynamical low-rank approximation and momentum-based optimization to design optimizers that respect the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating faster convergence, and stronger validation metrics at given parameter budgets.

A geometric framework for momentum-based optimizers for low-rank training
Steffen Schotthöfer" Timon Klein and Jonas Kusch
Abstract
Low-rank pre-training and finetuning have recently emerged promising tech- niques for reducing the computational and storage costs large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge ometry of the underlying optimization landscape. To address this, we introduce novel training strategies that combine dynamical low-rank approximation with momentum-based optimization, explicitly accounting for the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating stronger validation metrics at given parameter budgets.
local optimum due to the ge-
Introduction
Deep learning models have achieved remarkable success across natural language processing and computer vision tasks, but their deployment remains computationally expensive due to the large number of trainable parameters. To address this, parameter-efficiet strategies have been developed to reduce memory and compute requirements during training. Common approaches include sparsifi- 6. and layer factorization. The latter has gained considerable
cation 9 22 12 quantization 37
attention for pre-training [36] 16, 28
41 and especially finetuning [13] 34 42 [10 43 20, (27). Layer factorization represents weights adapters) low-rank matrices, allowing only the low-rank factors to be trained. This significantly reduces both memory usage and computational cost.
In the class of low-rank layer factorizations, applied typical optimizers such as stochastic gradient descent (SGD) or Adaptive Moment Estimation (Adam) [17] directly to the low-rank factors. The combination of these optimizers with LoRA does not guarantee an optimal optimization trajectory [27] 42 To overcome the former challenge, various improvements have been proposed: For example, LoRA+ 10] proposes the of separate learning rates for different components of the low-rank decomposition, Dora 21 normalizes the factor matrices and introduces magnitude parameter. Furthermore, to overcome the challenge of tuning the rank of the LoRA ansatz, AdaLoRA [42) adaptively allocating the parameter budget during training, by masking off rows and columns of the adapter matrices. Other low-rank methods focus entirely on the optimizer states: Galore (43) projects full-rank weight gradients into a low-rank subspace to reduce the memory footprint the optimizer state. GaLore 77] generalizes this technique to high-order mansor-parameterizee models, further improving efficiency for large-scale architectures. Adafactor [1 approximates the memmettmoment matrix using a low-rank decomposition of Adam with low-rank factors.
of the most popular methods is LoRA 115, which
"Computer Science and Mathematics Division; Oak Ridge National Laboratory; Oak Ridge, TN 37831 USA; Mail correspondence: achotthofers@omn.. gov Department of Mathematics; Otto von Guericke University Magdehurg: 39106 Magdeburg: Germany Scientific Computing: Norwegian University of Life Sciences; Drobakveien 31, 1433 As; Norway
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

A particularly promising class of layer factorization strategies is dynamical low-rank training (DLRT) which has been introduced in [28] and has since been used in various tasks 44
29 27 S
DLRT projects the gradient flow dynamics onto the tangent space of the manifold of low-rank parameters, thereby achieving convergence guarantees to low-rank optimal weights (27 However,
these projections are inherently non-smooth, leading to ill-conditioned optimization landscapes and requiring smaller learning rates for stable training [28]. To ensure robust integration of such projections, DLRT constrains movement to flat subspaces within the low-rank manifold, enabling stable convergence to low-rank optima. The method is rank adaptive- using a basis augmentation and subsequent singular value-based truncation criterion to adapt the rank of the low-rank factorization. further enables extensions to increase adversarial robustness of the compressed neural networks by enforcing orthonormality of the low-rank bases and projecting onto well-conditioned manifold [26 or regularizing the condition number of the factor matrix (30]
Despite its efficiency, current DLRT approaches primarily rely on SGD, and it remains unclear how adaptive optimizers such as Adam [7 can be effectively applied. This is a significant limitation, as many state-of-the-art models depend on momentum-basee optimizers like Adam and its variants for performance and stability. Therefore, the extension of DLRT to such optimization techniques is crucial.
To bridge this gap, we introduce a novel momentum- based optimization framework for low-rank pretraining and finetuming. The method integrates adaptive momentum techniques into the framework of dynamical low-rank training (DLRT), preserving the low-rank structure of model weights while enabling stable and efficient updates. This establishes a link between low-rank optimization and adaptive gradient methods, yielding both theoretical insights and practical improvements. Beyond the method derivation, we analyze why LoRA-style adapters- andirs--anddddd and low-rank parameterizations more broadly require momentum-basee based optimizers that are aware of the geometry of the low-rank parameter space, i.e., the underlying manifold structure. Naively applying standard optimizers such as heavy ball to low-rank parameterizations can produce updates that do not correspond to gradient flow leading to a low-rank optimum. As result, these methods may fail to converge to valid low-rank gradient flows
solutions. We show how DLRT can be adapted to approximate seometry-respectingg that consistently drive convergence toward low-rank optima. Together. these contributions lay foundation for more robust and efficient training of large-scale models under low bow-rank constraints.
Compared to low-rank methods that act on the optimizer states only 43 31 we provide holistic interpretation of a low-rank optimization algorithm that adaptively compresses the network weights, gradients and optimizer states simultaneously, achieving superior compression performance at high validation metrics. We remark that the method is directly extendable for tensor-valued neural networks using, e.g., low-rank Tucker factorization.
The paper is structured as follows: We first discuss limitations of naive momentum methods and show how to adapt the underlying gradient flow to achieve convergence to low-rank optimum in Section 2 While the adapted gradient flow facilitates convergence, constructing robust numerical optimizers from it is challenging due to its inherent stiffness. We propose a low-rank heavy ball optimizer in Section 3 which integrates the adapted gradient flow robustly. In Section we construct fully low-rank Adam optimizer by leveraging insights gained in Section 3 In Section we underline the efficiency of the proposed method through numerical experiments. In particular, we demonstrate fast convergence and superior validation accuracy at high compression levels for training from scratch, transfer learning, and low-rank finetuning of different neural network architectures and benchmarks.
2 Momentum- low-rank training
We consider low-rank neural network of the form
N(x)= oLU ( ==)1
where z1-1 (x) is defined recursively by
(0())= = x. E Ro0 and (x) = 07 (U.SVV) 2 =x-((x)) E RR, =I L.
(1)
Here, the weight matrices are defined as W; = U, E RuXm--, where U, E Raxx and Vi E - are orthonormal low-rank factors, and S R is the coefficient matrix. Thus,
Wi lies in the manifold of rank 7 matrices which we denote by Mr Additionally, or represents

---

the activation function of layer I. For simplicity of notation, we do not consider biases, but model with biases can always be expressed as Eq. ( by folding biases into weights and creating an input dimension that is always one. Several methods have been proposed to train the low-rank weight matrices Wi to minimize a given loss function L. Among these, the simplest training rule is the steepest descent method, which, for a fixed layer 1 with weights W = USVT = W reads
U U -
S = AVsL", V V \VvL".
where À is the learning rate, the index n denotes the training iteration, and we have used the shorthand notation _ == L(U SmV2..T) ). more general framework that facilitates the numerical analysis and construction of novel numerical methods interprets the steepest descent method as an explicit Fuler time discretization of the continuous gradient flow equations
(t) vu,, -st): v(t)
-vv,
where L == ((U())((())))))) Then, the steepest descent update equation for the individual factors can be obtained through forward Euler discretization of the pseudo-time f. E.g., the update equation for U can be retrieved through Ü(x)) R {(Un+1 - U")) and L R L While steepest descent methods offer a simple strategy to drive the parameters to a locally optimal point, one of the most widely adopted strategies for updating factorized parameters W, along with their factorized momentum terms V = UySy VV" involves momentum based optimization. For instance, in the case of the heavy ball method applied to all low-rank factors individually, e.g., in LoRA [15), the associated update equations take the form
Um+1= U + AUB
UB+1 = (1 - Xx) UB A VoC,,
VeL = (V"""")""(S"),, (3a)
p*+1 = 1" + + A
F = (1 - AV
VvL" = [V (3b)
gm+1 = + X SSD
Sp+1 =(1 -1- Xr) S3 A VsL",
VsL = (VV))(VV""")""" (3c)
The associated gradient flow equations are given by:
Ú Uv.
Up+ + VuL=0, VeL (VwL)VST
(4a)
V = Vv, + + VyC=0, VvL = C)
(4b)
$ = Sv, Sp+78y VsC =0, VsL = U VWLV.
(4c)
where y denotes the momentum decay parameter. While these equations are optimal when treating all low-rank factors in isolation, they do not account for the fact that factors change simultaneously. For example, Eq. (4a) expected to drive the solution to an optimum, only if (1) and ((I) remain constant in time. When accounting for the dynamics of all three low-rank factors, the resulting gradient flow for W(I) = ((V) takes the form
W = UVSVT + UU + +USVJ and V + 3+V = -VULSoVY ULV, - -U
(5a)
-(V SoVJ - -UU VWCVV - =: WPWW
U -
(5b)
This can easily be shown with the product rule, e.g., = ÜSTT + +USV + +UST and plugging in time derivatives from Eq. ④ These evolution equations for W and V are fundamentally different from the momentum-based gradient flow equations of the full-rank problem
Wian = Viau , Viall + Vaall = - V L(Waa)).
(6)
Indeed, a proper formulation of Eq. (6) that drives the weights of a heavy ball method local optimum while preserving the low-rank representation of W, requires that
W = P(W),, シサンV -PW))W
(7)
where for W = T the projector onto the tangent space is given by P(W) = UU Z(1
VVT) + ZVVT, see 18 Lemma 4.1] and Figure |
for geometric interpretation. Note that in
*We restrict the discussion to single layer without loss of generality, following the arguments of (27 Appendix l].
3

---

abuse of notation, we have recycled W and V hhere to denote the weights and momentum terms following Eq. 7 instead of the naive Eq. 4. Then. the time evolution will drive W into a O, see Theorem [ This steady
low-rank steady state (W*, , such that PV ((W*)) state thus fulfills the optimality condition of a local optimum, see, e.g. 25 Theorem 3.4]. Such a condition is not ensured by the simultaneous descent equations of since, in general, W & P(W)V and P(W, VL & P(W)V L., see Theorem Therefore, training low-rank factors with conventional momentum methods does not necessarly ensure convergence to low-rank optimum.
We aim to derive a numerical method that is consistent with the optimal gradient-llow equations 77. A central limitation of Eq. 7 is that it does not preserve the low- rank structure of the momentum term V. thus leading to prohibitive computational costs and memory requirements. Instead, we aim to derive method that fulfills
VwL
V3 c
TM.
v' / V
(8)
which preserves the low-rank structure of the momentum term. Indeed, the factorized solution of Eq. (8) fulfills (see Theorem 2
Uy = -- -TT V V LV.S''
(9a)
Vy- - -V-
U,S
(9b)
Sy = - - V
(9c)
with initial condition (0(0= = (0) and V (O).
Figure I: Geometric interpretation of Al- gorithm We compute the parametriza- tion of the tangent plane TM.. Then, we compute the projected gradient V y to construct the low-rank momentum up- date. The momentum optimizer is then applied to the low-rank weight coeffi- cient S. Lastly, we retract the updated coefficients back onto the manifold M, The interpretation of Algorithm ii anal- ogous. LoRA-like methods do not em- play orthogonal projections onto TM.., but instead map the full gradient Vw C implicitly onto M, The linear map (dis- played as the wavy orange line) may map the gradient direction far away from the properly projected gradient flow, leading to suboptimal descent directions.
While this formulation and in particular Eq. 7 provide good basis for constructing numerical methods, it also in- troduces the inverse terms ST and S5 on the right-hand side, rendering the system highly stiff, especially when these matrices are ill-conditioned. This stiffness can treated through robust time integrators [2 3) developed in the field of dynamical low-rank approximation [18) which
have also been used for stochastic-gradien descent meth-
ods in dynamical low-rank training [2 41
29 27
13 30
In the following section, we formulate an algorithm that provably approximates the gradient flow of Eq. (9) by following ideas of (2). It turns out that this method is consistent approximation of the optimal gradient flow of Eq. 7 under mild assumptions.
3 A low-rank heavy ball method
To approximate Eq. we start with the first time step from by = 0 to 4 = > and note that by definition of the initial condition U : (to) (yoo and y := V(t.) = (vo).). Let us first construct an augmented basis to ensure that the range and co-range of V are fully spanned. To ensure robustness to small singular values, we introduce change of variables and evolve the basis along Ky(t)= = (t)) and [t( = (t)()Sy(tt) for E [fo, f1] while keeping V = (/(()) and T = (too fixed, respectively
Then, using the product rule and derivatives from Eq.
get
kt(() = (ttt))=(((tt/+++0v00) UV))))
(
Kv(to) = U 50.
iv(() = ()())s)))))) + (1)
U') Lv((o) v S..
As no ill-conditioned S terms affect the dynamics, one can use a forward Euler step to update Ky and by from to to the next time step 4. Thus, we get for K ~ Kv((n), L3 Re (t,) and
T = (t.)
KG = KB - AV L(W
with KΦ = 50
1 = 1. - A W T with LG Vd
4

---

Algorithm I: Single iteration of the dynamical low-rank momentum method.
The functions masissaaumentationn and truncation are detailed in Algorithm in the appendix.
Input: Initial orthonormal bases U, V E R and coefficients S, Sy E R singular value threshold for rank truncation;
X: learning rate.
Evaluate LUUSVT)
/* Forward evaluate */
Go VoL(USV Gy + VVL(USVT)
/* Backprop */
[ Ü -
Gv)
3
+ basis_ Dugaisaagmentation((.
/* in parallel */
S Ū UTUSV + UTUS
Evaluate L(USVT)
/* Forward evaluate */
Gs VzL(USVT)
/* Backprop */
AGs: 8 ASy 8 S, + trrnnation... VVV.)
/* coefficient update */
Denoting orthonormalization algorithm like Gram-Schmin ortho, K3 and L therefore spanned by
U ortho Vw )( V oV, (VwL(WW)) T
Following [40 Cor. 2.2], we note that VoL LVS and VvL= (Vw UUS, meaning that Ö and can be rewritten as
Ü = OUt VuC(W))), V = or,,o VvL(WW)).
We note here that this choice of the updated bases for V also approximates the updated parameters (t11) of the equation W V. With an implicit Euler time discretization, we have
W = SO. + AÜSI
ÜÚ T V +
ÜS
where 3' = 0 A and S is the time- updeted coefficient matrix of V which we will derive in the following: Solving Eq. (9c) with fixed bases Ü and V yields
( ( Z(W
Sy(ta) = G. u"
Here, choose (ta) as the coefficient matrix S projected the updated bases Ô and V. This choice is crucial as it ensures the momentum term to be spanned with the updated basis. Using forward Euler time discretization yields
$ = (1-))001 T" Xo Vw
T W SO70 V
Let us note that with $
have V L VV
Thus, with Sy = :0 s vo, 0 V, the final coefficient updates (including the update for S) are
$ = - -) AVgL(STT
31 5 A.
We note that the basis for W and remain identical after one time update, thus the above derivation holds for general time updates from by to to+1- Since, by construction, the updated bases Ü and V have doubled in rank compared to 1 and V", perform a truncation step. The truncation can be performed back to the original rank r, or formulated with a relative truncation threshold for given tolerance parameter & T|| 2 or rank budget 42], enabling rank adaptive method.
5

---

Algorithm 2: Single iteration of the low-rank Adam method. The functions basis masis_aumeentae
and truncation are detailed in i the appendix.
Input: Initial orthonormal bases U, V E guxr and coefficients S, Sy, Sk E R
T: singular value threshold for rank truncation;
A: learning rate;
31.82: .2: Adam momentum parameters;
Small stability constant.
Evaluate L(USVT)
/* Forward evaluate */
2 Gu + VuC(USV Gv + VVC(USVT)
/* Backprop */
Ü + basis_ dasis_augmentationl...o
/* in parallel +/
1 + basissaugmentation//
S ÜTSSVV Sv U Sx (iTUVSENTD)*
5 Evaluate [(USVT)
/* Forward evaluate */
Gs VzL(ÜS²T)
/* Backprop */
7 Sy B + (1 (Gs)²
Sk B +
b Modifications for adaptive update
Sy 53 Sx
S
/* Bias correction */
1-
1-37
10 St
/* Adaptive coefficient update */
U,S,V, Sx trruncation(3,3,, 0x,
One step of the resulting method using relative truncation threshold is summarized in Algorithm main distinction of this method from a naive application of heavy ball method to DLRT 28 is that 1) our method uses the same bases for parameters and momentum terms, 2) our method does not use momentum method to update the bases, but uses a classical basis augmentation instead, and 3) the method projects momentum terms onto the new bases after the basis update. These three choices ensure that the method approximates the optimal gradient flow of Eq. 7 independent of the condition number of S and S,,'' when the truncation tolerance is sufficiently small, which we make rigorous in Theorem
Theorem 3 shows that the low-rank momentum method produces solutions that are close to solutions of full-rank (baseline) trained neural networks. Thus, we expect the validation accuracy of the trained low-rank networks to match the full-rank baseline. This is empirically confirmed in Table
4 A low-rank Adam method
While heavy ball methods are the cornerstone of momentum-basee optimization, they are commonly outperformed by momentum-bant based optimization methods that include stepsize control. Among these, perhaps the most popular method is Adaptive Moment Estimation (Adam), which was introduced in [17 and has significantly impacted the machine learning community; see Algorithm for reference formulation of Adam. While Adam exhibits superior performance, the non-linearities it introduces and missing gradient flow formulation make rigorous derivation of extension to LoRA-type training difficult. In the following, we use the insights gained from the heavy ball method to construct low-rank Adam optimizer. Adam's 's main distinction from heavy ball methods is the adaptive stepsize control, which determined from the exponentially weighted moving average of the first and second moment of the gradient, denoted by V and K.
naive update of the exponentially weighted moving average of the first and second moment of the gradients with respect to the coefficients S, which we denote as Vs, Kig 222rx2r would read as
V 3.V3 + (1 SL and
k
32K3 + (V)()

---

The bases U, V of the weight factorization could be updated as in Algorithm 1 However, this naive approach does not account for the fact that K2 and k respectively V and V belong different bases, which are updated between optimization steps by the augmentation-trunaaion mechanic.
Instead, we leverage the discussion in Section and propose to project the previous weighted moving average to the updated bases. We note that through the construction of the low-rank bases, no step-size control is required to update U and V since the basis spans weights and moments the old and current time steps and linear combinations in between. To facilitate the discussion, we denote the first low-rank moment Sy E IR*r", analogously to the momentum term in Algorithm and note that the update of Sy can be performed using the strategy of Algorithm I with Sy -0 U vo.T V and 5 == 17 (TT055 V i.e.
S
+ (1-
The dynamies of the second moment not follow the gradient flow directly, but have non-linear dynamics depending on the square the gradient. We remark that, the (full-rank) second moment Æ element-wise positive, which is important for taking the square root in the (full-rank) Adam update step W = W
Denoting the second moment of the coefficient matrix by Sx E R
VR+C
which projects the element-wise square root of the second moment of the low-rank coefficient onto the new basis and subsequently squares the elements. This ensures element-wise positivity of the second low-rank moment. The augmented low-rank moment Sx is subsequently updated with the standard Adam update scheme
we propose a modified projection Sx = VST
k 825x (1 (3)) (v&C(USTT))
We apply the Adam bias correction at iteration n. to the low-rank moments, i.e.,
Sy and 5x 1 52
Sp
5, 1 1-3m
and update steps for the coefficient S, i.e. S
8y
VBx+t
The resulting method is summarized in Algorithm 2
We wish to remark that the extension to Adam straightforward: The regularization term added to S directly instead of being combined with the gradient for the moment updates.
Computational and Memory Efficiency
We briefly analyze the computational and memory efficiency of the low-rank Heavy Ball method and 2) the low-rank Adam method. To update a matrix W E Rax" the full-rank (baseline) Heavy Ball method requires (3n?) floats to store the weight W, its gradient Vw and the momentum V. The computational cost is of the same order. In contrast, the low-rank Heavy Ball method (see Algorithm 1 requires 02nr++r2 floats to store the low-rank factorization USV+. and O(2nr+ for the gradients VvL. V sL. Since the bases U and V are shared between the weight and momentum, the momentum term requires only O additional floats. The extra computational costs are (nrr) for orthonormalization during basis augmentation and O(r3) for truncation, both negligible when r & n. Similarly, the full-rank Adam/Adam
method requíres (442²) floats to store W. Vw C, and the two momentum terms V. K, with comparable compute cost. The low-rank Adam method uses (2nr + 2²) for USV and the same for the gradients. The two momentum terms add only ((2r²) due shared bases. The computational cost is computed analogously that of the low-rank Heavy Ball method.
Extension to Tensor-valued layers
The proposed optimizer is directly extendable to tensor-valued neural network layers, e.g. convo- lutional layers, following the extension of the SGD-bbased DIRT method from matrices 28 to tensors 41 To that end, we remark that c.g. 2d, convolution can be formulated as operation
7

---

on an order four tensor W E output data, and sw,, Sp the width and height of the sliding window of the convolution acting on the spatial dimensions h of the input image E Rexhxd, Consider a low-rank Tucker factorization
where dj and de are the channels of the input and
of the weight, i.e. W = C x/=1 U,, with core tensor C' E X and U, E TR.., with n, E {d,, do Sw, 8 . Using the extension of the DLRT method to Tucker tensors [40] and applying Algorithm| respectively Algorithm to to the tensor update yields the desired method.
Numerical Results
Table I: UCM, Cifarl0 and Cifar100 benchmark; Low- rank compressed transfer learning. Accuracy means and std. devs. of 10 stochastic trainings using AdamW. The LoRA ranks are set up to match the compression rate of the results of Algorithm 2 Algorithm 2 achieves higher accuracy at higher compression rates across all benchmarks, compared to DLRT(28 w/o momentum term projection and LoRA-based pretraining w/ momentum terms.
In the following, we showcase numer- ical experiments for low-rank trans- fer learning (Section [.T]. finetuning (Section 5.2 and pre-training tasks
(Section 5.3 Details on training, data
and additional experiments are listed
in Section 10
UCM
Cifart0
Cifar100
Acc [%]
c.r. [5]
Arc [
ca.[5%]
Ace [%]
car [%]
Our primary baseline is full-rank training, where all model parameters are updated without any structural constraints. We compare this against 1) the low-rank finetuning strategy in-
VGG11 VG016
Buseline
98.40:10:22
0.0
89.82_00.45
0.0
65.21
0.0
Alpocithm
94.6110.35
95.84
39,9910.55
95.30
61.3810.46
95.54
D3LRT wio pio
89 52::0.93
93.56
85.00=00228
94.84
60 (0.48:111.7
9X.71
LoRA pretrain
90.6 90.64:2.27
93.57
88.43- 0.25
94.80
63:63:40:46
96.58
Bascline
94.23:00.71
0.0
08.34:0.49
0.0
63.13:041
0.0
Alpycttin
Fl
03.70++-077
94.89
88, 13-00.56
95. 13
60,84+0 40
95.00
DIRT phn.
88 -3-330
96.35
81. 11.95-0.25
97.08
61. 61.59+00.25 25
95.99
troduced in Algorithm 2) naive ap- plication of DLRT without the projec- tion of the momentum terms, and 3) simultaneous direct application of the Adam optimizer on the low-rank factors U, V, as typically done in LoRA [15]. a low-rank adap- tation technique originally designed for parameter-efffcient finetuning of large transformers. For A-ased experiments, we calibrate the per-layer rank hyperparameters to match the overall compression ratio achieved by Algorithm ensuring fair comparison at fixed parameter budgets.
LoRA pretrain
90.1412.56
94.72
$6.63.10.29
94.57
59:51110:00
94,78
Baseline
96.72=0.36
95.42=0.35
0.0
90.34+0.44
0.0
Alpocithm
83.
68.4610.53
DLRE Touj.
84.91
91,95-00.00
54.95
03.09
75.83
LoRA pretrain
86.542291
84.94
94.10120.56
50.78
76.76:30.33
74.36
The compression ratio is defined as: C.I.-
# paramms low-rank model
x 100.
# params baseline model
5.1 Low-Rank Compressed Transfer Learning
VGG16, VGG11, and ViT-B.16 on SON/CIFAR-10/CTAARC across three network architectures, VGG16 and VGG11, and the VIT-B.16 16 Vision Transformer on WWW/CIIAR-LOCCIAAR--AA
We evaluate performance
The convolutional VGG11 and VGG16 networks are selected to validate the performance of Algorithm 2 on tensor-valued layers, here given by the convolutional layers. We use the low-rank Tucker tensor format to compress and train the convolutions; for details, we refer to 30 40
Table 2: DeBERTaV3-base finetuning on GLUE We compare with full finetuning (Full FT), Houlsby adapter 14] (HAdapter). Pfeiffer adapter (3 (PAdapter), LoR A [15), Adal.oRA 42), GeoL A (7).
DoRA 21), LoRA+[10] and Bitfit We report target metrics and computational performance (higher is better) for the median of 5 runs using different random seeds. Best results per dataset are shown in bold. Results for BitFit, HAdapter, and PAdapter were taken from 42). " "AdaLoRa matched"
has the rank budget adapted to approximately match the final parameter count of Algorithm 2
Method (f Params)
SST-2
CoLA
QOP
QNLI
RTE
MRPC
STS-B
Mean
(Ace]
(Mec)
(F1)
(Acc)
(Acc)
(Ace)
(Clor)
Full FT (154M)
95
69.19
89.30
94.03
83.75
89:46
91.68
87.63
BaFi (OLIM)
94.84
66.96
84.95
92.24
78.20
87.35
91.35
85.25
HAdaaper (1 1.222M)
95.53
68.64
89.27
94,11
$4.45
89.95
91.48
87.63
PAdupter (118MM)
95.61
68.77
89.40
94.29
$5.20
89.46
91.54
87.75
LaRA 1-8 (L.33M)
95.29
68.57
90.61
93.01
$5.50
89.75
89.10
87.53
LaRAs and (1.33M)
95.37
69,22
90.3
93.96
$5.50
89.55
88.07
87.49
DoRA (1.35))
16
68.50
90.71
94.31
$5.05
89.32
91.38
87.65
AdILoRA (1.27M)
95
68.76
90.63
94.11
36.00
89.44
91.41
88,00
AdLLoRA, matched
95 64 (
68. 59 (1.07M)
90
(0.72M)
90.93 (0.72M
85.92 (L.15M)
58.21 (0.74M)
90.9110.7:M
87 66 (M91MM
GeoL.oRA
699,03 (0.95M)
90.55 (0.09M)
(0.70M)
(1.19M)
(0.71M)
86. (0.55M)
Algoriiin
96.112
(1-11M)
69.58
90.62
(0.76M)
(0.700))
(1.19M)
90.84
(0.766M)
91.5
88.7500899M)
8

---

We initialize all convolutional networks with PyTorch ImagenetIK weights and V1T-B.16 with Huggingface Imagenet21K weights. Stochasticity during training stems from randomized mini- batching. For each experiment, we report the mean performance across 10 independent training runs with different random seeds. We observe in Table [ that Algorithm 2 matches the validation accuracy of the baseline network in most test-cases, and surpasses the baseline in e.g. UCM/VGG16 while achieving compression rates of up to 95%. We point out that a naive implementation of DLRT without the proposed adaptation of the optimizer states causes performance drops of 5 to 13%. LoRA also struggles to achieve high accuracy at the prescribed compression rates.
5.2 Low Rank Adaptation for Parameter Efficient Finetuning (PEFT)
VEBERTaV33-aalt on GLUE We fine-tune the 183M parameter De transformer model [TT] on the GLUE benchmark suite [35]. The corresponding results are summarized in Table Overall, Algorithm 2 consistently outperforms competing methods, especially other rank adaptive methods as GeoLoRA 177 and AdaLoRA 42 on most tasks, achieving stronger validation metrics. The required number of trainable parameters is substantially lower than the compared fixed-rant methods. The average score higher than the reference methods, and the average parameter count for the finetuning tasks is lower than the next best method, which is GeoLoRA.
Llama2 7b-chat-hf on BoolQ and PIQA We compare Algorithm 2 with LoRA on Llama-2- 7b-chat-hf 33 across reasoning benchmarks,
Table 3:
Llama2 7b-chat-hf (3) finetuning soning datasets. We compare with LoRA [5] and report the best accuracy and the wall-time. The wall-time is reported for three epochs with batch size 12 and maximal sequence length of 640 tokens on a single NVIDIA H100.
including Q 4 and PIQA [1. as reported in Table 3 Inputs consist of either a pas- sage-question pair or standalone question with multiple-choice answers, and evaluation is based on answer accuracy. We also report wall-clock time on a single NVIDIA H100 GPU, showing negligible runtime overhead of Sigorithm@@over LoRA. Algorithm@outpee outperforms LoRA configurations with matching initial rank and with rank chosen to approximately match the final parameter count.
BoolO
PIQA
[18]
Arc (%)
Wall-Tmme
[%)
Wall
Algunilbe
99.73%
$4.89
186uin
00.75%
76.77%
225min
LoKA tpulT
95 82%
62.17 18
175mm
09.829
225mm
LaRA (--10))
99.70%
60.17 98
184mii
99.70%
50.45%
225mis
Table 4: Pretraining GPT-2 [24] repro- duction (124M) from scratch on Open- WebText g for 15,000 iterations.
5.3 Low Rank Pretraining
GPT2 on Web We pretrain Karpathy's 's re- production of the 124M-parameter GPT-2 model (24)
from seratch on the OpenWebTexx dataset [8] using next-
Method
c.r. [%]
validation loss [%]
word prediction. As seen in Table 4 and Figure 4 our
Baseline
0
3.2313
method significantly outperforms LoRA pretraining (best validation loss 3.4642 vs. 4.8141), while incurring only moderate increase relative the full-rank baseline (3.4642 vs. 3.2313). Algorithm 2 achieves compression rate of 60.61% ,, compared to 60.79% for LoRA- LoRA-Pretrair o
Algorithm 2
60.61%
3,4642
LORA Pretrun
79.
7,0242
Thus, our approach enables substantial compression of GPT-2 (with the potential for reduced inference time). whereas LoRA yields significant degradation in validation loss.
6 Conclusion
We introduced a principled and provably robust framework for momentum- based low-rank optimiza- tion that is both rank-adaptive and jointly compresses weights, gradients, and optimizer states. Our analysis reveals that the proposed method is resilient to the conditioning of the training problem, while maintaining fidelity full-rank momentum trajectories. Through extensive experiments on pretraining, transfer learning, and finetuning, we demonstrate that our approach consistently achieves stronger generalization performance under tight parameter budgets, outperforming existing low-rank techniques. These results position our method as strong foundation for efficient deep learning, offering a scalable and theoretically grounded alternative to full-rank training across diverse regimes. The accomplished faster convergence and higher compression of the optimizer states during training enable broader applications of machine learning on resource- cooourceconstraiieed devices. Furthermore, the
https:///itthu: httpggitthi..mccaaaattettteeeeee com//karpatay//nnoodrr
We note that approximately 31% of the total parameters are in the encoding head of the model which is not compressed.

---

method is computationally efficient and scalable. These achievements also enhance computational and memory efficiency, positively positvvvell impacting society.
Funding Acknowledgements
The work of Steffen Schotthöfer is sponsored by the Applied Mathematics Progrm at the Office of Advanced Scientific Computing Research, U.S. Department Energy, and performed at the Oak Ridge National Laboratory, which is managed by UT--Battelle LLC under Contract No. DE-AC05- 00OR22725 with the U.S. Department of Energy.
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-000R22225 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research accordance with the DOE Public Access Planhhttp://enereee
plan).
This project has received funding from the European Regional Development Fund (grants timing- Matters and IntelAlgen) under the European Union's 's Horizon Europe Research and Innovation Program, from the German Research Foundation DFG within GRK 2297 'Mathematical Complexity Reduction', and from the German Federal Joint Committee (Grant 01 VSF23011,, which we gratefully acknowledge.
10

---

References
[1] Y. Bisk, R. Zellers, R. Le bras, J. Gao, and Y. Choi. PIQA: Reasoning about Physical Com- monsense in Natural Language. Proceedings of the AAAI Conference Artificial Intelligence. 34(05):7432-7439 Apr. 2020.
[2] G. Ceruti, J. Kusch, and C. Lubich. A rank-adaptive robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics, pages 1-26, 2022.
[3] G. Ceruti, J. Kusch, and C. Lubich. A parallel rank-adaptive integrator for dynamical low-rank approximation. SIAM Journal on Scientific Compuling, 46(3):B205-B223 2024.
[4] C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions, 2019.
[5] D. Coquelin, K. Flügel, M. Weiel, N. Kiefer, C. Debus, A. Streit, and M. Götz. Harnessing orthogonality to train low-rank neural networks. In ECAI 2024, pages 2106-2113, IOS Press, 2024.
[6] M. Courbariaux, Hubara, Soudry, R. El-Yaniv, and Y. Bengio. Binarized Neural Net- works: Training Deep Neural Networks with Weights and Activations constrained to +1 or -1. arXiv:1602.02883 020000000 2016.
[7] R. J. George, D. Pitt, Zhao, Kossaifi, C. Luo. Y. Tian, and A. Anandkumar. Tensor-GaLore: Efffire-fffiieent Training via Gradient Tensor Decomposition, 2025.
[8] A. Gokaslan and V. Cohen. OpenWebText Corpus.
htttp://Sky1ion007. github..hh...... io/
OpenieebeextCorpus 2019.
[9] Y. Guo, A. Yao, and Y. Chen. Dynamic Network Surgery for Efficient DNNs. Advances in neural information processing systems, 29, 2016.
[10] S. Hayou, N. Ghosh, and B. Yu. LoRA+: Efficient Low Rank Adaptation of Large Models, 2024.
[11] P. He, J. Gao, and W. Chen. Debertav3: Improving DeBERTa using ELECTRA-Style Pre- Training with Gradient- DistentgdDDe Embedding Sharing. 2023.
[12] Y. He, Zhang, and Sun. Channel Pruning for Accelerating Very Deep Neural Networks. In Proceedings of the IEEE international conference on computer vision, pages 1389- 1397, 2017.
[13] A. Hnatiuk, J. Kusch, I.. Kusch, N. R. Gauger, and A. Walther. Stochastic Aspects of Dynamical Low-Rank Approximation in the Context of Machine Learning. Optimization Online, 2024.
[14] N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Efficietr-Efficeen Transfer Learning for NLP. In K. Chaudhuri and R. Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learn- ing, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR, 09-15 Jun 2019.
[15] E. J. Hu, Y. Shen, P. Wallis, 7. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3, 2022.
[16] M. Khodak, N. Tenenholtz, L. Mackey, and N. Fusi. Initialization and Regularization of Factorized Neural Layers. In International Conference on Learning Representations, 2021.
[17] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. In Y. Bengio and LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
[18] O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 454, 2007.
[19] Kusch, S. Schotthöfer, and A. Walter. An Augmented Backward-Correeted Projector Splitting Integrator for Dynamical Rank Training, 2025.

---

[20] V. Lialin, S. Muckatira, N. Shivagunde, and A. Rumshisky. ReloRA: High-rank training through low-rank updates. The Twelfth International Conference on Learning Representations, 2024.
[21] Y. Mao, K. Huang, C. Guan, G. Bao, F. Mo, and J. Xu. DoRA: Enhancing Parameter-Efiie Efffciient Fine- Tine-Tuning with Dynamic Rank Distribution. In L.-W. Ku, A. Martins, and V. Srikumar, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 11662-11611 1165, Bangkok, Thailand, Aug. 2024. Association for Computational Linguistics.
[22] Molchanov, S. Tyree, T. Karras, T. Aila, and J. Kautz. Pruming Convolutional Neural Networks for Resource Efficient Inference. In International Conference on Learning Representations, 2017.
[23] J. Pfeifferr A. Kamath, A. Rücklé, K. Cho. and I. Gurevych. Adapterfusion: Non-Destructive Task Composition for Transfer Learning, 2021.
[24] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language Models are Unsupervised Multitask Learners. 2019.
[25] H. Sato. Riemannian optimization and its applications, volume 670. Springer, 2021.
[26] D. Savostianova, E. Zangrando, G. Ceruti, and F. Tudisco. Robust low-rank training via approximate orthonormal constraints. Advances in Neural Information Processing Systems, 36:66064 66083, 2023.
[27] S. Schotthöfer, E. Zangrando, G. Ceruti, F. Tudisco, and Kusch. GeoLoRA: Geometric integration for parameter efficient fine-tuning. In The Thirteenth International Conference Learning Representations, 2025.
[28] S. Schotthöfer, E. Zangrando, J. Kusch, G. Ceruti, and F. Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equations. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh. editors, Advances in Neural Information Processing Systems, volume 35, pages 20051- Curran Associates, Inc., 2022.
[29] S. Schotthöfer and M. P. Laiu. Federated Dynamical Low-Rank Training with Global Loss Convergence Guarantees, 2024.
[30] Schotthöfer, H. Yang, and S. Schnake. Dynamical low-rank compression of neural networks with robustness under adversarial attacks, 2025.
[31] N. Shazeer and M. Stern. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost. In
J. Dy and A. Krause, editors, Proceedings of the 35th International Conference Machine Learning. volume 80 of Proceedings of Machine Learning Research, pages 4596 4604. PMLR, 10-15 Jul 2018.
[32] P. Singh, G. Bachmann, and Hofmann. Analytic insights into structure and rank neural network Hessian maps. In Advances in Neural Information Processing Systems, volume 34, 2021.
[33] H. Touvron, L.. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bashlykov, S. Batra, Bhargava, Bhosale,
Bikel, L. Blecher, C. C. Ferrer, M. Chen, Cucurull, Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshom, S. Hosseini, R. Hou, H. Inan, M. Kardas, V. Kerkez, M. Khabsa, Kloumann, A. Korenev, P. S. Koura, M.-A. Lachaux, T. Lavril, J. Lee, D. Liskovich, Y. Lu, Y. Mao, X. Martinet, T. Mihaylov, P. Mishra, Nie, A. Poulton, Reizenstein, R. Rungta, Smith, R. Subramanian, X. F. Tan, B. Tang, R. Taylor, A. Williams, J. X. Kuan, P. Xu, Z. Yan, I. Zarov, Y. Zhang, A. Fan, M. Kambadur, S. Narang, A. Rodriguez, R. Stojnic, S. Edunov, and T. Scialom. Llama 2: Open Foundation and Fine-Tune' Chat Models, 2023.
I. Molybog,
Saladi, A. Schelten, R. Silva, E. M.
[34] M. Valipour, M. Rezagholizadeh, I. Kobyzev, and Ghodsi. Dylora: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Lank Adaptation, 2023.
12

---

[35] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A Musk Benchmark and Analysis Platform for Natural Language Understanding, 2019.
[36] H. Wang, S. Agarwal, and D. Papailiopoulos. Pufferfish: Communication-eficieett models at no extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
[37] J. Wu, C. Leng, Y. Wang, Q. Hu, and J. Cheng. Quantized Convolutional Neural Networks for Mobile Devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4820 4820-4828, 2016.
[38] Y. Yang and S. Newsam. aog-of-visual-worde and spatial extensions for land-use classification. In Proceedings of the 18th SIGSPATIAL International Conference on Advances in Geographic Information Systems, GIS '10, page 270-279, New York, NY, USA, 2010. Association for Computing Machinery.
[39] E. B. Zaken, S. Ravfogel, and Y. Goldherg. Bitfit: Simple Parameter-effiieet Fine-tuning for Transformer-based Masked manguage-modette 2022.
[40] E. Zangrando, S. Schotthöfer, G. Ceruti, J. Kusch, and F. Tudisco. Geometry -wwrr training of factorized layers in tensor tucker format. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang, editors, Advances in Neural Information Processing Curran Associates, Inc., 2024.
Systems, volume 37, pages
[41] E. Zangrando, S. Schotthöfer, G. Ceruti, Kusch, and F. Tudisco. Rank-adaptive spectral pruning of convolutional layers during training. In Advances in Neural Information Processing Systems, 2024.
[42] Q. Zhang, M. Chen, A. Bukharin, He, Cheng, W. Chen, and T. Zhao. AdaLoRA: Adaptive Budget Allocation for Seaaeetr--Effcien Fine-Tuning. In The Eleventh International Conference on Learning Representations, 2023.
[43] J. Zhao, Zhang, B. Chen, Z. Wang, A. Anandkumar, and Tian, GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection, 2024.
13

---

7
Notation
Table 5: Summary of notation used throughout the paper.
Notation
Definition
Model & Training
Wi E RERR 1
The weight matrix for layer
L
The loss function.
W
The gradient of the loss with respect to the full matrix W. The learning rate of stochastic gradient descent.
A
Low-Rank Factorization
M,
The manifold of matrices of rank Orthogonal projection of matrix onto the tangent space of M, at W. Low-rank decomposition of W, where U, V orthonormal. Orthonormal basis matrices for the column and row spaces.
P(W)Z
W USVT
E R V E Rui-1Xr
SERrX
The core tensor or coefficient matrix.
Gradients with respect to the low-rank factors U,S, V.
Momentum Terms (Heavy Ball)
The momentum term (a full-rank matrix). Low-rank decomposition of the momentum term. The coefficient matrix for the momentum term.
UVS.VT Sy E 1R"x
The momentum decay parameter.
Momentum Terms (Adam)
The momentum term (a full-rank matrix).
UVSVVT Sy E RR
Low-rank decomposition of 1st moment term (moving average of gradients). Coefficient matrix for the 1st moment (moving average of gradients). The momentum term (a full-rank matrix).
K.
SK
Low-rank decomposition of the 2nd moment.
Sc Rexr
Coefficient matrix for the 2nd moment (moving average of squared gradients).
$1,82
Exponential decay rates for the moment estimates.
Algorithm-Specifec
U,V
Augmented bases after incorporating gradient information. Coefficient matrix projected onto the augmented bases Ü, V. Momentum coefficients projected onto the new augmented bases. The final updated coefficient matrix within a single optimization step. Relative and absolute truncation tolerance parameter for rank adaptation.
Sv
S
Algorithms
We list the helper functions of Algorithm I and Algorithm 1 in Algorithm 3 The baseline (full-
rank) Heavyball method is listed in algorithm 6 and the (full-rank) Adam method is listed in Algorithm 4 The naive application of the DLRT method (40 for Adam is listed in Algorithm 5
Note that the coefficient matrices of the momentum terms are not updated after augmentation and truncation. Analogously, the naive implementation of DLRT with momentum omits updating the momentum coefficient matrix. The implementation of LoRA with momentum or Adam simply applies Algorithm 4 to each of the LoRA factors without regard for the underlying manifold representation.
Numerical Analysis
Theorem (Convergence). Let W(t) the solution Eq. 7 and let C be bounded from below. Then, W converges to a W* which fulfills the low-rank optimality condition
P( 0.
(10)
14

---

Algorithm 3: The functions basis_ augisssauuentt,ionn and truncation on of the used algorithms.
def basis_ angiinnttnmnnt B: old basis, Gg: basis dynamics):
B t ortho([GB B)) /* orthonormalizationn, e.g. return B
Gram-Schmidt */
def truncation(S: augmented coefficient, Sy: augmented momentum, U: augmented busis,
V: augmented co-basis ):
En, Qn t truncated svd(5) with threshold to new rank Y1 U UP + -QQ,
/* Basis update */
U UUTÜS
/* Coefficient update */
return V,
def truncation(S: augmented coefficient, Sy: augmented momentum, Sx: augmented 2nd
momentum, Ú: augmented basis, V: augmented co-basis ):
Pr. Er, Qr truncated svd(S) with threshold to new rank
11
U + UP,; V t VQr,
/* Basis update */
St E-E U UTÜS V: (UTÖVSEÍTV)*
12
/* Coefficient update
*/
13
return U,S.V S.S Sk
14 def wwwnnaaiinnnaaieeeeoooaoo augmented coefficient, 0: augmented basis, V: augmented co-basis
Pr. E Qn truncated svd(5) with threshold new rank T
16
U UPri V < VQn
Basis update */
17
S S
/* Coefficient update */
Algorithm 4: Single iteration of the (full-rank version of) Adam.
Input: : W E R=x", Weight matrix;
V R=x*, Initial 1st moment;
K. E Rex*; Initial 2nd moment;
X: learning rate;
31, 32: Adam momentum parameters;
e 0: Small stability constant.
Evaluate L(W)
2 g + C(W)
Compute gradient
3 V + 81V+ + +1 (1-31)g
/* 1st moment estimate */
4 K. + B2K + (1 82)
/* 2nd moment estimate (element-wise square) */
V v
I- k+ K
/* Bias correction
1-35
W + W - A
/* Parameter update */
R+E
Proof. Let us define the energy as
=()=== + 1000..
The time derivative is given by
E(t) = V w L(W())), (()) + (V(t), = w (t) (()()))(1) + (V(0) P(
Since P is self-adjoint this directly gives
(t)) = (() v(0) + (V00--7(C (() - P(W())=<C(W())
=
15

---

Algorithm 5: Single iteration of the naive low-rank Adam method. The functions basis_ crrriss_ummentaiiio and truncation, naive are detailed in B the appendix.
Input: Initial orthonormal bases U, V E guxx and coefficients S, Sy, Sk E R
T: singular value threshold for rank truncation;
A: learning rate;
31.82: .2: Adam momentum parameters;
Small stability constant.
Evaluate L(USVT)
/* Forward evaluate */
2 GU + VoC(USV Gv + VVC(USVT)
/* Backprop */
Ü
basis_ masiss_agmentttionn(,, (U,Go)
/* in parallel +/
+ basis_ Digiissaageenttion(..ccc
S + ÜTUSVTV
5 Evaluate ((ÜSVT)
/* Forward evaluate */
Gs + VgL(ÜS²T)
/* Backprop */
7 Sy -is + +1 11--110 -s)
Sx + 32Sx + (1 (1- (Gg)²
b Modifications for adaptive update
83 Sy - 1- Sx
Sg
/* Bias correction */
--
8t 5 Sy
/* Adaptive coefficient update */
+x
11 U,S.V, Sv. Sk + truuncationnnaiieffff
Algorithm 6: Single iteration of the (full-rank version of) the Heavy-Ball SGD method.
Input:
Initial parameter vector W R"x";
V: Initial velocity (momentum term);
Gradient g Vw-)(W);
X: learning rate;
y: momentum coefficient.
Evaluate L(W)
g f C(W
Compute gradient
3 V + [1-7)V- Ag
/* Update velocity */
W W +V
Parameter update +/
Hence, if L is bounded from below, this means that lim E(I) = Eoo with Ere finite and
Exp E(0) -
|V(t)]|ddt: dt.
This implies that tom V(t) = 0 and thus lim ((t) lim, (() = 0. Hence, since (tt), ((t) converge to a steady state and lim V(t) 0, the evolution equation for V gives V ((W = 0 as t + oc.
We can obtain a similar, but not equivalent, result when solving low-rank gradient flow of the form instead:
9
Theorem (Convergence of low-rank factors). The low-rank gradient flow
Ün - (I- UvUJ V CVVS Vi -I- VV C
(1
(11b)
Š, 7Sy Li LWV.V
(11c)
fulfills
v= - - P.V)
16

---

Proof. By the product rule we have
=Ü + UvSVKi + UvSDKy
--VV- -I- VW - UVUy CW - UDU V
= -
(12)
Theorem 3
For an integer k. lett kX. Let (t) be the solution of Eq. 7 and
ler W,, Ve be the factorized low-rank solution after k steps with Algorithm Assume that for any Z E Mr in a neighborhood of WT, we have I| ((Z)) < & and [!, V.V.V. V.V.VVV S 0. where || denotes the Frobenius norm. Moreover, assume that the gradient is bounded and Lipschitz continuous. Then,
[ - WT S ci cgd/++ c///,,
(13)
where the constants 012 C22 C3 are independent of singular values of S and Sp
Proof. We start by bounding the local error. That is, we assume that W (4)) W and V(to) Vo where Vi is the momentum of the low-rank method. By definition of Ū we have (I 00 ) and thus
I1 I O0 WO-T & [" ('I (T)())) P(
Using the boundedness of normal components and Taylor expansion around to gives for E h]
P( = VV)((W + O(e)) V C() + ((++))
(14)
Hence, with V(s) = ((6o) + +(A++
[I - -
-
0(22 + As)
= (I - VVVV-007VVV(((()))>+++++0000
By construction of Ū we have 0 = (1-00))0-Q((6))=0-00000
hence
[|( |[0-000)00| S
From this, we directly conclude
II ( W II(T 00 (W(()) + /"' ds)ll + +22))
An analogous derivation for the co-range gives
(W( - 00 W(t + - 00 ((G]) | DTTAI
=0 + X²5).
Next, we need to bound
|100 100T W(G) - -3' /
(15)
We note that from Eq. 14 we have with Wo = (tto) and Vo V(to)
0 V
Ū T (Wo
- (111-111,)))
Vw + 0(2 + +E)
= - Xy - Xo T V Vw + + + +s),
where 5= -. W and S, VTV. f2. By definition of the S-update of Algorithm 2 we have
3 = 3 + A(1 7)S,- X CZ
Thus, since V C(Ū(SVV therefore the local error is bounded by
V L(Wo) we have ( - 11 =0/2 + +ee and
[W - WIll < (W - 00 W(t)) -T 117 0
= -
From the truncation tolerance 3. the bound on the truncation of V, and the stability of the exact flow, can obtain the desired error bound for the global error using Lady Windermere's 's fan. D
17

---

S V.V. is fulfilled for user determined 0, e.g. 1 = 12, by increasing the new rank "1 in the truncation step of Algorithm B necessary. However, since V - when the method reaches a steady state, the effect of this error term is expected to be limited. We remark that the main motivation present Theorem 3 is to rigorously demonstrate the robust treatment of stiff terms in the gradient flow ( The main component in our construction of the algorithm, which removes these stiff terms in our error bound, the construction of the augmented basis matrices U and V.
We remark, that we can always ensure that condition V. U,U,
Theorem 4. The conventional low-rank gradient flow equations can fail converge to a point fulfilling P(W) L(W) = O.
Proof. The potential lack of convergence be proven with counter-example. Consider state where the momentum term is zero, i.e., V = O, by choosing the momentum factors as Uy = -U, Sy O, and Vy V. Now, consider any weight matrix W that is not low-rank optimum, meaning V(W)Vw * O, but for which the naive update term in Eq. (5b) happens to be zero: P(W, L(W) = O. In this scenario, the naive gradient flow equations would identify V) stationary point, since W = 0 and V O. However, this point is not valid low-rank optimum because the true projected gradient P(W)V ((W) is non-zero. In contrast, (W, V) is not stationary point of (4) and the gradient flow (4) will provably drive the system state where P( W(W)) L(W) = O.
We remark that a large class of matrices fulfills P(W) Vw w ((W) + and P(W, V O. For instance, any W USVT where the gradient is non-zero in the range of V but zero in the range of U (i.e., U V w L(W) = O but VwL(WW) + 0) would satisfy this. This can be easily verified: Since = 0, we have
((W UU V VWZ(W)VT =0,
but
P(W)) Vw #0.
10 Details to the numerical experiments of this work
is important to note that the accuracy-vs-e cisturacyvs-coopresttii comppesssoo trade-off varies by application. While low-rank methods excel in finetuning and transfer learning tasks (sometimes even improving upon the baseline), pre-training a network from scratch on a complex dataset often involves balancing memory savings against a potential drop in accuracy.
10.1
ImageNet-Ik, UCM and Cifar Benchmarks
10.1.1 Network architecture and training details
In this paper, we use the pytorch implementation for neural network training. We take pretrained weights from the imagenetlk dataset as initialization, except for the long-term training study using ViT-small, which is randomly initialized. The data-loaded randomly samples batch for each batch- update which is the only source of randomness in our training setup. Below is an overview of the used network architectures
VGG16 is deep convolutional neural network architecture that consists of 16 layers, including 13 convolutional layers and 3 fully connected layers.
VGG11 a convolutional neural network architecture similar to VGG16 but with fewer layers, consisting of 11 layers: 8 convolutional layers and 3 fully connected layers. It follows the same design principle as VGG16, using small 3x3 convolution filters and 2x2 max-pooll max-pooling layers. ViT-B.16 is a Vision Transformer with 16 x 16 patch size, deep learning architecture that leverages transformer models for image classification tasks.
ViT-small is a compact vision transformer with patch size 8 x 8, and an embedding dimension of 512. The model comprises six attention layers, each equipped with two heads, followed by a ResNet block and dropout layer.
18

---

Table 6: Training hyperparameters for the UCM, Cifar10, Cifar100 and ImageNetlk Benchmark. The first set hyperparameters apply to both DLRT and baseline training, and we train DLRT with the same hyperparameters as the full-rank baseline models. The second set hyper-parametent specific to DLRT. The DLRT hyperparameters are selected by an initial parameter sweep. We choose the DLRT truncation tolerance relative to the Frobenius norm of S, i.e. 0 r S|| suggested in (88]
Hyperparameter
VGG16
VGG11
ViT-B.16
ViT-small
ViT-L.32
Batch Size (UCM)
16
16
n/a
n.a.
Batch Size (Cifar10)
128
128
128
256
Batch Size (0)far100)
128
128
128
n.a
n.a.
Batch Size (ImageNet)
256
Learning Rate
0.001
0.001
0.001
0.0001
0.001
Number of Epochs (UCM, Cifar10)
20
20
5
450
Number of Epochs (Cifar100)
30
30
20
n.a
n.a
Number of Epochs (ImageNetIk)
n.a
n.a
n.a
n.a
10
L2 Regularization
0
0.001
0.01
0.0001
Optimizer
AdamW
A
AdamW
Adam
Adam W
DLRT rel. truncation tolerance Coefficient Steps
0.05
0.08
0.05
0.013
10
10
10
75
75
Initial Rank
150
150
150
200
200
Parameters
138M
132M
86M
50M
304M
VIT-L.32 is a Vision Transformer with 32x32 patch size, deep learning architecture that leverages transformer models for image classification tasks. We use the Imagenet21K k weights from the weight initialization.
huggingface endpoint
The full training setup described in Table 6 We train DLRT with the same hyperparameters as the full-rank baseline models. It is known 27 that DLRT methods are robust w.r.t. common hyperparameters as learning rate, and batch-size, and initial rank. The truncation tolerance T is chosen per initial parameter study. These values are similar to default values reported in recent literature 30 32 In general, there is a trade-off between target compression ratio and accuracy, illustrated c.g. (28 for mutrix-valued and 32 for censor-valued (CNN) layers.
10.1.2
UCM Data
The UC Merced (UCM) Land Use Dataset 38 a standard benchmark in remote sensing and computer vision. It consists of 2,100 high-resolution aerial RGB images, each of size 256 x 256 pixels, organized into 21 land use classes with 100 images per class.
We normalize the training and validation data using channel-wise means [0.485, 0.456, 0.406] and standard deviations [0.229, 0.224, 0.225]. Convolutional neural networks (CNNs) are applied directly to the original 256 x 256 image resolution. For the Vision Transformer (ViT), the input images are resized 224 224 pixels within the data pipeline.
10.1.3 CIFAR-10 Data
The CIFAR-10 dataset comprises 60,000 RGB images of size 32 x 32 pixels, uniformly distributed across 10 object classes.
We apply standard data augmentation techniques to the training set, including random horizon- tal flipping followed by normalization with mean [0.4914, 0.4822, 0.4465) and standard deviation (0.2470, 0.2435, 0.2616]. The test set is only normalized. The same augmentation strategy is applied CIFAR-100, using mean (0.5071, 0.4867, 0.4408) and standard deviation [0.2673, 0.2564, 0.2762).
CNNs are trained on the original 32 x 32 resolution, while ViT models receive images resized to 224 x 224 through the data pipeline.
19

---

Training Accuracy [%]
Training Loss
1.0
0.8
0.6
Algorithm
0.2
LoRA pretraining
Full-rank Adam
0.0
1500
2002
2500
3000
36000
4002
1000
1500
2000
2500
3000
3500
4003
Optimizer Step
Optimizer Step
Figure 2: VIT-L.332 on ImageNetlk, pretrained from scratch low-rank and full-rank baseline format for 4000 iterations. Training loss and accuracy of Algorithm is close to the full-rank baseline, whereas LoRA pretraining struggles to converge within the training time budget.
10.1.4 ImageNet-Ik 1k Data
The ImageNet dataset consists of 1000 classes and over 1.2 million RGB training images, with a standard resolution of 224 224 pixels. We follow the standard data augmentation pipeline for ImageNet, which includes a random resized crop to 224 x 224, and normalization using mean [0.5, 0.5, 0.5] and standard deviation [0.5, 0.5, 0.5]. The test set is only resized and center-cropped to 224 x 224, followed by normalization.
10.1.5 Additional Results- Transfer Learning
VIT-L.32 on ImageNetlk We repeat the experimental setup on VÝT-L.32 on the ImageNet-lk dataset, where ViT- L.32 is initialized with Huggingface Imagenet21 K check- point. We compare the baseline model, LoRA- -based simul- taneous descent pretraining, and Algorithm 2 We observe that Algorithm is able to recover the baseline accuracy up small margin whereas LoR A-RAeb train- ing exhibits decreased Top-1 and Top-5 accuracy. Finally, we remark that the slightly lower compression rate ex- pected since the hidden dimension of ViT-L.32 (1024) is
Table 7: Results on ImageNet-Ik Ik with ViT-L.32 (304M parameters). Compres- sion rate (c.r.) is reported in percent.
in Table
7
Method
[%] Top-1 [S] Top-5
Raseline
74.37
72.27
LaRA Poutrin
close to the number of ImageNet classes (1000), thus there less redundancy in the model compared to other reported benchmarks.
10.1.6 Additional Results Transfer Learning: Low-Rank Heavyball Method
VGG16 on Cifar10 We consider VGG16 C0 with Heavyball SGD using the same hyperparameters as described in Section [0.1.1] We choose ~=0.9 and train
Table 8: Results on Cifar10 with VGG16 using low-rank Heavyball SGD. Com- pression rate (c.I.) is reported in percent.
a (full-rank) baseline, LoRA pretrain and Algorithm [
The compression rate of LoRA pretrain fixed to match the final compression rate of Algorithm [ In Table 8
Method
C.r. [%]
Acc. [%]
we observe similar performance of Algorithm [ to the Baseline we saw for Algorithm to the Adam Baseline,
Baseline
78.98
Algorithm
94.35
79.01
whereas LoRA pretrain exhibits a slight drop accuracy.
LoRA transter learning
93.72
75.12
10.1.7 Additional Results Low-Rank Pretraining
ViT-small on Cifar10 We consider compact Vision
Transformer architecture for the CIFAR- 10 dataset, see Section TO. for details on training and architecture. We compare baseline full-rank training with LoRA pretraining, Algorithm [2] and the naive implementation of Adam with DLRT (28). Instead of low-rank finetuning, we pretrain network from scratch and initialize the weights with normal distribution. The low-rank methods
factorize the fully-connected layers, while keeping the attention layers, which are typically high-rank,
in baseline format. The LoRA rank is chosen match the final compression rate of the rank adaptive
20

---

siation Assuray [N]
Validation
Thaining Loss
4.00
1.50
Alperihmm
pretannngg
rank Adam
DURT Adam nate
20
180
100
200
200
250
400
200 3.5
100 200 205
350
300
450
Figure 3: ViT-small on Cifar10, pretrained from scratch in low-rank and full-rank baseline format for 450 epochs. Median trajectory over 5 runs. Algorithm 2 and LoRA pretraining initially converge faster than the full-rant baseline. After the initial warm-up phase, Algorithm exhibits steeper convergence slope than LoRA. Moreover, Algorithm P achieves lower loss and higher validation aceuracy than LoRA, even surpassing the baseline. naive DLRT implementation with Adam leads to slower convergence and over 10% drop in validation accuracy.
naive DLRT and Algorithm| 2 method, which achieves compression rate of 67%. Remark that the compression rate is lower, since the attention matrices remain full-rank.
The goal of this test is compare the long-term convergence behavior of all four methods, presented in Figure 3 We observe that Algorithm 2 and standard LoRA pretraining first converge faster than the baseline training. The non-orthooonal bases A, B of LoRA and the corresponding non-orthogonal projection onto the low-rank manifold cause LoRA to plateau, whereas Algorithm ( achieves lower loss values and higher validation accuracies, even surpassing the full-rank baseline in this test case. Naive implementation of DLRT with the Adam optimizer causes a more than 10% reduction in validation accuracy and slower convergence.
10.2
GLUE Benchmark
10.2.1
Dataset description
We present the benchmark overview in Table 9 We evaluate Algorithm 2
against several recent
Table 9: Summary of GLUE benchmark tasks
Corpus
Task
#Train
#Dey
#Test
#Label
Metrics
Single-Sentence Classification (GLUE)
CoLA
Acceptability Sentiment
8.5k
1k
lk
2
Matthews corr
SST2
67k
872
1.8k
2
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLI
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
F1
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy
QNLI
QA/LLI
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS-B
Similarity
7k
1.5k
1.4k
1
Pearson/Spearman cor
finetuning methods on the General Language Understanding Evaluation (GLUE) benchmark (35 GLUE a standard benchmark comprising a diverse set of natural language understanding tasks that assess model's 's ability comprehend and process human language. It provides broad evaluation by including tasks covering various linguistic aspects such as entailment, sentiment, and semantic similarity. The benchmark comprises the following nine tasks:
CoLA (Corpus of Linguistic Acceptability): Determines if sentence is grammatically accept-
able.
SST-2 (Stanford Sentiment Treebank): A binary sentiment classification task distinguishing
between positive and negative sentiment. MRPC (Microsoft Research Paraphrase Corpus): Identifies whether two given sentences are paraphrases.
21

---

STS-B (Semantic Textual Similarity Benchmark): Measures the semantic similarity of two
sentences on a continuous scale from 5.
QQP (Quora Question Pairs): Assesses whether two questions are semantically equivalent.
QNLI (Question Natural Language Inference): Determines if context sentence correctly
answers a question.
RTE Reecgnnizing Textual Entailment): A binary entailment classification task.
Specific Focus: MRPC (Microsoft Research Paraphrase Corpus)
The Fl score, used for evaluation, is computed from the precision P and recall R follows. The precision P is defined as
PT
P:
(16)
PT P
where Pr denotes the number of true positives and Pp the number of false positives. The recall R is given by
PT
R
(17)
Pr
NF
where Np represents the number of false negatives. The Fl score is then the harmonic mean of P and R:
2PR
F1 P+ R
(18)
10.2.2 Reference implementations
The standard approach in transfer learning, where the model is initialized with pre-trained weights and all parameters are updated via gradient descent.
Full Finetuning (FT):
Bitfit 39 Finetuning where only the bias terms are updated while all other parameters remain fixed.
Involves inserting two-layer adapter modules within transformer blocks. In [14,, adapters are placed between the self-attention and feed-forward modules with a residual connection (denoted HAdapter). In 223] adapters are inserted after the feed-forward and layer normalization modules (denoted PAdapter), following the notation of 42.
Adapter Tuning [14 [23::
LoRA 15 Applies low-rank additive updates to selected weight matrices, modeled as
[W a AB [
(19)
where A. B Raxr, We apply LoRA to the attention matrices Wg. WR. W., and the cerw-forward matrices WA, and Wfz- Learning rates and optimizers follow the setup in [42]. Appendix D-F.
Results for FT, Bitfit, Adapter tuning, and LoRA in Table are reproduced from [42) The perfor-
mance of DoRA, LoRA, LoRA+, and AdaLoRA is computed using the HuggingFace implementations of these adapters.
DoRA [21] A low-rank adapter similar in structure to LoRA, but with normalized AB matrices and an additional magnitude parameter. Unlike LoRA, DoRA initializes the adapter with the pre- trainrd weights Wo. rather than zero.
LoRA+ [10]: Differs from LoRA in the assignment of learning rates: separate learning rates are used for A and B, with a fixed ratio ÅB/ A 1.1.
AdaLoRA 42
Introduces adaptive low-rank updates to selected weight matrices:
= σ (Wxx+ 2 -SV
(20)
with frozen base weights Wi pc x rank-r adapters U, V E gaxr, and scaling matrix S E grxr, The rank is determined using either SVD-based truncation or sensitivity analysis of the singular vectors. AdaLoRA is applied to W,, Ws. W., W,, and W,, with an orthogonality regularization coefficient = 0.1.
When comparing to AdaLoRA, we align the total parameter budget with LoRA by setting the final budget ((T) to 576, and initialize with (()) 1.5 x ((T)
22

---

We also compare AdaLoRA using budget schedules obtained via Algorithm ensuring that 6(7) approximately matches the parameter count of the final models trained using Algorithm
GeoLoRA 2
GeoLoRA integrates the projected gradient flow Equation ( in a parallelizable
single-step scheme, including rank adapative ingmentation-trunaali scheme the proposed method. However, the method is only applicable for stochastic gradient descent, and not yet extended momentum-based approaches. We use the hyperparameter choices reported in [27
use the implementation of 442 Appendix CJ to compute the results for the presented reference methods. We set the exponential moving average parameters 31 and Bg of Adam their pytorch default value. We select the learning rates as denoted in Table[T0] selected by an initial hyperparameter sweep.
We implement Algorithm 2 similar as possible to the reference models to achieve fair comparison. That is, we add an adapter of the form z (W + USV x) to the key WA. query W,, and value W matrices of all attention blocks, and both feed-forward layers W and fa- For each adapter, we employ Algorithm 2 to update the layer weights and ranks.
Table 10: Hyper-pane parameeer setup for the GLUE benchmark, determined by an initial hyperparameter
sweep.
Dataset
Learning Rate
Batch Size
# Epochs
init. rank
Adapter dropout
weight decay
RTE
1,2
32
20
0.075
0.01
QNLI
5 10-4
64
0.05
0.2
MRPC
1 10
64
0.05
10
0.15
0.05
QQP
1 10
64
5
0.05
10
0.15
0.05
SST-2
1 10
64
10
0.05
10
0.05
0.01
CoLA
5 10
32
25
0.05
10
0.1
0.01
STS-B
1 10
128
30
0.05
10
0.05
0.1
10.3
Llama2 7b-chat-hf on BoolQ and PIQA
BoolQ is reading comprehension dataset consisting of naturally occurring yes/no questions paired with passages from Wikipedia. Questions are drawn from real Google search queries, and each is annotated with answer by human raters, making it benchmark for natural, open-domain question answering.
PIQA (Paragraph-level In-context QA) is dataset designed for evaluating in-context learning in long-form reading comprehension. It provides paragraph-length passages with associated questions and answers, emphasizing models' ability to extract relevant information from extended contexts rather than isolated sentences.
Table 11: Hyper- parameter setup for Algorithm 2 for the reasoning benchmark Table by an initial hyperparameter sweep.
determined
Dataset
Learning Rate
Batch Size
Epochs
init. rank
Adapter dropout
weight decay
BoolQ
1.76
12
3
0.0696
PIQA
0.0838
Table 12: Hyper-parameter setup for LoRA for the reasoning benchmark Table determined by an initial hyperparameter sweep.
Dataset
Learning Rate
Batch Size
# Epochs
init. rank
Adapter dropout
weight decay
BoolQ
4.47 x e- 1.76 e-
12
3
None None
6/10
0
0.1
PIQA
2.04 x 1.36 e
12
3
6/10
0
0.1
10.4 GPT2 on OpenWebText
Open WebText is an open-source dataset constructed replication of OpenAl's WebText. was created by scraping URLs shared on Reddit. The dataset contains web pages spanning diverse topics,
23

---

Nigorthm LoRA pretraining
Algorthmm
LoRA pratraining
Training Loss
Validalion Loss
50
100
150
200
250
300
350
400
450
50
100
150
200
250
300
350
400
450
Epoch
Epoch
Figure 4: GPT2 reproduction on Open pretrained from scratch low-rank, full-rank baseline and Algorithm for 15000 iterations. Algorithm@mette significantly outperforms LoRA pretraining (best validation loss 3. 3.4642 vs. 4.8141), while incurring only a moderate increase relative the full-rank baseline (3.4642 3.2313).
filtered remove duplicates and non-English text, and is commonly used as large-scale corpus for training and evaluating language models.
Table 13: Hyperparameter configuration for pretraining GPT-2 (124M) on OpenWebText (see Table
Dataset
Learning Rate
Batch Size
# iteration
7
init. rank
Adapter dropout
weight decay
OpenWebText
[6c-1, 66
64
15 000
0.05
135
0.1
10.5 Computational hardware
All experiments in this paper are computed using workstation GPUs. Each training run used a single GPU, except for GPT-2 pretraining, which performed two NVIDIA H100 GPUs. Specifically, we have used 5 NVIDIA RTX A6000, 3 NVIDIA RTX 4090, and NVIDIA H100.
24

---

NeurIPS Paper Checklist
The checklist is designed encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and societal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question is Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justificatin right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include it (after eventual revisions) with the final version of your paper, and its final version will be published with the paper.
The reviewers of your paper will be asked to use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", is perfectly acceptable to answer "[No] pprovided proper justification is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering "[No] or "[NA] is not grounds for rejection. While the questions are phrased in binary way, we acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or the supplemental material, provided in appendix. If you answer [Yes] to question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS paper checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We describe our contribution in the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize other settings.
It is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer:
[Yes]
We discuss the underlying assumptions of the method the analysis of the corresponding theorems.
Justification:
25

---

Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: We discuss the underlying assumptions of the method in the analysis of the corresponding theorems.
Guidelines:
The answer NA means that the paper does include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
All experimental details are provided in the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
26

---

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. w
For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification:
We provide the open source code upon paper acceptance
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (nttps://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ 7/nips. /ttppp.cc/pbliccguideeee http://oeeuuiiissooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
27

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide the training details and hyperparameters in the appendix.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: We provide error bars and report the mean and median over different initializa- tions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-simma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The used compute resources are reported in the appendix
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
28

---

The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethies
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neur/i: hctpp//ic////ttteeatiaiati
Answer: [Yes]
Justification: The paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The impacts are discussed in the conclusion
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This work is algorithmic and does not release special data or models.
Guidelines:
The answer NA means that the paper poses such risks.
29

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification: Not needed for this work
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Not needed for this work
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Not needed for this work
30

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Not needed for this work
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
31


Reviews:
[{"review_id": "7FqjgGFBP5", "reviewer": "Reviewer_sz9N", "summary": "This paper theoretically demonstrates that conventional momentum-based optimizers like Heavy Ball and Adam fail to converge to local optima in low-rank neural network training due to ignoring the geometric constraints of the low-rank manifold's tangent space. To address this, it extends the Dynamic Low-Rank Training (DLRT) theory and propose a geometry-aware optimization framework that explicitly enforces momentum updates to remain within the tangent space. It achieves 95% parameter compression in both image classification (pretraining) and natural language processing benchmarks (finetuning) with comparable performance.", "questions": "Mentioned in Strengths And Weaknesses.", "limitations": "Mentioned in Strengths And Weaknesses.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths\n- Theoretical Analysis of Limitations: The paper theoretically demonstrates that conventional momentum-based optimizers fail to converge to low-rank optima due to geometric constraints.\n- Memory Efficiency with Performance: Achieves comparable accuracy to full-rank baselines at huge compression ratio across multiple benchmarks (image domain and NLP).\n\nWeakness\n- Type (Line 67): `optimizer` should be corrected to `optimizer`.\n- Clarity on gamma in equations: I think that gamma represents the damping coefficient but there is no explicit definition.\n- Ambiguous Reference to equations: It’s better to use expressions such as Eq. (5) or Equation (5) rather than (5). It makes use be confused\n- Theorem Placement: Theorems are briefly mentioned in the main text without explanation; integrating key theorems into the main body could improve readability.\n- Missing Wall Time Comparison: The paper analyzes the computational efficiency, but real-world training time (wall clock time) is absent.\n- Theoretical Convergence Rate: A direct convergence rate comparison with existing methods helps the claim of faster convergence mentioned in the abstract.\n- ImageNet Pretraining: Experiments are conducted on smaller datasets (CIFAR, UCM), but validation on large-scale datasets like ImageNet is missing. Pretraining, in this context, refers to training on large datasets to enhance generalization performance. Training on large-scale datasets like ImageNet could strengthen the persuasiveness of the paper.\n- The overall notation explanation needs to be improved for better clarity and readability.", "was_revised": true, "final_justification": "After rebuttal and discussion, the authors clarified the review’s concerns, addressed typographical issues, and committed to improving equation referencing, which resolved several presentation concerns. The paper provides valuable and novel insight into the limitations of momentum-based optimizers and demonstrates impressive compression performance with competitive accuracy, highlighting its potential impact in both theory and practice. While the empirical validation is currently limited to small datasets without large-scale experiments, and real-world wall-clock training times, direct convergence rate comparisons, and more accessible theorem explanations would further strengthen the work, these limitations appear addressable in future revisions. Overall, the contributions are significant and the results compelling, making this a promising paper that is well worth borderline acceptance.", "rebuttal": "Thank you very much for the encouraging feedback and for your useful\ncomments and constructive suggestions. We believe the reviewer’s comments and suggestions have helped greatly in improving our manuscript.\nWe will upload the revised version as soon as the review process allows.\n\n1. We have corrected that typo.\n\n2. $\\gamma$ denotes the momentum decay parameter. A clarifying remark has been added after Eq. (3) in the revised manuscript, and Section 7 in the appendix now includes a table with all notation definitions.\n\n3. All occurrences of *(num)* have been replaced by *Eq. (num)* in the revised manuscript.\n\n4. We chose to included theorems in the appendix to provide interested\nreaders with theoretical intuition for our algorithmic design while not\nhindering readability of our work. However, our main contribution\nis the statement of Algorithm 2 and its empirically demonstrated\nperformance. If you believe it increases readability to directly state\ntheorems in the main paper, we are happy to include these.\n\n5. We fully agree that it was unclear how additional operations increase the overall runtime compared to standard LoRA training. Therefore, we decide to provide a wall clock comparison of a bigger benchmark to demonstrate that aditional operations do not significantly increase the runtime. We provide a wall clock comparison of finetuning  Llama2 7b-chat-hf on BoolQ and PICA benchmarks\nwith standard LoRA [Hu et al., 2021] and Algorithm 2 in the table below. The wall-time is reported for three epochs with batch size 12 and maximal sequence length of 640 tokens on a single NVIDIA H100. We observe singificant outperformance of Algorithm 2 at a negligible wall-time overhead for all test cases.\n\n| **Method**              | **BoolQ Acc [%]** | **Wall-Time** | **Params** | **PIQA Acc [%]** | **Wall-Time** | **Params** |\n|----------------------------|-------------------|---------------|------------|------------------|---------------|------------|\n| Algorithm 2                | 84.09%            | 186min        | 0.270%     | 76.77%           | 228min        | 0.247%     | \n| LoRA (r=6)                 | 62.17%            | 173min        | 0.179%     | 52.18%           | 225min        | 0.179%     |\n| LoRA (r=10)                | 62.17%            | 184min        | 0.299%     | 50.43%           | 225min        | 0.299%     |\n\nHyperparameters are determined by initial hyperparameter sweeps (which were performed for LoRA and Algorithm 2 individually). Hyperparameters for Algorithm 2 are reported below:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**      | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|----------------|------------|-----------|--------|-------------|------------------|---------------|\n| BoolQ   | 1.76e-4         | 12         | 3         | 0.0696 | 6           | 0                | 0.1           |\n| PIQA    | 1.36e-4         | 12         | 3         | 0.0838 | 6           | 0                | 0.1           |\n\nHyperparameters for LoRA:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**      | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|---------------|------------|----------|------|-------------|------------------|---------------|\n| BoolQ   | 4.47e-4 / 1.76e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n| PIQA    | 2.04e-4 / 1.36e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n\n\n6. First, we remark that we only make claims about the empirical convergence rate. \nTo that end, we have provided a convergence rate comparison\nin Figure 2, where we train a ViT model on Cifar-10 from scratch with\n\n    (a) full-rank (baseline) Adam (Algorithm 4), \n\n    (b)  the LoRA-based Adam, i.e. Algorithm 4 applied to all factors of the low-rank factorization individually, \n\n    (c) the proposed Adam-DLRT (Algorithm 2), and \n\n    (d) a naive implementation of Adam for DLRT, that does not change the mo-\nmentum states. \n\nAs observed in Figure 2, the proposed Algorithm 2\nhas a fast initial convergence, and a higher final accuracy than the\ncomparison methods. For empirical soundness, we have added the\nstandard deviation to the image in the revised manuscript.\n\n7. We provide numerical results for the ImageNet-1k (1.2 Million im-\nages), where we compress the ViT-L32 vision transformer (304 Million\nparameters). We first conduct a hyperparameter sweep to determine\nstandard hyperparameters as learning rate, weight-decay and batch\nsize: ViT-L32 is trained on ImageNet with a batch size of 256, learning rate 0.001, for 20 epochs using AdamW optimizer and L2 regularization of 0.0001; DLRT uses a relative truncation tolerance of 0.013 and initial rank 200.\n\n    We compare then the baseline model, LoRA-based simulateneous descent pretrianing, and Algorithm 2  in the table below.\nWe observe that Algorithm 2 is able to recover the baseline accuracy up to a small margin whereas LoRA pretraining exhibits decreased top1 and top5 accuracy.\n\n\n   | **Method**        | **c.r. [%]** | **Top-1 Acc. [%]** | **Top-5 Acc. [%]** |\n   |---------------|----------|----------------|----------------|\n   | Baseline      | 0        | 74.37          | 92.20          |\n   | Algorithm 2   | 61.45    | 72.27          | 90.19          |\n   | LoRA Pretrain | 60.00    | 63.20          | 84.81          | \n\n   Finally we wish to remark that the slightly lower compression rate is expected since the hidden dimensions of ViT-L32 of 1024 is close to the number of ImageNet classes (1000), thus there is less redundancy in the model compared to other reported benchmarks. \n\n\n8. We unified notation across momentum and Adam equations, using consistent symbols for the first moment terms in the revised manuscript. A Notation table is now included in the appendix. We believe that this makes it much easier to follow our manuscript and we thank the reviewer for pointing this out."}, {"review_id": "5QEMbgU6ml", "reviewer": "Reviewer_Q6vZ", "summary": "Momentum-based updates for low-rank training needs to be geometry-aware for correctness. The paper shows how naive implementation of LoRA with heavy ball method (independently updating each of the factors in Equation 3) does not ensure that update dynamics stay on the low-rank manifold. The paper gives a framework for the evolution of the factorized weights $W$ and factorized momentum terms $\\mathcal{V}$. Using this framework, the paper provides update steps and algorithms for two methods: low-rank heavy ball method and low-rank Adam method. The paper also addresses two numerical challenges in their methods: evolving $W$ in factorized form to maintain low-cost and avoiding stiffness in the dynamical system through the avoidance of inverting $S_v$ and $S_v^\\intercal$. Experimental results show that the proposed Algorithm 2 (low-rank Adam) gives both higher accuracy and higher compression rate than existing methods and naive implementation of low-rank momentum.", "questions": "- Do the convergence results given in Theorems 1 and 2 for continuous-time dynamics carry over to discrete algorithms (such as Algorithms 1 and 2)? \n- Is there an extra e in Lady Winderemere's fan on line 430?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n - The paper addresses a short-coming of other LoRA-style methods: the updates may leave the low-rank manifold. It does so by developing a rigorous framework for momentum-based optimization methods on the manifold of low-rank matrices.\n - Proposes low-rank versions of two popular methods: Adam and Heavy Ball. The algorithms keeps inner product calculations only on low-ranked factorized matrices and thus is efficient.\n - Empirical results look good: the proposed method achieves high accuracy with high compression rates.\n\nWeaknesses\n - The experiments do not show any wall-clock time measurements. Although theoretically, all operations on done on much smaller factorized matrices, but even small operations (e.g. the truncation step in Algorithm 2) could have significant cost in the aggregate.", "was_revised": true, "final_justification": "Because the additional experimental results are preliminary and because the theoretical results in the paper does not necessarily characterize the proposed algorithms, I will maintain my rating at 5 (Accept).", "rebuttal": "We thank the reviewer for their insightful comments.\n\n1. **Wall-clock Timing (W1):**  \n     \n    We fully agree that it was unclear how additional operations increase the overall runtime compared to standard LoRA training.  Therefore, we decided to provide a wall clock comparison of a bigger benchmark to demonstrate that aditional operations do not significantly increase the runtime. We have added wall clock comparisons for finetuning Llama2 7b-chat-hf on BoolQ and PICA benchmarks with standard LoRA [Hu et al., 2021] and Algorithm 2 in the table below. The wall-time is reported for three epochs with batch size 12 and maximal sequence length of 640 tokens on a single NVIDIA H100. We observe singificant outperformance of Algorithm 2 at a negligible wall-time overhead for all test cases.\n\n| **Method**              | **BoolQ Acc [%]** | **Wall-Time** | **Params** | **PIQA Acc [%]** | **Wall-Time** | **Params** |\n|----------------------------|-------------------|---------------|------------|------------------|---------------|------------|\n| Algorithm 2                | 84.09%            | 186min        | 0.270%     | 76.77%           | 228min        | 0.247%     | \n| LoRA (r=6)                 | 62.17%            | 173min        | 0.179%     | 52.18%           | 225min        | 0.179%     |\n| LoRA (r=10)                | 62.17%            | 184min        | 0.299%     | 50.43%           | 225min        | 0.299%     |\n\nHyperparameters are determined by initial hyperparameter sweeps (which were performed for LoRA and Algorithm 2 individually). Hyperparameters for Algorithm 2 are reported below:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**      | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|----------------|------------|-----------|--------|-------------|------------------|---------------|\n| BoolQ   | 1.76e-4         | 12         | 3         | 0.0696 | 6           | 0                | 0.1           |\n| PIQA    | 1.36e-4         | 12         | 3         | 0.0838 | 6           | 0                | 0.1           |\n\nHyperparameters for LoRA:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**      | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|---------------|------------|----------|------|-------------|------------------|---------------|\n| BoolQ   | 4.47e-4 / 1.76e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n| PIQA    | 2.04e-4 / 1.36e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n\n2. **Convergence Results Transfer (Q1):**  \n   The convergence result for Theorem 2 directly carries over to the discrete setting of Algorithm 1. The reason for this is that a first order time discretization will not add error terms that are asymptotically bigger than the current error rates.\nThe situation of Algorithm 2 become much more challenging, since there does not exist a direct interpretation of ADAM as a time dis-\ncretization of a gradient flow. We point this out in Section 4, Line 185. Theorem 1 does not directly translate into a discrete setting and a discrete version would, most probably, require additional conditions on the learning rate.\n\n3. **Typo Fix (Q2):**  \n   You're right — it should be *Windermere*. We've corrected it in the revised manuscript."}, {"review_id": "ULbzAhSujT", "reviewer": "Reviewer_yZDw", "summary": "This paper studies the momentum-based algorithm when applying to low-rank weight optimization in neural network training. The paper identifies a condition where the heaby-ball method converges to the low-rank optimum solution, and shows that directly applying momentum to the factorization form of the weight does not satisfy this condition. Based on this condition, the paper derives the heavy-ball and Adam method for low-rank problems. The paper presents experimental results in the main text, and a convergence guarantee in the appendix to corroborate the effectiveness of the proposed algorithm.", "questions": "Does the proposed algorithm also work for tasks like matrix completion and matrix sensing?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strength**\n\n1. The algorithm design comes from a solid theoretical principle and looks quite convincing.\n\n2. Per experimental result, the performance is better than the chosen baseline for a significant amount.\n\n**Weaknesses**\n\n1. The motivation of using the proposed algorithm is a bit weak. The paper points out the issue with directly applying momentum to DLRT by showing that the direct application does not satisfy (4). However, in the appendix it is only shown that (4) is a sufficient condition, but it is not clear whether (4) is also a necessary condition. Therefore, I believe that the explanation of why direct application of the momentum to DLRT is inconclusive.\n\n2. I am not sure whether the experimental result in Table 1,2 are measured in epochs or in total training time. If the performance is based on a fixed number of epochs, then the comparison can be unfair as the proposed algorithm imposes a constant times of additional per-iteration cost. \n\n3. Convergence guarantee in Theorem 3 is not strong enough. In particular, Theorem 3 bounds the difference between the algorithm trajectory and an ideal trajectory that converges to the low-rank optimum. However, the upper bound depends on $\\hat{\\nu}$, which is the gap between the projection of the momentum onto the current basis and the augmented basis. Intuitively, the augmented basis should explore more possibility, so this difference can be quite large.", "was_revised": true, "final_justification": "The rebuttal resolves most of my concerns. The only thing that is kind of weak from my perspective is the theoretical convergence guarantee of the algorithm. Despite the empirical choice of $nu$ that could lead to a meaningful convergence result, the paper did not provide a strong enough convergence guarantee.", "rebuttal": "We wish to thank the reviewer for their thoughtful and constructive feedback. We are encouraged that the reviewer found our algorithm design to\nbe based on a solid theoretical principle and that our experimental results\nshow significant performance improvements.\nThe reviewer raised three main points regarding the motivation, experimental setup, and the interpretation of our convergence guarantee. We\naddress each weakness (W) and question (Q) below.\n\n---\n\n1. **W1 (On the motivation for the proposed gradient flow):**\n\n    The reviewer correctly points out that we showed that satisfying equation (4) is a *sufficient* condition for convergence to a low-rank optimum, but we did not explicitly demonstrate that naive momentum methods can fail to converge to a point fulfilling $P(W)\\nabla_W \\mathcal{L}(W)=0$. We thank the reviewer for highlighting this, as a more concrete discussion improves the paper's motivation.\n\n    The potential lack of convergence for a naive application of momentum methods can be proven with a simple but illustrative counter-example. Consider a state where the momentum term is zero, i.e., $\\mathcal{V}=0$, by choosing the momentum factors as $U_v = -U$, $S_v = 0$, and $V_v = V$. Now, consider any weight matrix $W$ that is *not* a low-rank optimum, meaning $P(W)\\nabla_W \\mathcal{L}(W) \\neq 0$, but for which the naive update term happens to be zero: $\\hat{P}(W,\\mathcal{V})\\nabla_W \\mathcal{L}(W) = 0$.\n\n    A large class of matrices fulfills this. For instance, any $W=USV^\\top$ where the gradient is non-zero in the range of V but zero in the range of U (i.e., $U^\\top\\nabla_W \\mathcal{L}(W) = 0$ but $\\nabla_W \\mathcal{L}(W)V \\neq 0$) would satisfy this. This can be easily verified: Since $\\mathcal{V}=0$, we have\n\n$$\n\\hat{P}(W,\\mathcal{V})\\nabla_W \\mathcal{L}(W) = UU^\\top\\nabla_W \\mathcal{L}(W)VV^\\top = 0,\n$$\n\nbut\n\n$$\nP(W)\\nabla_W \\mathcal{L}(W) = \\nabla_W \\mathcal{L}(W)VV^{\\top}\\neq 0.\n$$\n\nIn this scenario, the naive gradient flow equations would identify $(W, \\mathcal{V})$ as a stationary point, since $\\dot{W}=0$ and $\\dot{\\mathcal{V}}=0$. However, this point is not a valid low-rank optimum because the true projected gradient $P(W)\\nabla_W \\mathcal{L}(W)$ is non-zero. In contrast, $(W, \\mathcal{V})$ is not a stationary point of (4) and the gradient flow (4) will provably drive the system to a state where $P(W)\\nabla_W \\mathcal{L}(W)=0$.\n\nThis demonstrates a fundamental geometric inconsistency in the naive approach: it can prematurely halt at suboptimal points that do not correspond to true low-rank critical points. **We added this counter-example and discussion to the appendix of the revised manuscript to make our motivation clearer and more compelling.**\n\n\n\n2. **W2 (On the experimental setting and runtime comparison):**\n\n    The details for the numerical experiments are provided in the Appendix in Table 3 and 5, respectively. We set an epoch limit for all experiments in the paper, where we have chosen the epoch number such that all methods have sufficient iterations to converge. This is a default test case setup.\n\n\n    To underline that Algorithm 2 does not require a significantly increased training time, we provide a wall clock comparison of finetuning  Llama2 7b-chat-hf on BoolQ and PICA benchmarks\n    with standard LoRA [Hu et al., 2021] and Algorithm 2 in the table below. The wall-time is reported for three epochs with batch size 12 and maximal sequence length of 640 tokens on a single NVIDIA H100. We observe singificant outperformance of Algorithm 2 at a negligible wall-time overhead for all test cases.\n\n| **Method**              | **BoolQ Acc [%]** | **Wall-Time** | **Params** | **PIQA Acc [%]** | **Wall-Time** | **Params** |\n|----------------------------|-------------------|---------------|------------|------------------|---------------|------------|\n| Algorithm 2                | 84.09%            | 186min        | 0.270%     | 76.77%           | 228min        | 0.247%     | \n| LoRA (r=6)                 | 62.17%            | 173min        | 0.179%     | 52.18%           | 225min        | 0.179%     |\n| LoRA (r=10)                | 62.17%            | 184min        | 0.299%     | 50.43%           | 225min        | 0.299%     |\n\n\n\nHyperparameters are determined by initial hyperparameter sweeps (which were performed for LoRA and Algorithm 2 individually). Hyperparameters for Algorithm 2 are reported below:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**    | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|----------------|------------|-----------|--------|-------------|------------------|---------------|\n| BoolQ   | 1.76e-4         | 12         | 3         | 0.0696 | 6           | 0                | 0.1           |\n| PIQA    | 1.36e-4         | 12         | 3         | 0.0838 | 6           | 0                | 0.1           |\n---\n\nHyperparameters for LoRA:\n\n| **Dataset** | **Learning Rate** | **Batch Size** | **# Epochs** | **τ**    | **Init. Rank** | **Adapter Dropout** | **Weight Decay** |\n|---------|---------------|------------|----------|------|-------------|------------------|---------------|\n| BoolQ   | 4.47e-4 / 1.76e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n| PIQA    | 2.04e-4 / 1.36e-4      | 12         | 3        | None | 6 / 10      | 0                | 0.1           |\n\n2. **W3 (On the derived error bound):**\n\n    We appreciate the reviewer's detailed question regarding Theorem 3. We wish to note that our algorithm can be straightforwardly modified to provably control the term $\\widehat \\vartheta$ (as underlined in the comment following Theorem 3 in our manuscript in Line 431). In this case, $\\widehat{\\vartheta}$ becomes a user-determined parameter. We chose to present the main algorithm without this explicit modification because our analysis and empirical results suggest it is unnecessary and adds computational overhead for no tangible benefit. To name an example, we obtain 84.15% (instead of 84.09%) validation accuracy when finetuning Llama2 7b-chat-hf on the BoolQ benchmark when controlling the projection term by choosing $\\widehat{\\vartheta}=0.1$, which is fulfilled automatically for almost all iterations.\n\n    Intuitively, the influence of this term is naturally limited, as it is expected to diminish during training. This expectation is based on the dynamics of the optimization process: as the optimizer converges to a local minimum, the momentum term $\\mathcal{V}$ itself must converge toward zero. \n    This is a direct consequence of Theorem 1 in the Appendix. \n    Consequently, the difference in its projection onto two closely related subspaces will also naturally tend to zero. This intuition is strongly backed by the consistent high performance of our method across all experiments, where this term was not explicitly controlled.\n\n    To perhaps clarify the context of our response, we feel it's helpful to underline the role of Theorem 3 within our work. We included this analysis in the appendix to provide readers with theoretical intuition for our algorithmic design—specifically, to show how our integrator navigates the numerical instability associated with inverting the $S_v$ matrix. It serves as a motivational stepping stone for our final algorithm, which is the paper's primary contribution. Our goal is a novel, efficient method derived from a geometric framework, and the ultimate validation for such a method lies in its empirical performance. We believe the current algorithm, which, as the reviewer noted, shows significant performance improvements, is the strongest version to present, as it achieves these results at minimal cost without unneeded operations.\n\n    We will revise the discussion around Theorem 3 in the appendix to better articulate this interplay between theoretical motivation and practical algorithm design.\n\n\n3. **Q1 (Applicability to matrix completion/sensing):**\n\n    Absolutely. While our paper focuses on neural network training, the proposed geometric framework is general. The core of our method is a momentum-based optimizer for problems with low-rank constraints. Applying it to classic low-rank optimization problems like matrix completion and matrix sensing is a promising and natural extension. We hypothesize that our approach could offer similar benefits in convergence speed and stability for these tasks, particularly in challenging, ill-conditioned scenarios.\n\n    We remark that we do not aim for a state-of-the-art matrix completion algorithm as there exists a rich corpus of specialized methods that are tailored to the specific structure of a matrix completion problem and can therefore outperform more general methods in this specific task.\n\n\n\nWe hope these clarifications, along with the planned additions to the appendix, fully address the reviewer's concerns. We are confident that these improvements will strengthen the paper and hope the reviewer will reconsider their rating."}, {"review_id": "oeh9kPINfY", "reviewer": "Reviewer_9kiS", "summary": "This paper introduces a new momentum based optimization framework for low rank pre-training and fine-tuning. The proposed method combines dynamic low-rank approximation and momentum methods. Compared to existing layer factorization method DLRT, the proposed method extend the applicable optimizers from SGD to Heave-ball momentum gradient descent and Adam.\n\nThe proposed framework is supported with theoretical convergence and error bound guarantees, the authors also provide extensive experimental results on pre-training, fine-tuning and transfer learning to demonstrate the performance of proposed method generally matches the full rank baseline.", "questions": "1. This paper mainly provides experimental results for Algorithm 2, how does the performance of Algorithm 2 compare to that of Algorithm 1?\n2. In the experiment \"Pretraining from scratch of ViT-small on Cifar10\", algorithm 2 exhibits faster convergence and smaller validation loss compared to the full rank baseline. Could the authors provide some insights or explanations for this observed behavior?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n1. This paper extends DLRT to momentum optimizers, including both Heavy-ball momentum gradient descent and Adam, bridging an existing gap in the literature.\n2. The authors provide a comprehensive treatment of the proposed framework, including detailed derivations, clear computational complexity and memory analysis, as well as theoretical convergence and error bound guarantees.\n3.  This paper is well structured and easy to follow\n\nWeaknesses\n1. To enhance the readability of the paper for general readers who may not be familiar with LoRA or DLRT, it would be beneficial to include more explanations of the benchmark methods used in the experiment results, adding algorithmic descriptions for DLRT w/o proj and LoRA pretrain", "was_revised": true, "final_justification": "The authors have addressed my questions, my rating remains", "rebuttal": "We thank the reviewer for their constructive comments and appreciate that they found the paper comprehensive, well structured and that it bridges a gap in the literature.\n\n\n1. (W1) We fully agree and have added the Algorithmic descriptions for both methods in the Appendix in the revised manuscript and will upload the version as soon as the review process allows it. \n    \n    For completeness we point out the differences of the methods below.\n    \n     - DLRT w/o projection does not perform the projection update of $S_{\\mathcal{V}}$ in Line 7 and 12 of Algorithm 3 in the supplementary material. Thus it creates an inconsistency of the coefficient representation $S$ and the corresponding optimizer states, since they are now spanned by different bases.\n     - LoRA pretrain essentially applies Algorithm 4 to the LoRA factors $A,B\\in\\mathbb{R}^{n\\times r}$, where $r$ is fixed, as stated in Line 244. \n\n2. (Q1) The Dynamical Low-Rank Adam method is the main result of the paper, as the (full rank) Adam optimizer is currently the most widely used training method for neural networks - to the best of our knowledge. Thus we focused our evaluation on this setting. \n\n    For completeness we added VGG16 on Cifar10 with SGD using the same hyperparameters as in the experiments in table 1. We choose momentum=0.9 and train a (full-rank) baseline, LoRA pretrain and Algorithm 1. The compression rate of LoRA pretrain is fixed to match the final compression rate of Algorithm 1. We observe similar  performance of Algorithm 1 to the Baseline as we saw for Algorithm 2 to the Adam Baseline, whereas LoRA pretrain exhibits a slight drop in accuracy. \n\n| **Method**         | **c.r. [%]** | **Accuracy [%]** |\n|----------------|------------------------|--------------|\n| Baseline       | 0.00                   | 78.98        |\n| Algorithm 1        | 94.35                  | 79.01        |\n| LoRA Pretrain  | 93.72                  | 75.12        |\n\n3. (Q2) In our experience, and as investigated in recent literature [a], Vision Transformers train relatively slow on small datasets, e.g. Cifar10, and exhibit worse performance than ViT models pretrained on large datasets (ImageNet size). We think that encouraging low-rank solutions by training with Algorithm 2 helps eleviate this issue to some extend - explained by an Occam's Razor argument: A low-rank solution expresses a certain simplicity bias imposed on the very general transformer architecture. \n    One can see that the low-rank initialization leads to a fast loss decay in Figure 2b, whereas the validation loss of the full vision transformer stalls first. Interestingly the LoRA loss, also with a low-rank initialization decays fast at first, but then stalls. This might be attributed to the missing convergence guarantees to a low-rank optimum (see Section 2). \n    \n    A similar effect is observed in the GLUE benchmark portfolio, where low-rank finetuning methods (LoRA-based and DLRT-based) outperform full-fine tuning of certain sub-benchmarks. \n    \n    \n    To the best of our knowledge, this often seen observation is currently not rigorously understood and future research may shed more light onto this phenomenon.\n\n\n    [a] H. Zhu, B. Chen, and C. Yang. Understanding why vit trains badly on small datasets: An intuitive perspective, 2023."}]

Author rebuttal:

Meta-review: This paper proposes a method for low rank training of neural networks, an important problem in the fine-tuning of foundation models. For the most part the reviewers praised both the theoretical and the empirical results, with the main concerns being about clarity and wallclock improvements. Given that these have been addressed with proposed modifications and new results, I recommend acceptance.
Decision: Accept (poster)