NeurIPS 2025

A geometric framework for momentum-based
optimizers for low-rank training
Steffen Schotthöfer∗Timon Klein† and Jonas Kusch‡
Abstract
Low-rank pre-training and finetuning have recently emerged as promising tech-
niques for reducing the computational and storage costs of large neural networks.
Training low-rank parameterizations typically relies on conventional optimizers
such as heavy ball momentum methods or Adam. In this work, we identify and
analyze potential difficulties that these training methods encounter when used to
train low-rank parameterizations of weights. In particular, we show that classical
momentum methods can struggle to converge to a local optimum due to the ge-
ometry of the underlying optimization landscape. To address this, we introduce
novel training strategies that combine dynamical low-rank approximation with
momentum-based optimization, explicitly accounting for the intrinsic geometry
of the parameter space. We validate our methods through numerical experiments,
demonstrating stronger validation metrics at given parameter budgets.
1
Introduction
Deep learning models have achieved remarkable success across natural language processing and
computer vision tasks, but their deployment remains computationally expensive due to the large
number of trainable parameters. To address this, parameter-efficient strategies have been developed
to reduce memory and compute requirements during training. Common approaches include sparsifi-
cation [9, 22, 12], quantization [37, 6], and layer factorization. The latter has gained considerable
attention for pre-training [36, 16, 28, 29, 41] and especially finetuning [15, 34, 42, 10, 43, 20, 27].
Layer factorization represents weights (or adapters) as low-rank matrices, allowing only the low-rank
factors to be trained. This significantly reduces both memory usage and computational cost.
In the class of low-rank layer factorizations, one of the most popular methods is LoRA [15], which
applied typical optimizers such...