GeoLoRA: Geometric integration for parameter efficient fine-tuning
Authors: Steffen Schotthöfer
Emanuele Zangrando
Gianluca Ceruti
Francesco Tudisco
Jonas Kusch
Venue: ICLR 2025 Poster (2025)
Keywords: Low Rank
Finetuning
Robustness
Rank Adaptive
LoRA
Low Rank Adapter
Primary area: optimization

Abstract: Low-Rank Adaptation (LoRA) has become a widely used method for parameter-efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness, and computational efficiency during the fine-tuning process. We introduce GeoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single backpropagation pass over the small-rank adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such as AdaLoRA. This allows GeoLoRA to efficiently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error-bound theoretical guarantees. The resulting method is not only more efficient but also more robust to varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several state-of-the-art benchmarks, showing that it outperforms existing methods in both
accuracy and computational efficiency

Published as a conference paper at ICLR 2025
GEOLORA: GEOMETRIC INTEGRATION FOR PARAME- TER EFFICIENT HINE-TUNING
Steffen Schotthöfer', Emanuele Zangrando', Gianluca Ceruti F Francesco Tudisco Jonas Kusch
Computer Science and Mathematics Division, Oak Ridge National I Laboratory, USA
2 Gran Sasso Science Institute, L.' L'Aqulla, Italy
3 *Department of Mathematics, University of Innsbruck, Austria
"School of Mathematics and Maxwell Institute, University of Edinburgh, UK
6 Miniml Ltd, UK
Department of Data Science, Norwegian University of Life Sciences, Norway
Link to source code
ABSTRACT
Low-Rank Adaptation (LoRA) has become a widely used method for parameter- efficient fine-tuning of large-scale, pre-trained neural networks. However, LoRA and its extensions face several challenges, including the need for rank adaptivity, robustness. and computational efficiency during the fine-tuning process. We intro- duce GcoLoRA, a novel approach that addresses these limitations by leveraging dynamical low-rank approximation theory. GeoLoRA requires only a single back- propagation pass over the small-rant adapters, significantly reducing computational cost as compared to similar dynamical low-rank training methods and making it faster than popular baselines such AdaLoRA. This allows GeoLoRA to effi- ciently adapt the allocated parameter budget across the model, achieving smaller low-rank adapters compared to heuristic methods like AdaLoRA and LoRA, while maintaining critical convergence, descent, and error- Frror-bound thcoretical guarantces. The resulting method is not only
efficient but also more robust varying hyperparameter settings. We demonstrate the effectiveness of GeoLoRA on several sfae--of-the-ar benchmarks, showing that it outperforms existing methods in both accuracy and computational efficiency.
INTRODUCTION
Large-scale pre-trained and fine-tuned models have significantly advanced the performance of deep learning models in assisting various natural language processing and computer vision tasks. However, their deployment often incurs substantial computational and memory costs due to the enormous number of trainable parameters. To address this, parameter-efficient fine-tuning (PEFT) methods have been developed, which modify subset of model parameters while keeping the frozen. Among these, low-rank adaptation (LoRA) (Hu et al.) (02TT) has emerged as prominent approach, allowing efficient fine-tuning by injecting low-rank updates into pre-trained model weights. Despite its efficiency, LoRA faces limitations in adaptively distributing the parameter budget across weight matrices, and its performance sensitive the choice of hyperparameters (Zhang al. 2023).
Recent works, such as AdaLoRA (Zhang et al. 2023). DyLoRA (Valipour et al. 2023), and ReLoRA (Lialin al. 2023), have attempted improve LoRA by dynamically adjusting the rank of the low-rank adapters during training. While these methods enhance parameter efficiency, they are constructed as simultaneous descent methods and therefore do not guarantee convergence to optimal low-rank adapters. Methods that guarantee convergence to optimal adapters cxist (Schotthöfer et al., 2022] Schotthöfer & Laiu| 2024 Zangrando et al. 2024). However, these require several gradient tapes per iteration and, therefore, have an intrinsically higher run time per training step.
In this paper, we introduce GeoLoRA (Geometric Low-Rank Adaptation). a novel dynamical low-rank training method for parameter-ede efficieent fine-tuning. GeoLoRA leverages the dynamical low-rank approximation theory from matrix differential equations (Koch & Lubich 20076] Ceruti et al.) 2022 2023) and exploits the intrinsic low-rank geometry of the weight matrices to allocate the

---

Published as a conference paper at ICLR 2025
parameter budget across the model adaptively. This dynamic allocation is facilitated by novel training strategy that updates the low-rank factors in parallel, contrasting with other recent methods based dynamical low-rank approximation theory (Schotthöfer et al. 2022 Schotthöfer & Laiu 2024 Zangrando et al., 2024), which require individual gradient tapes computed sequentially per each low-rank factor. Instead, GeoLoRA requires a single backprop pass over the small-rank adapters. limiting its computational cost and making it faster than popular baselines such as AdaLoRA (Zhang al. 2023). Moreover, GeoLoRA maintains the exact orthonormality of the low-rank factors, avoiding the ill-conditioning issues associated with well-known high-curvature challenges arising in low-rank optimization (Schotthöfer et al. 2022).
Through extensive experiments on the GLUE benchmark, Vision Transformers, and Stable Diffu- sion, we show that GooL outperforms existing PEFT methods both in terms of accuracy and computational efficiency.
Along with the experimental evaluation, we provide thorough convergence analysis, showing convergence to stationary points under standard assumptions, and a detailed error-bound analysis, demonstrating that GeoLoRA's low-rank adaptation remains close to its full-rank counterpart through- out the training process. This robustness is critical in ensuring that the fine-tuning process does not diverge, even under challenging conditions.
Overall, the main contributions of this work are as follows:
We show that standard common training methods for low-rank adapters do not necessarily reach a local optimum. (Section 3)
We propose GeoLoRA, a dynamical low-rank training method for low-rank adapters that leverages low-rank geometry and matrix differential equations achieve adaptive parameter allocation. (Section 44)
GeoLoRA only requires single gradient tape and one small-size SVD per training step, making it competitive with existing baselines such as AdaLoRA.
We provide a convergence analysis and error bound guarantees for GeoLoRA, ensuring robust training behaviour and convergence to a stationary point. (Section
Extensive experimental results demonstrate the superior performance of GeoLoRA over existing methods, with improved accuracy and training speed. (Section
2 RELATED WORK
The growing size of neural networks has led to significant computational and memory challenges during both training and deployment. Several strategies have been proposed to mitigate these issues, including sparsification (Guo et al. 2016 Molchanov et al.) 2017 He et al., (017)) and quantization (Wu et al. 2016 Courbariaux et al. (016). Among these, layer factorization has gained traction as an effective approach to reducing memory requirements. Layer factorization techniques have been applied successfully in both pre- rre-training (Wang et al. 2021 Khodak et al. 2021: Schotthöfer et al.)
Schootccht & Laiu| 2024 Zangrando et al. 2024
Zhao et al.] 2024) and fine-tuning scenarios
(Hu et 2021 Valipour al.] 2023) Zhang et al. 2023
Hayou al. 2024 Zhao et al. 2024 Lialin
ct al. 2023), demonstrating their versatility across various tasks.
Low-rank adapters such as LoRA (Hu al. 2021) have become a standard approach for PEFT by applying low-rank corrections to pre-trained models. LoRA introduces a low-rank decomposition to the weight matrices of the model, significantly reducing the number of trainable parameters while preserving performance. Despite its efficiency, LoRA's effectiveness heavily relies on the selection of hyperparameters such as learning rates and parameter budgets (Zhang al.) [023 Hayou et a 2024). These limitations have spurred the development of rank aada-tiee methods. AdaLoRA (Zhang et al. 2023) is a popular extension of LoRA, which dynamically adjusts the rank of the low-rank adapters during training. By incorporating an orthogonality regularizer and SVD-like adaptation, AdaLoRA aims to address the challenges rank selection and adaptation. It outperforms static low-rank methods by automatically allocating parameter budgets based on the importance of each matrix component. DyLoRA (Valipour et al. 2023) provides an alternative approach that hierarchically adjusts the rank during training, demonstrating that higher-rank adapters lead to better performance than very low-rank ones. DoRA (Mao et al.) 2024) proposes to sample a set of rank-1 updates for each LoRA layer and to combine them into rank-r update. Optimal rank-1
2

---

Published as a conference paper at ICLR 2025
components are chosen during fine-tuning using importance score based on the norm of the LoRA layer.
Beyond fine-tuning, low-rank methods have been successfully applied during the training and pre- training phases of neural networks. Techniques such as Pufferfish (Wang et al. 202T), intrinsic dimension reduction (Aghajanyan large deep learning models have an inherently low intrinsic dimensionality, making them amenable to low-rank approximations. These methods propose reducing the number of parameters during training, potentially improving both efficiency and generalization. Recent works in dynamical low-rank training have explored the use of geometric properties of the low-rank parameter space improve training stability and convergence. For example, the geometry-aware training approach for tensor layers in Tucker format (Zangrando et al., 2024) dynamically adapts the rank of the factorized layers, ensuring robust convergence even when the initial rank estimation is inaccurate. This method leverages the Riemannian geometry of the parameter space to avoid the ill-conditioning commonly encountered in low-rank training. ReLoRA (Lialin et al. 2023) introduces parameter-efficien training method by using multiple low-rank updates to effectively train high-rank networks. This method allows training larger models with significant memory savings and training speed improvements compared to conventional methods. GaLore (Zhao et al. 2024) introduces a memory-effictt training strategy by projecting gradients onto low-rank subspace. This approach achieves significant memory savings while maintaining performance.
al. 2020), and DLRT (Schotthöfer et al. (022) suggest that
3 LOW-RANK OPTIMIZATION: WHAT CAN GO WRONG
This section aims to discuss the nature of the critical points and optimization trajectories obtained when using gradient-based strategies for low-rank parameters, and why straightforward application of gradient-bar based steps to factorized adapters may lead to suboptimal results.
Consider neural network layer of the form
z oW pix USV x),
where a is an arbitrary activation function, Wpt E Ru are the frozen pre-trained weights, and U, V E Raxr, S E Rrxr are the rank-r adapter weights, with input x. For simplicity, omit the bias term. Low-rank adapters of the form W = USVT E Rax* have gained popularity in recent approaches such (Zhang et al.) (023), although our discussion extends other equivalent formulations like W = AB (Hu et al. 2021). The objective of the training process is to minimize loss function L(W) to find an optimal adapter weight W.. For full-rank matrices (r = n), optimality requires that Vw C(W.) = O. However, when r < n. this condition is generally unattainable due to the reduced parameter space. In this scenario, we seek matrix W, that is locally optimal within the low-rank parameter space, meaning no further reduction in the loss function L is possible in the neighborhood of W. necessary condition for local optimality can be expressed as P(W.) (.W.)VC(K C(W,) )) = O, see e.g., Sato 2021 Theorem 3,4). For orthonormal U and V. the projection operator P(USVT)Z == UU 201
+ ZVVT represents the orthonormal projection of Z onto the tangent space at USV If W. is not saddle point, then this condition ensures that no search direction within the tangent space of W, can further decrease the loss. See also Appendix | Note that this only guarantees local optimality, a limitation shared by all maddient-based optimizers.
Current training methods for low-rank adapters aim to optimize the low-rank factors with a single backpropagation pass to compute all the required gradients simultaneously. This boils down to integrating the following gradient flow equations for each individual factor
ú = VoL =-VV
V V. UUS
$ - V = -UT
where we use the chain rule and the decomposition W = USVT to derive the expressions for Vu,s.v.. Here, we have omitted the dependence on the time variable t, i.e., U,S. S(t, Vtt) for improved readability, and use dots to denote time derivatives. An explicit time discretization with a time step size equal to the learning rate À leads to the simultancous gradient descent updates commonly employed in conventional training methods for LoRA. At first glance, this
3

---

Published as a conference paper at ICLR 2025
VL(W.)
Tw,M M
Tw, M
W(9
M
(a) Gradient flow of simultaneous gradient descent.
(b) Riemannian gradient flow.
Figure Illustration of simultaneous vs. Riemannian gradient flow. The projector of the simultaneous gradient flow converges point W, such that P(W.)VZ projection, the gradient is not orthogonal to the tangent plane, i.c., W, is suboptimal. For Riemannian gradient flows, the adapter converges to a point W. such that P(W.)VC = O. Since P is the orthogonal projection tangent space Tw.M, which further decrease the loss. Here, . denotes the space of low-rank adapters, and Tw. M represents the tangent space the optimal adapter weight W..
Since P not an orthogonal
the tangent space, W. is local optimum, i.e., directions exist in the
procedure appears effective, single update step will decrease the loss if we freeze all but of the fawtooss However, in practice, LoRA training modifies all low-rank factors simultaneously. raising the question of how this affects the overall optimization trajectory.
To address this, consider the evolution equation for W = USVT, derived directly using the chain rule and eq. (2)
W = ÜSV +UŠVT + UUST
VWLVS SV UU VWVVVV - USS W
(3)
The operator P(USV : ZVS SVT + UU ZVVT + USSTUT Z again represents linear mapping onto the tangent space at W = USVT Note that this projection depends on the individual low-rank factors U. S, and V. but use the notation P(W) for brevity. Simultaneous descent methods approximate the gradient flow of eq. (3). which ideally converges to a solution W,, such that
O. However, P is orthogonal only when U and V are orthonormal and I, where denotes the identity matrix. these conditions are not met, the resulting optimization process may not find an optimal weight within the low-rank parameter space. This is because P(W.)V((W..) 0 does not imply P(W,) VL(W.) O, thus there could still be some decrease direction along the tangent space as depicted in Figure L
To construct methods that converge to an optimal low-rank solution, alternative approach is to evolve the adapter W along the projected gradient flow W(t) - V In this case, the corresponding evolution equations for the low-rank factors take the form
Ú -I- - UU V LwS-
V' -1- TTST
= -U- VWLV,
assuming that U and V are orthonormal Koch & Lubich 2007b)
While the evolution defined in eq. guarantees convergence to an optimal low-rank adapter, the presence of the S-1 term on the right-hand side introduces stiffness in the gradient flow. This stiffness can significantly slow down convergence, especially when the singular values in vary greatly in magnitude. Robust solutions to address the stiffness problem and ensure convergence without being hindered by the S term have been proposed Schotthöfer et al. (002)); Zangrando et al. (0024); Schotthöfer & Laiu 2024). However, these methods require multiple gradient tape evaluations per training update, which makes them computationally more expensive than traditional LoRA training techniques with simultaneous updates.
4

---

Published as a conference paper at ICLR 2025
To overcome these limitations, we propose GeoLoRA, a novel training method for low-rank adapters that only requires single gradient tape evaluation per update while ensuring convergence to an optimal low-rank solution, following the projected gradient flow in eq. 4.. This approach retains the computational efficiency of conventional LoRA methods while achieving comparable or even superior performance. By eliminating the need for multiple gradient tape evaluations, GeoLoRA offers a practical and scalable solution for training low-rank adapters effectively.
Before presenting the proposed training method, we illustrate different behaviours of different low- rank adaptation strategies using toy example. Consider the problem of matching a rank-r target matrix W earget E Raxx with low-rank adapter W. formulated as:
min W WI W 2
(5)
We compare the convergence behavior of six different training methods for n 5000, r and learning rate of x = 0.1:
104
10a
1. Full fine-tuning (FT) (blue).
10-1
DLRT from (Schotthöfer et al. 2022) (orange),
10
Training Setup
3. The proposed GeoLoRA method (green),
Full
4.
Fixed rank LoRA from Hu et al. (202T) (red),
10-3
DURT FT
GeoLoRA
5. AdaLoRA from (Zhang et al. 2023) (brown),
LoRA FT
Fixed rank AdaLoRA (purple).
10-3
LORA
AdaLoRa FT
10
10a
10a
10a
Iteration
In this experiment, the fixed rank approaches (4, 6) use a rank of 50. All adapters W are initialized to zero, with Sn for the SVD-based methods (2, 3, 5, 6) and B 0 for LoRA-based methods (4).
The results show that the proposed GeoLoRA method (3) converges as quickly as full fine-tuning. In contrast, method (2) (DLRT) takes approximately twice long due to the sequential updates of the basis and coefficient matrices]] LoRA-type methods (4, 5, 6) exhibit slower convergence due to the suboptimality of the underlying gradient flow defined in eq. 2 AdaLoRA (5) solves the same gradient flow as method (6) but plateaus at loss of 5 x 10 corresponding to the regularization parameter for the terms | U - T + ||V
Tll that enforce the orthonormality of U and V. We had to fix the minimum rank to 5 for Adal LRR to prevent stalling of the optimization due to rank underestimation, an issue not observed methods (2) and (3), where rank augmentation avoided this problem.
4 THE PROPOSED METHOD
In this section, we introduce GeoLoRA (Geometric Low-Rank Adaptation) a novel low-rank fine- tuning method that integrates rank adaptivity, low-rank optimality, and memory and compu- tational efficiency. Our method builds upon the parallel geometric low-rank integrator originally designed for model order reduction in high-dimensional PDEs (Ceruti al. 2023). and is equipped with loss descent, approximation bounds, and convergence guarantees. Notably, it improves upon existing dynamical low-rank methods, c.g. (Schotthöfer & Laiu, 2024 Zangrando et al., 20224] [Schot] thöfer et al. 2022) by updating basis and coefficients in parallel opposed to sequential basis update and coefficient step. Moreover, only a single backward pass per iteration step required through a novel evaluation strategy of robust gradients, thus doubling the wall-time performance. GcoLoRA is, therefore, the first low-rank training method solving the optimal gradient flow eq. (4) with training times per iteration comparable to standard simultaneous descent approaches low-rank adaptation such as LoRA and AdaLoRA (Hu et al., 2021 Zhang et al.) 2023). In particular, it improves upon these methods by providing robustness and convergence guarantees and demonstrating an overall improved performance and robustness to hyperparameters in numerical examples.
'The loss plateaus appear since the loss value remains constant during basis update and only decreases during coefficient update. This accounts for the fact that two gradient tapes need he computed.
5

---

Published as a conference paper at ICLR 2025
Algorithm 1: Single iteration of GeoLoRA.
The functions opt imi _er_stee masissaugmentttionn and truncati are detailed
in Algorithm 2 in the appendix.
Input: Initial orthonormal bases U, V E guxr and diagonal S E 1R**";
7: singular value threshold for rank truncation;
A: learning rate.
Evaluate ((USVT)
/+ Forward evaluate */
GU + Vv.(USS) Gs + VsC(USV Gv VvL(USVT)
/* Backprop */
Snew + opppimizer_stee(S..G.
Knew < optiimizer_stepUUS G
/* in parallel x/
T. 1 opt.imizer_step((S GySS-,
U - besiis_augenntttion( K_ew)
/* in parallel */
V + basis aossls_augmentationn.....
Snow
L new,
S
E 2rx22 /* Assemble new coefficient matrix */
UTK Keew
U,S,V,S etruncation(. I V V)
Starting from an initial factorization Uo, Vo, So with initial rank ro, where Sp diagonal and full-rank, GeoLoRA performs the following steps (also summarized in Algorithm [
Perform a (stochastic) gradient step to compute the new variables Snew E R XTo L and K Raxro, as follows:
Snew So AV CV
Keew UnSo AV Vo )ST
(6)
new = VoSS Vo
We will see in Theorem 3 that using these variables mitigates the stiffness of the system in eq. while approximating the optimal gradient flow. Note that the right-hand side gradients V. V and V SL can be evaluated with only backward pass through the network using standard algo- rithmic differentiation techniques. halving the computational cost of existing geometric methods such (Schotthöfer al. 2022 Zangrando al. 2024). Evaluation of the inverse Sp induces no computational overhead since So is diagonal at the start of each iteration.
Augment the current bases Ug. Vo to twice their rank using the gradient dynamics of the loss, which is encoded in Kneew and new i.e.
i -U; U] = orto Kmew]) x and r = [Vo V = oroho I, nx202
(7)
Here "ortho" denotes column orthonormalization procedure such the QR-algorithm. This augmentation step provides the low-rank adapter with a larger search space to increase the rank of its adaptation if the initial rank-guess ro was insufficient to fully capture the problem. Doubling the rank implies that in log(n) training iterations any rank can be captured by a rank one initialization, eliminating the need for tuning r as hyperparameter, see Figure
3. Assemble the augmented coefficient matrix
3+
Snew U K new
I new,
RR 122rxx2r
(8)
where we obtain the block entries Snew new and Dew from eq. 6
4. Truncate redundant singular values 8; of S and the corresponding singular vectors, i.e. basis
functions of Ü, V. using the criterion
2r
Σ
22 < 0,
(9)
i=

---

Published as a conference paper at ICLR 2025
where ri is the new rank of the factorization and 1 is tresholding hyperparameter. The singular values s; are obtained via the SVD of S PEQ E R Then determine the new factorization as Ruxr, U1 Ü (......)))) E Raxri and V1 (Q E R The truncation threshold 1 is chosen relative to the nuclear norm of the specific layer's 's current singular values, i.c. v ||| f Other norms, such as the I-norm of the singular values s,, possible as
S1 - diag($) $1, ..... : 5y1 $1,........
well. Thus, the truncation threshold determines how aggressively to prune each layer individually. Analogously, the following global threshold similar to the one used in e.g. (Zhang al. 2023
Ghadiri
et al. 2023 Idelbayev & Carreira- Pereinan 2020)
L
2rc
L
ML
Σ
7
(10)
t=1 i=1+1
can be considered by summing the singular values across all the layers 6 = 1, ,.... L. To directly control the parameter budget, order <²,, by descending by magnitude and selecting the largest ones first until either eq. (10) is violated or the budget is depleted.
4.1 PARAMETER INITIALIZATION
LoRA A-tpee adapters (uu et al. 202T) initilize the low rank matrices B, with zero initialization of B, and Gaussian initialization of A. This ensures that the fine-tuning indeed starts at the pretrained state of the network, i.., o (W pt + S. A. x) = o (Wp: For consistency with this initialization, the bases Uo and Vo can be initialized as random but orthonormal, whereas the coefficient matrix So has zero-initialization. In this first solve of eq. (6). we set S as the identity matrix. As a result, the first solve of eq. (6) is inconsistent with the optimal dynamics of eq. (4). However, all following iterations evolve the Tow-rank trajectory according to the optimal gradient flow. Since in the first iterations of LoRA fine-tuning, the adapter is typically close the original solution but far from the fine-tuning optimum, this inconsistency is irrelevant to the overall convergence behavior of the method. Alternativly., the required gradients can be computed with three individual gradient tapes in the first iteration, which does not require the inversion of So.
The proposed method can readily be used for dynamic low-rank compression (Schotthöfer et al. 2022 Zangrando et al. 2024 of pre-trained networks, where we consider layer z = a(W x), and approximate W = UnSoVo Here, the initial parameters Up, Su, Vú are obtained by truncated singular value composition of W. Finally, for low-rank pre-training of untrained network with given architecture, i.e. predetermined layer dimensions n. but unknown rank r, the factors Uo. Vo are initialized randomly, but orthonormal and So is initialized randomly, but diagonal for easy initialization of S.
4.2 ANALYSIS
In the following, we analyze Algorithm I Tunder the general assumption that L is L-smooth with constant L and bounded with constant B.
For brevity of exposition we denote = U,S,V, as the low-rank factorization at iteration t evaluated with Algorithm whereas W. denotes the full-rank solution obtained by "full fine-tuning" with stochastic gradient descent. Further, we denote by S(WT, .t) the stochastic gradient of the network loss L w.r.t the low-rank weight W at iteration t, obtained by batch aatch-gradient descent. The i.i.d random variable & models the randomness in the training data batch at iteration t. Lastly, recall that P(WT)) Z denotes the orthogonal projection of the matrix Z onto the tangent plane of the manifold of rank-r matrices at the point WT.
Algorithm is an optimizer on low-rank manifolds: Theorem shows, that the proposed scheme with stochastic gradients indeed decreases the training loss in each iteration, while optimizing manifold, and Theorem yields stochastic convergence to a locally optimal stationary point. with stochastic (mini-batch) gradients fulfills
Theorem 1 (Stochastic descent estimate). Algorithm 7
Eg.., [C(W++)) ((WT) - (1 >
LA²
Eg. (WW)) ((WT/((II1111111 LIEg, [I[ W
2
(11)
where WT, WE, W are the low-rank weight matrices at the start of iteration t + 1, before, and after the truncation step, respectively.
7

---

Published as a conference paper at ICLR 2025
The proof is provided in Appendix D The above theorem yields loss descent guarantee up the two last terms on the right-hand side. The first term of the right hand side induces the step size criterion À I f. which resembles the step size criterion of full gradient descent, where the two right hand side terms read -À(1 2 fllff(
This shows that the low-rank optimizer allows similar learning rates full fine-tuning setup, eliminating the need for the - scaling parameter of LoRA. The last term models the error introduced by the truncation step and is bounded by the uiserdeterminee cutoff threshold d, as Egg ||| W II] R 9. As the solution stabilizes in rank, the error term vanishes, and we obtain the following main convergence result:
Theorem 2 (Convergence). Let L 2 and WT. ......, Wi be the solutions generated by Algorithm 7 over T steps. Let the learning rate sequence {Re] satisfy the Jobbins-Monne conditions
=
E. +00
and each step x the step size restriction > S J. Further assume B Will S D
00, i.e. after some time, the solution W. is contained in a manifold of rank r. Then we have 0,
lim inf [B[]|
T-000
where the expected value is taken over all &t-
The proof is provided in Appendix E. Additionally, the solution trajectory of Algorithm ! is close to the (full-rank) trajectory of the dynamical system
W(t) - Ww (t)),,
(12)
i.e., the gradient flow of full training or fine-tuning:
Theorem 3 (Error-bound). For inleger k; lef = kX. Let (() be the solution of eq. 12). and let WT be the factorized low-rank solution after is steps with Algorithm Assume that for any Z in neighborhood of W (t), we have [|(1 P(Z))V
E, i.e. the gradiens flow close to T M,.
Then,
[[ - Will S Ci + + cg0/
(13)
Moreover, let W (1) denote the solution of the Riemannian flow of eq. (4) Then,
[[RF((t) W S C45 + c22 + cg0/X where the constants C12 C2. C32 Cq depend only on L and B.
(14)
The proof is provided in Appendix G
We refer to Appendix F for an interpretation of Algorithm aas an integrator of the gradient flow of eq.
Finally, we point out that the single-layer case discussed so far is not restrictive, and all the theo- retical results above can be directly transferred the multilayer setting by means of the following proposition:
Proposition (Global structure preservation). The application of Algorithm for multiple LoRA lay- ers corresponds to the numerical integration of an augmented single matrix system on the adjacency matrix of the computational graph
W P(-)(W))V
Where I1 is a linear projection that depends only on the structure of the neural network architecture. Moreover, the application of Algorithm 7 to this system, leads to the global truncation strategy proposed in Section
The proof of Proposition can be found in Appendix Stogether with the relative derivation of the global truncation strategy.
5 NUMERICAL RESULTS
DeBERTa for GLUE. We evaluate the performance of GeoLoRA by fine-tuning the 183 million parameter transformer DeBERTaV3-bass (He et al., 2023) on the GLUE Benchmark (Wang et al.. 2019) and compare the results in T For details the methods, implementation, hyperparameter choices, and benchmark setup, please refer to Appendix B.I In most cases, GeoLoRA outperforms other methods on the benchmark, achieving better metries with significantly fewer trainable parame- ters. This reduction in trainable parameters allows GeoLoRA to process substantially more samples during training and evaluation compared to AdaLoRA.
8

---

Published as a conference paper at ICLR 2025
Table 1: Method comparison for low-rank finetuning, We compare the computational cost of a single training step for an n X layer matrix of rank In the table, "local optimality" refers to the property
P(W.)V = O for the computed adapter W., as discussed in Section 3
Method
compute (per lerration)
memory (per iteration)
gradient evals.
rank adaptive
local opcimality
Pull
((w²)
Oln")
GRA
Ofber + (2n + 1):2 +
OfAar + 36
AdaLoRA
Zhanp et al 2023
O/2er
(2n 1)-2
O 3r
DLKI
Schoimate at 2022
Oltor
(2n + 5jr 9
Olter 3r9)
LoRA
TEU ar.
((2rr)
O(2or)
Table 2: DeBERTaV V3-base fine-tuning on GLUE We compare with full fine-tuning (Full FT),
Houlsby adapter (Houlsby et al. 2019)) (HAdapter). Pfeitfer adapter (Pfeiffer al.) [2021) (PAdapter),
LoRA (Hu et al. 2021), AdaLoRA (Zhang et al., 2023). DoRA (Mao et a (024)). LoRA+
al. 2024). and BithitZakken et al.) 2022). We report target metries and computational performance (higher is better) for the median of runs using different random seeds. Best results per dataset
are shown in bold. Results for BitFit, HAdapter, PAdapter were taken from (Zhang et al. 2023).
"AdaLoRaa matched" has the rank budget adpated to match the parameter count of GeoLoRA.
Mathaal Piramt
MNNII
CuLAA
QQP (F1)
QNLI (A60]
RTE (Aoc)
MRTC (Acc)
Men
(hoc)
[Acc]
(Moc)
Fa (194M)
39.90
95.61
50.19
52.50
94.03
$3.75
89.36
91.80
87:02
(0.1M
89.37
04.84
66.96
$4.95
92.24
78.70
87.25
91.35
85,77
HAdayiiz (1.27M)
95:15
46.64
39:17
$4:48
09:05
87:04
PAdapess (LI1M)
90.33
68.77
$5.30
99:46
$3.07
LoRA (1550)
90.29
05.20
06.57
90.61
99.11
50
19-75
W7.57
LaRt 168 (1.33M)
90.31
05.37
69.22
98.53
93.96
$5,50
89.55
$8.07
87.55
BaitA mi 11M)
S0.11
94 30
68 90.90
M3-71
04.31
$5.08
19:33
91 01.58
877 13-56
Adsi Fumpet (7M)
98.68
99.64
90.65
36.00
09:44
91.41
AdLoRs, mutched
90.11 (0.7584)
(1.27M)
(1.07M)
90.48
(0.73M)
55.92 (1.160)
88.21 (0
(4300
(9M)
Eniiuation and bais time comparisn
AdLLRR cevaltrann [iboo]
A2
17466-7
24.6%.
9,2/3.2
10,3332
21.1/85
13.754445
GeoLaiRA covalbrsan [sshoo]
17.149
21.383
37.4%9.
12/03.8
S.91L.
13.29.7
126337
17.05.
Performance analysis. The proposed method from Algorithm I combines low-rank optimality guarantees with computational efficiency gains compared to existing low-rank optimization methods, as shown in Table For rank r adapter, the computational cost of gradient evaluation (i.e., eq. (6) is equivalent to that of AdaLoRR, which updates U. S, and V directly, and is similar to a standard LoRA update. The cost of basis augmentation is (nr²? > due to the QR decomposition in eq. 7 comparable to evaluating the orthonormality regularization terms in AdaLoRA. Rank truncation is performed via an SVD of S at a cost of (rr3), where typically r < n. The complexity analysis shows comparable per-iteration costs for LoRA, AdaLoRA, and GeoLoRA. In Table also report the number of iterations computed per second during training and evaluation for both GeoLoRA and AdaLoRA, demonstrating that GeoLoRA outperforms AdaLoRA across almost all GLUE benchmarks. We note that training and inference speed depend on both layer ranks and sequence lengths, and the performance difference is less pronounced for benchmarks with longer sequences.
Vision transformer for object classification. We compare GeoLoRA and AdaLoRA on fine-tuning the tit-base-patchll 2-224 Vision Transformer, pre-trained on the Imagenet- dataset, and fine-tuned on 10 Cifar and Imageem GeoLoRA A "local" uses layer-wise rank truncation, and "global" uses the same global rank budget as AdaLoRA of 200 ranks. Details on implementation and typerparameters are provided in Appendix B.2, Table 3 shows that GeoLoRA achieves higher validation accuracy than AdaLoRA, while using fewer trainable parameters.
Ablations. In Figure 2. examine how the performance of GeoLoRA is influenced by the initial rank and learning rate. Figure a. b) demonstrate that GeoLoRA dynamically recovers the intrinsic rank of the low-rank adaptation, regardless of the initial rank, highlighting the robustness of the method with respect to this hyperparameter. Notably, Geol . can extend the adapter rank to full rank if necessary within logarithmic time, while truncating in constant time (in terms of optimization iterations). We provide a detailed discussion of the rank distribution across transformer layers in Appendix B.2 Similarly, Figure , d) show that GeoLoRA is less sensitive to learning rate variations compared to AdaLoRA.
Dreambooth stable diffusion. We test GeoLoRA on fine-tuning Stable Diffusion (Rombach et al., 2021) using Dreambooth (Ruiz al. 2023) their original datasets. Implementation details
9

---

Published as a conference paper at ICLR 2025
Table 3: Vit-base-pathh6-222 fine-tuning on Cifar10, 100 and Imag-em We compare LoRa, AdaLoRA to GeoLoRA with local and global budgeting reporting the median runs using different random seeds. GeoLoRA "local" uses layer-wise rank truncation, and "global" uses the same global rank budget as AdaLoRA.
Table 4: Stable Diffusion on Dreambooth benchmark. We compare LoRA and GeoLoRA re- porting the median of 5 runs. For AdaLoRA, A, ro is the initial and r is the target rank.
Method
Citar 10 [66]
Cifar 100 [S]
Timy-Imagenet (%)
Method
Val. Less
Pumms
[S]
LoRA (r
112755
3.0
LoRA (r
0.281
1.8
LeRA
0.47M (y=3)
98.47
0.47M (m=3)
0.9MM (1=6)
AdiLoRA
0.245
4.7M
oRA
0.47M
98.
0.45M
91.44
DOM
87.21
0.347
GeoLoRA, local
0.47M
98.55
0.35M
91.43
0.92M
88.09
GooLoRA, global
0.4SM
98.51
0.47M
91.62
0.75M
88.07
GeoLoRA
GRSLaRA 0.02)
0.342
520000
500000
90.0
460000
460000
Intal
Vel
07.8
420000
97.8
400000
16
880000
97.4
360000
Epochs
Epocts
(a) Validation accuracy over epochs.
(b) Trainable parameters over epochs.
Val. Aoc (%)
Val. Acc (%)
200
24.11
70.52
200
22.22
rank budpot
300
28.
75.41
97.20
rank budgot
300
nan
nan
97.21
400
27.34
4,0
nan
nan
500
26.53
500
600
31.88
89.58
97.12
600
11.05
nan
76.95
0.05
0.025
0.05
0.025 leaming rate
learning rate
(c) Geol.oRA
(d) Adal oRR
Figure 2: Top panels (a, b): GeoLoRA-adapted ViT-32b fine-tuned Cifar10 with different initial layer ranks, using a learning rate of 3e-3 and t = 0.3. The total number of trainable parameters converges to similar steady state, regardless of the initial rank. The differences in validation accuracy between runs are smaller than the variance observed within individual setups. Bottom panels (c, d): GeoLoRA- and AdaaoReddd ViT-32b fine-tuned Cifar10 with different rank budgets and learning rates. Fields marked with nan indicate that training diverged within the first epoch. GeoLoRA demonstrates significantly greater robustness than AdaLoRA, particularly with high learning rates.
provided in Appendix B.5) As displayed in Table GeoLoRA consistently achieves lower validation loss with fewer trainable parameters.
CONCLUSION
We introduced GeoLoRA (Geometric Low-Rank Adaptation), a novel adaptive low-rank fine-tuning method that combines computational efficiency with robustness. Based on geometric principles from dynamical low-rank approximation theory, the method comes with guarantees of convergence and local optimality. By leveraging a parallel update strategy of the low-rank adapters, the method requires only single backward pass per iteration, achieving inference and training speed comparable or superior to existing baselines such as AdaLoRA, and much more efficient than previous geometric- aware strategies. Our experiments on the GLUE benchmark, Vision Transformers, and Stable Diffusion demonstrate that GeoLoRA outperforms existing PEFT methods in both accuracy and efficiency, with fewer trainable parameters. These results, alongside strong thcoretical guarantees, position GeoLoRA LoRA A
robust solution for efficient model adaptation.
10

---

Published as a conference paper at ICLR 2025
FUNDING ACKNOLEDGEMENTS
This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-000R22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government re- tains non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the pub- lished form of this manuscript, or allow others do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored re- search in accordance with the DOE Public Access Plan(ht /tep:r///eeeg...ee wtw.gooowwwwoooo downloads/ doe-puan che-public access plann.
The work of E. Zangrando was funded by the MUR-PNRR project "Low-parametric machine learn- ing". Francesco Tudisco is partially funded by the PRIN-MUR project MOLE code: 20222K5ME7 MUR D.D. financing decree n. 20428 of November 6th, 2024, CUP B53C2400641000 and by the PRIN-PNRR project FIN4GEO within the European Union's Next Generation EU framework, Mission 4, Component 2, CUP P2022BNB97.
REFERENCES
Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning, 2020.
Gianluca Ceruti, Jonas Kusch, and Christian Lubich. A rank- ada-tive robust integrator for dynamical low-rank approximation. BIT Numerical Mathematics, 2022. URL https:://oi..or/////
1007/s10543 021-00907-1
Gianluca Ceruti, Jonas Kusch, and Christian Lubich. A pank integraor for dynamical low-rank approximation, 2023. URL batt:ss//arrav...rr/aaa/////// 05660
Matthieu Courbariaux, Itay Hubara, Daniel Soudry, Ran El- Yaniv, and Yoshua Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained to+
or-1. arXiv:1602.02830. 2016.
Mchrdad Ghadiri, Matthew Fahrbach, Gang Fu, and Vahab Mirrokni. Approximately optimal core shapes for tensor decompositions, In International Conference on Machine Learning, pp. 11237-11254. PMLR, 2023.
Yiwen Guo, Anbang Yao, and Yurong Chen. Dynamic network surgery for efficient dnns, Advances in neural information processing systems, 29, 2016.
Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models, 2024.
Pengcheng He, Jianfeng Gao, and Weizhu Chen. Debertav3: Improving deberta using electra-style pre-training with saddient-disentangled embedding sharing, 2023.
abs/ 0aas/211..955 09543
Yihui He, Xiangyu Zhang, and Jian Sun. Channel pruning for accelerating very deep neural networks. In Proceedings of the IEEE international conference computer vision, pp. 1389-1397, 2017.
Arsen Hnatiuk, Jonas Kusch, Lisa Kusch, Nicolas R. Gauger, and Andrea Walther. Stochas- tic aspects of dynamical low-rank approximation in the context of machine learning. Op- timization Online, 2024. doi: http://pwmmmiaaatoooooonnne///// optimization on www.onninnn...moomoooo wwt.iinn.org//22/////22
URL https://
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 2790-2722720 2799. PMLR, 09-15 Jun 2019. URL https://procedingg. mlr
press/v97 /oou sbyl 9a html
11

---

Published as a conference paper at ICLR 2025
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint wrXXv:2100600665. 2021.
Yerlan Idelbayev and Miguel A. Carreira- Carreira-Perpinan. Low-rank compression of neural nets: Learning the rank of each layer. In Proceedings the IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), 2020)
Mikhail Khodak, Neil Tenenholtz, Lester Mackey, and Nicolo Fusi. Initialization and regularization of factorized neural layers. In International Conference on Learning Representations, 2021.
O. Koch and C. Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 2007a. ISSN 0895-4798.
doi: 10.1137/050639700. URL
https: orrp::////oio.///00.../ 0ttp:/00..1177000333333
Othmar Koch and Christian Lubich. Dynamical low-rank approximation. SIAM Journal on Matrix Analysis and Applications, 29(2):434 454, 2007b.
Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates, 2023.
Yulong Mao, Kaiyu Huang, Changhao Guan, Ganglin Bao, Fengran Mo, and Jinan Xu. Dora: Enhancing parameter-efffcient fine-tuning with dynamic rank distribution. 2024. httptt
//api. sem.reettetttttcccooooooo
ID: 12:70006262
Molchanov, Tyree, Karras, Aila, and J Kautz. Pruning convolutional neural networks for resource efficient inference. In International Conference on Learning Representations, 2017.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Iryna Gurevych. Adapter- fusion: Non-destructive task composition for transfer learning, 2021. URL httts:://axptit oxc/abs/2005.000
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation, 2023. URL aittps://arxlle wtt/
http//b//22200 12242
Hiroyuki Sato. Riemannian optimization and its applications, volume 670. Springer, 2021.
Steffen Schotthöfer, Emanuele Zangrando, Jonas Kusch, Gianluca Ceruti, and Francesco Tudisco. Low-rank lottery tickets: finding efficient low-rank neural networks via ma- trix differential equations. In Advances in Neural Information Processing Systems, 2022. URL httppe:/pproceeiingg.. _tt.utteee//epoeeee.........p..p htpp
file F Fi1e7e38800eerrcee00555555555553 http:////cpppeeieeeeeeaeeaaaeeee wwtp:/ffweeeeeeeeeeeeeeeeeeee
pddf
Steffen Schotthöfer and M. Paul Laiu. Federated dynamical low-rank training with global loss 17887
convergence guarantees, 2024. URL
Mojtaba Valipour, Mehdi Rezagholizadeh, Ivan Kobyzev, and Ali Ghodsi. Dylora: Parameter efficient tuning of pre-trained models using dynamic search-free low-rank adaptation, 2023.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019. URL
https: org.
/bwp////b/////// abs/ 1804 07461
Hongyi Wang. Saurabh Agarwal, and Dimitris Papailiopoulos. Pufferfish: Communication--fficinen models at extra cost. Proceedings of Machine Learning and Systems, 3:365-386, 2021.
Gerhard Wanner and Ernst Hairer. Solving ordinary differential equations II, volume 375. Springer Berlin Heidelberg New York, 1996.
12

---

Published as a conference paper at ICLR 2025
Jiaxiang Wu, Cong Leng, Yuhang Wang, Qinghao Hu, and Jian Cheng. Quantized convolutional neural networks for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2820-4828. 2016.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. Bitfit: Simple parameele---ffiiiettinnn-e for transformer-based masked language-models, 2022. URL https://arxi...rr// 2106. 10199
Emanuele Zangrando, Steffen Schotthöfer, Gianluca Ceruti, Jonas Kusch, and Francesco Tudisco. Rank-adaptive spectral pruning of convolutional layers during training. In Advances in Neural Information Processing Systems, 2024.
Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effiiiet fine- tuning, 2023.
Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection, 2024.
13

---

Published as a conference paper at ICLR 2025
Algorithm 2: Various auxiliary functions
def opt imi zer zer_st((:::: param, G: gradient, learning rate):
pnew
t P XG
/* May use uomentum and weight decay x/
return pnew
def basis SurgisssammeettiionnB old basis, GB: basis dynamics):
[B B] qr([B Gøl)
return
def truncation($: augmented coefficient, 0: augmented basis, V. augmented co-basis ): Er On &- truncated svd(S) with threshold i to new rank
U +- ÜPr V < VQn
/* Basis update x/
10
S E Stw f- 2
/ Coefficient update with diagonal En, */
return U,S, Sia 11
A ALGORITHM FOR AUXILIARY FUNCTIONS
We present the auxiliary function for Algorithm Tin Algorithm
B ADDITIONAL INFORMATION FOR THE NUMERICAL TEST CASES
B. GLUE BENCIMARK
B. 1.1.1 DATASET DESCRIPTION
We compare GeoLoRA to several fine-tuning methods from recent literature the General Language Understanding Evaluation (GLUE) benchmark (Wang et al.) 2019). The GLUE benchmark is collection of diverse natural language understanding tasks designed to evaluate the performance of models in comprehending and processing human language. GLUE provides a comprehensive assessment by including tasks that cover range of linguistic phenomena, such as textual entailment, sentiment analysis, sentence similarity, and more. The benchmark consists of nine different tasks:
CoLA (Corpus Linguistic Acceptability): Classifying whether a sentence is grammatically
correct not.
SST-2 (Stanford Sentiment Treebank): Sentiment analysis task where the goal is to classify the sentiment of sentence positive negative.
MRPC (Microsoft Research Paraphrase Corpus): Identifying if two sentences are paraphrases of each other.
STS-B (Semantic Textual Similarity Benchmark): Measuring the degree of semantic similarity between two sentences on scale from 1 5.
QQP (Quora Question Pairs): Determining if pair of questions are semantically equivalent.
MNLI (Multi-Gen Genti-GGGreeee Natural Language Inference): Classifying the relationship between pair of sentences (entailment, contradiction, neutral).
QNLI (Question Natural Language Inference): Determining if sentence provides correct answer to given question.
RTE (Recognizing Textual Entailment): Binary classification task for entailment and contradiction.
WNLI (Winograd Schema Challenge): Resolving pronoun reference ambiguity in sentences. Specific Focus: MRPC (Microsoft Research Paraphrase Corpus)
We present the benchmark overview in Table To recapitulate, the F1 score is defined in dependence of precision score and recall score R. The model precision given by
Pr
P:=
(15)
Pr PF
14

---

Published as a conference paper at ICLR 2025
Table 5: Summary of GLUE benchmark tasks
Corpus
Task
#Train
#Dev
#Test
#Label
Metries
Single-Sentence Classification (GLUE)
CoLA
Acceptability
8.5k
lk
lk
2
Matthews corr
SST
Sentiment
67k
872
1.8k
2
Accuracy
Pairwise Text Classification (GLUE)
MNLI
NLI
393k
20k
20k
3
Accuracy
RTE
NLI
2.5k
276
3k
2
Accuracy
QQP
Paraphrase
364k
40k
391k
2
Fl
MRPC
Paraphrase
3.7k
408
1.7k
2
Accuracy
QNLI
QA/NLI
108k
5.7k
5.7k
2
Accuracy
Text Similarity (GLUE)
STS-B
Similarity
7k
1.5k
1.4k
1
Pearson/Spearman cor
where Pr is the number of true positive and Pp the number of false positive examples, The recall R is the ratio
Pr
R
(16)
Pr
NF
where Np the false negatives. The Fl score combines these two metrics to
2PR
F1 P+R
(17)
B.1.2 REFERENCE IMPLEMENTATIONS
Full finetuning (FT):: This is the most common approach for model finetuning and transfer learning. Here, the model initialized with pre-trained weights and all model parameters are updated with gradient descent.
Bitfit (Zaken et al. 2022): Here, the model is initialized with pre-trained weights, but only bias terms updated with gradient descent.
Two-layer adapters are inserted between transformer blocks. In (Houlsby al.[220 2019), the adapter inserted between the self-attention module and the feed-forward module and equipped with a residual connection. In (Pfeiffer et al.. 2021), the adapter applied after the feed-forward module and the layer-norm module. To maintain conistency with the notation of (Zhang et al. [223)) we call the method of (Houlsby et al. 2019) HAdapter and the method of (Pfeiffer et al. 2021) PAdapter.
Adapter tuning (Houlsby et al.] 2019; Pfeiffer et al., 2021
LoRA (Hu et al. 2021): As stated in Section 3, LoRA applies additive corrections to selected weight matrices, i.c. z o[W ppp + @ ABB x) for A "xxr We apply LoRA key Wi. query W, and value W., matrices of all attention blocks, and to both feed-forward layers W,, and W f2- We chose the learning rates and optimizer as described in (Zhang et al. 2023), Appendix D-F.
The values in Table for FT, Bitfit, Adapter tuning, and LoRA are taken from (Zhang et al. (023). We compute the results for the methods DoRA, LoRA. LoRA+, and AdaLoRA using the HuggingFacc open source implementations of the respective adpaters.
DoRA Mao al. 2024): DoRA an low-rank adapter similar to LoRA. The Main difference is that the AB matrices normalized and and additional magnitude parameter is included. Further, the adapter is initialized with the pretrained weights Wo instead of zero initialization found LORA.
LoRA+ (Hayou et al. 2024): The key difference between standard LoRA and LoRA+ + is in how learning rates are set. With standard LoRA, the learning rate is the same for A and B. In LoRA+, different learning rates set for A and B, where the learning rate for B is set a multiple of that of A. The choice of the learning rates the same of AdaLoRA (next paragraph), with a ratio An XA 1.1.
15

---

We compare against AdaLoRA, where we first match the total parameter budget to that of LoRA, i.e. choose the final budget ((T)) of ooRA as 576. Then we set ((0) as 1.5 times ((T)) In addition to the hyperaparmeters chosen above, we compare AdaLoRA with budget levels obtained by GeoLoRA, Published as a conference paper at ICLR 2025
AdaLoRA (Zhang et al. 2023): As stated in Section to selected weight matrices, i.c. z = OW USS x) with arbitrary activation 0, frozen pre-trained weights Pp E _ rank r adapter weights U,V E Ra** R An SVD-based truncation mechanism is used to select layer ranks. Alternatively, the loss-sensitivity of singular vectors can be used for layer rank selection. Just like LoRA, we apply AdaLoRA to key Ws. query Wg and value W, matrices of all attention blocks, and both feed-forward layers W and f2-
AdaLoRA applies additive corrections
We use the implementation of (Zhang et al. 2023 Appendix C) to compute the re- sults for the presented reference methods and of their Git Repository https: /ttp::/gitttuut com wwwp:/iggw/...aaagggaoooooo Bataaa//gg/ALaaRttteeeeeeeeeeeee e10f5ebeel6ce aetbbeeelcc78fa224aaaa.e.
the reported hyper-parameter choices
We set the ex- ponential moving average parameters 31 and B2 of AdamW as their default value 0.85. We select the learning rates as denoted Table and the regularization coefficient y as 0.1.
where again tune the final budget ((7) approximately match the parameter count of GeoLoRA.
B.1.3 IMPLEMENTATION DETAILS
We implement GeoLoRA as similar as possible as Adalora to achieve a fair comparison. That is, add an adapter of the form z pWppx + USV x) the key WN. query W; and value W matrices of all attention blocks, and to both feed- forward layers W,, and Wy,- For each adapter, we employ Algorithm[[]] update the layer weights and ranks. All hyperparameters (except for the truncation tolerance т, which is unique to GeoL are identical to the hyperparameters in Lora+, DoRA, and AdA.. The truncation tolerance chosen as 0.15 across all datasets, i.e., singular values below this weighted threshold are set to zero. Note that all other low-rank adapters have similar hyperparameters to define the compression ratio specified above in the respective sections.
In Table 6 we display the hyper-parameter choices of GeoLoRA, Lora+, DoRA and AdaLoRA.
Table Hyper-parameter setup for GLUE benchmark. Learning rate, batch size, and number of epochs are adopted from the GitHub repository of AdaLoRA.
Dataset
Learning Rate
Batch Size
Epochs
(GeoLoRA)
inital rank (GeoLoRA)
Aa (LoRAt)
MNLJ
5 x 10-4
32
7
0.15
10
1.1
RTE
1.2 10
32
50
0.15
10
1.1
QNLI
1.2 10
32
5
0..15
10
1.1
MRPC
1 30
32
30
0.15
10
1.1
QQP
5 10-4
32
0.15
10
SST-2
x 10
24
0.15
10
CoLA
5 10
32
25
0.15
10
1.1
STS-B
2.2 10
32
25
0.15
10
1.1
B.2 OBJECT CLASSIFICATION BENCIMARKS FOR THE VIT-BASE-PATCHI1---2 VISION TRANSFORMER
We present in Table 3 results for finetuning the vit- bi--base-ptchhll 1- vision transformer, which is pretrained on the imagenet- imagenet-Ik-dataset The pretrained weights are downloaded from the torch-vision python package. For both AdaLora and GeoLoRA, we augment the key, query, and value matrices from attention layers as well as the three fully connected layers of each transformer block with a low-rank adapter. The biases of each layer are trainable. Additionally, the classifier is augmented with low-rank adapter. The classifier is low-rank by construction, and we fix the rank as the number of classes. We fine-tune the vision transformer Cifar10, Cifar100 and -iag-em
The hyperparameter settings to generate the results of Table Table 7
Figure and Figure 4 are given in
Figure 3 and Figure show the rank distribution across layers for both AdaLoRA and GeoLoRA with global budget, for learning rate À 1e-3 and X 1e-4 and budgets ranging from b 200, .., 600
16

---

Published as a conference paper at ICLR 2025
Table 7: Hyper-parametee setup for fine-tuning vit-base-patch665 patch16-224 vision transformer with Ge- oLoRA. AdaLoRA uses the same hyperparameters and the same rank budget for the global truncation as GeoLoRA.
Dataset
Learning Rate
Batch Size
Epochs
r (local truncation)
rank budget (global truncation)
inital rank
Cifar10
x 10
256
0.2
200
Cifar100
10 a
TinylmageNet
10
256
0.15
300
32
total ranks for the network. Both methods prefer to allocate higher ranks to the deeper layers of the vision transformer, and prefer fully-connected layers over attention layers. Both methods pefer the first fully connected layer of transformer block over the second. Overall GeoLoRA tends to assert higher ranks to single layers, compared to AdaLora, that distributes ranks more heterogeneously. The effects more pronounced for smaller learning rates.
B.3 ABLATION STUDY FOR THE INITIAL RANK FOR VIT
In addition to the results in Figure 2 we show in Table & that for Cifar10, Cifar100 and Tiny- Imagenet,, GeoLoRA is robust with respect to the choice of the initial rank. We train using the hyperparameter of Table 7 but adapt the initial rank and fix the local truncation criterion 7 = 0.15.
Table 8: Vit-base-pate 16-224 fine-tuning on Cifar10, 100 and Tmny-Imagenet. We report the median of runs using different random seeds. GeoLoRA uses the layer-wise ("local") rank truncation with tolerance 7 0.15.
Method
Cifar 10
[%]
Cifar 100
[%]
Tiny-Imagenet [%]
Params
Acc [%]
Params
Acc [%]
# Params
Acc [%]
GeoLoRA, local (r=10, 0.15)
0.472M
98.52
0.351M
91.60
0. 0.904M
88.08
GeoLoRA, local (r=16, 0.15)
0.472M
98.55
0.357M
91.63
0.909M
88.10
GeoLoRA, local (r=32, 0.15)
0.473M
98.54
0.362M
91.63
0.921M
88.09
B.4 Loss CURVES FOR VIT
We consider the C0 test case with the settings of Table but adapt the learning rate to find the critical point, where GeoLoRA still converges well, but AdaLoRA diverges. grid search between 5e-2 and 1e-4 for the learning rate yields -. 3. We display the corresponding training loss curves in Figure 5 For learning rates larger than -e-3, 3, AdaLoRA diverges, whereas GeoLoRA remains stable for the entire range of learning rates. This observation agrees with the results of Figure 2 we observe similar stability behavior in the example of Cifar10. We remark that AdaLoRA performs well for well- tuned learning rates, but the range of "good" learning rates is bigger for GeoLoRA in comparison with AdaLoRA.
where
B.5 STABLE DIFFUSION ON DREAMBOOTHI,
In this numerical example, we apply low-rank adapters all linear and attention layers of the U-Net and the text encoder networks. The hyperparameters for LoRA and GeoLoRA are the same, apart from fact that we start with adapters rank 8 for both Unet and text encoder. train for full epochs, using adam W optimizer, with (81, 82) (0.9, 0.999), initial learning rate 5 x 10 and weight decay set to 10
17

---

Published as a conference paper at ICLR 2025
M
QUAry
Quary
VSlUe
2 2
2
Value
4
122
122
242
ouc
2
3
6
6
10
11
2
3
6
10
11
(a) GcoLoRA: mean b=200
(b) AdaLoRA: mean b=200
QUery
querr
Value
2
value
7
5
kt
4
12.000 241
10
11
10
(c) GeoLoRA: A: mean b=300
(d) ooaLoRA mean b=300
QUary
quary
Mian
I:1
1:1
RLAi
(e) GeoLoRA: mean b=400
(I) AdaLoRA: mean b=400
key
QU2rY
9
2
4
2
query
3 4
value
Mom
value
FCL
5 C
122
122
(g) GeoLoRA mean b=500
(h) AdaLoRA R.: mean b=500
QU3ry
Quary
vilue
5
2
2
value
2
2
2
2
8
8
122
3
7
8
I22
2
9
(i) GeoL: mean b=600
(i) AdaL.oRA. mean b=600
Figure 3: Rank distribution of Vit-32b finetuned on Cifarl0 10 for 5 epochs at learning rate 1 using GeoLoRA and A.A
18

---

Published as a conference paper at ICLR 2025
QUAry
Quary
VSlUe
2
2
2
2
Value
2
2
122
122
242
2
5
4
6
8
10
11
2
3
6
10
11
(a) GcoLoRA: mcan b=200
(b) AdaLoRA: mean b=200
QUeYY
query
VAlue
2
value
Micn
kt
200
12,040
11
10
(c) GeoLoRA: mean b=300
(d) AdaLoRA: mean b=000
QUary
2
Quary
5
I:3
RLAi
(e) GeoLoRA: mean b=400
(I) AdaLoRA: mean b=400
key
QU3rY
7 8 4
8
query
4
4
value
Mon
value
FCL
0 2
RL
4
122
(g) GcoLoRA: mean b=500
(h) AdaLoRA: mean b=500
key
Quary
value
velue
2 0 2
8
2
8
f2
I2
9
2
7
(i) GeoLo mean b=600
(i) AdaL.R.A mean b=600
Figure 4: Rank distribution of Vit-32b finetuned on Cifarl0 for 5 epochs at learning rate 1e-4 using GeoLoRA and A..
C OVERVIEW FOR THE NUMERICAL ANALYSIS
C.1 NOTATION
We provide overview of the notation used throughout the main manuscript and the appendix.
19

---

Published as a conference paper at ICLR 2025
GeoLoRA AdaLoRA
5
Training Loss
2
20
40
60
80
100
Batch Update
Figure 5: Loss evolution ViT trained on Cifar100.
W E R is the full-rank weight matrix of a neural network layer or adapter.
Rxxn is an arbitrary matrix.
M, {Z E Rax= Zan = = is manifold of rank matrices.
M, is the tangent space M, at Z for any Z E R=*x
W USVT E M, is rank-r approximation of a matrix W.
W ÜSVT E M, is rank-r approximation of a matrix with augmented basis.
U, V E Re is the orthonormal basis and co-basis M,.
Ū = Uo. U E Rn 2r is the augmented basis. (Analogously for V.)
Uo E =ax is the basis at the beginning of the iteration. (Analogously for Vo-)
E R"** is the basis augmentation, obtained by [Uo U1 ortho((U,, keew]) E Rnx22r, (Analo-
gously for V.)
" is the coefficient matrix so assemble the low-rank approximation W from U, V. P(Z) is the orthogonal projection onto Mr.
Py UUT is the orthogonal projection onto the range of orthonormal U E Raxr
Py VVT is the orthogonal projection onto the range of orthonormal V Raxr.
When applied to vectors, |-|l denotes the Euclidean norm (f2-norm). When applied to matrices, II-Il
denotes the Frobenius norm.
g) denotes the loss function dependent on weight matrix W and data sample randomness g.
Commonly abbreviated by L(W).
f( = V w L(W; 5) is the negative stochastic loss gradient w.r. W. Commonly abbreviated by f(W).
F(W) = (e[f(W, {)]
the expectation of the random loss gradient, called the deterministic gradient.
C.2 RECAP OF COMMONLY USED PROPERTIES
We recapitulate repeatedly used properties of the mathematical objects and notations introduced above.
Per definition, we have for any Z E nnx
P(W") UUT ZVVT UU ZVVT
(18)
Since M2,,, is subspace of Rxxn for any Z MM can decompose the gradients F and f into F(Z) = M(Z) R(Z) and f(Z) = f(Z) + r(() for any Z E Ruxn, where ( (( ZZ) Tz.M.
C.3 GLOBAL ASSUMPTIONS
The following provides comprehensive overview of the global assumptions for made in the analysis section and proofs of the provided theorems. The assumptions are common in literature, see e.g. (Hnatiuk et al. 20244)
Assumption 1. There is an such that || [|r( S for all Z M, Assumption 2. and f are bounded by a constant > 0 and L-continuous w.et. ||+| with constant
I L>0 0.
20

---

Published as a conference paper at ICLR 2025
Assumption 3. There is constant C such that [[F f(z) SC. Assumption 4. At initial time, assume that the difference of a full-rank weight matrix Wo and its low-rank counterpart W6 is bounded by IW6 Yol S for 8 0.
Assumption 5. For all times, we have w.l.o.g (()) > 0.
D DESCENT DIRECTION
We first state a few auxiliary lemmas, which provide common inequalities that will be used in the following analysis.
Lemma 1. (Hnatiuk al. 2024 Lemma 5.2) For any two matrices Y1, Y2 E Raxx and an L-smooth L with constant L it holds
-Yn ((Y)) ((Y2) < - - -2, + L Y Y21l²
(19)
where f(Y) -
v L(Y).... Furthermore, it holds
L ((X)) _(Y)) S - (Yi Y2, F(X,)) + 2 1 12112
(20)
where = -EV
The following results are primarily based on (Hnatiuk al. 2024) and use the reformulation of truncated terms as proposed in Zangrando et al. 2024). For ease of notation, we use f(WT) = f(W7, Hence, randomness not explicitly stated in notation. Note that in this case, the factorized solution WT. = U1Si V,T is random since it depends on f(WT). When using expected values, explicitly write down the corresponding random variable. That is, Eşl is the expected value for a random variable ç. We denote the random variable in step T as Er and denote B(.) = Eg (1
Theorem 4. (Restatement of Thhorem 71 Algorithm 7 with stochastic (mini-batch) gradients fulfills
EE ()) L(W)) - (1
LA²
E FW + LEg. [I]
W II]
2
(21)
where WT, WT. W are the low-rank weight matrices at the start of iteration before, and after the truncation step, respectively. The step size is given by X.
Proof. Without loss of generality, we restrict ourselves to time steps and write f(W)) shorthand for f(WT T &.). By definition of the coefficient matrix assembly in eq. (8). we get respectively
GūT vWW:) for the right hand side of the Snew block
* UpU fWW,)VT for the right hand side of the new block
VW for the right hand side of the new block
and zero for the lower right block.
Since the augmented bases orthonormal, we write for W UnSoVo
W
WH + AUOU (6W, + NOOT S(W3)ooV AUDUO V(W:)
WV AUOU f(W7) Vo -0OO ((W))ooV NUOU PTT
-W AUOUD VoV, AF(WW)VoVV + AUOUO f(W))
W (AW)W))(WW).
21

---

Published as a conference paper at ICLR 2025
By Lemma we have
L W woll.
((W)) ((W)) S (J(W)), W w
(22)
Therefore, plugging the above equation into eq. (2 yields
TX² [IP(W))((WT)))
<(W)) C(WB) - X((W)),
(23)
= (WWW)) + LXX²
2 [P(W))(WW)))
(24)
- --x IX²
IP(W))
(25)
where the second line is obtained by definition of the orthogonal projection. Comparing the loss before W and after WI truncation yields for some (0, 1) using the mean value theorem and the Cauchy-Schwarz inequality,
((WT) s C(W, + (VC(sWT + (1-s)W") WT W S ((W)) + LIIT Woll (26)
Plugging eq. (26) into eq.
23 then gives
C(W)) ((W)) < - -1
LA
|PP(W))((W))) + LII - Woll.
2
where is the Lipschitz constant F. Hence, taking the expected value yields
Be, [e,(C(W])) < ((WT) - > 111
Ba, Bell) LEs, [I[ W6ll]-
E CONVERGENCE
and WT.... WT be the solutions generated by Algorithm over T steps. Let the learning rate sequence {}e] satisfy the Robbins-Monro conditions:
Theorem 5. (Restatement of Theorem 2 Let L = 0
E, X? < +00
Further assume En E[Il WIll S D oc, i.e. after some time, the solution WT is
contained in a manifold of rank r. Then we have
lim T-000
B[]] (WWA)
where the expected value is taken over all &
Proof. By taking the expected value over EI,...,Er Er in eq. and denoting the corresponding expected value [E[] we get
LX.
E(C(WT -E/((())))E(((W)))E (
EIIP(W))((W))1
+LEI Will
= ^ (1 (1- LA
(WPP())))(W2)))) WIll.
Using telescoping until t T then yields
T-1
LA:
-((Y) & B[C(WT) ((%) - - Σ
A
1
EIIP(W2))(W2)11)
2
t=1
T-1
L
Σ
BillWT+ -W7ll-
22

---

Published as a conference paper at ICLR 2025
Rearranging gives
LA (-)
1
E ((IP(())/((]))))
+ Σ B W4alll
< LD.
Using the assumptions [IP(WT)) [WW)) B and Σ Wi+1 W41lll S D. Now, when - oc, then the right-hand side remains bounded, implying that
lim lim (WW T-X
F EFFICIENT EVALUATION OF THE RIGHT HAND SIDE OF THE LOW-RANK DYNAMICS
Algorithm creates trajectory the low-rank parameter space, that robustly follows the full-rank solution of the gradient flow of the neural network training. In particular, Theorem 6 yields time-continuous representation Algorithm [
Theorem The evolution equations eq. are explicit Euler discretizations of a dynamical system which is equivalent to
$ V=C(UOS(IVV))
S(t 0) So
K ))C(K()))
K(t = 0) = UpSS0.
(27)
= -VCCCO((()
L(t= = S Vo:
where C is the stochastic loss given random data samples.
Proof. Consider the continuous time dynamics of K, where omit explicit time dependence U, U.S.V V and K for the sake of brevity, i.c..
K (US)
= ÚS + UŠ
- WW
(28)
= =(UUT - -I
Further, using the chain rule, we observe
VoL(USVT) CWCLUSVTVV
LI
Thus, - VULLUSTT)SST = LUS = K. Lastly we have by the chain rule K =
LV -VxC(USVV) ) which yields
= -VVUUU TAACUUSVTS-TT=-- -Vx==VV
Analogously obtain for i
-= (=-Vv((UVV))+= -V-=LLV)('''),
which concludes the proof.
Note that using an explicit Euler time discretization for eq. (27) directly yields eq. (6) the update step of L...
23

---

Published as a conference paper at ICLR 2025
G ROBUST ERROR BOUND OF THE LOW-RANK SYSTEM
We show the robust error bound for A applied to single layer, and then extend the result network containing multiple layers treated with Algorithm
Theorem 7. (Restatement of T'hoore 3 For an integer k. ler t = kd. Let (t)) be the solution of eq. 1 and let W be the factorized low-rank solution after steps with Algorithm / Assume that for any Z in a neighborhood of (()), we have IK P(Z))VL(Z)II < &, i.e., the gradient flow is close to My. Then,
[I - Will S 01 + C2) + c28+
(29)
Moreover, let W (t) denote the solution of the Riemannian flow of equation 4 Then,
| (F)(t)= Wi S CEE + c2 + c30/A
(30)
where the constants C22 c2. C32 C2 depend only L and B.
Proof. Let us first investigate the local error. That is, we choose the solution at given time to of the full-rank gradient flow of eq. 12). denoted as ((t)), as a given iteration of GeoLoRA, which denote as W Hence, ((t)) W Wo E M, We are then interested in bounding the distance between the full-rank flow at 4 = to + to the Geol LoRA solution after a single iteration
with learning rate A. To simplify notation, we denote = [U, E Tx222 V = [V V E R and denote the projections onto these augmented basis vectors Pa UÜ and Py VQT. Moreover, e denotes a generic constant that only depends on I. and B. It is important to note that this constant does not depend SF' since never perform Taylor expansions of the individual low-rank factors.
Let us denote the augmented solution of GeoLoRA before truncation as W WT is the truncated solution after iteration 1. Then, the local error is bounded by
[[W(tt) - Will = [[((t)) (t)) +
(P)) - WT W- WT
In the following, we bound the three norms individually in three corresponding steps. Step 1 Bounding ([t( (t) ||: Using the triangle inequality, obtain
Ph < W (/ Pp) = =1| - Pp) || -| -P))ll
using orthonormality of Ū.
First term: Consider the first term with the dynamics wi) - f(W) in mind,
[[| - P4))(
A WV
(I - PW + Xf(Wo))] +
[[ PW XP(W)))(W)) + X(L PWL))))))) cx2
S
A Pτ) +| [ +
|| (T - |
+
Al((1 [WW))
2 || - (WW.)) + + cX
using Taylor expansion in (I), Wo E M, in (II, Assumption in (III, and eq.
18 in (IV).
By construction of the basis augmentation, obtain
(I - P2) (I -I- -O) 0.
(31)
24

---

Published as a conference paper at ICLR 2025
From eq. 31 we can directly conclude that - ((U-PP)
0. Thus obtain
(T Po)s(Wo)(TTT]
= | (P))))
|| + > [I PP))(W
S Ae,
where we used for the second term that V is in the orthogonal complement of Vo. Hence,
- P/)-P))(())) s ex² + xe
Second term: The same derivation for the co-range using the evolution for T(l) yields
[[ Po)l S cx² te.
Step 2 Bounding P. - - We have by the assembly of the augmented S matrix in
eq. 8
W = ÜSV = Snew Vo - UU Kne Vo Ug T VVT,
from which we obtain the error bound between the projected Wt,)) and W":
PoW(()) UnSnewyy VoT UÚ k Up new, VVT PoW(h)) Wr
_ ( - Swew|| || + | (ti) Knww
keew
+ Uo Luw. 1/ U W(
where we orthonormality of Ú.V in (I). All terms the right-ham side can be bounded by 2 and e terms:
First term: We have
|10 W(1)Vo - Snew
Ug (I(W(1)) ((W)())-(W)))V)))))
to
IAB IAB)
(I)(W(t)) f(Wo)l dt
(
(I)(W(tt)) (Ww)ll di + ex²
(V)
where we use in (I) Snew = So UD Vw <(W:: V, = -U (W.) We use the orthonormality of Up Vo in (II), perform Taylor expansion of the full-rank flow (III), and finally use that ((ta) = s ( (ta) in (IV).
Second and third term: We have
| W(1)V - ŪT new
(TT()((()))))))))))))))))(()))) - - di
lg
(1)
lIf(W((t)) f(Wo)l dt +x²
where we use the K-step of GeoLoRA in (I) and Taylor expansion of the full-rank flow in (II).
Ug (WTWWVVV1 - Leew, T V can be bounded analogously.
25

---

Published as a conference paper at ICLR 2025
Fourth term: Lastly, obtain for the fourth term,
ü (TT(t) = üT w [" ((w(t)
UJ S(W(()) de
60
a Ü ((W( di cx² S As + cd2.
with ((a) = 0 by the construction of the augmented matrix S used in (I), and in (II,, we use Assumption
Step 3 Bounding of W" [W---p
WT||: By construction of the truncation step we directly obtain
W" WT| 2
In conclusion, we obtain for single iteration of Algorithm
!
[[W(t)) - W Σ [W(t)) (t)
(OTTTWL w(G) - W' W wT|
<ā -222 +
To conclude, the global error in the training epochs follows by using the Lipschitz continuity of the gradient flow: We move from the local error in time to the global error in time by standard ODEs argument of Lady Windermere's fan (Wanner& & Hairer 1996 $II.3); With = kλ and denoting the adapter computed with GeoLoRA at iteration
W." then have
[t W7l S C1 c2A
This bounds the distance between the full-rank flow GeoLoRA. The result trivially extends to the Riemannian flow of equation 4 Denote by WRF(t) the solution of the Riemannian flow
WRF (t) = VW C C(WRF (t)). Then, since ||W(()) WRF (t)ll it directly follows that
[[)F(() Will c4c +2. cz0/A.
H VISUALIZATION OF THE STIFFNESS OF THE BASIC LOW-RANK SYSTEM
Consider Equation 5 in the case for n
= 20. We set the target matrix
0 15
2 0
W
0
0
E R 20x20
which has rank r = 2 and singular values 0i = 15 and 02 = 2 We compare SVD-Iora, AdaLora, and GeoLoRA, both with an ansatz form Wms USV initialized as
10
0
0
0
U.V
!
R 20x4
S
le 2
0
4x4
0
1e
4
0
1e
6
where U.V orthonormal, and the S matrix has fast decaying singular spectrum.
AdaLora and GeoLoRA use a relative singular value truncation threshold 7 = 1.15 for rank truncation. We found that learning rate 0.178 is the maximal learning rate before Adal orra and SVD-Loo
26

---

Published as a conference paper at ICLR 2025
become unstable, whereas GeoLoRA allows for arbitrary large learning rates, and set X = 0.1. We present the trajectories of the S-matrix elements the corresponding methods in Figure for up to 1000 iterations until single precision accuracy is reached. As seen in Figure AdaLora and SVD--Lor exhibit heavy oscillations in the trajectories of the S-mmtrix elements leading to slow convergence. Adalora although using orthonormalization by regularization of the low-rank basis is not able to stabilize the training, leading to overestimation of the rank, which is = 5 at final time and final loss value of 1.6. Similarly SVD-Lora exhibits even stronger oscillations and is not able to find the right matrix approximation. In contrast, GeoLoRA identifies the correct rank r = 2 and the corresponding correct singular values 15 and 2.
(a) SVD-Lora, S matrix
(b) AdaLora, S matrix
(c) GeoL,RA, S matrix
Figure 6: Time-trace of the matrix elements of SVD-Lora (a) AdaLora (b) and the proposed method GeoLoRA (c) to solve Equation ( SVD-Lora was trained with learning rate À 0.00178, which is the largest learning rate for which the optimization remained stable, GeoLoRA allows larger learning rates, set to À 0.1. GeoLoRA converges fast to single precision accuracy, whereas SVD-LORA still has a loss value of 1.7 after 1000 iterations, due to the heavy oscillations in it i's S matrix trajectory (a). Adalora reduces the oscillations, however incorrectly identifies the rank and fails to converge due to the influence the additional singular values.
I STRUCTURE PRESERVATION
The goal of this section is to clarify the formulation of Algorithm [ in relation with the previous related literature. In particular, we want to show that the proposed algorithm can be seen as an efficient structure preservation formulation of projected gradient flow (Koch & Lubich (200a)) for training neural networks. In this section, to achieve full generality, we will denote with Y, either the pretrained matrices the low-rank adapters.
As already mentioned the previous section, gradient descent can be seen an forward Fuler dis- cretization of the gradient system
y, C(Y,, Yr.,, = =
The neural network fyi....Y YY naturally induces weighted graph, where nodes are neurons and weights are connections among them and for which the adjacency matrix can be written as:
0
Y1
0
0
Y2
y
:
:
E X
0
0
Vr.
where = -1 d, is the total number of neurons of the neural network. =
The matrix )' now represents the adjacency matrix of the computational graph, and the block structure given by the layers. A model with a general full adjacency matrix V, would in general have non-zero connections between two generic layers i,j, descriebed by the block Y.g. Let's consider the model
fx(2) = (x), 2x)x) = x, ++1
< Yietiz*
Notice that for y upper diagonal, the previous model would be a feedforward network. Given this observation, under the assumption that (y(x)) is well defined eventual fixed point, can
27

---

Published as a conference paper at ICLR 2025
the loss function function of the full adjacency matrix, with abuse of notation will call it again C())). Usual training would superimpose the sparse graph with the same structure of y, but let's consider for moment the gradient flow
" = -((()
Clearly, the flow does not preserve the sparsity of the adjacency matrix V. even for sparse initial conditions. Using the theory developed in (Koch & Lubich, 2007a) directly on this gradient flow would lead to neural networks with non-feedlorward topology. Moreover, given the size of [ for modern neural networks, it can be expensive to compute QR or SVD decomposition of the basis matrices. Luckily, the sparsity structure is simple linear constraint represented by the mask matrix
0
11
0
0
M
:
E R
0
0
11
0
0
and the linear operator IT(A) M A.
A system preserving the sparsity pattern is given naturally by the ODE
y -=-IVVCO)
However, it is not obvious that by projecting this last system on the manifold of rank-r matrices M, the block structure is preserved. Fortunately, it is indeed the case, described by the following lemma: Proposition 2. (Block structure preservation of the flow) Consider the gradient flow with sparse initial condition
y -P(Y))VZ(() 2(() Vo E raaggeII)
Then (t) (range(II) for all t 2 0.
Proof. It is necessary and sufficient to prove that (()(t) E range(II) for all t 2 ic. that IIP(()))LVVZ(((())
((())) The key to prove this is to observe that for 7 E raIg we have Z E raIge(II). In fact, given Z e range( (T), we can write a SVD
of Z
0
U1
0
...
0
...
0
I
U2
5
0
V
0
0
Z
:.
:
0
0
0
0
UL
0
SL-1
0
0
0
0
V
L-1
0
I
0
0
0
St
0
V2
and we have UUT VVT E range(I). Thus, by direct calculation we can show that
UUT 7, ZVVT, U ZVVT range([I) and thus Z UU Z -VVT-UU - ZVVT (11) Since (I))())(t) E rlnge( by construction for all t > 0. we get the desider result. Thanks this last proposition, following again the line of work in (Koch & Lubich 2007a), it pos- sible to restrict the parameterization in the tangent space to In this way, we get the following coherence theorem: Proposition 3. Consider the gradient flow with sparse initial condition
-U
one as in Proposition
U ύ (()))VC(()) V(0) Vo range(II)
Consider now the parametrization y = USVT with
0
U1
0
0
0
U2
0
S
V
:
:
:.
:
:
:
0
0
0
0
U1
0
0
0
St-1
V1
0
0
0
0
0
St
28

---

Published as a conference paper at ICLR 2025
where U, U, V, V; I. Then, by imposing the Gauge conditions U, U, =0, V, 0, the projected flow y P()))TV be rewritten in block fashion follows:
(1(1) -U, (UV,
Ü,(t) = (1 - Pv.e) Vy, (VV,UU00))(((VV(()))))
Vi(t) = - Pr(t)
1=
Proof. Thanks to the previous proposition, we know that the variation P())ITVZ(()) range(IT) for all 2 0. Then, we have (tt E range (I1) for all times, and thus we can decompose it using a block SVD as described in the statement of the proposition. Moreover, by the self-adjointness of 11, Galerkin condition can be written as:
()' + VLO)). (ÜSVT USST +USV + VLO)). =0, Aq TyM, range
Since E TyMM, n (I)) we can represent it as = SUSV + USSVT + USSV with 5U, 8V, 888 with the same block structure of U.S and V. By writing the last conditions basis of TyM, O range(TI), we get
(USVT +USVT +USV + V2()),USVV =
(ÜSVT +USVT +USV
(ÜSV +USV +US VE()), =0
Thanks to the Gauge conditions ÚTU = VTV = 0 and to the properties of the Frobenius inner product, the last system becomes
(ÜSST + VLOUSS 80) =0
(UTUŠV +U VC))V, 85) = US+ - V'V'),,,, SS + VL =0
and from this equations we get the known
Ú - -I- UUT
Š = -U VUVW
V = - VV V
By writing this equations block-by-block, we get the desidered result.
This last proposition clarifies how connect the single matrix setting with the multi-matrix setting, showing that the presentation of Algorithm [ is in fact coherent with the single matrix setting. Moreover, investigation of this setting leads naturally to the global truncation strategy.
1.1 TRUNCATION STRATEGY
The global truncation strategy proposed in the main manuscript is in fact coherent with the single matrix formulation presented in the previous section. fact, one can assemble the rank augmented matrix as:
S
0
0
S₂
0
0
S(t = 1)
"t.
0
SL-1
0
32
and then truncate the smallest singular values up required precision. This be efficiently done by computing an SVD on each diagonal block, giving effectively an SVD of the global matrix. In particular, if S; P.E.Q. we get that
$ blockdiag(P..... ... Pr. Pocckiccdi E1,,... Er Moc.MMokoggQ......... Q1)
29

---

Published as a conference paper at ICLR 2025
Since the matrix blockdiag(E1.... Σz) is effectively diagonal, by assuming the diagonal is in- creasingly ordered, it is natural to globally truncate the ranks according to the minimal such that
Σ 2i=k+1 of Σ of
1-T
Which corresponds in throwing away the smallest singular values of until we reach the desired relative error. By rewriting this criterion on the singular values of each matrix S,, we get exactly the global criterion proposed in Section
OPTIMALITY ON THE LOW-RANK MANIFOLD
We remark below that if W, is a local minimum, then P(W.)VC(WW 0. In particular, since P(W)VL(() is the Riemannian gradient with respect to the ambient metric, then the following holds by definition of the gradient:
Oaw L(W) (P())VC(W),
where Osw L(W) is the directional derivative of C along the direction 8W. Thus, (WW))C((.)) if and only if 03w L(W) for all óW E T and this happens and only if VL(W) (TwM) So geometrically, if W. is local minimum, then all available directions, there are none that decrease the loss. For simultaneous descent, the points satisfy P(W.)V
0 means that among
condition doesn't 't hold, in fact, the algorithm's stationary which given the non-orthogynality does not, general, imply ((W.) = 0, so there could be descent directions unexploited by the method.
30


Reviews:
[{"review_id": "yCP4XF0woS", "reviewer": "Reviewer_kJ1Q", "summary": "This paper come up with a more efficient and robust variant of low-rank adaptation, GeoLoRA. The intuition is straightforward — existing implementation of low-rank adaptation using gradient descent, which update U, S, V together and the update of S cause the algorithm slow to converge.  \n\nThis method proposed GeoLoRA, which integrates rank adaptivity, low-rank optimality, and memory and computational efficiency. This method controls the update of S, supported by theoretical convergence guarantee and error analysis, and various experiment results.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- [Major] Easy to follow, well-written\n- [Major] The intuition is straightforward, and the method is simple yet effective\n- [Major] The proposed method is supported by many evidence, including theoretical convergence guarantee and error analysis, and various experiment results.\n- [Major] The experiments are diverse, including both language tasks and vision tasks.", "weaknesses": "- [Major] While the figure between L222 and L232 exhibit great performance of GeoLoRA, the improvement caused by GeoLoRA seems to be marginal, especially in Table 2.\n- [Major] Some experiment settings are not well-justified. For instance, in Table 2, the authors report the experiment results of GeoLoRA individually, while for others, it seems not. It looks like an unfair comparison, and the reason why GeoLoRA are reported individually should be justified. For Table 3, the results of LoRA are missing; for Table 4, the results of AdaLoRA are missing. More discussion are needed to justify the experiment results.\n- [Medium] More baselines are needed, since there exists many LoRA variants.\n\nPersonally speaking, I like this paper. However, given the current limitations of the experiments, I can only give 5. If authors are able to address my concerns listed above, I am willing to increase my score to accept this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qT87i58fUg", "reviewer": "Reviewer_MVJ1", "summary": "The popular finetuning method LoRA is learned as an optimization problem with some variant of gradient descent. However, these may not arrive at the optimal low-rank solution. This issue has been looked into before, but that solution provides a stiff gradient flow. This work first reparameterizes the gradient flow to be non-stiff. Further, GeoLoRA allows for the rank of the LoRA adapters to vary, eliminating the need to specify that hyperparameter. This parameterization only requires a single backward pass rather than multiple which are needed by other methods. Based on a clear theoretical backing, theoretical results follow, with loss descent being guaranteed.", "questions": "- Determining if the rank should be changed is dependent on calculating orthonormalizations of n-by-2r0-sized matrices and SVDs of r-by-r matrices. With r significantly less than n, is it possible to use approximate methods such as power iteration to accelerate GeoLoRA further?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This work provides a novel way of solving the stiffness/convergence issues in existing LoRA solvers, finding the LoRA factors at around the same rate as full fine-tuning. \n- GeoLoRA, while being more efficient than standard LoRA training, also includes adaptive rank finding, which is the most common alternative AdaLoRA has been shown to do but at a significantly greater cost than GeoLoRA. Finding LoRA-like fine-tuning factors can be done without specifying the rank a prior at nearly the same if not fewer training iterations.\n- Experiments showing GeoLoRA converging to around the same number of trainable parameters, with both starting with lesser and greater parameters, depict how this method does work towards decreasing rank if the extra dimensions are unneeded.\n- Timing experiments showing a direct efficiency improvement over AdaLoRA in Table 2 strongly defend that this method is an improvement over its alternatives.", "weaknesses": "- For clarity, the motivation (Section 3) is currently a little lengthy. When reading, there was some expectation that one of these major derivations was part of the GeoLoRA algorithm, and it only became clear that the algorithm's description happened a few pages later. Including either a more complete description of Section 3 at its beginning, another subsection title, or moving some of Section 3 to the appendix/another section would help with readability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NcAHLAxvvX", "reviewer": "Reviewer_XLNG", "summary": "This paper proposes a new parameter efficient fine-tuning (PEFT) technique based on low-rank adaption with adaptive rank allocation. Existing PEFT methods with adaptive rank allocation either lack optimality guarantees or require multiple backward passes per step---the proposed method, GeoLoRA, only requires a single gradient tape per step and comes with optimality guarantees. GeoLoRA does this by using the low-rank geometry of the adapters and work on matrix differential equations. Empirically, GeoLoRA is more efficient than existing adaptive PEFT techniques and yields higher performance on a variety of benchmarks.", "questions": "I am confused about the results in Table 2. Why do adaptive methods like AdaLoRA only have one parameter count for all datasets? Is this the budget, or is the same configuration used for all datasets?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well-written and the technique is overall well-motivated. \n- The theoretical support for the proposed method is solid -- the majority of the theoretical results are built on the single-layer case, but Proposition 1 extends this to the general multilayer case. If I understand this result correctly, this implies that GeoLoRA can obtain the optimal rank configuration for arbitrary architectures -- this seems like a particularly strong result to me. \n- GeoLoRA appears to robustly outperform or match existing PEFT methods, and it also seems faster in terms of wall-clock time compared to existing adaptive PEFT techniques. \n- GeoLoRA is robust to changes in the hyperparameters, which is already a substantial improvement over existing techniques.", "weaknesses": "- This paper could benefit from more discussion of the details of existing methods, and why the best adaptive methods require multiple gradient tapes and what types of guarantees they have. This information is currently summarized in a table. \n- In Table 2, it appears that the same rank allocation configuration is used for every dataset, for each of the baselines, whereas the parameter count of the proposed method varies for different datasets. Can the authors comment on this? \n- In Section 4.1, the authors claim that GeoLoRA can be used for low-rank pre-training, but unless I've missed something, the authors do not present any empirical validation of this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Is2F7V8IK0", "reviewer": "Reviewer_Ht32", "summary": "This paper proposes a new LoRA variant, GeoLoRA, which can adaptively allocate rank budget and reduce the computational overhead. They analyze the limitation of low-rank optimization  and imporve LoRA according to the Riemannian gradient flow analysis. They also provide theoretical results about convergence and error bounds. The experiments show that GeoLoRA can improve accuracy and training speed compared to AdaLoRA.", "questions": "1. How to choose a proper $\\tau$ and initial rank for GeoLoRA? In Table 7, it seems that you chose different $\\tau$ and initial rank for different dataset. Does this suggest that GeoLoRA is sensitive to hyperparameters on visual experiments?\n2. The authors provide a toy example to show the  behaviours of different low-rank adaptation strategies. How does they behave in more practical settings? (for example, the settings of Table 2)", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The analysis of Section 3 clearly point out the limitation of current low-rank optimization, i.e., the optimality condition for low-rank optimization is that the projection of the gradient to 0. Further analysis shows that parallel optimization may not be able to achieve such optimality condition, indicating the urgency of more advanced methods.\n2. The paper describes their methodology very clearly. The readers can easily understand how GeoLoRA is implemented.", "weaknesses": "1. GeoLoRA needs to be validated on larger datasets and models, such as MetaMathQA task and  LLAMA. The authors claim GeoLoRA can converge to an optimal low-rank solution, so they should evaluate GeoLoRA under the settings where LoRA's performance is worse than full finetuning to validate their claim. \n2. The improvements seems to be minor, in some subsets of GLUE, GeoLoRA performs even worse than vanilla LoRA. Moreover, the authors should include more baseline methods (DoRA, LoRA+, PiSSA) in their experiments to show the necessity of adaptive rank allocation (TABLE 2,3,4). \n3. The paper lacks an intuitive explanation of their method. Although there are some formal connections between  Section 3 and Section 4, it's not clear to me why the authors design GeoLoRA. Thus, the theoretical contribution of GeoLoRA is not clear. \n\nI would consider increasing the score if the authors can explain the methods more clearly and provide more solid experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)