StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
Authors: Zhizhong Li
Sina Sajadmanesh
Jingtao Li
Lingjuan Lyu
Venue: NeurIPS 2025 spotlight ()
Keywords: LoRA
Stiefel Manifold
Geometric Optimization
Primary area: deep_learning
TL;DR: Use Stiefel manifold to constraint the three-factor low-rank matrix in LoRA and improve the performance.

Abstract: Low-rank adaptation (LoRA) has been widely adopted as a parameter-efficient technique for fine-tuning large-scale pre-trained models. However, it still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses a three-factor decomposition $USV^\top$. Analogous to the structure of singular value decomposition (SVD), it separates the adapter's input and output subspaces, $V$ and $U$, from the scaling factor $S$. Our method constrains $U$ and $V$ to lie on the Stiefel manifold, ensuring their orthonormality throughout the training. To optimize on the Stiefel manifold, we employ a flexible and modular geometric optimization design that converts any Euclidean optimizer to a Riemannian one. It enables efficient subspace learning while remaining compatible with existing fine-tuning pipelines. Empirical results across a wide range of downstream tasks, including commonsense reasoning, math and code generation, image classification, and image generation, demonstrate the superior performance of our approach against the recent state-of-the-art variants of LoRA. Code is available at https://github.com/SonyResearch/stella.

StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold
Zhizhong Li, Sina Sajadmanesh, Jingtao Li, Lingjuan Lyu"
Sony Al
Zurich, Switzerland
{zhizhong. li sina. jinaaaaanne,,innnee, 1i, www.gguuann... Satiann.vI@aaaa wevvvvon..ccommm com
Abstract
Low-rank adaptation (LoRA) has been widely adopted as parameter-efficient technique for fine-tuning large-scale pre-trained models. However, still lags behind full fine-tuning in performance, partly due to its insufficient exploitation of the geometric structure underlying low-rank manifolds. In this paper, we propose a geometry-aware extension of LoRA that uses three-factor decomposition USV T. Analogous to the structure of singular value decomposition (SVD). it separates the adapter's input and output subspaces, V and U, from the scaling factor S. Our method constrains U and to lie on the Sticfel manifold, ensuring their orthonormality throughout the training. To optimize employ flexible and modular geometric optimization design that converts any Euclidean optimizer Riemannian while remaining compatible with existing fine-tuning pipelines. Empirical results across wide range of downstream tasks, including commonsense reasoning, math and code generation, image classification, and image generation, demonstrate the superior performance of approach against the recent state-of-the-art variants of com hat.a/...oooaaaaaaaaaarcs//ttll
the Stiefel manifold, we
enables efficient subspace learning
LoRA. Code is available at hhtps::://. /topss://ee
Introduction
The rise of large foundation models 2 21 has transformed machine learning, driving breakthroughs
across a diverse range of applications 55 65 41 These models, often with billions of parameters,
show exceptional performance fields such as natural language understanding [22], computer vi- sion [76], and multi-modal learning [67]. Yet, their substantial scale results in massive computational and storage costs, limiting broader adoption via task-specific fine-tuning
20)
To address this challenge, parameter-efficient fine-tuning (PEFT) methods, such as prefix tuning 35) prompt tuning [50]. and adapter tuning (28 22), have gained considerable attention. Among these, Low-Rank Adaptation (LoRA) [29] is widely adopted due to its ability to efficiently adapt pre- trained models to downstream tasks without altering the original network architecture incurring extra inference costs. LoRA operates by learning low-rank adaptations to a selected set of linear layers while keeping the original weights frozen. Given pre-trained weight matrix W mm LoRA computes the adapted weight as W + BAT, where E Rmxr and A E RRar Choosing small & min(m (m, n) significantly reduces the number of trainable parameters. Nevertheless, performance gap often exists between LoRA and full fine-tuning due to the limited capacity of the low-rank matrices A and to capture complex updates required for optimal task performance 23
common strategy to improve LoRA's performance is to guide its initialization by leveraging structural insights from the pre-trained model [73]. Particularly, singular value decomposition
"Corresponing author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(SVD) has been used for uncovering informative subspaces in weight matrices, activations, or gradients 4g 64 46]. SVD factorizes a given matrix M into M UEV where the columns of U and V form orthonormal bases for the output and input spaces of M, respectively, ordered by the importance dictated by the singular values Σ. This property makes SVD well-suited for identifying low-dimensional subspaces that preserve meaningful information from the pre-trained model.
While SVD-based initialization methods for LoRA have shown promise, their influence is limited to the start of training, offering minimal guidance for the subsequent optimization process. This naturally leads the question: whether explicitly optimizing the subspace throughout training can yield further performance improvements? Moreover, the existence of various heuristics for subspace selection such as focusing on leading ys. trailing components [43 63), or considering weights vs. manual subspace selection is suboptimal. This motivates a more principled approach to learn subspaces from data throughout training.
gradients 43 64] suggests that intuition-dre
To bridge this gap, we propose a novel approach that directly optimizes the input-output subspaces of LoRA during training. Our key insight is to mirror the structure of the truncated SVD by representing the low-rank adaptation for weight matrix W as a three-factor formulation,
W + USV
where U Rm and V Rexr define the orthonormal bases for the output and input subspaces, respectively, and S E Rrxr captures the transformation between them. To ensure that the subspace parameters U and V remain orthonormal during training. we constrain them lie on the Stiefel manifold-the set of matrices with orthonormal columns. This leads to our method, Stiefel Low-rank Adaptation (StelLA), which performs LoRA using a subspace-awe aware formulation. By leveraging the geometric optimization on the Stiefel manifold. StelLA maintains the geometric structure of the subspaces in training. allowing for principled and effective learning of low-rank adaptation.
Recently, methods such as DoRA 40 found that decomposing the weight into magnitude and direction components
improve LoRA's performance. This aligns with our approach, as the Stiefel manifold constraint on U and V captures the direction, and models the magnitude. Besides, maintaining the orthogonality of low-rank matrices U and V during training is also beneficial for certain downstream analyses, such as adversarial robustness 55 57].
We benchmark StelL across a diverse set of domains, including natural language understanding, natural language generation, visual understanding, and visual generation. Compared with the state-of- the-art LoRA variants, StelLA consistently achieves superior performance across all evaluated tasks. Relative to the strongest baseline, StelLA achieves notable improvements: up to +1.3 accuracy points on commonsense reasoning, +2.33 math and code generation, +0.25 on image classification, and 7.11 point reduction in FID for text-to-image generation.
Our contributions are summarized as follows: (1) We propose a novel three-factor representation for LoRA incorporating Stiefel manifold constraints, enabling the optimization of LoRA's input and output subspaces directly during training. (2) We present a flexible geometric optimization algorithm for the Stiefel manifold, allowing seamless integration with existing Fuclidean optimizers. (3) We conduct ablation studies on the design choices and evaluate the impact of different geometric struc- StelLA across a variety of tasks and models, encompassing natural language understanding and generation, image classification, and text-to-image generation.
tures. (4) We verify the effectiveness of
2 Related Work
LoRA Initialization Methods and the Use of Several recent works explore improved LoRA
initialization using matrix decomposition techniques like SVD better align the adaptation subspace with pretrained weights. PiSSA [43] and LaMDA 4 initialize adapters using the leading singular vectors from SVD of the pretrained weights, while MiLoRA (33 uses the trailing r singular vectors.
EVA [6)] applies SVD to activations and adapts rank based on the spectrum, while LoRA-GA [64] uses SVD on gradients to align with tesk-relevant directions. Beyond SVD, OLoRA [0] employs QR decomposition and selects the first columns from the orthonormal matrix. These approaches exploit meaningful subspaces to improve initialization, whereas ours directly learns the optimal subspace during training, allowing for more flexible and effective task-specific adaptation.

---

Geometric Constraints for LoRA. Zhang and Pilanci [72 use Riemannian geometry to precondi- tion the LoRA optimization, whereas we optimize the adapter's subspaces using the Stiefel manifold. OFT [5 learns orthonormal rotations of pretrained weights via Cayley parameterization. Spectral Adapter 73 fine-tunes within the top spectral subspace, with variant that rotates leading singular vectors using orthonormal matrices. In contrast, our approach enforces orthogonality directly on the input and output subspace projections via Stiefel manifold optimization, offering a more flexible and expressive adaptation mechanism. Concurrently, PoLAR [39] also sets U and V on Stiefel manifold, but it uses a landing algorithm with infeasibility penalty instead of retraction for optimization.
LoRA with Three Factors. TriLoRA 20 and MoSLoRA [66] use the three-factor formulation for LoRA, but they do not keep the orthogonality of the two projections during training. LoRA-XO A-XS (5) also uses the three-factor formulation, but they freeze the two projections and only train the middle r x r matrix for extreme efficiency. Both AdaLoRA (74] and GeoLoRA 58 adopt a three-factor formulation, with U and V constrained to be orthogonal matrices. AdaLoRA achieves orthogonality via regularization, while GeoLoRA uses gradient flow. However, their focus is to achieve rank adaptability by inspecting the singular values of S. StelLA can be readily combined with AdaLoRA's rank adaptation strategy by constraining S to be diagonal, which we leave to future work.
Other LoRA Variants. rsLoRA 30 introduces a scaling factor based on the square root of the rank for better stability. LoRA+ 24 accelerates convergence by applying separate learning rates to the two matrices. DoRA [40] and DeLoRA (7] decouple LoRA updates into magnitude and direction. ReLoRA (88 periodically merges adapter weights to improve expressiveness. QLoRA (I5) integrates quantization with LoRA to minimize memory usage, These efforts are orthogonal to our work. Besides LoRA, there are other adaptation methods like Prefix-Tuning (36), Prompt Tuning (33
Adapters (28) and others [17]. which we do not cover in this paper.
3 Preliminaries
Here we briefly review the related concepts in differential geometry and geometric optimization. For more detailed introduction, we refer the reader to Edelman et al. [8]. Absil et al. [1. Li al. (37. Roy
et al. [54), Becigneul and Ganea [6]. We also prepared an intuitive example in low dimensions in
Appendix A to help readers understand the geometric concepts. The Stiefel manifold, denoted Si(k, n). is the set of all n X k matrices with orthonormal columns, i.e., St(k, n) = (Y E Ra Y J. To optimize function f St(k, n) R. require tools from the Riemannian geometry. The tangent space at a point Y E St(k, n), denoted Ty Stt(k, r2). consists of matrices A E R satisfying Y AAAAAY space, and the canonical metric in Stiefel manifold is gr (Ai, A2) tr tr(AT (Ln, 2 (Y))) The Riemannian gradient grady w.r.t. function f can be computed from the Euclidean gradient Vy by
0. A Riemannian metric defines an inner product on the tangent
grady = Vy - YVJY.
(2)
If the Riemannian gradient is modified, e.g., by adding momentum, it may no longer lie in the tangent space of the manifold. To perform geometric optimization, we need to project it back to the tangent space using the projection Ty
Ty Bank + TySt(k, n).
(A) = A Ysymm(Y A).
(3)
where symm(A) (A (A+AA) AT) symmetrizes matrix. After taking a step along a tangent vector A, the resulting point y² + A typically leaves the manifold. Hence, a retraction is used to map it back. In this work, we use the polar retraction, defined as
(A = uY + A),
where uf()) returns the orthogonal matrix in the polar decomposition.
4 Subspace Learning in LoRA Using Stiefel Manifold
We now present Stiefel Low-rank Adaptation (StelLA), which learns the input and output subspaces of the adapter directly during fine-tuning. Given a pre-trained weight matrix W Rmmx, the goal is to fit it to a downstream task using three-factor low-rank adapter.
W W + a USV
(5)
r
3

---

Algorithm/ StelLA: Stiefel Low-Rank Adaptation
Require:
Pre-trained weight W R scale factor a, number of iterations T.
, loss function C. Euclidean optimizer's step function 'step', rank
1: Randomly initialize Uo E St(r, m) and Vo E &St(r; nt),
set Sot t- 4.
2: for 4- T do
3: Compute loss: L. + L(W + U.S.V,
4: Compute Euclidean gradients: VE,, Ve,, V..
via automatic differentiation
Convert Euclidean gradients Riemannian:
Equation
grady, < Vu, U.V., Us, grady, <- Vv. VVVV Va.
6: Take tentative steps using the given optimizer's step function:
e.g using Adam
Ū141 t step(U,, grady;), V4+1 + step(Vi, grady,): S411 t step(St, Vs,).
7: Project the perturbed gradients U1+1 - Uc, Ve41 - V/2 back to the tangent space: Au. nu, (U1+1 Ut), An TV, (Ve+1 Vi). 8: Update and retract back the manifold: U++1 + pv. (Avz): Vi+1 + pv. (Av.).
Equation
Equation
9: end
10: return Adapted weight: W - W BUrSrV
where U E St(r, m) and Ve St n) provide orthonormal basis for the output and input subspaces, respectively. R xx learns mapping, with the rank r < min n). can also be further constrained be diagonal reduce the number of parameters.
R is scale hyperparameter.
Let L denote the task-specific loss. We optimize the following objective (for notational brevity, we express the objective for one StelLA layer):
min UESt(r.m), SeR VeSt(r.)) 2 < (UVTT)
a
(6)
leading to constrained optimization over Stiefel manifolds for and V, and an unconstrained optimization for S.
The optimization is carried out using Algorithm [ The algorithm begins by initializing Up. Vo- and So (line T). At training iteration f, the loss is computed based on the adapted weight in the forward pass (line B), and Euclidean gradients with respect to U,. St, and V are obtained (line 4 in the backward pass using standard automatic differentiation in deep learning frameworks. Since U, and V/2 must remain on the Stiefel manifold, their gradients are converted to Riemannian gradients using Equation (2) (line These gradients, along with the gradient of St, are passed to an existing Euclidean optimizer's step function, e.g., using SGD, Adam, or RMSProp, to generate tentative updates (line 6). The perturbed gradient by the optimizer is reverse-engineeree by examining the difference Ü++1 U.. Since the perturbed gradient may not lie on the tangent space of the manifold, it projected back to the tangent spaces at the current points (line 77. Then, the update performed by using the polar retraction (line (). The algorithm ensures that U and V evolve in the Stiefel manifold throughout training. After T steps, the final adapted weight is returned (line 10).
Initialization. Both U and V are initialized with random column- we initialize S with the identity matrix. In prior work 45), when the adapter is initialized with non-zero values, the initialization is subtracted from the pre-trained weight matrix to simulate zero-initialized adapter. We empirically find this trick to be unnecessary our case. An in-depth ablation of initialization strategies is studied in Section 5.5
matrices [56), and
Implementation. Algorithm| D is designed be modular and flexible, enabling integration with any existing Euclidean optimizer. The 'step' function abstracts the optimizer's 's internal logic, such as momentum updates or adaptive learning rates, allowing StelLA to seamlessly integrate with wide range of optimizers, including SGD, Adam, and others, while cleanly separating geometric constraints from the choice of optimization algorithm. We implement StelLA in PyTorch [48] using optimizer hooks. Specifically, line 5 implemented as pre-hook to the optimizer step, while lines 7 are implemented as a post-hook. Our implementation is readily integrable with HuggingFace PEFT library [42]. enabling easy adoption by the community.

---

The polar retraction is computed via SVD, which is the most expensive operation in the algorithm. To improve efficiency, we stack all Us and Vs with identical shapes across different layers and apply a batched SVD. This batched strategy yields 15-20x speed up in our experiments, effectively eliminating the computational bottleneck when scaling to large models with many adapted layers. Please refer to Appendix D for detailed discussion.
Complexity. StelLA adds r(m + n) + r² parameters per adapted layer, which is r² parameters more than LoRA. This additional memory footprint is negligible since < min(m, n)). In our exper- iments, we show that the number of parameters in StelLA is comparable to LoRA and DoRA [40]. Nevertheless, in Section 5.5 we show that the superior performance of StelLA is not merely the result of this tiny increase in the number of trainable parameters but our geometric formulation by relaxing the orthonormality constraints. Regarding inference, similar to LoRA, we can merge the adapted weights into the original ones, resulting in a single weight matrix with no overhead.
Gradient Scaling. Previous work such as LoRA+ (24] showed that setting different learning rates for B and A in LoRA can improve convergence speed. The core idea is to ensure that both B and A are efficiently updated during training. Motivated by their insights, we balance the learning speed of U and V in StelLA via gradient scaling. For a random unit vector æ E R", it is easy to show that the variance of each element m Since the columns of and V are unit vectors, their individual entries are expected to have magnitudes on the order of 1/vm and 1/V7, respectively. Adam-style optimizers, however, normalize gradients so that their coordinate- wise variance ((1) 69]. This means that the learning speed- the ratio between the average magnitude of the effective gradient and the average magnitude of the parameter-for the entries of and are different when m + n. This imbalance is particularly pronounced in feed-forward layers of LLMs, where the hidden dimension is typically enlarged and then shrunk by a factor of 4 [9]. causing the taller matrix to learn two times faster. To compensate for this difference, before applying the projection operation (line 7 in Algorithm [ scale the gradients of U and V by V4/m and v vd/n, respectively, where d is hyperparameter. In our experiments, we set d equal to the hidden dimension of the input tokens.
Comparison to Existing Geometric Optimizers. Existing geometric optimization methods, such as Riemannian SGD 54 and Riemannian Adam (6), can handle optimization over generic manifolds. However, these methods are intrinsically coupled specific optimizers, as they rely on accessing and manipulating internal states such as momentum or adaptive learning rates. For instance, Riemannian Adam parallel transports the momentum vector across tangent spaces to maintain consistency in updates. This tight integration limits their generalizability to other optimization algorithms. In contrast, our approach decouples geometric constraints from the choice of base Euclidean optimizer by treating its update direction as perturbation to the Riemannian gradient. The projection operator (line 7 in Algorithm [ ensures that the perturbed gradient mapped back the tangent space of the current point. This allows us to use any Euclidean optimizer without modifying its internal logic.
5 Experiments
We conduct extensive experiments to evaluate the performance of StelLA on various domains: natural language understanding (commonsense reasoning), natural language generation (math and code generation), visual understanding (image classification), and visual generation (text-to-image), using diverse set of model architectures, including LLaMA2 60), LLaMA3 (21). ViT [6). and Stable Diffusion (53] We compare StelLA with diverse set of low-rank adaptation baselines, covering various methodological categories: LoRA [29] as the standard baseline; DoRA
40 as strong LoRA variant; PiSSA [4] and OLoRA [1O] for SVD- initialization; TriLoRA (2) and MoSLoRA [66] for three-factor decompositions: and SdaledAdamm (72] to represent geometry- optimization. Finally, we perform ablation studies to analyze the effect of different components in StelL..
5.1 Commonsense Reasoning
Models and Datasets. We evaluate the performance of StelLA on the commonsense reasoning benchmark, which assesses the reasoning capabilities of large language models across 8 sub-tasks. Following the setup of Liu al. [40), we train on the combined data from all sub-tasks and evaluate on the test set. We fine-tune two popular LLM checkpoints, LLaMA2-7B (60) and LLaMA3-8B 21
5

---

Table Accuracy on the commonsense reasoning benchmark. All results are averaged over runs.
Model
Method
Params (%) BoolQ
PIQA
SIQA
HellaS.
WinoG,
ARC-e
ARC-c
OBQA
Avg.
LoRA
0.826
72.02
83.46
79.87
90.44
82.69
84.83
71.19
81.53
80.76
DoRA
0.838
72.67
83.48
79.82
90.82
83.58
85.16
71.27
81.20
81.00
PISSA
0.826
71.16
83.89
79.19
91.00
82.87
85.09
69.48
83.93
80.83
OLoRA
0.826
71.11
82.70
78.64
89.41
81.48
83.58
68.17
80,20
79.41
LLaMA2-7B
TriL.RA
0.828
71.23
80.96
78.33
80.91
77.59
81.76
66.69
79.80
77.16
MoSLoRA
0.828
71.54
83.84
79.60
90.50
83.19
84.40
69.96
80.47
80.44
ScaledAdam
0.826
72,20
83.86
79.67
90.80
82.43
85.55
70.59
81.93
80.88
StelLA
0.828
73.62
84.87
80.64
91.44
84.50
86.43
72.84
84.33
82.33
LoRA
0.700
75.16
80.18
95.41
86.74 74
90.84
78.70
87.00
85.27
DoRA
0.710
75.38
88
79.94
95.35
86.29
90.54
79.69
86.07
85.16
PISSA
0.700
74.67
80,50
94.98
85.22
78.87
85.60
84.76
LLAMA3-8B
OLoRA
0.700
74.41
87.68
79.55
94.79
85.40
90.04
78.24
85.00
84,39
Tril.oRA
0.702
73.09
86.64
78.64
93.40
82.88
87.76
75.26
84.30
82.74
MoSLoRA
0.702
74.88
88.43
80.31
95.50
86.26
79.86
85.80
85.13
ScaledAdamW
0.700
75.24
88.57
80.21
95.81
85.11
80.55
86.60
85.40
StelLA
0.702
75.91
89.86
81.68
96.41
87.82
91.98
82.34
87.80
86.72
Implementation Details. We compare StelLA against all the aforementioned baselines. For all methods, we insert low-rank adapters into the Q, K, and projections the self-attention layers and the up and down projections of the feed-forward layers. For fair comparison, we fix the rank to 32, a 64, batch size 16, weight decay to O, dropout 0.05, and train for 3 epochs using AdamW. The learning rate is separately tuned for each method and follows a linear decay schedule.
Results. Table shows that StelLA yields consistent, model-agnostic gains on every commonsense sub-task. StelLA the average accuracy LLaMA2-7B from 81.0% (the best baseline) 82.3% and that of LLaMA3-8B from 85,4% to 86.7%, corresponding to absolute improvements of about 1.3 points on both models. Crucially, the benefit is uniform: StelLA attains the top score on all eight datasets for both model sizes, showing that its geometry-aware adapters generalize across binary, causal and multiple-choice commonsense formats.
5.2 Math and Code Generation
Models and Datasets. To assess the generative capabilities of StelL.,, we conduct experiments on two representative natural language generation (NLG) tasks: mathematical reasoning and code generation. For math-related tasks, the models are fine-tuned on MetaMathQA [70] and evaluated on
GSM8K [13] and MATH [26]. For code-related tasks, we train on CodeFeedback (75 and evaluate
on Evaa M and MBPP 3
Implementation Details. We benchmark StelLA against three strong baselines: LoRA 29 DoRA [40], and PiSSA 43 We follow the experimental protocol of Meng et al. 43 Each method applies low-rank adaptation all linear transformations in both self-attention and feed- forward modules. For consistency, we standardize the training configuration across methods: rank is set to 128, a to 128, LoRA dropout and weight decay are both zero, and models are trained with AdamW W optimizer for I epoch using a batch size of 128. The learning rate follows a cosine decay schedule and is tuned individually per method to ensure optimal performance.
Results. As summarized in Table _ StelLA delivers the strongest overall performance (39.30 on average) across the math and code generation benchmarks, surpassing all three established baselines by comfortable margin (up to +2.69 absolute points on average over DoRA). Importantly, the only method that ranks first or second every benchmark, confirming StelLA's versatility and effectiveness in challenging natural language generation scenarios.
5.3 Image Classification
We assess the performance on image classification tasks using the Vision Transformer [16] pretrained on ImageNet-2IK [14]. We fine-tune the Base and Large ViT on 8
Models and Datasets.

---

Table 2: Results on math and code generation. All results are averaged over 3 runs.
Math
Code
Model
Method
Params (%)
Avg.
GSM8K
MATH
HvmanEval
MBPP
LoRA
4.531
64.67
15.33
30.07
41.00
37.76
DoRA
4.550
64.87
15.32
27.00
39.27
36.61
LLaMA2-7B
PISSA
4.531
63.73
16.64
31.10
38.20
37.41
StelLA
4.581
65.43
16.55
33.73
41.47
39.30
Table 3: Comparison of image classification accuracy (%) on datasets averaged over runs.
Model Method Params (%)
DTD EuroSAT Flowers Food101 OxfordPets Sun397 CIFARIO CIFARI00 Avg.
LoRA
0.72
79.20
98.98
99.22
90.07
92.94
75.62
98.92
92.60
90.94
VIT Base
DoRA
0.75
78.56
98.91
99.19
90.15
93.40
75.67
98.96
92.78
90.95
PiSSA
0.72
77.29
98.93
98.88
89.99
93.21
75.41
98.92
92.28
90.61
StelLA
0.73
79.89
98.91
99.22
90.17
93,54
76.28
98.88
92.72
91.20
LoRA
0.53
80.11
99.11
99.32
91.30
94.41
77.08
99.23
94.06
91.83
VIT
DoRA
0.55
80.21
99.00
99.28
91.27
94.30
77.19
99.29
94.11
91.83
Large
PiSSA
0.53
80.21
99.11
99.20
91.10
94.33
77.15
99.11
93.99
91.78
StelT.A
0.54
81,54
99.17
99.19
91.33
94.14
77.51
99.16
93.92
92.00
datasets: Caltech101 19
CUB200 622. Cars 196
31.
Flowers 45), Food101 8 Oxford-
Pets 47
Sun397 68 CIFARIO and CIFARI00 32 and measure the validation top-1 accuracy.
Implementation Details. We include the following baselines for comparison: LoRA 229] DoRA [40], and PiSSA [43]. We adapt only the query and value matrices of the attention lay- ers in the ViT model. For all methods, fix the rank and scaling factor at 16, use batch size of 128, apply no weight decay, set dropout to 0.1, and train for 10 epochs with the Adam optimizer and linear learning-rate schedule. The learning rate itself is tuned independently for each model to ensure a fair comparison.
Results. Table S summarizes the results of our experiments. We observe that StelLA delivers the best average performance using both ViT-Base and ViT-Large models, outperforming all other methods across most datasets. Using ViT-Base. StelLA attains the highest accuracy on five of the eight datasets and posts the best overall mean of 91.20%, surpassing the strongest baseline, DoRA, by +0.25 points. Scaling up to ViT-Large, StelLA achieves the best accuracy on four datasets and mean of 92.00%, outperforming the best baseline, LoRA, by +0.17 17 points. These results demonstrate that Stell.A is a strong contender for image classification tasks.
5.4 Text-to-Image Generation
Models and Datasets. To explore the effectiveness of StelLA in generative vision tasks, we fine- tune text-to-image models five stylistically diverse datasets sourced from CivitA/ [12]: Barbie, Cyberpunk, Elementfire, Expedition, and Hornify. Each dataset includes captions generated using the BLIP model [34]. We conduct experiments two commonly used latent diffusion [53] based models-Stable Diffusion v1.5 and v2.0, which are equipped with a U-Net architecture composed of ResNet blocks 25
Implementation Details. We follow standard LoRA fine-tuning recipes in Diffuseess 61 by inject- ing LoRA parameters into the cross-attention layers of the U-Net. For benchmarking, we compare StelLA with LoRA, DoRA, and PiSSA and report CLIP score [52] and FID (27], two metrics that respectively measure semantic alignment and visual fidelity. For all methods, we the rank and scaling factor at 4, use batch size of 8, apply 0.01 weight decay and train for 100 epochs with the Adam W optimizer and cosine learning-rate schedule. The learning rate itself is tuned independently for each model to ensure fair comparison.
7

---

Table Tmxe- quantitative results on finetuning SD 1.5 and SD 2.0 to downstream tasks. In most cases, StelLA achieves the best FID and on-par CLIP scores.
Params (%)
BarbieCore
Cyberpunk
ElementFire
Expedition
Hornity
Model
Method
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP
FID
CLIP t
w/o finetune
208.11
30.84
145.37
27.49
253.66
27.78
180.18
27.99
212.90
27.98
LoRA
0.093
175,48
30.31
127.50
27.62
202.49
27.80
156.34
27.64
180.48
27.24
SD 1.5
DoRA
0.104
175.04
30.36
127.11
27.61
200.77
27.78
155.80
27.65
179.58
27.26
PiSSA
0.093
299.49
17.32
269.44
16.88
303.18
18.82
291.22
17.55
295.15
17.29
StelLA
0.093
170.25
30.15
124.46
27.73
194.41
27.83
146.12
27.56
167.53
27.23
w/o linetune
210.45
31.12
158.68
27.42
256.01
27.82
180.74
27.86
214.41
28.15
LoRA 0.096
171.68
30.72
140.77
27.85
194.53
27.91
159.66
27.53
188.11
27.20
SD 2.0
DoR A PISSA
0.107
176.16
30.87
140.71
27.94
197.12
27.92
159.54
27.54
185.40
27.19
0.096
315.64
16.86
272.60
15.81
284.23
17.96
267.87
16.71
294.59
15.71
StelLA
0.096
171.83
30.92
135.05
27.83
194.71
28.00
154.06
27.34
177.51
27.05
Train Examples
Prompt
wio finetune
LoRA
DoRA
PISSA
SteILA (Ours)
space, featuring spaceship who traveling, evokingg an adventurous fealing.
A futuristic mythology scene mountain, featuring
syber-dragon flying, evoking powerful feeling.
vintage scene
featuring blooming, avoking beautiful feeling.
cartoon portrait scene in desert, featuring camel who journeying, evoking determined feeling.
garden, featuring dragon who dancing. evoking a happy fooling.
Figure 1: Qualitative results of finetuning SD 1.5 on BarbieCore, Expedition and Hornify.
Results. Quantitative results in Table 4 show that StelLA consistently yields the lowest FID scores. often outperforming the baselines by notable margin- on datasets like Expedition and Horuify-whil maintaining competitive CLIP scores. In addition, qualitative results shown in Figure reveal that StelLA generates images with strong stylistic consistency (e.g., preserving the original background aesthetics in BarbieCore) and high perceptual quality, demonstrating its strength adapting generative models to downstream domains.
5.5 Ablation Studies
In this section, we analyze the design choices StelLA. We compare the performance of alternative
geometric structures, different initialization strategies, and study the effectiveness of the additional gradient scaling for the Adam optimizer. We also evaluate the effect of parallel transport introduced in Riemannian Adam [6] and the effect of alternative choice for the retraction operator. The

---

Table 5: Ablation study on design choices using the Commonsense with LLaMA3-8B. GS and PT denote gradient scaling and parallel transport. respectively. Results are averaged over 3 runs.
Geometry Initialization
GS
PT BoolQ PIQA
SIQA
HellaS.
WinoG. ARC-e ARC-c OBQA
Avg.
StelLA
Non-zero
/
75.9
89.9
81.7
96.4
87.8
92.0
82.3
87.8
86.7
Euclidean
74.0
88.0
80.4
94.9
85.1
89.5
78.1
85.1
84.4
Non-zero
N/A
Quotient
75.7
88.6
81.0
96.1
86.9
90.9
80.0
86.4
85.7
Zero
75.9
89.2
81.6
96.3
87.4
91.7
81.8
87.9
86.5
Pseudo-zero
72.8
87.1
80.8
94.8
86.0
89.3
78.2
84.8
84.2
StelLA
SVD-major
76.1
89.8
81.5
96.5
87.5
92.3
81.9
88.3
86.7
SVD-minor
75.9
89.8
81.9
96.4
87.2
92.4
82.0
87.5
86.6
StelLA
Non-zero
x
x
76.1
89.7
81,2
96.3
87.4
92.0
80.9
87.6
86.4
StelLA
Non-zero
76.2
81.4
96.4
87.4
81.7
87.1
86.5
Commonsense reasoning benchmark with the LLaMA3-8B model is used for all ablation studies, and the results summarized in Table
Geometric Structures. For each weight matrix W to be adapted, there are three learnable matrices U. S. and V in StelLA. Since U and V are constrained lie the Stiefel manifold, StelLA's geometry is product of two Stiefel manifolds and Euclidean manifold, namely, St(r, m) x Rrxr, x (r, n). To show the effectiveness of this geometry, we compare it with the following alternatives:
R R"x x R"*r, The simplest geometry is a product of Euclidean spaces, which does not impose any orthonormality constraints on the factors. used in previous works such as TriLoRA 20 and MoSLoRA (66).
Euclidean:
St(r, m) x Rrxr, x St m)/(O(r) x ((r)). The three-factor decomposition of low-rank matrix M USV not unique due to the equivalence relationship (U, S, V) ~ (UO,, ST VO2). VO,,,,0,,C O2 E (()), where (r)) is the orthogonal group. Factoring out this symmetry, we get quotient space where each low-rank matrix is uniquely represented. We adapt StelLA to use this geometry along with the Riemannian metric defined in [44) Details of the geometry and its optimization are discussed in Appendix B
Quotient:
Comparing the performance of these geometries in Table 5 we observe that the product space St(r, m) Rexr St(r, n) for StelLA consistently outpertorms the other two geometries across subtasks. This evidences that (1) The Euclidean three-factor geometry is not as effective as StelLA's A's geometry, implying that the orthonormality constraints U and V are beneficial for the low-rank adaptation task, and (2) the StelL.''s geometry more effective than the quotient geometry, which is likely due to the fact that the Riemannian metric 44 the quotient space was initially designed for the low-rank matrix completion problem rather than low-rank adaptation.
Initializations.
We refer the initialization of StelLA introduced in Section 4as non-zero initial- ization, and compare it with the following initialization strategies:
Zero. Initialize S to be zero, and and to be random column-orthononn orthonorral matrices.
Pseudo-zero. Same the non-zero initialization, except that the original weight matrix is modified by subtracting the adapter's initialization.
SVD-major. Initialize U and V to be the leading left and right singular vectors of the pretrained weight matrix, respectively. S is initialized as the identity matrix. This setting aligns with the philosophy PiSSA (33 and can be interpreted as StelLA + PiSSA.
SVD-minor. Same as SVD-major, except that the trailing r singular vectors are used. This setting aligns with the philosophy in MiLoRA [63] and can be interpreted as StelLA + MiLoRA.
From the results Table 5 we observe that (1) zero initialization is not as effective as the non- zero initialization. We hypothesize the small value of S creates small gradients for U and V at the beginning of training, leading to slow convergence. (2) Pseudo-zero initialization leads to the worst performance, presumably because it contaminates the pretrained weights with the adapter's

---

Table Comparison of the polar retraction with the exponential map on the commonsense reasoning benchmark, Results are averaged over 3 runs.
Geometry Retraction
BoolQ
PIQA SIQA
HellaS,
WinoG.
ARC-e
ARC-c
OBQA
Avg.
StelLA
Polar
75.91
89.86
81.68
96.41
87.82
91.98
82.34
87.80
86.72
StelLA Exponential Map
75.98
89.52
81.10
96.42
88.27
91.83
82.85
88.13
86.76
initialization. (3) SVD-majo and SVD-minor initializations show similar performance non-zero initialization. This showcases the robustness of our geometric optimization, as it can effectively learn the suitable subspaces regardless of the initialization.
Gradient Scaling (GS) and Parallel Transport (PT). As we have used the Adam W optimizer, we evaluate the effectiveness of the gradient scaling strategy introduced in Section 4 and the parallel transport introduced in Riemannian Adam [6] indicaesss that our gradient scaling slightly improves the average accuracy from 86.4% 86. 7%, showing its effectiveness in balancing the learning rates of U and V. Moreover, implementing the parallel transport results in the vanilla Riemannian Adam optimization. However, it does not lead to any performance gain compared to treatment of converting a Euclidean Adam into a Riemannian one in Algorithm |
Alternative Choices for Retraction. Other than the polar retraction in Equation 4 the exponential map offers an alternative way of retraction. The exponential map is a locally defined operation that maps tangent vector at the given direction for a unit time. Intuitively, it traces the shortest curve on the manifold starting from point U along the direction of tangent vector A. Formally, let QR == A UU TA be the QR decomposition of A UU TA Then, the exponential map expo (A) can be computed by (A) UM + QN, where
point to new point the manifold by following the geodesic in
M N
U R
R
!))
exp
and exp is the matrix exponential (not be confused with the exponential map expy)
While the exponential map is geometrically well-founded, it is more expensive to compute than the polar retraction. We compare the performance of polar retraction and the exponential map in Table o The two methods achieve nearly identical results (86.72 vs. 86. 76). We adopt the polar retraction as the default choice in StelLA due to its lower computational cost.
6 Limitation and Future Work
While StelLA demonstrates strong empirical performance across range of models and tasks, also comes with several limitations. First, did not explore combining StelLA with complementary LoRA variants such as AdaLoRA [74]. These methods introduce orthogonal improvements such as rank scheduling. Second, our approach may be extended tensor-valued layers using the low-rank Tucker format (71 straightforwardly, via the product Stiefel manifolds structure. Finally, due to time and resource constraints, leave the evaluation of StelLA more model families, and at very large scales, to future work.
7 Conclusion
We introduced StelLA, subspace-aware extension of Low-Rank Adaptation (LoRA) that explicitly learns input and output subspaces through a three-factor decomposition USVT, with U and V constrained on Stiefel manifolds. By combining Riemannian optimization with a modular training algorithm that supports arbitrary optimizers, StelLA offers flexible and geometry-aware approach to parameter-efficient fine-tuning. We demonstrated the effectiveness StelLA on a variety tasks, including language modeling, image classification, and text-to-image generation. Our experiments show that StelLA consistently outperforms LoRA A and variants, achieving state-of-the-ar results on several benchmarks.
10

---

References
[1] P-A Absil, Robert Mahony, and Rodolphe Sepulchre. Optimization algorithms matrix manifolds. In Optimization Algorithms Matrix Manifolds. Princetoon University Press, 2009.
[2] Josh Achiam, Steven Adler. Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08 08774, 2023.
[3] Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Hearyk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models, arXiv preprint XrXi:: 100 07732, 2021.
[4] Seyedarmin Azizi, Souvik Kundu, and Massoud Pedram. LaMDA: Large model fine-tuning via spectrally decomposed low-dimensional adaptation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 9635-9644 Miami, Florida, USA, 2024. Association for Computational Linguistics.
[5] Klaudia Balazy, Mohammadreza Banaci, Karl Aberer, and Jacek Tabor. Lora-xs: Low-rank adaptation with extremely small number of parameters. arXiv preprint MXXi:2240.116000 2024.
[6] Gary Becigneul and Conference on Learning Representations, 2019.
Ganea, Riemannian adaptive optimization methods. In International
[7] Massimo Bini, Leander Girrbach, and Zeynop Akata. Decoupling angles and strength in low-rank adaptation. In The Thirteenth International Conference on Learning Representations, 2025.
[8] Lukas Bossard. Matthieu Guillaumin, and Luc Van Gool. Food-101 - mining discriminative components with random forests. In European Conference on Computer Vision, pages 446-461. Springer, 2014.
[9] Tom Brown, Benjamin Mann. Nick Ryder, Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Fric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, lya Sutskever, and Dario Amodei. Language models few-shot learners. In Advances in Neural Information Processing Systems, pages 1877-1901. Curran Associates, Inc., 2020.
[10] Kerim Büyükakyüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2406.0177 2024.
[11] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray. Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings. Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert- Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji. Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra. Evan Morikawa, Alee Radford, Matthew Knight, Miles Brundage, Mira Murati, Katic Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojeiech Zaremba. Evaluating large language models trained on code, 2021.
[12] Civitai Team. Lora datasets / training data list civitai dataset guide. httpp:://ctitiiiie http:p////ciiiaaaa.coma/aacillll www.////maatcleeeoeo 1L38/lora-datastt trainttnttttiiiinn-nen
guide 2025. Accessed: 2025-05-
08.
[13] Karl Cobbe. Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reüchhro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems, arXiv preprint arXiv:21 141668 2021.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. 2009 IEEE Conference Computer Vision and Pattern Recognition, pages 248-255, 2009.
[15] Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. QLORA: Efficient finetuning of quantized LLMs. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[16] Alexey Dosovitskiy. Lucas Beyer. Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
11

---

[17] Dexin Duan, Peilin Liu. Bingwei Hui, and Fei Wen. Brain-inspired online adaptation for remote sensing with spiking neural network. IEEE Transactions on Geoxcience and Remote Sensing, 2025.
[18] Alan Edelman, Tomás A Arias, and Steven T Smith. The geometry of algorithms with orthogonality constraints. SIAM journal on Matrix Analysis and Applications, 20(2):303-353, 1998.
[19] Li Fei-Fei, Rob Fergus, and Pietro Perona, One-shot learning of object categories. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 594-611. IEEE, 2006.
[20] Chengeheng Feng, Mu Hc, Qiuyu Tian, Haojie Yin, Xiaofang Zhao, Hongwei Tang, and Xingqiang Wei. Trilora: Integrating svd for advanced style personalization in eet-to-image generation, 2024.
[21] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint wrXiv:240.22178. 2024.
[22] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseekrr:: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501. 1eel.12940 12948, 2025.
[23] Zeyu Han, Chao Gao, Jinyang Liu, Jeff Zhang, and Sai Qian Zhang. Parameter-efficient fine-tuning for large models: comprehensive survey. Transactions on Machine Learning Research, 2024.
[24] Soufiane Hayou, Nikhil Ghosh, and Bin Yu, Lorat: Efficient low rank adaptation of large models. In International Conference on Machine Learning, pages 17783-17806. PMLR, 2024.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference computer vision and pottern recognition, pages 770-778, 2016.
[26] Dan Hendrrcks,, Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[27] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
[28] Neil Houlsby. Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, pages 2790-2799. PMLR, 2019.
[29] Edward Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen, LoRA: Low-rank adaptation of large language models. In International Conference on Learning Reprexentations, 2022.
[30] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732. 2023.
[31] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. object representations for fine- mn-ggrained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[32] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
[33] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter efficieen prompt tuning. Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages
Online and Punta Cana, Dominican Republic, 2021, Association for Computational Linguistics.
[34] Junnan Li. Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for unified langune understanding and generation. In International conference on machine learning, pages 1 11980. PMLR, 2022.
[35] Xiang Lisa Li and Perey Liang. teninnt Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language P Processing (Volume I: Long Papers), pages 4582- 4597, Online, 2021. Association for Computational Linguistics.
12

---

[36] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting the Association for Compulational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume I: Longg Pupers), pages 4582- 4597, 2021.
[37] Zhizhong Li, Deli Zhao. Zhouchen Lin, and Edward Y. Chang. A new retraction for accelerating the riemannian three-factor low-rank matrix completion algorithm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
[38] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky. ReloRA: High- training through low-rank updates. In The Twelfth International Conference on Learning Representations. 2024.
[39] Kai Lion, Liang Zhang, Bingcong Li, and Niao He. PoLAR: Polar decom-deco low-rank adapter represen- tation. ES- ES-FoMo III: 3rd Workshop on Efficient Systems for Foundation Models, 2025.
[40] Shih- Yang Liu, Chien-Y Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min- Chen. DoRA: Conference on Machine Learning, 2024.
low-rank adaptation. In Forty-first International
[41] Yun Long, Haifeng Luo, and Yu Zhang. Evaluating large language models in analysing classroom dialogue. npj Science of Learning, 9(1)60,, 2024.
[42] Sourab Mangrulkar. Sylvain Gugger. Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. ltttps://githb. com / hugg ingfgg/ peft 2022.
[43] Fanxu Meng. Zhaohui Wang, and Muhan Zhang. PISSA: Principal singular values and singular vectors adaptation of large language models. In The Tighth Annual Conference on Neural Information Processing Systems, 2024.
[44] Bamdev Mishra and Rodolphe Sepulchre. R3mc: A riemannian ttree-factor algorithm for low-rank matrix completion. In 53rd IEEE Conference on Decision and Control, pages 1137- 1142. IEEE, 2014.
[45] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. Proceedings of the Indian Conference on Computer Vision, Graphics and Image Processing, 2008.
[46] Fabian Paischer, Lukas Hauzenberger. Thomas Schmied, Benedikt Alkin, Marc Peter Deisenroth, and Sepp Hochreiter. One initialization to rule them all: Fine-tuning via explained variance adaptation. NeurlPS 2024 Workshop on Fine-Tunie TuninT in Modern Machine Learning: Principles and Scalability, 2024.
[47] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pages 3498-3505, 2012.
[48] Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary Vito,o, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017
[49] PyTorch. torch. linalg,svd - pytorch 2.7 documentation. https:/ wtt.o//cc.......c..oc.o. www.eepp.oooocoooooococcaaaee. org/docs/2.7 generated/torch... 1 linalg..nn............l......l .vd....................m.m...... www.hhttohohhhh.o.oo linalg- -iinal--vvd 2025. Accessed: 2025-05-10.
[50] Yujia Qin, Xiaozhi Wang, Yusheng Su, Yankai Lin, Ning Ding, Jing Yi, Weize Chen, Zhiyuan Liu, Juanzi Li, Lei Hou, Peng Li, Maosong Sun, and Jie Zhou. Exploring universal intrinsic task subspace for few-shot learning via prompt tuning. JEEE/ACM Trans. Audio, Speech and Lang. Proc., 32:3631-3643, 2024.
[51] Zeju Qiu, Weiyang Liu, Haiwen Feng. Yuxuan Xue. Yao Feng, Zhen Liu, Dan Zhang, Adrian Weller. and Bernhard Schölkopf. Controlling text-to-image diffusion by orthogonal finetuning. In Thirth-seventh Conference on Neural Information Processing Systems, 2023.
[52] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmLR, 2021.
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffiusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Partern Recognition, pages 10684-10695, 2022.
[54] Soumava Kumar Roy, Zakaria Mhammedi, and Mehrtash Harandi. Geometry aware constrained optimiza- tion techniques for deep learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4460 4469, 2018
13

---

[55] Dayana Savostianova, Emanuele Zangrando, Gianluca Ceruti, and Francesco Tudisco. Robust low-rank training via approximate orthonormal constraints, Advances in Neural Information Processing Systems, 36: 66064-66083, 2023.
[56] Andrew M Saxe, James I. McClelland, and Surya Ganguli. Exact solutions the nonlinear dynamics of learning in deep linear neural networks. arXiv preprint /3 2013.
[57] Steffen Schotthöfer, H Lexie Yang, and Stefan Schnake. Dynamical low-rank compression of neural networks with robustness under adversarial attacks. arXiv preprint arXiv:2505. 08022, 2025.
[58] Steffen Schotthöfer. Emanuele Zangrando. Gianluca Ceruti, Francesco Tudisco, and Jonas Kusch. Ge- oloRA: Geometric integration for parameter efficient fine-tuning. In The Thirteenth International Confer- ence on Learning Reprexentations, 2025.
[59] Karan Singhal, Tao Tu, Juraj Gottweis, Rory Sayres, Ellery Wulczyn, Mohamed Amin, Le Hou, Kevin Clark, Stephen R Pfohl, Heather Cole-Lewis, et al. Toward expert-level medical question answering with large language models. Nature Medicine, pages 1-8, 2025.
[60] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[61] Patrick von Platen. Suraj Patil, Anton Lozhkov, Pedro Cuenca, Nathan Lambert, Kashif Rasul, Mishig Davaadorj. Dhruv Nair, Sayak Paul, William Berman, Yiyi Xu, Steven Liu, and Thomas Wolf. Diffusers: State-of-the-art diffusion models. https:// http://ggtttb..... com wtt.:cco.oohhcgeeee hwgginggacceeee//ecoeooooooooeah http///aac/afffurr/////e 2022.
[62] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 1 dataset. In California Institute of Technology, 2011.
[63] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. MiL.RA: Harnessing minor singular components for parameter-cffcient LLM finetuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguisties: Human Language Technologies (Volume 1: Long Papers). pages 4823-48366 Albuquerque, New Mexico, 2025. Association for Computational Linguistics.
[64] Shaowen Wang, Linxi Yu, and Jian Li, LoRA-GA: Low-rank adaptation with gradient approximation. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[65] Shijie Wu, Ozan Irsoy, Steven Lu, Vadim Dabravolski, Mark Dredze, Sebastian Gehrmann, Prabhanjan Kambadur, David Rosenberg, and Gidcon Mann. Bloomberggpt: A large language model for finance. arXiv preprint arXiv:2303. 17564, 2023.
[66] Taiqiang Wu, Jiahao Wang. 7he Zhao, and Ngai Wong. Exture-of-ssbspecee in low-rank adaptation. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pages 7880-7899, Miami. Florida, USA, 2024. Association for Computational Linguistics.
[67] Yecheng Wa, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang. Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, Song Han, and Yao Lu. VILA-u: a unified foundation model integrating visual understanding and generation. In The Thirteenth International Conference on Learning Representations, 2025.
[68] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large- scale scene recognition from abbey to In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[69] Greg Yang and Etai Litwwin Tensor programs ivb: Adaptive optimization in the infihite-width limit, arXiv preprint arXiv:2308.0 01814, 2023.
[70] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
[71] Emanuele Zangrando, Steffen Schotthöfer, Gianluca Ccruti, Jonas Kusch, and Francesco Tudisco. Geometry-aware training of factorized layers in tensor tucker format, Advances in Neural Information Processing Systems, 37: 37:129743-129773 2024.
[72] Fangzhao Zhang and Ment Pilanci. Riemannian preconditioned lora for fine tine-tuning foundation models. In International Conference on Machine Learning, pages 19641-59669. PMLR, 2024.
14

---

[73] Fangzhao Zhang and Mert Pilanci. Spectral adapter: Fine-tuning in spectral space. In The Thirt Anmual Conference Neural Information Processing Systems, 2024.
[74] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In The Eleventh International Conference on Learning Representations. 2023.
[75] Tianyu Zheng. Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opennoddenterppetee:: Integrating code generation with execution and refinement. In Findings of Association for Computational Linguistics: ACL 2024, pages 12834-1222 12859, Bangkok, Thailand, 2024.
Association for Computational Linguistics.
[76] Weiming Zhuang, Chen Chen, Zhizhong Li, Sina Sajadmanesh, Jingtao Li, Jiabo Huang, Vikash Sehwag. Vivek Sharma, Hirotaka Shinozaki, Felan Carlo Garcia, et al. Argus: A compact and versatile foundation model for vision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, 2025.
15

---

A Intuitive Example about Stiefel Manifold
To help readers understand the Stiefel manifold, we provide an intuitive example in low dimensions. In Section 3, we provided general equations for various geometric concepts. Here, we apply those ingredients to the following specific example.
2),
TySt(1,
Y
Y
St(1,2)
Vy
St(1, 2)
A
grad Y
0
(a) The manifold St(1, 2) and the tangent space Ty ()SS(11,) at point Y.
(b) Converting a Euclidean gradient Vy to the Riemannian gradient grad Y is projection.
Sy,t 2)
SySt(1,
Y
St(1,2) 2)
A
St(1, 2)
A
Y
Iy A
Pr (A)
0
1
1
(c) Projecting tangent vector A E Ty-R. R² onto the tangent space of the manifold Ty St(1,2). 2).
(d) Retraction py (A) is the normalization of the vector Y + A to unit vector.
Figure 2: An example Stiefel manifold St(1, 2): the unit circle on the two-dimensional plane.
Manifold. Consider St(1,2) = (Y = [v v2] E R = yî + y2 = 1}, the Stiefel manifold of orthonormal 1-frames in R22 which consists of all unit vectors in the plane, iee, the unit circle. See Figure 2a for an illustration.
Tangent Space. For point Y [y,, v2 V2 E St(1, 2), tangent vector A [01, 62] must satisfy the constraint YT A + ATY 0, which leads to the condition U1 + y2 O. namely Y 1 A. The tangent space at Y is the set of all vectors that are orthogonal to Y. See Figure 2a for an illustration.
Riemannian Metric. For two tangent vectors A Ag at point Y, the canonical metric defines the inner product
gr (A,, A2) = tr(A] (T, 1 (YYT)A)) = tr(A] 42) 2 tr A2) = ATA2, (8)
which equals the standard inner product in R The last equation holds because YT A2 0 and A A2 is scalar.
Riemannian Gradient. Suppose the Euclidean gradient of function at Y is VY which is abbreviated as Vy. Then the Riemannian gradient grad fYY), abbreviated as grad Y, is given by
grad Y Vy
YVTY Vy (V.Y),,
(9)
16

---

which is the projection of the Euclidean gradient onto the tangent space at Y. The second term (V)Y) is the component of the Euclidean gradient that is not tangent the manifold. See Figure 2b for an illustration.
Tangent Space Projection.
The projection vector A onto the tangent space at Y is given by
Ty (A) = - Y symmmm ( Y) A (A YYY,
(10)
which the component of that is tangent to the manifold. The second term (A (YTY)Y is the component of A that is not tangent the manifold. See Figure 2c for an illustration.
Retraction. For a tangent vector A at Y, the retraction is given by
Y+
py (4) uf(Y A)
(11)
|Y+
which in the (1, 2) case equivalent to normalizing the vector Y + A to a unit vector. See Figure 2d for illustration.
B Details of the Quotient Geometry
The fhree-factor decomposition of a low-rank matrix M = USVT is not unique due to the equivalence relationship (U, V) (UO Of SO2, vO,),vO,,00 O2 E (r),, where ((r) is the orthogonal group. Each rank-r matrix has infinite representations. For example,
(UO)) (oo) ((0:0)))(0::3))V== USV
(12)
Factoring out this symmetry, we get a quotient space St m) x xx x m)(rO)) O(r))
where each low-rank matrix is uniquely represented.
Mishra and Sepulchre [44] proposed a Riemannian metric for this quotient space. We adapt StelLA to use this Riemannian metric and refer this setting as the Quotient geometry. The Riemannian metric in 44 is induced by the block approximation of the Hessian of ||USV* Wll For two tangent vectors (Ev, Ev), (yu, ns, nv) point (U,S, V), the Riemannian metric is defined
((Er,,) Ev), (Mu Vs, m))) Lr (SST <Tm))) + Lr E + tr (ST SETW),
(13)
where SST and S S act as preconditioners which improve the convergence in the low-rank matrix completion problem. Details of the geometric optimization in this space discussed in [44), including the equations for the Euclidean gradient to Riemannian gradient conversion, the tangent vector projection V)), and the retraction V)). Note that Mishra and Sepulchre [44] used the conjugate- gradient optimization algorithm to solve the low-rank matrix completion problem. However, with the operators such as (TU.S.V) and P(U.S.V) defined, it is straightforward to adapt to the Riemannian Adam algorithm following Algorithm 1. Specifically, the Euclidean gradient to the Riemannian gradient, [44 Eq. (5)) for the projection onto tangent space, and 44 Eq. (7)] for the retraction.
(44 Eq. (9)) to convert
C Details of Hyperparameters
We provide the detailed hyperparameters used in our experiments to ensure full reproducibility. For each benchmark, all compared methods share common set of hyperparameters such as rank, batch size, weight decay, and training schedule which which are outlined in Section 5. The only exception is the learning rate, which we individually tune for each method to ensure fair comparison. this section, we list the specific learning rates used for each algorithm across benchmarks.
Commonsense Reasoning. We explore the learning rate in (0.0005, 0.0001, 0.00005, 0.00001} for all algorithms, with the chosen values shown in Table 7
Math and Code Generation. The search space for the learning rates is (0.0005, 0.0002, 0.00002) for all methods. We list the selected learning rates in Table g
17

---

Table Learning Rates for Commonsense Reasoning Experiments.
Model
Method
Learning Rate
LoRA
0.0001
DoRA
0.0001
PISSA
0.00005
OLoRA
0.00005
LLaMA2-7B
TriLoRA
0.00001
MoSLoRA
0,0001
ScaledAdam
0.0001
StelLA
0.0005
LoRA
0.0001
DoRA
0.0001
PiSSA
0.00005
OLoRA
0.00005
LLaMA3-8B
TriLoRA
0.0001
MoSLoRA
0,0001
ScaledAdam
0.00005
StelLA
0.0005
Table 8: Learning Rates for Math and Code Generation Experiments.
Model
Method
Math
Code
LoRA
0.0002
0.0002
DoRA
0.0002
0.0002
LLaMA2-7B
PISSA
0.0002
0.0002
StelLA
0.0005
0.0005
Image Classification. The learning rate is tuned in 10.0001,0.0005, 0.001, for all methods. The chosen learning rates are summarized in Table 9
Text to Image Generation. We search the learning rate over {0.0001, 0. 0.0002, 0.0004, 0.0008} for all methods, with the selected values reported in Table I0 For each method, use the same learning rate for all five CivitAI datasets.
D Details of the Computational Cost for Algorithm
In Algorithm for each of the Stiefel parameter (i.e., and V), has the following operations:
1. Converting the Euclidean gradient to the Riemannian gradient using Equation (2).
2. Projecting the update direction onto the tangent space using Equation (3).
3. Retracting the updated point back onto the Stiefel manifold using Equation (4).
Among these, the first two steps involve only basic matrix multiplications and additions, and thus incur negligible computational overhead. The dominant cost arises from the retraction step, which requires a polar decomposition of the matrix Y + A. This polar decomposition is typically computed via singular value decomposition (SVD) as follows.
Suppose the SVD of (Y + A) E pmxrr with m > r is expressed as Y + A = UVV where U E Rx and V E T are orthogonal matrices, and E grxr is diagonal matrix. Then the orthonormal factor in the polar decomposition is given by
u + A) = UV.
(14)
Thus, the retraction step has a cost dominated by computing the SVD of an m x rectangular matrix.
Theoretically, the computational complexity of the polar retraction step is 0(mr²) for tall matrices (m > r) due to the SVD computation. But practically, there are efficient SVD algorithms for tall matrices that can reduce the computational cost significantly. Specifically, use the geavda
18

---

Table 9: Learning Rates for Image Classification Experiments.
Model
Method
DTD
EuroSAT
11owers102
Food101
OxfordPets
Sun397
CIFARIO
CIFAR100
LoRA
0.005
0.005
0.005
0.001
0.005
0,001
0.001
0.001
VIT Base
DoRA
0.005
0,005
0,005
0,001
0.005
0,001
0.001
0.001
PISSA
0.005
0.005
0.005
0.001
0.001
0.001
0.0005
0.001
StelLA
0.005
0.005
0.005
0.001
0.005
0.001
0.001
0.001
LoRA
0.001
0.005
0.001
0.0005
0.001
0.0005
0.001
0.001
VIT Large
DoRA
0.001
0.005
0.001
0.0005
0.0005
0.0005
0.001
0.0005
PiSSA
0.001
0.001
0.001
0.0005
0.0005
0.0005
0.0001
0.0005
StelLA
0.005
0.005
0.001
0.001
0.001
0.001
0.001
0.0005
Table 10: Learning Rates for Text to Image Generation Experiments.
Model
Method
Learning Rate (same for all 5 tasks)
LoRA
0.0001
DoRA
0.0001
SD 1.5 & SD 2.0
PISSA
0.0001
StelLA
0.0008
solver [49]. which is a CUDA- SVD implementation that can handle tall matrices effi- ciently. Below is ihero-bennchark of different SVD solvers for the matrix shapes used in StelLA for the commonsense benchmark on LLaMA3-SB. Results in Table Maleelsseee show that the gesvda solver significantly faster than the default GPU solver in PyTorch.
The speed could be further improved by using batch processing to increase parallelism. Specifically, can stack all the low-rank matrices U and V with the same shape into batch, and then perform the polar retraction step on the batch of matrices. Table [2] is a nicro-benchhark of the SVD using batch processing on the matrix shapes used in StelLA for the commonsense benchmark on LLaMA3-8B (there are 192, 64, and 64 matrices with shapes 4096x32, 1024x32, and 14336x32, respectively). The results show that the batched SVD is significantly faster than the sequential one, and the polar retraction step is no longer the bottleneck of the training time.
In practice, training LoRA-adapted LLaMA3-8B model on commonsense reasoning benchmark takes approximately 4.5 hours on a single H100 GPU, whereas training the same model with StelLA takes around 5.2 hours, about only 15% slower than vanilla LoRA.
E Discussion on Scale Stability
The scale stability of LoRA refers to the property that neither the activations nor the gradients explode vanish the rank r. input dimension and output dimension grow infinity [30 64). More frecisely,forward stability is achieved if, assuming the input to the adapter has i.i.d. entries with second moment 0r,m,n (1), the output of the adapter also maintains second moment of the same order. Similarly, backward stability is achieved if, when the gradient of the loss w.r.t. the adapter output has second moment Or.m, (1), the gradient w.r.t. the adapter input also remains at m,, (1). Letting denote the scaling coefficient applied to the low-rank update, osLoRA (30) shows that the original LoRA choice of 7 _ +(+) is not rank stable. Instead, they propose that y should scale as ((I/VF) maintain stability.
In StelLA, we initialize U and V as random orthogonal matrices and to the identity matrix. This setup satisfies the main assumptions in Theorem 3.2 of (64), which analyzes the scale stability of LoRA. Following their analysis, we assess the scale stability of StelLA at initialization as follows.
Let the adapter compute y T a for an input vector where is the scaling factor. Since is initialized to be identity, this simplifies to y = NUV z.. The forward pass can be expressed component-wiss as
Un Vajaki
1 1 m. (Forward)
(15)
19

---

Table 11: Runtime of different SVD solvers in Py PyTorch on H100 GPU.
Matrix Shape
Default Solver (ms)
GESVDA Solver (ms)
Speedup of GESVDA
x0
0.938
0,641
1.46x
30
0.795
0.632
1.26x
x4336x332
1.530
0.654
2.34x
Table 12:
Runtime of SVD with/without batch processing on H100 GPU. The GESVDA solver is used for both batched and sequential processing.
Matrix Shape
Batch Size
Batched (ms)
Sequential (ms)
Speedup Batched
4096x30
192
5.80
123.1
22.22
x4 x024x32
64
1.62
40.4
24.94
64 x 14336x32
64
2.89
41.8
14.46
To analyze the scale, we compute the second moment:
E =
El Vasi Ts, Uni Tkzl
(16)
11 j2= b Rg
Blu U [B[V
Veuja B|x Tk2l
(17)
ElU U BB V
(18)
E E
(19)
22r Or.m.n (1).
m mm (1)
(20)
Hence, the forward pass remains scale-stable in the beginning of the training if 7 = @(v 10
Equation 18) is derived from the independence of Jk, and Tkg, where Biax, Tk5] = 0 when kn 7 k2-
Equation 19 and Equation (20) derived from the fact that for random unit vector R."
B|a,a,] = 0 when i + j and E
For the backward pass, the gradient with respect to the input is given by e Oc/0y. Then,
04 dx
= -VUTu u, where
m ViJKasuk
9.
isism. n. (Backward)
(21)
k=1
and the second moment becomes:
m
m
Elghl
Vos Ukiji Uk, Viga Ukajz Uka
(22)
in
Ja=1
R1 k===1
m
E Elu JE E[12
(23)
j=1
22r θ mm (1).
2 θ mm (1)
(24)
72 m
n
which implies the backward pass is stable in the beginning of the training if @(v²).
In practice, we adopt the coefficient 7 - to be consistent with LoRA. While this choice does not guarantee theoretical scale stability, it does not negatively affect training in practice as a can be adjusted empirically. Since m, n, and r are fixed for given model, tuning a allows us to ensure stable and convergent training across tasks.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims made in the abstract and introduction are discussed in Section and Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We have a limitation section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
We do not have theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide enough experimental details for each set experiments, including hyperparamters, and setting detail in appendix to help reproduce the results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
We will open source the code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7/nips.com wttp:ccpppbccccceee guwde//ddeeeeee
siiiisisccyyyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We provide information of the datasets and training details that are necessary.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: For experiments, provide average of runs for most of our experiments, however, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
23

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπхо rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: Our experiments cover different tasks with different models and datasets, which would require vastly different resources. Hence verbose to give an estimation on the computation resources that are needed for all the experiments we have.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httphh/.t.gg.....s......../////p http://ccppccccccicccceeeeeeeeee
Answer: [Yes]
Justification:
We cannot identify any violation.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
We cannot identify any new social impact in addition to the existing literature.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We respected the license of all codes, datasets, and models used in the paper experiments, which can be found in the appendix.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
26

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification:
We sufficiently described the usage of LLMs in experiments.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neuree 22tppp/p/..///oofeeeeee////////0 for what should or should not be described.
27


Reviews:
[{"review_id": "Gf2ZACd6Jy", "reviewer": "Reviewer_Qyxg", "summary": "This paper proposes StelLA,  a geometry-aware extension of LoRA that uses decomposition **USV** instead of traditional **BA**. During the training process, StelLA constrains **U** and **V** to lie on the Stiefel manifold. A significant advantage of stelLA is flexibility.  It decouples geometric constraints from the choice of optimizer. The authors conduct experiments across multiple domains, demonstrating the superiority of  StelLA over vanilla LoRA and other variants.", "questions": "(1) Compared to traditional optimizers, StelLA introduces a significant amount of additional computation. Theoretically and practically, how much would the proposed method slow down the model training speed?\n\n(2) By adding strict constraints during training, could the convergence speed become slow or even fail to converge? Convergence speed is already a pain point for LoRA, and modification in this direction will further reduce the convergence speed.\n\n(3) Is there potential for algorithmic or hardware-level optimizations to speed up Stella?\n\n(4) I expect to get some intuitive explanations: Why does applying SVD decomposition with orthogonal constraints during training improve model performance? Could this operation be merely playing a similar role to simple regularization?\n\n(5) Why are the rank settings different in each scenario? The choices lack justification.", "limitations": "The limitations section is missing.", "rating": 4, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths\n\nS1: It is an interesting and well-motivated idea to use a three-factor decomposition **USV** and constrains **U** and **V** to lie on the Stiefel manifold.\n\nS2: The extensive experiments, covering multiple scenarios, demonstrate the effectiveness of the proposed StelLA.\n\nS3: The paper is well-written, with a reasonable structure and a clear algorithm flow, making it easy for readers to understand the work presented.\n\n\nWeaknesses\n\nW1: The experiment only compares the number of trainable parameters, but lacks comparison of computational overhead or training time.\n\nW2: Will there be a major drawback to computational complexity? Will the training cost be significantly increased due to the involvement of complex operations?\n\nW3: References should be in numerical form according to NeurIPS 2025 Paper Formatting Instructions.\n\nW4: There is no discussion of limitations.\n\nW5: Why maintain the orthogonality of subspaces? This approach lacks motivation.\n\nI have limited understanding of the manifold direction and find it difficult to determine whether the formulas in the article are accurate.", "was_revised": true, "final_justification": "The main reason I raised the score is that the authors' response addresses my concerns. In the rebuttal, the authors elaborated on the training overhead and explained the value of maintaining matrix orthogonality.\n\nThe reason I assigned a low confidence score is that I am concerned about whether this line of work has strong practical value. In practice, imposing constraints on the training matrix often has a negative impact on convergence (although the authors claimed otherwise). In addition, the improvement in performance may not necessarily come from the decomposition of direction and magnitude as claimed by the authors.", "rebuttal": "Thanks for the detailed review and constructive comments.\nWe have addressed all the comments and suggestions in the revised version of the manuscript.\nBelow we clarify the questions and concerns in detail.\n\n> W1: Lack of comparison of computational overhead or training time.\n\nand\n\n> W2: Will there be a major drawback to computational complexity? Will the training cost be significantly increased due to the involvement of complex operations?\n\nand\n\n> Q1: Theoretically and practically, how much would the proposed method slow down the model training speed?\n\nand\n\n> Q3: Is there potential for algorithmic or hardware-level optimizations to speed up Stella?\n\nThanks for the comments and questions regarding the computational overhead and training time, which is indeed an important aspect of our proposed method.\nWe would like to point out that a detailed discussion of computational cost of Alg. 1 has already been provided in the appendix Sec. D.\nWe will move this information from the appendix to the main text for better visibility.\n\nWe summarize the key information in the appendix here:\n- The main computational overhead of Alg. 1 comes from the SVD operations in the polar retraction step, which can be reduced by using efficient CUDA-accelerated SVD implementation [e].\n- Training the LLaMA3-8B model with StelLA using Alg. 1 takes 6.5h for the commonsense benchmark, which is approximately 1.4x the time of LoRA (4.5h) on one H100 GPU. It can be further improved, e.g., by increasing parallelism with batch processing (details below), which reduces the training time from 6.5h to **5.2h**, about only 15% slower than vanilla LoRA.\n\nDetailed discussion:\n\nTheoretically, the computational complexity of the polar retraction step is $O(mn^2)$ for $m\\times n$ shaped tall matrices ($m > n$) due to the SVD computation.\nBut practically, there are efficient SVD algorithms for tall matrices that can reduce the computational cost significantly.\nSpecifically, we used the `gesvda` solver [e] in StelLA, which is a CUDA-accelerated SVD implementation that can handle tall matrices efficiently.\nBelow is a micro-benchmark of different SVD solvers for the matrix shapes used in StelLA for the commonsense benchmark on LLaMA3-8B.\nThe results show that the `gesvda` solver is significantly faster than the default GPU solver in PyTorch.\n\n**Table: Runtime of different SVD solvers in PyTorch on H100 GPU**\n\n| Matrix Shape | Default GPU Solver (ms) | GESVDA Solver (ms) | Speedup of GESVDA |\n| ------------ | ----------------------- | ------------------ | ----------------- |\n| 4096x32      | 0.938                   | 0.641              | **1.46x**         |\n| 1024x32      | 0.795                   | 0.632              | **1.26x**         |\n| 14336x32     | 1.530                   | 0.654              | **2.34x**         |\n\n\nThe speed could be further improved by using batch processing to increase parallelism.\nSpecifically, we can stack all the low-rank matrices $U$ and $V$ with the same shape into a batch, and then perform the polar retraction step on the batch of matrices.\nThis reduces the training time from 6.5h to **5.2h**, about 15% slower than vanilla LoRA.\nBelow is a micro-benchmark of the SVD using batch processing on the matrix shapes used in StelLA for the commonsense benchmark on LLaMA3-8B (there are 192, 64, and 64 matrices with shapes 4096x32, 1024x32, and 14336x32, respectively).\nThe results show that the batched SVD is significantly faster than the sequential one, and the polar retraction step is no longer the bottleneck of the training time.\n\n**Table: Runtime of SVD with/without batch processing**\n\n| Matrix Shape | Batch Size | GESVDA with Batch (ms) | GESVDA without Batch (ms) | Speedup of Batched Processing |\n| ------------ | ---------- | ---------------------- | ------------------------- | ----------------------------- |\n| 192x4096x32  | 192        | 5.80                   | 123.1                     | **22.22x**                    |\n| 64x1024x32   | 64         | 1.62                   | 40.4                      | **24.94x**                    |\n| 64x14336x32  | 64         | 2.89                   | 41.8                      | **14.46x**                    |\n\n\n> W3: References should be in numerical form according to NeurIPS 2025 Paper Formatting Instructions.\n\nThanks for pointing this out; we have fixed the reference format in the revised manuscript.\n\n> W4: There is no discussion of limitations.\n\nWe have discussed the limitations in appendix Sec. G, which we will move to the main text for better visibility.\n\n> W5: Why maintain the orthogonality of subspaces? This approach lacks motivation.\n\nand\n\n> Q4: I expect to get some intuitive explanations: Why does applying SVD decomposition with orthogonal constraints during training improve model performance? Could this operation be merely playing a similar role to simple regularization?\n\nThe columns of the orthogonal matrix form an orthonormal basis for the subspace, which represents the \"direction\" of the subspace and is decoupled with the \"magnitude\" of the representation.\nMaintaining orthogonality is to ensure that the learning focuses on optimizing the \"direction\" of the subspace rather than their \"magnitudes\".\nSpecifically in StelLA, $V$ represents the \"direction\" of the input subspace, and $U$ represents the \"direction\" of the output subspace, and $S$ represents the \"magnitude\" of the adapter.\n\nThe separation of direction and magnitude is a useful regularization in many areas of machine learning, such as in weight normalization [a] (separates direction and magnitude of network weights), CosFace [b] (normalize both features and class representative vector to focus on learning directions), and Grassmann Class Representation [c] (represent class prototype by subspaces, which ignores the magnitude).\nResearch in these areas shows that decoupling direction and magnitude can improve the performance of the model.\nRecently, DoRA [d] also shows that this separation also benefits low-rank adaptation.\nAs such, we investigated whether explicitly optimizing the direction of subspaces during low-rank training can further improve LoRA's performance.\nOur empirical results confirm that preserving orthogonality throughout training can indeed lead to performance gains.\n\nAs reviewer CC7z pointed out, maintaining U and V on the Stiefel manifold is also beneficial for certain downstream analyses, e.g. adversarial robustness.\n\nWe will enhance the discussion of the motivation for maintaining orthogonality in the revised version of the manuscript.\n\n\n> Q2: By adding strict constraints during training, could the convergence speed become slow or even fail to converge? Convergence speed is already a pain point for LoRA, and modification in this direction will further reduce the convergence speed.\n\nWe didn't observe any convergence issues in our experiments.\nWe used the same number of training epochs for both StelLA and all baselines.\nAt the end of training, StelLA achieves better performance than other methods, which indicates that the added constraints do not hinder convergence but instead contribute to more effective optimization.\n\nGenerally speaking, literature on adding orthogonal constraints [c, f, g, h, i] in neural networks shows that the convergence speed is improved with the orthogonality constraints.\n\nWe will add comparison of convergence plots to the revised manuscript.\n\n> Q5: Why are the rank settings different in each scenario? The choices lack justification.\n\nFor each learning task, we carefully followed the same setting as in previous work for fair comparison.\nFor example, for the commonsense reasoning benchmark, we followed the setting in DoRA (L201), and for the math and code generation benchmarks, we followed the setting in PiSSA (L224).\n\n> Difficult to determine whether the formulas in the article are accurate.\n\nThanks for raising this concern.\nWe provided an illustrative example in the appendix Sec. A, where a two-dimensional case is used to explain the concepts related to Stiefel manifold and geometric optimization.\nThe equations are simplified in two-dimensional space, hoping to help sanity checking the workflow of Alg. 1.\n\n- [a] Weight normalization: A simple reparameterization to accelerate training of deep neural networks, NIPS 2016.\n- [b] Cosface: Large margin cosine loss for deep face recognition, CVPR 2018.\n- [c] Get the Best of Both Worlds: Improving Accuracy and Transferability by Grassmann Class Representation, ICCV 2023.\n- [d] DoRA: Weight-Decomposed Low-Rank Adaptation, ICML 2024.\n- [e] https://docs.pytorch.org/docs/stable/generated/torch.linalg.svd.html\n- [f] Unitary Evolution Recurrent Neural Networks, ICML 2016.\n- [g] All you need is beyond a good init: Exploring better solution for training extremely\n deep convolutional neural networks with orthonormality and modulation. CVPR 2017\n- [h] Can we gain more from orthogonality regularizations in training deep networks? NeurIPS 2018.\n- [i] Orthogonal Convolutional Neural Networks, CVPR 2020."}, {"review_id": "7THti8OAFI", "reviewer": "Reviewer_Bh7E", "summary": "This paper enhances LoRA by applying an SVD-inspired three-factor formulation. Unlike prior methods that only use SVD for initialization, this approach keeps U and V orthonormal throughout training by constraining them to lie on the Stiefel manifold. Empirical results show the proposed method improves performance across a range of language and vision tasks.", "questions": "See Weaknesses", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe core novelty lies in applying LoRA decomposition $USV^T$, where the input/output subspaces U and V are constrained to lie on the Stiefel manifold. This allows the model to maintain orthonormality during training, unlike prior methods such as TriLoRA [1] and MoSLoRA [2], which do not keep the orthogonality during training.\n\nThe proposed method consistently outperforms other approaches on commonsense reasoning tasks, demonstrating its effectiveness in language understanding tasks.\n\nThe paper is well-written and easy to follow.\n\nWeaknesses:\n\n1) The paper lacks a comparison of computational efficiency. While the proposed method introduces additional computation for manifold operations (e.g., projection and retraction), which may impact training speed. A runtime or efficiency analysis would strengthen the empirical evaluation.\n\n2) The method is related to LoRA-XS [3]. However, the paper does not include experimental comparisons against LoRA-XS, weakening the completeness of the evaluation.\n\n3) It would be beneficial to compare the orthogonality of the proposed method to other methods like TriLoRA and LoRA-XS, as this would strengthen the effectiveness of the proposed Stiefel manifold constraints.\n\n4) The performance improvements on image classification benchmarks are limited. This raises questions about the robustness and generalizability of the proposed method\n\n[1] Trilora: Integrating svd for advanced style personalization in text-to-image generation\n\n[2] Mixture-of-subspaces in low-rank adaptation.\n\n[3] LoRA-XS: Low-Rank Adaptation with Extremely Small Number of Parameters", "was_revised": true, "final_justification": "Thank authors for their reply. I will maintain my score.", "rebuttal": "Thanks for the insightful review and constructive comments.\nWe have addressed all the comments and suggestions in the revised version of the manuscript.\nBelow we clarify the questions and concerns in detail.\n\n> The paper lacks a comparison of computational efficiency. While the proposed method introduces additional computation for manifold operations (e.g., projection and retraction), which may impact training speed. A runtime or efficiency analysis would strengthen the empirical evaluation.\n\nThanks for the suggestion.\nA detailed discussion of computational cost of Alg. 1 is provided in the appendix Sec. D.\nWe will move this information from the appendix to the main text for better visibility.\n\nHere we summarize the key information:\n- The main computational overhead of Alg. 1 comes from the SVD operations in the polar retraction step, which can be reduced by using efficient CUDA-accelerated SVD implementation.\n- Training the LLaMA3-8B model with StelLA using Alg. 1 takes 6.5h for the commonsense benchmark, which is approximately 1.4x the time of LoRA (4.5h) on one H100 GPU. It can be further improved, e.g., by increasing parallelism with batch processing (details in our response to Reviewer Qyxg), which reduces the training time from 6.5h to **5.2h**, about only 15% slower than vanilla LoRA.\n\nPlease kindly refer to our response to Reviewer Qyxg for more details on the computational cost, training time and how to improve them.\n\n> The method is related to LoRA-XS [3]. However, the paper does not include experimental comparisons against LoRA-XS, weakening the completeness of the evaluation.\n\nThanks for the suggestion.\nBelow we list the results on commonsense reasoning benchmark using LLaMA3-8B model from LoRA-XS's paper (they followed the same experiment setting as ours, so the results are directly comparable.)\nThe result shows that StelLA outperforms LoRA-XS by a large margin.\nThis is understandable because the number of learnable parameters in LoRA-XS is much smaller (only the $S$ matrix is learnable) than that in StelLA.\n\n**Table: Comparison of StelLA and LoRA-XS on the commonsense reasoning benchmark using LLaMA3-8B model.**\n\n| Method  | BoolQ | PIQA | SIQA | HellaS. | WinoG. | ARC-e | ARC-c | OBQA | Avg. |\n| ------- | ----- | ---- | ---- | ------- | ------ | ----- | ----- | ---- | ---- |\n| LoRA-XS | 66.6  | 85.8 | 79.4 | 90.1    | 85.2   | 87.0  | 76.5  | 81.8 | 81.6 |\n| StelLA  | 75.9  | 89.9 | 81.7 | 96.4    | 87.8   | 92.0  | 82.3  | 87.8 | 86.7 |\n\n> It would be beneficial to compare the orthogonality of the proposed method to other methods like TriLoRA and LoRA-XS, as this would strengthen the effectiveness of the proposed Stiefel manifold constraints.\n\nThanks for the suggestion.\nWe have briefly discussed the difference between StelLA and TriLoRA/LoRA-XS in the related work section (L94-97).\nHere we reiterate the key points and will add more details in the revised version:\n- TriLoRA does not initialize the low-rank matrices with orthogonal matrices (they initialize U with Gaussian, and V with zeros), and it does not maintain orthogonality during training.\n- LoRA-XS initializes the low-rank matrices with the leading $r$ singular vectors and the singular values of the weight matrix (Suppose $M=U_r S_r V_r^T$, then $U:=U_r$ and $V=S_r V_r^T$), in which only one of the low-rank matrices $U$ is orthogonal.\nBoth $U$ and $V$ are frozen during training, which limits the expressiveness of the low-rank adaptation.\n\nWe summarize the usage of orthogonality in StelLA, TriLoRA, and LoRA-XS in the table below.\n\n**Table: Comparison of orthogonality in StelLA, TriLoRA, and LoRA-XS**\n\n| Method  | Orthogonality of $U$ | Orthogonality of $V$ | Maintains Orthogonality in Training | $U, V$ Learnable | Commonsense Accuracy (avg.) |\n| ------- | -------------------- | -------------------- | ----------------------------------- | ---------------- | --------------------------- |\n| TriLoRA | $\\times$             | $\\times$             | $\\times$                            | $\\checkmark$     | 82.7                        |\n| LoRA-XS | $\\checkmark$         | $\\times$             | $\\checkmark$ (by freezing $U$)      | $\\times$         | 81.6                        |\n| StelLA  | $\\checkmark$         | $\\checkmark$         | $\\checkmark$                        | $\\checkmark$     | 86.7                        |\n\n> The performance improvements on image classification benchmarks are limited. This raises questions about the robustness and generalizability of the proposed method\n\nThanks for raising the concern.\nThe relatively limited performance improvements on image classification benchmarks are due to the fact that the performance might have being saturated on the benchmark: the accuracy of LoRA is above 90% on 6/8 of the datasets.\nYet, the performance improvements of StelLA over LoRA are significant on the two less saturated datasets: DTD (79.2% -> 79.89%) and Sun397 (75.62% -> 76.28%).\nDespite facing the challenge in further improving performance, our method still outperforms the two SOTA baselines, DoRA and PiSSA."}, {"review_id": "LUIFXMgRwm", "reviewer": "Reviewer_eSmE", "summary": "The work proposes a new LoRA method for fine-tuning large language models. It \n\t\tconsists of exploiting three matrices $U \\in \\mathbb{R}^{m \\times r}, \n\t\tV \\\\in \\\\mathbb{R}^{n \\\\times r}, S \\\\in \\\\mathbb{R}^{r \\\\times r}$ in the following minimization \n\t\ttask \n\t\t\\begin{equation*}\n\t\t\t\\\\mathcal{L}\\\\left( W + USV^\\\\top \\\\right)\t\\\\to \n\t\t\t\\min\\limits_{U \\\\in \\\\mathrm{St}\\\\, (r, m), \\\\; S, \\\\; V \\\\in \\\\mathrm{St}\\\\, (r, n)},\n\t\t\\\\end{equation*}\n\t\twhere $W \\\\in \\\\mathbb{R}^{m \\\\times n}$ is a frozen weight matrix and \n\t\t$\\\\mathrm{St}\\\\, (r, m)$ is the Stiefel manifold. The authors \n\t\tpresent a new method (Algorithm $1$). They compute\n\t\t\\\\begin{equation*}\n\t\t\t\t\\\\hat{U}\\_{k + 1} \\\\leftarrow \\\\mathrm{Optimizer}\\\\,(U_k, \\\\mathrm{grad}\\\\, U_k), \n\t\t\t\t\\\\hat{V}\\_{k + 1} \\\\leftarrow \\\\mathrm{Optimizer}\\\\,(V_k, \\\\mathrm{grad}\\\\, V_k), \n\t\t\t\t\\\\hat{S}\\_{k + 1} \\\\leftarrow \\\\mathrm{Optimizer}\\\\,(S_k, \\\\mathrm{grad}\\\\, S_k).\n\t\t\\\\end{equation*}\n\t\tThe directions $\\\\hat{U}\\_{k + 1}, \\\\hat{V}\\_{k + 1}$ do not \n\t\tnecesserally belong to the manifold \n\t\t(since the optimizer does not guarantee this). So the next step is to obtain a Riemannian\n\t\tdirection via the projection onto a tangent space \n\t\t$\\\\mathrm{proj}\\_{U_k} (U_k - \\\\hat{U}\\_{k + 1})$ and $\\\\mathrm{proj}\\_{V\\_k} (V\\_k - \\\\hat{V}\\_{k + 1})$. \n\t\tOnce the direction is found, the following steps correspond to vanilla Riemannian optimization techniques.", "questions": "- The mentioned quotient structure allows for decreasing the ambiguity of the \n\t\t\tchoice for $U$ and $V$ matrices, but according to the text such an approach \n\t\t\t\"is not designed for low-rank adaptation\". Could you clarify the details for the optimization method that you use in this case?\n- Did you try to use basic Riemannian optimization algorithms for \n\t\t\t$\\\\mathrm{St}\\\\, (r, m) \\\\times \\\\mathbb{R}^{r \\\\times r} \\\\times \\\\mathrm{St}\\\\, (r, n)$ and \n\t\t\t$(\\\\mathrm{St}\\\\, (r, m) \\\\times \\\\mathbb{R}^{r \\\\times r} \\\\times \\\\mathrm{St}\\\\, (r, n)) /(\\\\mathcal{O}(r) \n\t\t\t\\\\times \\\\mathcal{O}(r))$ without mixing them with the Euclidean optimizers?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n- The proposed method allows for using existing Euclidean optimizers, which \n\t\t\t\tis natively connected to the HuggingFace's PEFT library.  \n- The proposed method imposes stability due to the Stiefel manifold.\n\n\nWeaknesses\n- A single initialization strategy is used (no combinations with Pissa, etc). This might additionally boost the performance.\n- The method implies using additional computation overheads in the retraction process (which is common for all Riemannian approaches).", "was_revised": true, "final_justification": "The authors addressed my questions and concerns. The idea seems to be interesting and quite generally applicable.", "rebuttal": "We would like to thank the reviewer for the constructive comments and suggestions.\nWe have addressed all the comments and suggestions in the revised version of the manuscript.\nBelow we clarify the questions and concerns in detail.\n\n> A single initialization strategy is used (no combinations with Pissa, etc). This might additionally boost the performance.\n\nThanks for the suggestion.\nWe indeed have experimented with different initialization strategies, including the combination of StelLA with PiSSA and MiLoRA in the ablation studies.\nIn Table 5 of the main paper, the row marked as \"SVD-major\" is in fact StelLA + PiSSA, and the row marked as \"SVD-Minor\" is StelLA + MiLoRA.\nWe will clarify this matter in the revised version to avoid confusion.\n\nFor easier reference, we summarize the results in the table below.\n\n**Table: Comparison different initializations on the commonsense benchmark using LLaMA3-8B.**\n\n| Initialization | Accuracy (Avg.) | Comments on Initialization |\n| -------------- | --------------- | -------------------------- |\n| SVD-major      | 86.7            | StelLA + PiSSA             |\n| SVD-minor      | 86.6            | StelLA + MiLoRA            |\n| Non-zero       | 86.7            | StelLA                     |\n\nThe results show that StelLA is robust to different initialization strategies.\nThe subspace learning powered by Riemannian optimization can effectively learn the optimal subspace regardless of the initialization strategy (L312-313).\n\n> The method implies using additional computation overheads in the retraction process (which is common for all Riemannian approaches).\n\nThanks for raising the concern.\nThe retraction step is indeed computationally expensive, but with speed optimization including both algorithmic and hardware acceleration techniques, we can mitigate this issue and let StelLA be trained with only 15% additional training time compared to LoRA.\nPlease kindly refer to the appendix Sec. D and our response to Reviewer Qyxg for the detailed and thorough discussion of computational cost of StelLA.\n\n> The mentioned quotient structure allows for decreasing the ambiguity of the choice for U and V matrices, but according to the text such an approach \"is not designed for low-rank adaptation\". Could you clarify the details for the optimization method that you use in this case?\n\nThanks for the question, we will clarify this detail in the main text of the revised version.\nWe use Alg. 1 to optimize the quotient structure, same as in StelLA.\nThe difference between the quotient structure and StelLA is that the quotient structure is equipped with the Riemannian metric proposed in \"R3MC: A Riemannian three-factor algorithm for low-rank matrix completion\" [36].\nThis Riemannian metric is designed for the low-rank **matrix completion** problem, which is different from the **low-rank adaptation** (LoRA) problem.\nWe provided the details of R3MC's Riemannian metric in the appendix Sec. B.\nSpecifically, as noted in the appendix L34-39, the Riemannian metric is induced by the block approximation of the Hessian of the loss function in the low-rank matrix completion problem: $||USV^T-W||_F^2$.\nHence, we say that it is not designed for low-rank adaptation, because the Hessian of the loss function in low-rank adaptation is different from that in low-rank matrix completion.\n\nNevertheless, the quotient structure can be optimized using Alg. 1, as long as the three equations for\n- converting Euclidean gradients to Riemannian gradients (see [36, Eq. 9]),\n- projecting a vector onto the tangent space (see [36, Eq. 5]), and\n- retracting a point onto the manifold (see [36, Eq. 7]),\n\nare computed accordingly for R3MC's Riemannian metric.\n\n> Did you try to use basic Riemannian optimization algorithms for $\\mathrm{St}\\, (r, m) \\times \\mathbb{R}^{r \\times r} \\times \\mathrm{St}\\, (r, n)$ and $(\\mathrm{St}\\, (r, m) \\times \\mathbb{R}^{r \\times r} \\times \\mathrm{St}\\, (r, n)) /(\\mathcal{O}(r) \\times \\mathcal{O}(r))$ without mixing them with the Euclidean optimizers?\n\nYes, we have tried using the Riemannian Adam optimizer [6] on StelLA in the ablation studies.\nThe results are shown in the last row of Table 5, where the parallel transport is enabled.\nThe setting of StelLA + parallel transport is equivalent to using the basic Riemannian Adam optimizer on the product manifold $\\mathrm{St}\\, (r, m) \\times \\mathbb{R}^{r \\times r} \\times \\mathrm{St}\\, (r, n)$.\nWe will make this information explicit in the discussion of Table 5 in the revised manuscript."}, {"review_id": "vugNkaLYpA", "reviewer": "Reviewer_CC7z", "summary": "The paper introduces Stella,  an low-rank optimization framework designed for factorizations of type $USV^T$ with orthonormal $U,V$. \nThe proposed optimizer performs a Riemannian optimization on the Stiefel manifolds for U,V and a standard gradient descent step for S. \nThe authors report superior performance than LoRA, which uses standard gradient descent for USV, which was the declared motivation to design the manifold constraint optimization.", "questions": "1. Although the authors give empirical evidence for improved evaluation metrics for a given rank budget, it is not clear to me why the manifold constraint should yield this. There is evidence and analyses that the plain LoRA does not preserve first-order optimality on the manifold of rank r matrices, see [3]. \n[3] Constructs a method that also preserves orthogonality of U,V -  as a consequence first order optimality conditions and convergence criteria on the rank r manifold is available. \n However in the proposed work, the authors consider the stiefel manifolds of U and V seperately, which gives a different view on optimality and convergence. This should be discussed. \n\n[3] Schotthöfer et al., GeoLoRA: Geometric integration for parameter efficient fine-tuning, 2024.\n\n2.In the gradient scaling section: Why is the \"learning speed\" an issue? Your method considers the training of U,S,V in some sense individually, i.e. each object is optimized in their own manifold/space. An example where USV=W is optimized as an element of a rank r manifold is given, e.g. in [3]. \n\n3. It would be interesting to compare the convergence plots of LoRA, the proposed method and full-rank training using the same optimizer (e.g. Adam) with the same learning rates and hyperparameters. \nA simple test case could be compressing (not finetuning) the fully-connected layers (everything but key,query,value) of Vit-b/16 on Cifar100. This should train to convergence in about 500epochs, if you just use the pretrained imagenet weights, but converges slowly enough so that the difference between LoRa and your \"proper\" method should be visible. Finetuning of low-rank adapters may converge too fast to see the difference. Rank 50 should be a good initial guess. \n\n4. There are methods e.g. AdaLoRa or [3] that are rank adaptive. Can your method handle adaptiveness? \n\n5. Can your method be extended to tensor-valued layers, e.g. using low-rank Tucker format, see e.g. [4]\n\n[4] Zangrando et al., Geometry-aware training of factorized layers in tensor Tucker format, 2024.", "limitations": "Some limitations are well discussed, e.g. the computational cost of the retraction. Others need to be added, see questions.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "The paper gives a good introduction to manifold constraint optimization and refers to relevant literature for further research. \nThe method is clearly structured and the implementation is well described. \nProviding optimization methods that yield U,V from a Stiefel manifold is benefitial for certain downstream analyses, e.g. adversarial robustness [1,2] making this method relevant\n\n[1]  Savostianova et al., Robust low-rank training via approximate orthonormal constraints, Advances in Neural Information Processing Systems 36, 66064–66083, 2023.\n[2] Schotthöfer et al., Dynamical Low-Rank Compression of Neural Networks with Robustness under Adversarial Attacks, arXiv preprint arXiv:2505.08022, 2024.\n\n(These can be included in the introduction, but I leave that to the discretion of the authors, as it is a downstream application)\n\n\nSome questions and critiques can be found below:", "was_revised": true, "final_justification": "The authors have addressed my concerns and questions - and provided additional datapoints as requested. \nTherefore I have raised my score to 4 (Borderline accept)", "rebuttal": "Thanks for the in-depth review and constructive comments.\nWe have addressed all the comments and suggestions in the revised version of the manuscript.\nBelow we clarify the questions and concerns in detail.\n\n> Optimization methods that yield U,V from a Stiefel manifold is beneficial for certain downstream analyses, e.g. adversarial robustness [1,2]. These can be included in the introduction.\n\nThanks for the suggestion on potential applications.\nWe will add these references to the introduction section in the revised version.\nSpecifically, we will add the following sentences to L50:\n\n\"Besides, maintaining orthogonality of the low-rank matrices $U$ and $V$ during training is also beneficial for certain downstream analyses, e.g. adversarial robustness [1,2].\"\n\n\n> Although the authors give empirical evidence for improved evaluation metrics for a given rank budget, it is not clear to me why the manifold constraint should yield this. There is evidence and analyses that the plain LoRA does not preserve first-order optimality on the manifold of rank r matrices, see [3]. [3] Constructs a method that also preserves orthogonality of U,V - as a consequence first order optimality conditions and convergence criteria on the rank r manifold is available. However in the proposed work, the authors consider the Stiefel manifolds of U and V separately, which gives a different view on optimality and convergence. This should be discussed.\n\nThanks for the pointer to the work of GeoLoRA [3].\nwe will add GeoLoRA to the related work section and discuss the differences between GeoLoRA and StelLA.\n\nRegarding the convergence and optimality of StelLA, if we consider the tuple $(U, S, V)$ as a point on the product Riemannian manifold $\\mathrm{St}(r, m) \\times \\mathbb{R}^{r \\times r} \\times \\mathrm{St}(r, n)$, then the convergence can be analyzed using the Riemannian optimization theory, which is well-established in the literature.\nFor example, we refer the readers to [a] and [b] for the convergence analysis of Riemannian SGD and Riemannian Adam optimization algorithms.\nBecause of the product structure, the components $U$ and $V$ can be optimized separately, i.e., $U$ and $V$ are optimized on **their own respective Stiefel manifolds**, leading to our Alg. 1.\nThis treatment is mathematically equivalent to optimizing on the product manifold directly, thus preserving the convergence properties.\nTo avoid confusion, we would like to emphasize that all $U$s and $V$s in StelLA are optimized **jointly** in the same forward and backward pass, as described in Alg. 1.\n\nFurthermore, we would like to clarify that the product Riemannian manifold of StelLA is not the same as the manifold of rank-r matrices in GeoLoRA because each rank-r matrix has infinitely many representations in the product manifold due to the equivalence relationship by orthogonal groups (L287-289).\nNote that this does not affect the optimality because an optimal solution in the product manifold also corresponds to an optimal rank-r matrix.\n\n> In the gradient scaling section: Why is the \"learning speed\" an issue?\n\nThanks for the question on the motivation of gradient scaling, we will clarify this detail in the main text of the revised version.\nPrevious works such as LoRA+ [c] demonstrated that using the same learning rate for $A$ and $B$ matrices in LoRA does not allow efficient feature learning.\nThey showed that setting **different learning rates** for $A$ and $B$ with a fixed ratio can improve performance.\nThe core idea is to ensure that both $A$ and $B$ are efficiently updated.\nHowever, $U$ and $V$ in StelLA are constrained to be orthogonal matrices, while $A$ and $B$ in LoRA+ are unconstrained matrices, so the technique in LoRA+ cannot be directly applied to StelLA.\nMotivated by their insights, in StelLA, we ensure that both $U$ and $V$ are efficiently learned by balancing their effective learning speed via gradient scaling, achieving the same goal as LoRA+ but in a different way.\n\n> It would be interesting to compare the convergence plots of LoRA, the proposed method and full-rank training using the same optimizer (e.g. Adam) with the same learning rates and hyperparameters.\n\nThanks for the suggestion on the comparison.\nWe will add the convergence plots of StelLA, LoRA, and full-rank training in the appendix of the revised version.\n\n> A simple test case could be compressing (not finetuning) the fully-connected layers (everything but key,query,value) of Vit-b/16 on Cifar100. This should train to convergence in about 500epochs, if you just use the pretrained imagenet weights, but converges slowly enough so that the difference between LoRa and your \"proper\" method should be visible. Finetuning of low-rank adapters may converge too fast to see the difference. Rank 50 should be a good initial guess.\n\nThanks for the suggestion.\nUnfortunately, we do not have experience in the experiment setting of compressing and didn't find reference code for benchmarking.\nDue to the limited time, we are unable to implement this experiment.\nWe would like to kindly note that we have already conducted extensive experiments (and confirmed by Reviewer Qyxg) on the commonsense reasoning benchmark, math and code reasoning, image classification, and image generation benchmarks, which cover a wide range of tasks and models.\n\n> There are methods e.g. AdaLoRa or [3] that are rank adaptive. Can your method handle adaptiveness?\n\nThanks for the question regarding rank adaptiveness.\nYes, StelLA is compatible with AdaLoRA with minimal modifications.\nIn AdaLoRA, the adapter is represented as $U\\Lambda V^T$, where $U$ and $V$ are orthonormal matrices and $\\Lambda$ is a diagonal matrix.\nValues in $\\Lambda$ are used to control the rank allocation of the adapter.\nIn StelLA, we can also constrain the matrix $S$ to be diagonal by setting all off-diagonal elements of $S$ to be zero during training.\nWith this modification, StelLA can be used with AdaLoRA to adaptively adjust the rank of the adapter during training.\nHowever, it is hard to apply the rank adaptation approach in [3] to StelLA, because [3] needs expanding the rank of the adapter during training, which is not supported in StelLA (there is no rank augmentation step in StelLA).\n\n> Can your method be extended to tensor-valued layers, e.g. using low-rank Tucker format, see e.g. [4]\n\nThanks for the question regarding the extension of StelLA to tensor-valued layers.\nYes, StelLA can be extended to tensor-valued layers straightforwardly as follows.\nThe Tucker decomposition of a rank-$(r_1,\\dots r_d)$ tensor is written as $C\\times_1 U_1\\cdots \\times_d U_d$, where $U_i$ are column-orthonormal matrices.\nTherefore, the tuple $(C, U_1, \\dots, U_d)$ lies in the product Riemannian manifold of $\\mathbb{R}^{r_1\\times\\cdots r_d}\\times \\mathrm{St}(r_1, n_1)\\times \\cdots \\times \\mathrm{St}(r_d, n_d)$.\nWe can use the same approach as in StelLA to optimize the low-rank Tucker decomposition, because each component $U_i$ can be optimized in its own Stiefel manifold thanks to the product manifold structure.\n\n- [a] Bonnabel S. Stochastic gradient descent on Riemannian manifolds. IEEE Transactions on Automatic Control, 2013.\n- [b] Becigneul G, Ganea O E. Riemannian Adaptive Optimization Methods. ICLR 2019.\n- [c] Hayou S, Ghosh N, Yu B. Lora+: Efficient low rank adaptation of large models. ICML 2024."}]

Author rebuttal:

Meta-review: This paper introduces StelLA, a principled extension of LoRA that leverages a three-factor decomposition with orthonormal subspaces constrained on the Stiefel manifold. The work empirically demonstrates that these constraints ensures geometric consistency and stability of adaptation while remaining compatible with standard optimizers. Reviewers highlighted several strengths: (i) a clear and well-motivated novelty in constraining LoRA factors on the Stiefel manifold to address a limitation of prior works, (ii) a well-written, accessible presentation with sound algorithmic design, and (iii) experiments across diverse tasks (classification, language modeling, and text-to-image generation). Overall, the paper advances the field of parameter-efficient fine-tuning, and given its relevance to both research and practice, I recommend acceptance.
Decision: Accept (spotlight)