GoRA: Gradient-driven Adaptive Low Rank Adaptation
Authors: haonan he
Peng Ye
Yuchen Ren
yuan yuan
LuyangZhou
ShucunJu
lei chen
Venue: NeurIPS 2025 poster ()
Keywords: LLMs
LoRA
PEFT
Primary area: deep_learning
TL;DR: GoRA improves LoRA by adaptively assigning ranks and initializing weights using gradient info, achieving better performance while staying efficient. It outperforms LoRA and full fine-tuning on GLUE (+5.88) and GSM8k (+5.13) benchmarks.

Abstract: Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework—**GoRA** (**G**radient-driven Adaptive L**o**w **R**ank **A**daptation)—that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches—which often focus on either rank selection or initialization in isolation—but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers.

GoRA: Gradient-driven Adaptive Low Rank Adaptation
Haonan H.
Peng Ye Yuchen Ren Yuan Yuan Luyang Zhou', Shucun Ju', Lei Chen
UUniversity of Science and Technology of China Institute of Intelligent Machines, HFIPS, Chinese Academy of Sciences
3 Shanghai Artificial Intelligence Laboratory *Fudan University
The Chinese University Hong Kong University of Sydney
7 Anhui Disaster Warning & Agrometeorological Information Center hehn@mail ustc. edu. cn
Abstract
Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed improve performance by addressing one these aspects, they often compromise usability or computational efficiency. In this paper, we ana- lyze and identify the core limitations of existing approaches and propose a novel framework- GoRA (Gradient-drive) multaneously adapts both the rank and initialization strategy within unified framework. GoRA A leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive man- To our knowledge, GoRA is the first method that not only addresses the limitations prior approaches- initialization in isolation-but also unifies both aspects within single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-basee
Adaptive Low Rank Adaptation)-thar si-
often focus on either rank selection or
methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1- 1-8B--8BBBa for mathematical reasoning, GoRA achieves a 5. 1--pontt improvement over standard LoRA and even outper- forms full fine-tuning by 2.05 points under high-rank settings. Code is available http://ghwub..ooo/////ggggggaaa/ Transformerrs
Introduction
Open-source pre-trained large language models (LLMs) such as the Llama series 2 have demon- strated exceptional capabilities. Through supervised fine-tuning, these models can be adapted to various downstream tasks such as code generation [3] and mathematical problem solving [4] However, when the model has parameter size and uses FP16/BF16 mixed-precision training strategy [5] with the Adam optimizer [7], the parameters and gradients require 46 bytes of mem- ory, while the optimizer states require 126 bytes. Thus, the minimum memory usage, excluding activations, reaches 166 bytes. Such high memory demands limit the training of large language models under constrained resources. To reduce memory usage, Low-Rank Adaptation (LoRA) 8 decomposes the weight matrix W E Rmxn into W W, + AW Wa + sAB, where s is a scaling factor, and A Rmxx B E R="= , & min n)), shown in Figure a LoRA only updates the low-rank weights A and B, keeping the pre-trained weight Wn unchanged, thereby significantly reducing the memory footprint of optimizer states. Although LoRA performs well simple tasks, when applied to pre-trained large language models, its performance on more challenging tasks, such mathematical reasoning and code generation, still lags behind full fine-tuning
9 10
"Equal contribution.
Corresponding author: chenlei @ iim.ac.en.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

W
W
B
Ag o).
o
Miok
Allocating food every layers
Mesking relalively unimportant
(a) Vanilia LoRA
(b) Adaptive Rank Masking Methods
AgRo
An
W
W
B
Ag Nonzero, Bg Nonzero Allocating fixed for every layers
Ao o(),o"), - "",, Apll Allocating gradient- importance
(c) Nonzero Initialization Methods
(d) GoRA
Figure Illustration of (a) LoRA; (b) LoRA variants utilizing adaptive rank masking strategies; (c) LoRA variants employing nonzero initialization strategies; and (d) GoRA, which introduces adaptively leveraging the weight W's gradient to allocate the rank the low-rank adapter and initialize B. As and By denote the initialized value of A and B.
One the critical factors in LoRA is its rank. Kalajdzievski the rank of LoRA significantly improve performance when paired with appropriate scaling factor. However, a direct increase in rank leads to a substantial rise in memory requirement overhead, thus imposing constraints on rank selection. To address this, several studies [12] [3] propose to ensemble multiple low-rank subspaces, allowing for rank increases without proportionally increasing the number of trainable parameters. Nevertheless, these approaches often come the expense of usability due to their intrusion into architecture training processes. Another promising line of research explores adaptively assigning ranks to pre-trained weights based on importance. For example, AdaLoRA A [14) adaptively adjusts ranks by quantifying the importance of each rank during training and masking less significant ones, as illustrated in Figure ((b). However, this masking mechanism necessitates a larger parameter space (e.g., 3. 1.5 times), increasing the number of trainable parameters and limiting the upper bound of rank. Consequently, as demonstrated in Section [5.]] adaptively allocating ranks without significantly increasing the training cost remains an open challenge.
demonstrates that increasing
Another vital factor in LoRA is its initialization strategy. In vanilla LoRA, Ao initialized with a normal distribution (In the PEFT library, Ao is initialized with Kaiming distribution (15)) and Bo is initialized with zeros. This initialization method ensures that the weights Wo + Ao remain unchanged at the beginning of training. Besides, zero initialization is not the only option: When Bo is nonzero, manipulating the pre-trained weight by subtracting Bo from W, also ensures stability. Existing nonzero initialization methods can be categorized into experience-driven and data-driven methods. In experience-dree driven methods, PiSSA [66] and MiLoRA [7 employ decomposition techniques such as Singular Value Decomposition (SVD) to capture specific features of pre-trained weights. However, these methods are inherently agnossto, which limits their generalizability across diverse tasks. In contrast, data-driven methods incorporate task information. For example, LoRA-GA [18] uses the singular features of gradients initialize LoRA weights, minimizing the difference between LoRA and full fine-tuning. However, as illustrated in Figure (c) and Section [.2] existing nonzero methods require manipulating the pre-trained weights, resulting inference gap. Thus, designing nonzero initialization method without manipulating pre-trained weights remains an open problem.
in training-ifference
Given the challenges of adaptive rank allocation and nonzero initialization, we turn to gradients of pre-trained weights, which are crucial for assessing the importance of pre-trained weights and deeply related to LoRA adapters' optimization processes [19 20 As shown in Figure [ we propose GoRA. Specifically, before training, we compute gradients of pre-trained weights on a subset of training samples, using these gradients to assess the importance of each pre-trained weight. Given reference rank, we caleulate trainable parameter budget. Based on the normalized importance and the trainable parameter budget, we allocate trainable parameter count and corresponding rank for

---

each low-rank adapter, achieving adaptive rank allocation without significantly increasing the trainable parameter count compared LoRA and allowing for higher rank allocation upper bounds, as shown in Table 5 In GoRA's 's initialization, we maintain LoRA's initialization for A, while B is initialized using - (A An) Anj G, where G is the gradient of the weight W. This initialization ensures that the computation result of the low-rank adapter A Ao) AAj G & -G compresses the gradient optimally, setting solid foundation for further optimization without manipulating the pre-trained weight. Our key contributions are summarized as follows:
1. We conduct an in-depth investigation into LoRA's rank allocation and initialization strategy, uncovering the limitations of existing works. We propose GoRA, which achieves adaptive rank allocation and initialization without compromising usability and efficiency.
2. We use the gradients of weights to assess their importance and allocate ranks. We then initialize the low-rank weights using the pseudo-inverse of the compressed gradients, which enhances performance while ensuring training stability.
3. We conduct extensive experiments. demonstrating that GoRA consistently outperforms low-rank baselines and even rivals full fine-tuning in certain settings. For example, on the Llama3. 1-8B--8sss-- Base model fine-tuned for mathematical reasoning, GoRA achieves 5.13- point improvement over LoRA and even surpasses full fine-tuning high-rank configurations.
2 Related Works
2.1 Rank of LoRA
The choice of rank is crucial for LoRA, with higher ranks consistently yielding better outcomes [ However, increasing the rank raises the number of trainable parameters and corresponding memory usage overhead, making it challenging train with sufficient ranks limited hardware resources. Previous works [2 12 attempt to continuously merge and reinitialize low-rank weights during training to stack the overall rank. However, these methods often require resetting the states of the optimizer and learning rate scheduler during reinitialization to ensure that updates take place in distinct low-rank subspaces and ensure training stability, significantly increasing overall training complexity and making the training process unstable. MeL.RA (13) proposes aggregating multiple mini low-rank adapters diagonally increase the overall rank. Nevertheless, this approach requires modifying the structure of LoRA, limiting its usability.
At the same time, the significances of weights during training demonstrate heterogeneity, and an intuitive proposition is to assign larger ranks to relatively more important weights. Previous works [4 22 attempted to dynamically mask less important ranks during training to achieve adaptive rank adjusting. However, these methods require allocating larger matrices for low-rank adapters to reserve space for masked ranks, leading
increase in the number of trainable parameters, which compromises their operational efficacy and establishes limitations on the upper threshold of rank. IncreLoRA (23) introduces an approach that begins with single rank for each low-rank adapter and incrementally increases the rank during training. This method effectively addresses the challenge of large initial matrices. Nevertheless, this approach demonstrates suboptimal compatibility with distributed training architectures, notably FSDP[24] and ZeRO[25), which constitute essential infrastructural components for the effective training of large-scale models.
2.2 Initialization of LoRA
Parameter initialization represents a fundamental paradigm in deep learning methodologies. Well- established initialization protocols, such the strategy proposed by Xavier Glorot and Yoshua Bengio (26). facilitate the convergent training trajectories of deep neural networks. Similarly, appro- priate initialization strategies constitute a critical determinant for LoRA. Beyond zero initialization used by vanilla LoRA, some studies have explored different initialization strategies: PiSSA [I6] performs SVD on pre-trained weights and uses the most important singular features initialize low-rank weights; MiLoRA [17], in contrast to PiSSA, uses the least important singular features to initialize low-rank weights; similarly, OLoRA 27 uses QR decomposition of pre-trained weights to initialize low-rank weights; EVA 228 uses singular features of activations to initialize low-rank weights; and LoRA-GA 18] uses singular features of gradients to initialize low-rank weights. These methods can improve LoRA's performance to some extent.
3

---

Nevertheless, owing to the inherent zero initialization characteristics of these methodologies, they require subtracting the LoRA initialization results from the pre-trained weights to ensure correct forward and backward propagation during the initial phases of the training regimen, conse- quently creating a gap between training and inference. Recomputing the initialization result of these methods during inference is not feasible in cases involving randomness [16] or requiring original training data [18 28). And the initialization process requires significant time for methods such as MiLoRA 177. The most straightforward solution is to save not only the low-rank weights but also the manipulated pre-trained weights, but this sacrifices one of LoRA's significant advantages, namely, minimal checkpoint storage [29]. Another approach is to save the initialized LoRA weights and use block matrix multiplication to eliminate the gap, but this reduces usability.
Method
In this section, we will reinterpret LoRA adapters from the perspective of gradient compressors and introduce GoRA's gradient-driven adaptive rank allocation and initialization strategy.
3.1 View LoRA adapters as Gradient Compressors
The core idea of LoRA is to fine-tune a model by leveraging the intrinsic low-rank property of the update of weight matrix W e m=="" during training. Specifically, a pair of low-rank matrices A E mx and B E Rr** are initialized alongside the pre-trained weight Wu. During training, Wo remains frozen, while the model updated by training the low-rank matrices A and B, thereby reducing memory usage during training. For any training step t, the updated weight W, is given by ( where a is a tunable hyperparameter that ensures the scale of the LoRA computation depends only a and is independent of the rank r:
W, Wo AW; Wo A,
(1)
Specifically, given the training loss C, the gradient of the pre-trained weight Wo can be computed as dc Using the chain rule, the gradients of A and B are wWo in vanilla LoRA, Bo = 0.) Given a learning rate 17, the updates to the weight are as shown in
ac ac B, and A aW, respectively. (Note:
aW,
AB, 71
OWo AA, ー - Σ
OL.
T
OL,
B
(2)
aw
0
AW, A, Ao - LAAAAB -
Experimental results from LoRA-FA 30 have shown that freezing the randomly initialized matrix A and only training the matrix B can achieve performance close that of LoRA. When matrix A is frozen (AA, = 0), the weight update is given by (4). One can observe that matrix B accumulates the gradients compressed by A during training, and when multiplied by Ag. the compressed gradients are up-projected. Thus, the training process of LoRA-FA can be viewed as process of gradient accumulation and compression, with the compression matrix being the randomly initialized A.
AW, a -AAB - Σ AOA
T
OC. OW,
(4)
t=0
The update form of LoRA-FA provides significant inspiration. We hypothesize that vanilla LoRA has similar properties, i.e., LoRA adapters act as gradient compressors. Based on this hypothesis, we can allocate larger ranks to weights whose gradients and weights themselves contain more low-rank information and initialize LoRA parameters using compressed gradients.
3.2 GoRA's Adaptive Rank Allocation Strategy
Based on the hypothesis that LoRA adapters function similarly to gradient compressors, our adaptive rank allocation strategy aims to: (1) allocate rank based weight importance derived from pre- computed N-batch accumulated gradients G # Σn -i TWo before the formal training process; (2) complete rank allocation before training to avoid dynamic shape changes; (3) maintain a similar number of trainable parameters as LoRA (within 10%); and (4) preserve structural compatibility with LoRA for easy integration.
&CJ

---

To evaluate the importance of weights, we first consider the nuclear norm of the pre-computed gradient, which aggregates all singular values matrix and often used measure low-rank properties [I8]. However, as shown in T this metric does not effectively capture the importance of weights in practice. Instead, we adopt a sensitivity- importance metric commonly used in model pruning [3]] Specifically, we define the importance of weight matrix W as:
(5) where the operator @ denotes the Hadamard product and avg(-) computes the average value to yield scalar importance score. After computing the importance scores for all target pre-trained weight matrices, we form an importance set [) To facilitate adaptive rank allocation, we normalize the importance set to compute an advantage 0; for the i-th pre-trained weight W
I(W) avg ([W G
I(W))
(6)
E ((W)
With the normalized advantages computed, we next determine the total trainable parameter budget b for the model. Given reference rank yd the budget for single weight matrix W E m is estimated as:
b' = (vm + X ref
(7)
reflecting smoothed parameter budget under vanilla LoRA. Summing over all budgets, the total budget becomes = M.N and trainable parameter count p' for each weight matrix as:
b. D. Using this budget and the advantage a', we allocate the adapter rank yd
p
b a'
s.t.r S 5r' S 1 max
(8)
where the operator [.] denotes rounding to the nearest integer, and min y are hyper-parameter defining the allowable rank range. This formulation ensures that the total number of trainable parameters p = E ip' approximately matches that of standard LoRA with rank re closely.
In summary, before training begins, we use the N-batch accumulated gradients for all target weights These gradients are then used estimate the importance each weight matrix, based on which we perform adaptive rank allocation in GoRA, achieving all four objectives outlined earlier.
3.3 GoRA's Adaptive Initialization Strategy
Once ranks are allocated for each layer, it is crucial to initialize the low-rank weights properly. The compression form in (4) is suboptimal when A is randomly initialized and fixed; to achieve better alignment with the gradient dynamics, we can initialize B such that the computation of the low-rank adapter at the start of training closely approximates the N-batch accumulated gradient G. inverse of Ao (this computation
This optimal initialization can be derived using the requires negligible computational time as detailed in Appendix D.I]:
Bo -A, Ao) A AJ G, Bo Ao A, AA)-AA,G..
As shown in (9) initializing B as - -(A -A, G ensures that Bo provides the best low-rank approximation of G given fixed Ag, with proof provided in Appendix B.I
However, due to the properties pseudo-inverse computation, the magnitude of Ad does not exactly match that of G. Assuming both G E m and A Rm follow distributions with mean 0 and variance 1, the expected Frobenius norm of G, [[]]]]]]]] is /mn, while that of Ao B, || F]. is /rn, as detailed in Appendix B.2
To ensure that the initial computation of low-rank adapter approximates a single step of stochastic gradient descent with tunable step size 7, we introduce scaling factor & for Bo:
a V / G== Bo =
(10)
Thus, to make GoRA's initialization equivalent one step of gradient descent, & should be set to 2/y/mm Inspired by rsLoRA [T]] and to better utilize the larger ranks obtained through dynamic allocation, we modify the forward computation to W, W. + AW- = Wo of A, Br, which
adjusts & to 2/v/m Setting y to a relatively large value further improves performance and yields optimal results. The full algorithm is summarized in Algorithm D and Algorithm 2
5

---

Algorithm Rank Allocation and Initialization of GoRA under Single Training Worker
Input: Model f(-) with I. layers, pre-trained parameters 0 ( W gradient accumulation steps N, loss function F, scale factor Trainable parameter budget 6
Output: Initialized low-rank matrices {Aol (Bi]==
1: for | 1 to L do
2: G' avg f 0
Initialize gradients buffer in CPU memory.
3: i = to N do
Compute gradients without optimizer.
4: Randomly sampled mini-batch {x,
5:
ÿ < fzz 0), + F(v.))
6: for L do
7:
ac Accumulate on CPU: G' avg f G' avg + N aw;
8:
for 1 to I. do Compute importance / 7(W)) x- avg( W/ G avg
9:
10: for p=r-11 1 to L do
11:
Compute advantage a' Els ((W)
r(w))
12:
for = 1 to L do m, n f- size W
13:
14:
r' + -clip(round(
ba Vmn+n
min ).maa
Clip by m and ya to avoid extremum.
15:
A Ldkiiiing (m x v5)
Initialize Ab gmmxe' with Kaiming uniform.
16:
B/S -AA Ab) -1) G
> Initialize B/y E H x
B/b, 2 Bd
17:
18:
Return {Ab)=r BB
Experiments
We conducted comprehensive experiments comparing GoRA with baseline methods on natural lan- guage understanding (Section [.1)), generation tasks (Section 4.2) and image classification tasks (Section 4.3 For understanding tasks, we trained T5-Base
five tasks of GLUE 33 (MNLI, SST-2, CoLA, QNLI, MRPC) and reported accuracy on corresponding validation sets. For generation tasks, we fine-tuned Llama-3.1 3.1.---BBBs. [2] and Llama-2- 1aatma-2-7B-Baae Base [ on chat, mathematies, and cod- ing datasets, evaluating test performance on MTBench (34), GSM8k [35]. and HumanEval [36]. For image classification tasks, we fine-tuned CLIP-ViT-B/L B/1- 37 on seven datasets including Stanford- Cars (38], DTD (39) EuroSAT [4),, GT-SRB 41. RESISC45 [42] SUN397 [ and SVHN 44 and reported test accuracy. All experiments were conducted using single-epoch training across three random seeds, with results reported as mean values and standard deviations. Unless specified
otherwise, we set the LoRA rank or GoRA's reference rank re The hyperparameters of GoRA
are detailed in Appendix C.3
4.1 Experimental Results on Natural Language Understanding Tasks
Table 1: Performance fine-tuning T5-Bast on 5 sub-tasks of the GLUE benchmark. Bold and underline indicate the highest and second- -ighhsst scores of low-rank methods with or re 8.
Method
MNLI
SST-2
CoLA
ONLI
MRPC
Average
Full
0.00
74.71
80.70:10.24
93.19:0.22
84.56-10.73
87.91
LoRA
g
85.30+0.004
94.0440.11
69.35-40.05
92.96-40.09
68.38-40.01
82.08
Convergence Optimization Methods LoRA
IsLeRA !I
85 14.1..33
72.32+1.12
93.12-10.09
52.86:12.27
79.64
DoRA 45
85.67=10.09
94.04.1..33
72.04:10.4
93.04:10.06
08.05110.51
82.57
LoRA+
85.81
93.85-+0.24
77.53+0.20
93.14++0.03
1.34
84.95
Intrialization Optimization Methods for LoRA
PISSA [6
85.75:10.07
94.07-1.066
74.27-10.39
93.15:10.14
76.31:10.51
84.71
LoRA-GA
1B
85.70-0.09
94.1110.188
80.5710.20
93.18110006
85.29:10.24
87.77
Adaptive Methods for LoRA
AdaL.RRA 14
00.15
93.69
69.16-1.24
91.66-H0.05
68.14:+0.28
81.62
GoRA
8.091
94.68
79.86-10.35 93.27-10.08
86.10:10.20
87.96
Settings: We adopted baseline performances reported by LoRA-GA 18) maintaining their experi-
mental parameters for fair comparison: Adam optimizer (3) 0.9, B2 0.999, weight decay

---

0), batch size 32, cosine decay learning rate with a warmup ratioo of 0.03. We trained all linear layers except the language head using peak learning rate of le-4, a maximum sequence length of 128, and FP32 precision.
Results: Table compares GoRA against multiple baselines across five GLUE benchmark tasks. GoRA achieved superior performance on four datasets (MNLI, SST-2, QNLI, and MRPC), demon- strating exceptional adaptability and generalization. While slightly underperforming LoRA-GA on CoLA by just 0.71 percentage points, GoRA's average score (87.96) surpassed all baselines and even exceeded full fine-tuning (87.91). This confirms GoRA's ability to maximize model potential while maintaining parameter efficiency. Notably, GoRA showed particularly strong performance on MRPC and QNLI, highlighting its effectiveness in small-sample learning and sentence- pettence-pair tasks.
4.2 Experimental Results on Natural Language Generation Tasks
Settings: We trained mathematical, coding, and dialogue capabilities using 100K MetamathQA 47
100K Code-FeedBack [48] (code-only labels), and 52K WizardLM [49
subsets, respectively.
For experiments on Llama-3. 1-8B-base, training used Adam (50)
(31 0.9, 82 0.999,
weight decay = 5e - 4) with batch size 64, cosine decay learning rate (warmup ratio=0.03, de- cay ratio=0. 1), and BF16 mixed precision. For all methods, including GoRA, we trained attention modules' linear components with a peak learning rate of 5e-5 (5c-4 for AdaLoRA). Evaluation met- rics: mathematics- regex-extracted accuracy; coding PASS@ 1; dialogue- aveeraee scores (0-10) from GPT-4o [51] Gemini-L.5-PP [52]. and For experiments Llama-2-7B-Ban 2-seeBase, adopted baseline results from LoRA-GA (18), and maintained the same training and evaluation settings. Further details are provided in Appendix C.4
[2] using prompts from [34.
Table 2: Performance of fine-tuning Llama-3. 3.1 3.1-8B-Basc
Table 3: Performance of fine-tuning Llama-2-78 2-mma-2-BBBBse.
Method
MTBench
GSM8K
HumanEval
Method
BeBch
GSM8K
HumanEval
Full
5.88.0023
51.63-1.27
Full
5.30 1 0.11
59.36
1 0.35
35.31 2.13
LoRA
[
6.154-0.02
67.78+1.22
43.09-0.35
LoRA g
5.61 + 0.10
42.08
+ 0.04
14.76 0.17
LsLoRA
1
6.18::0.09
68 08.36:0077
48.78-2.80
rLLoRA
5.25 + 0.03
45.62
0.10
16.01
= 0.79
DoRA (45
6.244-0.12
69.1741100
1.57
DoRA E51
5.97 + 0.02
53.07
0.7.5
19.75
0.41
LoRA+ 40
6.35:10.10
71.29=0093
44.51=2.11 11
LoRA+ 46
5.71 + 0.08
62.11
18.17
0.52
OLoRA
6.13:10:01
08.5420.42
43.29=2.41
OLoRA 2
5.30
13.29
0.83
17.22
PISSA [6
6.08::0.09
68 06.56=1100 03
44.10=1.55 1.54
PISSA [6
5.30
0.02
44.54
+ 0.27
16.02
0.17
LoRA-GA
18
5.991:0.06
71.39=0.90
43.29=0.61
LoRA-GA
0.16
1.46
AdaLoRA
14
6.19.10.16
70.63.10.77
41.46-3.66
AdaLoRA 14
5.57 1 0.05
50.72
1 1.39
17.80
0.44
GoRA
6.34-10.04
2214
GoRA
54.04
GuRA,
6.21.10.10
75.59.11.04
$1.22-1.883
GoRA,
5.75 1: 0.06
56. 18
1: 0.10
26.83
2.84
GoRA,-
5.82-40.31
75.74-40.40
52.03-+1.41
Go
6.05
0.04
56.58
0.12
Results: Table 2 and Table 3 show the per-
MetaMothO@
formance GoRA and baseline methods on
0.7
fine-tuned Llama3. 1-8B-Base and Llama2-7B- Base. Specifically, GoRA demonstrated excep- tional performance the more challenging Hu- manEval and GSMSk benchmarks, substantially surpassing all baseline methods. For Llama3, 1- 8B-Base, on the GSM8k dataset, GoRA scored 72.91, outperforming LoRA-GA's 71.39 by 1.52 points; on the HumanEval dataset, GoRA achieved 48.98, surpassing rsLoRA's 45.78 by 3.20 points. On MTBench, GoRA slightly underperforms in terms of overall effective- ness, scoring 6.34, just 0.01 points lower than LoRA+'s 6.35. Notably, GoRA performed well across different rank allocation settings, For ex- ample, GoRA, achieved 75.74 and 52.03 on the GSM8k and HumanEval, respectively, surpassing full fine-tuning 's 73.69 and 51.63. Even the pe = 32 configuration of GoRA, while slightly underperforming re = 128, still outperformed full fine-tuning GSM8k. For Ilama2-7B-B Base, GoRA demonstrated similar
0.6
gatn
0.5
gota (tank]128]
0.3
0.2
300
1200
1500
CodeFeedlbcck
0.80
0.55
0.50
0.45
300
600
900
1200
1500
Figure 2: The training loss curves of full fine- tuning, LoRA, GoRA, and GoRA on Llama- 3.1-8B-Base. GoRA demonstrates lower start loss and faster convergence speed.
superior results compared to baseline methods. Especially, GoRA outperformed LoRA-GA by 4.99
7

---

and 0.44 on HumanEval and GSM8K, respectively. These results validate the effectiveness of GoRA across different LLMs and settings. The training loss curves of GoRA are depicted in Figure
Table 4: Performance of fine-tuning CLIP-VIP B/16 on 7 image classification tasks.
Method
Cars
EuroSAT
CTSRB
RESISC45
SUN,997
SVHN
Average
Zem--shat
63.75
44.39
42.22
35.22
56.46
62.56
15.53
45.73
Fall
84.23+00.06
77.440019
98.0940.03
94,31
93.95=0.00
75.35=0..11 10
93.04=0.15
88.06
LoRA 8
12.81=0.13
73.92:0.38
96.9341007
92.40=0.10
90.03=0.14
70.1240.18
88.0240.07
83,46
DoRA
[55
73.72:10.06
73.72:10.33
96.95:10.001
92.38:0.088
90.03-0.08
70.20:10.19
88.23:10.05
83.48
LoRA+ 44
72.87400.88
LoRA-Pro
53
85.87:10.08
78.64:00.85
98.46100.03
95.6610.05
94.75=0.21
76.42:10.14
94.63110.00
89.20
LoRA GA
85.18:10.41
77.50+10.12
98.05
95.28:00.10
94.43=0.119
0..44
93.68-10.35
88.51
GoRA
85.76++0.19
78.17:00.2
98.77:40.35
96.66:0.36
95.16-0.26
76.46-:0.08
95.32-10.13
89.47
4.3
Experimental Results Image Classification Tasks
Settings: We adopted baseline performances from LoRA-Pro (53, maintaining their experimental
hyper-parameters for fair comparison: Adam 7 optimizer (31 0.9, .3 0.999, weight decay 0), batch size 64, cosine decay learning rate with 0.03 warmup ratio. We trained all linear layers in the vision backend using peak learning rate of le-4, and FP32 precision. The classifier is obtained using prompts such as "a photo of a [class)."
As shown in Table GoRA outperforms baseline methods across all seven image classi- fication tasks. Specifically, GoRA outperforms full fine-tuning by margin of 1.01; outperforms LoRA-GA by 0.96 and outperforms LoRA-Pro by 0.27. These results demonstrate that GoRA exhibits superior performance across different models and modalities.
Results:
Discussions
In this section, present comprehensive of ablation studies to evaluate the effectiveness of GoRA's adaptive rank allocation and initialization strategy. We also examine the impact of GoRA's hyperparameters and diseuss their role in shaping performance. Additionally, explore hyperparameter auto-tuning strategies to improve usability.
31
91
0.00100
29
29
28
+0.0015
0.00073
21
21
17
20
17
19
15
0.005
17
19
0.00925
0.00000
15
19
-0.00050
-0.00075
090
wik
WVV
W0
Wiz
VV
160
39
U/k
WV
(a)
(b)
(c)
Figure 3: (a) Result rank distribution of fine-tuning Llama-3.1-888 1-8B-Base on the MetaMathQA- dataset using GoRA:(b) Difference values between GoRA and LoRA in directional updates of pre- trained weights after merging:(c) Difference values between GoRA and LoRA in magnitude updates of pre-trained weights after merging. Data points are presented for every two layers.
5.1 The Effect of the Rank Allocation Strategy
The rank allocation strategy is a crucial component that influences the performance of GoRA. As highlighted in Table conducted ablation studies to evaluate different rank allocation ranges. The results demonstrate that a broader rank allocation range consistently leads to superior performance. For instance, given 5e min 4, max 32) achieved score of 48.98 98 HumanEval, significantly outperforming both the fixed rank allocation strategy mm 8, max 8) and the more conservative allocation strategy min max 15).
Figure 3 illustratee the rank distribution of mi 4, m 32). Notably, most ranks are allocated to the wv layers, while the wq layers receive the fewest rank allocations. This observation aligns

---

with findings reported in prior work [8] Moreover, weights with higher ranks receive larger updates after merging the low-rank matrices. These observations underscore the effectiveness of our rank allocation strategy.
Table 5: Ablation study on hyperparameters. To maintain an approximately constant number of trainable parameters, the rank allocation upper bound was reduced the lower bound was increased.
Method
?
GSM8k
HumanEval
AdaLoRA
0
12
70.63:0.77
41,461:366
LoRA
8
8
0
67.78
43.09:10.35
GoRA
4
32
8e-2
72.91-10.76
16.54:11.54
GoRA
32
5c-2
02.88=+0.99
48.98=2.14
GoRA
4
32
3e-2
72.71+1.222
15.93-11.27
GoRA
32
0
72.45=1.14
46.34:0.61
GoRA
5e-2
72.83+0.80
46.1333.36
GoRA
4
32
5e-2
72.88:10.99
48.98
GoRA
6
15
5e-2
72.25:10.27
45.85
GoRA
5e-2
72.10+1.12
44.75+3.97
5.2 The Effect of the Initialization Strategy
Table also summarizes the results of ablation studies conducted with various scaling factors. Our experiments revealed that the choice of scaling factor y has a substantial impact on performance. Notably, GoRA achieves the best performance on HumanEval with y 5e 2. attaining score of 48.98. Meanwhile, GoRA with y & 2 slightly outperformed other configurations on the GSM8k, achieving a score of 72.91. Conversely, when 7 0, GoRA exhibited the weakest performance on GSM8k, scoring 72.45. A carefully selected scaling factor ensures that the initial low-rank adapter computation closely approximates a gradient descent step, establishing a robust foundation for subsequent optimization. This is crucial for maintaining training stability.
5.3 The Effect of Different Importance Metrics
Table compares several importance metrics: parameter sensitivity to loss (the metric used by GoRA), the nuclear norm of the gradient, and the nuclear norm of the parameter-gradient product. The results demonstrate that param- eter sensitivity consistently outperforms the other metrics on both GSM8k and HumanEval.
Table Ablation studies on different importance metrics, where || ||+ represents the nuclear norm.
Metric
GSM8K
HumanEval
avg W o GI)
72.88
10.99
48.9812.14
G|I. W + G||+
43.09 72.70+10.68
72.65:10.78
45.12::.117
Notably, on HumanEval, parameter sensitivity achieves a score of 48.9 98, surpassing the nuclear norm of the gradient (43.09) and the nuclear norm of the parameter-gradient product (45.
5.4 Hyperparameter Auto-tuning
Compared to vanilla LoRA, GoRA introduces two additional hyperparameters: the number of gradient accumulation steps N and the initialization scaling factor y. While hyperparameter tuning strategies for LoRA have been well established in prior work, we aim to minimize the additional tuning burden introduced by GoRA ensure its practical usability. To this end, we design lightweight, automated strategies that effectively eliminate the need for manual tuning of these new parameters:
(1) Adaptive Gradient Accumulation: During gradient accumulation, monitor the normalized layer-wise importance scores at each accumulation step. Once the change between consecutive importance score sets falls below a small pre-defined threshold (e.g.. 0.01), indicating convergence of layer importance, we terminate accumulation early and proceed to the parameter update. This adaptive stopping criterion automatically determines effective N the fly, removing the need for manual specification and often accelerating training.
(2) Adaptive Scaling Factor: We hypothesize that the optimal scaling factor 7 is the one that minimizes the loss of the first training step. To find it, we start from a relatively large initial value (e.g., 1.0) and iteratively reduce it by multiplying by decay factor (c.g., 0.9) until small lower bound (c.g., 5c-5)

---

is reached. For each candidate we evaluate the loss on the first training batch without performing backpropagation. The ? yielding the lowest loss is then used initialize formal training.
Importantly, the pre-defined constants used in these strategies (0.01), decay factor (0.9), and lower bound (5e-5) are empirically stable across tasks and model scales. In practice, they require little to no tuning, making GoRA highly usable out of the box while preserving its enhanced representational capacity.
as the convergence threshold
To validate the effectiveness and practicality of auto-tuning strategies, we conduct ablation studies on the scaling factor and the gradient accumulation steps N.. As shown in Table 7 our adaptive methods achieve performance com- parable or better than manually tuned base- lines, while eliminating the need for hyperpa- rameter search. Notably, the auto-selected ? values (0.0549 for CodeFeedback and 0.0858 for MetaMathQA) closely align with those reported in Table ( and 8e-2, respectively), confirming the reliability of our selection criterion.
Table 7: Performance comparison of GoRA with adaptive hyperparameters.
Method
GSM8k
HumanEval
Original
72.91+0.76
48.98+2.14
Adaptive N
72.96+t..19
46.85+2.11
Adaptive y
72.50+00.8
50.00++3.23
6
Computational and Memory Analysis
Table 8: Comparison of trainable parameter counts. For GoRA, we set min 4 and y max = 32
Model
Dataset
Target Modules
LoRA
AdaLoRA
GoRA
TS-Base
SST-2
all-linear
3.24M
4.86M
3.05M
Llama3. 1-8B-Base
MetamathQA
attention
6.82M
10.24M
7.00M
1lama2-7B-Bars
MetumathQA
all-lincar
19.99M
29.99M
20.18M
VIT--VTT-B/1
Cars
dision_modes
1.33M
2.03M
1.35M
During training, GoRA's computational and memory overhead is nearly iden- both
Table 9: Comparison of time and GPU memory costs during GoRA's initialization process and training process
tical to LoRA's, methods maintain a simi- lar number of trainable pa- rameters given reference rank. compares the trainable parameter counts of LoRA, AdaLoRA, and GoRA across various models, with all methods using rank or averagefreference rank of 8.
Model Size
Process Initialization
Time Cost
Memory Cost
23. 23.60s
56,139MB 73,295MB
8B
Training
37min54.222
Initialization
2min44.65s
64,821 MB 66,381 MB
32B
Training
197min52.15s
As outlined in Algorithm [ GoRA requires brief gradient pre-computing phase for rank allocation and initialization. However, this overhead is minimal in both computation and memory usage. First, gradient accumulation is performed over only small number of batcheb without any optimizer updates. Second, gradients are computed layer-by-laye and immediately offloaded to CPU memory, avoiding the storage of optimizer states. Moreover, Algorithm integrates seamlessly with distributed data Parallelism and can be readily extended to more complex parallel training setups. Specifically, rather than having each worker independently compute and transfer gradients to CPU memory. which would cause massive concurrent PCIe traffic and excessive CPU memory consumption, our method performs an immediate Reduce operation on per-layer gradients directly to rank 0 during the backward pass. This ensures that only single copy of the globally averaged gradient is ever transferred to CPU memory, drastically reducing PCIc bandwidth usage and CPU memory footprint.
To empirically validate the efficiency of GoRA, we measure both training time and memory consump- tion when fine-tuning Llama-3. -88-BBes and Qwen2.5-32B [54] on MetaMathQA using 8xA800 80GB GPUs. For Llama-3. 1-BBBBBBBBBBBBBB-- 1-8--Bass, we use GPr batch size of 8 (without activation check- point); for the larger Qwen2.5-332 model, we reduce the per-GPU batch size (with activation checkpoint (55]) due to memory constraints. We employ the adaptive N strategy described in Sec- tion 5.4 and all other experimental settings follow those of our main experiments. As shown in memories benefit from Liger kernel [56] and FlashAttention (57), GoRA incurs negligible additional time cost and no extra memory overhead compared to standard LoRA, even for models of vastly different scales, demonstrating its practicality for rorl-world deployment.
10

---

7 Acknowledgement
This paper is supported by National Key Research and Development Program of China (Grants No. 20.2222YFD200002 National Natural Science Foundation of China (Grants No.32471988, 92271981) and the HFIPS Director's s Fund (Grant No. Y2JJKX202401).

---

References
[1] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajiwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprini arXiv:2307.0922: 2023.
[2] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783 2024.
[3] Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, al. Code llama: Open foundation models for code. arXiv preprint arXiv:2308. 12950, 2023.
[4] An Yang, Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via seff-improvement. arXiv preprint arXiv:2409. /2/22, 2024.
[5] Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Gareia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. In International Conference on Learning Representations, 2018.
[6] Dhiraj Kalankar, Dheevatsa Mudigere. Naveen Mellempudi, Dipankar Das, Kunal Banerjee, Sasikanth Avancha, Dharma Teja Vooturi, Nataraj Jammalamadaka, Jianyu Huang, Hector Yuen, al. A study of 6 for deep learning training. arXiv preprint arXiv: 1905:1905.12 12322. 2019.
[7] Diederik P Kingma. Adam: A method for stochastic optimization. In International Conference on Learning Representations, 2015.
[8] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2022.
[9] Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. Transactions on Machine Learning Research, 2024.
[10] Sreyan Ghosh, Chandra Kiran Reddy Evuru, Sonal Kumar, Rananeswaran S, Deepali Aneja, Zeyu Jin, Ramani Duraiswami, and Dinesh Manocha. A closer look at the limitations of instruction tuning. In International Conference on Machine Learning, pages 15559-15588 PMLR, 2024.
[11] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora. arXiv preprint arXiv:2312.03732 2023.
[12] Vladislav Lialin, Sherin Muckatira, Namrata Shivagunde, and Anna Rumshisky, Relora: High-rank training through low-rank updates. In International Conference on Learning Repre- sentations, 2023.
[13] Pengjie Ren, Chengshun Shi, Shiguang Wu, Mengqi Zhang, Zhaochun Ren, Maarten Rijke, Zhumin Chen, and Jiahuan Pei. Melora: mini-ensemble low-rank adapters for parameter- efficient fine-tuning. In Proceedings of the 62nd Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Papers), pages
2024.
[14] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Pengcheng He, Yu Cheng. Weizhu Chen, and Tuo Zhao. Adaptive budget allocation for parameter-efficient fine-tuning. In International Conference Leurning Representations, 2023.
[15) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pages 1026-1034, 2015.
[16] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models. In Advances in Neural Information Processing Systems, volume 37, pages 12108- 2024.
[17] Hanqing Wang, Yixia Li, Shuo Wang, Guanhua Chen, and Yun Chen. Milora: Harnessing minor singular components for parameter-efficeent IIm finetuning. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational
Linguistics: Human Language Technologies (Volume 1: Longg Papers), pages 1823-4836, 2025.
12

---

[18] Shaowen Wang, Linxi Yu, and Jian Li. Lora-ga: ga: Low-rank adaptation with gradient approxima- tion. In Advances in Neural Information Processing Systems, volume 37, pages 54905-54931, 2024.
[19] Yongchang Hao, Yanshuai Cao, and Lili Mou. Flora: Low-rank adapters are secretly gradient compressors. In International Conference on Machine Learning, pages 17554-17571. PMLR, 2024.
[20] Jiawei Zhao, Zhenyu Zhang. Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficien Ilm training by gradient low-rank projection. In International Conference Machine Learning, pages
61143. PMLR, 2024.
[21] Xiangdi Meng, Damai Dai, Weiyao Luo, Zhe Yang, Shaoxiang Wu, Xiaochen Wang. Peiyi Wang, Qingxiu Dong, Liang Chen, and Zhifang Sui. Periodiclora: Breaking the low-rank bottleneck in lora optimization. arXiv preprint arXiv:2402.1614), 2024.
[22] Yahao Hu, Yifei Xie, Tianfeng Wang. Man Chen, and Zhisong Pan. Structure-awar low-rank adaptation for purameter-efficient fine-tuming. Mathematics, 11(20):4317, 2023.
[23] Feiyu Zhang, Liangzhi Li, Junhao Chen, Zhouqiang Jiang, Bowen Wang, and Yiming Qian. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning. arXiv preprint arXiv:2308. 12043, 2023.
[24] Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien- Chiin-Chinnue Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch fsdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304. /2277, 2023.
[25] Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimiza- tions toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1-16, 2020.
[26] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, 2010.
[27] Kerim Büyükakküüz. Olora: Orthonormal low-rank adaptation of large language models. arXiv preprint arXiv:2400..11775 2024.
[28] Fabian Paischer, Lukas Hauzenbergger, Thomas Schmied, Benedikt Alkin, Mare Peter Deisenroth, and Sepp Hochreiter. One initialization to rule them all: Fine-tuning via explained variance adaptation. arXiv preprint 2r1.v:2410.07170, 2024.
[29] Vlad Fomenko, Han Yu, Jongho Lee, Stanley Hsieh, and Weizhu Chen. A note on lora. arXiv preprint arXiv:2404.05008 050866 2024.
[30] Longteng Zhang, Lin Zhang, Shaohuai Shi, Xiaowen Chu, and Bo Li. Lora-fa: Memory-efficient low-rank adaptation for large language models fine-tuning, arXiv preprint arXiv:2308.0330; 2023.
[31] Qingru Zhang, Simiao Zuo, Chen Liang, Alexander Bukharin, Pengcheng He, Weizhu Chen, and Tuo Zhao. Platon: Pruning large transformer models with upper confidence bound of weight importance. International Conference on Machine Learning, pages 26809-26823, PMLR, 2022.
[32] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020.
[33] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019.
[34] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. In Advances in Neural Information Processing Systems, volume 36, pages 46595-46623, 2023.
[35] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiy preprint arXiv:2110.14166 2021.
13

---

[36]
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv: (tti::2107.0333: 2021.
[37] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning.
pages 1748-8765 PMLR, 2021.
[38] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 554-561, 2013.
[39] Mircea Cimpoi, Subhransu Maji, Iasonas Kokkinos, Sammy Mohamed, and Andrea Vedaldi. Describing textures in the wild. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 3606-3613, 2014.
[40] Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-2226, 2019.
[41] Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs real-world images: The german traffic sign detection benchmark. In International Join! Conference Neural Networks, 2013.
[42] Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1855-188,, 2017.
[43] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In 2010 JEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485-3492, 2010.
[44] Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Andrew Y Ng, et al. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[45] Shih- Shih-Yang Liu, Chien-Yi Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decompossed low-rank adaptation. In International Conference on Machine Learning, pages 32100-32121 PMLR, 2024.
[46] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+ efficient low rank adaptation of large models. In International Conference on Machine Learning, pages 17783-17806. PMLR, 2024.
[47] Longhui Yu, Weisen Jiang, Han Shi, Jincheng YU, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In International Conference on Learning Representations, 2024.
[48] Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreeer:: Integrating code generation with execution and refinement. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1 12834-1288 2024.
[49] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin. and Daxin Jiang. WizardIm: Empowering large pre-trained language models to follow complex instructions. In International Conference on Learning Representations, 2024.
[50] Uyya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[51] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidr, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
[52] Gemini Team, Petko Georgiev. Ving Ian Lei, Ryan Burnell, Libin Bai, Anmol Gulati, Garrett Tanzer, Damien Vincent, Zhufeng Pan, Shibo Wang, al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.0553. 2024.
[53) Zhengbo Wang, Jian Liang, Ran He, Zilei Wang, and Tieniu Tan. Lora-pro: Are low-rank adapters properly optimized? In International Conference Learning Representations, 2025.
14

---

[54] An Yang, Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang. Jianhong Tu, Jianwei Zhang. Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2025.
[55] Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. arXiv preprint /604. 2016.
[56] Pin-Lun Hsu, Yun Dai, Vignesh Kothapalli, Qingquan Song, Shao Tang, Siyu Zhu, Steven Shimizu, Shivam Sahni, Haowen Ning, and Yanning Chen. Liger kernel: Efficient triton kernels for IIm training. arXiv preprint arXiv:2410.1009 2024.
[57] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in Neural Information Processing Systems, volume 35, pages 06344-16355 2022.
[58] Tri Dao. Flashattention-2: Faster attention with better parallelism and work partitioning. In International Conference on Learning Representations, 2024.
[59] Kai Lv, Hang Yan, Qipeng Guo, Haijun Lv, and Xipeng Qiu. Adalomo: Low-memory optimiza-
tion with adaptive learning rate. In Findings of the Association for Computational Linguistics:
ACL 2024, pages 12486-12502. 2024.
15

---

A Notations
In this section, we summarize the notations used in the paper in Table I0
Table 10: List of Notations used in the paper
Symbol
Description
W E R
Full-rank weight matrix of a linear layer. Pre-trained weight matrix of linear layer in pre-trained model.
Wo E R
A AW& E Ruxn
Update of the pre-trained weight matrix after fine-tuning. Update of the pre-trained weight matrix at fine-tuning step Weight matrix of a linear layer at training step (W, = Wo+AW))
AW, E Rmxn
W E R
DEL
E R
Gradient matrix of the pre-trained weight Wo at step t.
UW
G gmxn
N-batch accumulated gradient matrix of the pre-trained weight Wo- Trainable low-rank matrices of LoRA adapter. Initial values of the low-rank matrices A and B. Trainable low-rank matrices of LoRA adapter step Input dimension of the matrix W.
B R Ao E Rmaxr
A E Rmxr
Bp E Rrxa
A, maxr
B, R
Output dimension of the matrix W. Rank of the low-rank adapter, with & min(m, n). Pre-defined maximum rank of GoRA adapter.
mux
min
Pre-defined minimum rank of a GoRA adapter.
ref
Reference rank of GoRA.
Hyperparameter LoRA and most of its variants.
y
Scaling hyperparameter of GoRA.
Scaling factor GoRA, automatically determined by 7. Kaiming uniform distribution. Learning rate used the optimizer (e.g., AdamW). Hadamard (element- wise) product of two matrices. Rounding to the nearest integer. Frobenius norm of a matrix.
L 4kaiming
7
F-]
F
Nuclear norm of a matrix. Average operation (element-wise for matrix).
avg(.)
B Proofs
B.1
Proof of optimal approximation of G given A.
Let G be the projection formula that minimizes the Frobenius norm of the error ||G G where G the optimal approximation of G in the column space of A, denoted as Col(A).
matrix, and A be
matrix where < min(m,n) n). We aim to derive
The best approximation G lies in Col(A), so we can express G as:
G = AB,
where B an r matrix of coefficients to be determined. Our goal is to find such that the error ||G - G||F minimized.
The error matrix is given by:
E G AB.
To minimize EE we take the derivative of | with respect to B and set it to zero. Expanding |
we have:
B Tr ABG (G -AB)),
where Tr represents the trace of matrix.
Expanding this expression:
E 2 Tr(G G) 2B
( +T A AB).
16

---

Taking the derivative with respect to B and setting it to zero:
-2A G 2A AB 0.
Simplifying:
A A A AB
Assuming A A is invertible, solve for B:
B (A ( ATG.
Substituting B into G AB, we get:
G A(A
Thus, the best approximation G is:
G = A(A'A)-AATG:
The matrix G A(ATA)-IATG is the projection of G onto the column space A, and minimizes the Frobenius norm of the error |G Gl]F.
B.2 Proof of Expectation of Frobenius Norm of AB.
Let A be a random Gaussian matrix of size m r, where each element of A is sampled independently from (0, 1). Let G be a random Gaussian matrix of size m n. where each element G is also sampled independently from N 1). Define:
B (ATA)-
and consider the product:
AB = A(A A) G. The goal is to compute the expected Frobenius norm |||| AB p]. where the Frobenius norm is defined
|B
(AB)?
1.j
First, observe that AB can be rewritten as:
AB A(A A)-LAGG.
Let = A(ATA)-IA- Note that P is projection matrix onto the column space of A, and thus P satisfies:
p P, P = P, and rank(P)
Substituting P into the expression for AB, we have:
AB PG.
The Frobenius norm of AB is given by:
|B
| PG |3 (T((G))PG))).
Since (PG) G TP, this becomes:
|B TTPGG
The matrix GGT an random Wishart matrix. When G is a standard Gaussian matrix of size m x n, the expected value of GG is:
E G n Ims
17

---

where Im is the m x m identity matrix. Substituting this result into the expression for |BB [p we
get:
E B I ETTT(PG P)] Tr(PEGGT
Using B|GG n Im this simplifies to:
E | |+]]
(r(P(m P) n Tr(P²).
Since p P. we have:
B (r(P).
The trace of P is equal its rank, which is the dimension of the column space of A. Since A is m x r matrix, we have:
Tr Tr(P) =..
Thus:
E AB
Taking the square root, the expected Frobenius norm of AB is:
B|||AB| F] vn-r.
C
Implementation Details
C.1 Baseline Methods
We compared GoRA with baseline methods to demonstrate the effectiveness of our approach:
a. Full: Trains all parameters the target layers, resulting in the highest memory consumption.
b. LoRA [8]: Introduces low-rank adapters into the target layers, significantly reducing the number of trainable parameters.
Convergence Optimization Methods for LoRA
-rsLoRA MT] Modifies the scaling factor in LoRA from to بे enabling better perfor- mance with higher-rank adapters and stabilizing the training processes.
Decomposes the weight updates of pre-trained weights into magnitude and direction components, and applies LoRA to update only the direction.
-DoRA 45
Addresses the imbalance between matrices and B in LoRA by assigning relatively larger learning rate to matrix B than to matrix A.
-LoRA+ 46
d. Initialization Optimization Methods for LoRA
OLoRA [27]: Initializes LoRA weights using the QR decomposition the corresponding pre-trained weights.
PiSSA [66] Initializes LoRA weights based the dominant singular vectors obtained from the SVD of pre-trained weights.
LoRA-GA [18]: Initializes LoRA weights using significant singular vectors derived from the SVD of gradients pre-trained weights.
e. Adaptive Methods for LoRA
AdaLoRA [4] Approximates the low-rank adapter structure using SVD, enabling dy- namic rank allocation through singular value masking. It also introduces an orthogonal regularization term to the loss function for enhancing orthogonality among features in the low-rank adapter.
C.2 Implementation Details for Baseline Methods
Several baseline methods introduce tunable hyperparameters compared with vanilla LoRA 8] To
ensure fair comparison, we adopt the optimal settings reported in the original papers whenever possible. Specifically, for LoRA+ [46]. we set the learning rate ratio of matrices A and B to 16. For LoRA-GA [18) we use the "stable" scaling method (the scaling hyperparameter 7 LoRA-GA is configured to 16) and manipulate the pre-trained weights during initialization. For AdaLoRA 14]. the initial rank is set to 12, the final rank to 8, with 4 150 and i 900. For PiSSA 16 number of iterations for fast SVD is set to 64.
18

---

C.3 Implementation Details for GoRA
For all experiments with r 8, except for the model trained on MetaMathQA [47]. we set the scaling factor ? to 5e 2. For the model trained on MetaMathQA, y is set 8e 2. For all experiments with r re = 32, the scaling factor is set to le - 2: and for r"ef = 128, we set the scaling factor 5e 3. This is because we observe that more gradient information is compressed by GoRA's initialization with higher rank, even if y can control the magnitude of the initialization results. To address the imbalance GoRA's matrices A and B, we set the learning rate of matrix to be 16 times that of matrix A Throughout the experiments, the mmar was empirically defined as 4 x rrej, the Twin was set to rref/2 /2 (as this setting can maintain a comparable parameter count compared to LoRA), and the gradient accumulation step for GoRA's initialization set to 64. In the ablation studies, we adhered to the same hyperparameter settings as the main experiments, unless otherwise specified.
Furthermore, Algorithm I presents a basic algorithm for non-parallel scenarios; however, parallel training strategies, particularly data parallelism, are widely employed for training large language models. We introduce the GoRA algorithm under distributed data parallelism in Algorithm 2 which is readily integrable into more complex parallel frameworks.
Algorithm 2 Rank Allocation and Initialization of GoRA under Distributed Data Parallelism
Input: Number of layers L, gradient accumulation steps N, model parameters 0 (W))2>>
current data parallel worker ID w, total data parallel workers W
Output: Initialized low-rank matrices {Ap] (B 1B
1: Initialize empty CPU buffers (G' avg } only worker
2: for i 1 to do
3: Sample mini-batch and compute loss C
4: for 1 to L do
5: Compute local gradient on GPU: G¹
8C OW!
6: if' w = 0 then
7: Accumulate CPU: G avg
G¹ avg N
G
8:
Release the GPU memory occupied by G:: G2 <- None
9:
if 0 then forl=1 L. do
10:
11:
Compute the importance 7 7(W)) detailed in Algorithm |
12: Broadcast the importance set {/(W))) to all other workers
13: else
14:
Receive (I(W)))/ from worker
15: for / = 1 to do
16:
if w then
17:
Load the gradient G from CPU to GPU Initialize A/. ByJ as detailed in Algorithm Clear the GPU memory occupied by G Broadcast initialized A/, By to all other workers
20:
21:
else
22:
Receive initialized A/, B/, from worker
23:
Return (Ab) (B))
C.4 Hyperparameters for Each Experiment
The hyperparameters used each experiment are summarized in Table For experiments on T5-Base and Llama2-7B-BBl -Bsee,, we adopt the settings from LoRA-GA 18 for CLIP-ViT-B/1C we follow LoRA-Pro 53 For experiments on GoRA,
including both baseline methods and of the most commonly adopted hyperparameter configurations.
19

---

Table 11: Hyperparameters used in experiments
Model
LR
LR Decay
Warmup
Optimizer
Betas
Weight Decay
Batch Size
T5-Base
le-4
0.3
Adam
0.9. 0.999
0
32
Llama3. Baatat3.---BBBs--
5e-5
0.1
0.3
AdamW
0.9. 0.999
5e4
64
Llama2-7B-Ban
2e-5
0
0.3
AdamW
0.9. 0.999
0
32
CLIP-ViT-B/16
le-4
0.3
Adam
0.9, 0.999
0 64
C.5
Training Environments
For natural language understanding tasks reported in section 4.1
conduct experiments using
the Huggingface Transformers framework for model and trainer implementation on a single RTX 4090 24GB. In contrast, for natural language generation tasks reported in Section 4.2 and Section 5 we utilize the DeepSpeed ZeRO2 [25] data parallel framework and FlashAttention-2 [58] mechanism, leveraging the power of 8 RTX 4090 24 GB GPUs 8 A800 80 GB GPUs. All codes of GoRA and baseline methods are implemented in PyTorch
D
Addtional Experiments
D.1
Computational Overhead of Pseudo-Inverse Initialization
The initialization strategy of GoRA involves computing a pseudo-inverse for each low-rank adapter matrix. Although the theoretical cost of pseudo-inversion scales cubically with rank (O(r)),, our implementation performs all operations on GPUs using optimized linear algebra routines, resulting in minimal practical overhead.
We benchmark the total initialization time for the Llama-3. 3.1 3.B-8B-JB 1--8B-Base model across different ranks following the settings of our main experiments. Results summarized in Table 12, Even at rank 128, where each pseudo-inverse requires approximately million FLOOs, the entire initialization completes in under 4 seconds. This amounts less than 0.1% of typical fine-tuning runtime, confirming that the pseudo-inverse step does not constitute computational bottleneck in practice.
Table 12: Pseudo-inverse initialization time for Llama-3-88 3-8B-Base (GPU, end-to-end).
Rank
Initialization Time
1.40s
1.56s
128
3.43s
D.2 Combining GoRA with QLoRA
To investigate whether GoRA can be effectively integrated with quantization techniques, we adopt the quantization method from QLoRA, which quantizes pre-trained weights to NF4 precision for storage and dequantizes them back to BF16 precision during computation. We evaluate this combined approach, which we term QGoRA, by fine-tuning the Llama-3. 1-8--Bas model and assessing its performance on math and code benchmarks, following the same protocol as our main experiments. As shown in Table 13 GoRA integrates seamlessly with QLoRA's quantization framework and consistently outperforms QLoRA across evaluated tasks.
Table 13: Performance of QGoRA and QLORA.
Method QLoRA
GSM8k
HumanEval
65.10 + 0.95
43.49 = 0.70
QGoRA 70.58 + 1.33
44.10 = 3.68
D.3 Gradient Reconstruction Accuracy of GoRA Initialization
To assess how well the GoRA initialization approximates the pre-computed gradients, we measure the reconstruction error between the initialized low-rank adapters and the full gradients across all adapted layers in the Llama-3. 1-1--88--Bas model. Results on two datasets are summarized in Table [4] These results suggest that the initialization preserves high proportion of gradient information (approximately 88-89% in relative terms).
20

---

Table 14: Gradient reconstruction error of GoRA initialization.
Dataset
Absolute Error
Relative Error
MetaMathQA
0.1147 0.0382 0.0322
Code-Feedback
0.1152
D.4 Multi-Step Gradient Initialization
We explored an alternative initialization variant, GoRA-pro, which estimates layer importance using multi-step stochastic gradients. In this approach, lightweight pre-training phase performs exploratory updates per layer using the AdaLomo [59] optimizer; gradients are accumulated on CPU and then discarded, and the original weights are restored before adapter initialization. This process uses approximately 23% less GPU memory than full fine-tuning (56.2 GB vs. 73.0 GB for 1--na-..1-B--ase
We evaluate GoRA-pro on GSM8K and HumanEval, comparing it to the standard GoRA (which one-step gradient accumulation). Results shown in Table 15] The comparable performance suggests that the multi-step gradient idea is viable and may offer further improvements with careful tuning of the exploration schedule, gradient normalization. We leave a systematic investigation of this direction to future work.
Table 15: Comparison of GoRA and GoRA-pu pro (Llama-3-88) 1-8B-Base).
Method
GSM8K
HumanEval
GoRA
72.91 + 0.76
48.98 2.14
GoRA-pro
72.30 + 0.30
47.36 = 1.54
E
Clarifications
E.1
Clarification on the training-inferend gap introduced by previous initialization methods
This is due to their reliance on manipulating pre-trained weights. Specifically:
Manipulation of Pre-Trained Weights: These methods are required to manipulate the value of pre-trained weights during initialization, A, * 0 and Wo Ag * Wo. As a result, during inference, the term Ao must be recomputed to properly reconstruct the adapted weight matrix for effective model deployment, which essential for correct model outputs.
During inference, is often infeasible to recompute the initialization results for methods that either rely randomness, such PiSSA(16) or require access to training data, such as LoRA-GA [8] and [VA Furthermore, for approaches like MiLoRA (17), the initialization process itself can be computationally expensive and time-consuming.
The Inconvenience of Recalculating Initialization Results:
Incompatibility Across Multiple Adapters: When multiple adapters are trained on different tasks using previous data-driven non-zero initialization methods, the pre-trained weights manipulated inconsistently. As the result of B, depends on the task. This makes challenging to serve multiple adapters simultaneously. limiting flexibility in multi-task scenarios.
Saving manipulated pre-trained weights sacrifices one of the key advantages of LoRA: While it is possible to merge the low-rank adapter weights into the pre-trained weights after training, saving the pre-trained weights post-merging sacrifices one the key advantages of LoRA: minimal storage requirements (e.g., 10MB compared to 14GB). Other potential approaches to eliminate the training-infeon inferrncceeeeee gap and their limitations are discussed Section 2.2
E.2
Compare GoRA's initialization strategy with LoRA-GA
Both GoRA and LoRA-GA leverage gradients to initialize low-rank adapters, but there several key differences:
1. Motivation:
LLRAAGA: Minimizes the difference updates of weights between LoRA and full fine-tuning.
21

---

GoRA: Views LoRA adapters, gradient compressors, and optimizes the compression form.
2. Scaling factor:
LoRA-GA: Inspired by rsLoRA, its scaling aims to stabilize training.
GoRA:: Scale the initialized product of adapters any desired magnitude flexibly.
3. Methodology:
LoRA-GA: Initialize the weights using SVD decomposed gradients.
GoRA: Initialize the weights of B using gradients compressed by pscudo-inverse of random initialized A.
E,3 Further compare with previous related works
Building upon previous works, GoRA makes several unique contributions:
1. First data-driven initialization method without manipulating pre-trained weights.
2. Efficient rank allocation strategy without additional trainable parameters and training com- plexity.
3. Unified framework for gradient-driven rank allocation and initialization.
F Limitations And Future Works
In this study, we demonstrate that GoRA outperforms baseline low-rank adaptation methods and achieves performance comparable to full fine-tuning. However, our evaluation has not yet extended to larger models and more extensive datasets. We hypothesize that for larger models, such as Llama- 3.1-70B (2). GoRA could more effectively leverage the pre-trained knowledge inherent in these models. Additionally, while this research primarily focuses on language models and natural language processing tasks, there is potential to generalize GoRA to a broader range of model types and tasks, such as visual language models and visual question answering.
Another limitation of this study is that the initialization of matrix A is not restricted to random initialization. Employing alternative methods, such as extracting distinguishing features from pre- trained weights to initialize the matrix A, could potentially enhance performance, the benefits of both experience-drive and data-driven initialization approaches. Furthermore, is worth noting that GoRA demonstrates theoretical compatibility with other LoRA variants, such as DoRA 455.
would combine
These promising avenues remain to be explored future research endeavors.
22

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Yes, and we believe GoRA can accelerate the research on LoRA.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have concluded our limitations in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
23

---

Justification:
Yes, we have provided proofs in the Appendix B
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Yes, we have provided experiment settings and our algorithm in the main paper and the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
24

---

Answer: [Yes]
Justification: Yes, we have provided the link to our official repository in the abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
http:///cpplicca..aaaaooo//eaaaa / Soaileesss/oSSSeeeiieeeeee
seipaiisssooooiooyyo. for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We have provided hyperparameter information in Section 4.1 Section 4.2 and
Section 4.3 Additional information is provided in Appendix C.4
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
Justification: We run each experiment with 3 different random seeds and report the mean and the standard deviation.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
25

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Yes, we have provided related information in the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:///arrr.....cppl////aaaaaaa
Answer: [Yes]
Justification: Yes.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: There is no societal impact of the work performed.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
26

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer:
Justification:
No risk of misuse in this work.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer:
Justification: Yes.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
27

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
Justification:
No new assets introduced in the paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
28

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA
that the method development in this research does involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should be described.
29


Reviews:
[{"review_id": "QqNWbEmzil", "reviewer": "Reviewer_4NWP", "summary": "The paper proposes **GoRA**, a new method for setting Initialization and Rank in LoRA finetuning. GoRA allocates LoRA ranks before training by scoring each weight matrix with a gradient-based importance metric, and derives a data-driven, non-zero initialization for the B matrix. Empirically, GoRA seems to improve over LoRA, LoRA+, AdaLoRA, and several other baselines on language (GLUE, MT-Bench, GSM8K, HumanEval) and vision while keeping trainable parameters and GPU memory similar to that of LoRA.", "questions": "See above", "limitations": "NA", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "### I. Strengths\n\n- **Unified treatment of rank and initialization** : GoRA re-uses the same gradient statistics to decide both how many LoRA ranks each layer receives and how the adapters are initialized. This avoids the masking overhead of AdaLoRA-style methods.\n\n- **Robust gains across modalities** The reported result show that GoRA wins or competes with the strongest baseline on most tasks tested. With \\(r_{\\max}=128\\) it even surpasses full fine-tuning on GSM8K and HumanEval, demonstrating that gradient-driven rank allocation scales to both language understanding and generation as well as vision classification.\n\n- **Ablations**:   The paper provides ablations results on the effect of rank bounds, alternative importance metrics, etc. These ablations clarify why the chosen design works and how sensitive it is to hyper-parameters.\n\n\n### II. Weaknesses\n\n- **Compute start-up cost**: GoRA always requires N full forward-and-backward passes *before* training starts. While negligible for multi-hour jobs, this warm-up can dominate short experiments or hyper-parameter sweeps. Could N be selected adaptively—e.g., stop once the layer-wise importance estimates converge?\n\n- **CPU off-load bottleneck**: Copying every gradient tensor to host memory each step may saturate PCIe on multi-GPU or FSDP setups. The paper shows single-GPU numbers only; bandwidth and wall-clock analyses for 2- and 8-GPU nodes would clarify scalability.\n\n- **Host RAM usage**: Off-loading stores one fp32 copy of all targeted weights (about 20 GB for a 70 B model). Techniques such as bf16 accumulation or on-the-fly reduction could cap memory, but these are not discussed.\n\n- **Hyperparameter sensitivity.**  GoRA introduces at least 4 other HPs $r_{min}, r_{max}, \\gamma, N$. Performance can shift by ~3 accuracy points when the scaling factor $\\gamma$ moves from 0 to 0.08. It is unclear how to set $\\gamma\\$ without an exhaustive grid search. \n\n-  **Pseudo-inverse cost**: Unless I didn't see it, the authors do not discuss the compute cost of pseudo inverse. The initialization step requires computing $\\bigl(A^{\\mathsf T}A\\bigr)^{-1}$ for *every* adapter. Although the matrix to invert is only $r \\times r$, the cost scales cubically with rank ($\\mathcal O(r^{3})$) *and linearly with the number of insertion points* (six per transformer block).  *For example*: on an 8 B model with 32 layers and six LoRA adapters per layer, moving from $r=32$ (≈ 32 k flops per inverse) to $r=128$ (≈ 2 M flops) increases total inverse cost by a factor of 64 -- non-trivial if performed on CPU. \n\n### III. Suggestions for improvement\n\nThere are a couple of things that could help improve the paper.\n\n1. **Adaptive or progressive probing.** Stop the warm-up when layer-wise importance estimates stabilize, or continue refining ranks asynchronously during early training.  \n2. **Guidelines for $\\gamma$ and $N$.** Provide practical heuristics to avoid grid search.", "was_revised": true, "final_justification": "During the author-reviewer discussion period, some of my concerns have been addressed. However, my main concern about compute cost of their method was not fully addressed. While I appreciate the authors willingness to implement more optimized approaches, I cannot factor that in the evaluation of the current paper since we do not have access to a revised version. I will therefore maintain my current score.", "rebuttal": "We sincerely appreciate the reviewers' recognition of GoRA's strengths, including its unified treatment of rank and initialization that avoids AdaLoRA-style masking overhead, its robust gains across multiple modalities (even surpassing full fine-tuning on GSM8K and HumanEval with rmax=128), and its comprehensive ablation studies. We also deeply value the reviewers' insightful concerns and valuable suggestions regarding computational efficiency (time/memory costs) and hyperparameter selection in GoRA. Below we provide detailed responses with experimental validations.\n\n## Experimental Setup\nAll experiments presented in this rebuttal were conducted on a computing node with:\n- **Hardware**:\n  - 8 × NVIDIA A800 80GB GPUs\n  - 105 × Intel(R) Xeon(R) Platinum 8336C @ 2.30GHz cores\n  - 1.875TB RAM\n- **Software Configuration**:\n  - model: Llama3.1-8b-Base\n  - Liger kernel for RMSNorm, fused linear cross-entropy loss, and MLP operations\n  - **Batch Configuration**:\n    - Global batch size: 64\n    - Per-GPU batch size: 8\n    - GoRA HP $N$: 8 (64 reported in our paper, where the per-GPU batch size is 1)\n    - Total number of samples for calculating gradients during GoRA initialization: 8 * 8 * 8 = 512\n    - No gradient checkpointing or accumulation\n  - Other configurations (including FlashAttention) match original paper settings\n\n## Q1: Compute Start-up Cost\nWe sincerely thank the reviewer for this insightful suggestion regarding initialization overhead. To address the computational cost of N full forward-backward passes, we have conducted additional exploration by implementing an adaptive convergence detection algorithm. This innovation automatically terminates gradient computation once layer-wise importance scores stabilize, significantly improving efficiency:\n\n| Method | Init Time (MetaMathQA) | GSM8K Accuracy | Convergence Criteria |\n|--------|-----------------------|----------------|-----------------------|\n| Fixed $N$=8 | 3min46.98s | 72.71±0.53 | - |\n| Dynamic (4 steps to converge) | 1min44.97s | 72.96±0.19 | <1% importance variation for 2 steps |\n\n**Impact**:\n- Full training time: 36min\n- Time saved: 7.5% total reduction\n- Accuracy maintained (±0.25 difference)\n\n## Q2: CPU Off-load Bottleneck\nWe thank the reviewer for this important observation. Our results indeed show the CPU offload overhead scales with GPU count:\n\n| GPU Count | Time/Step | Total Time |\n|-----------|-----------|------------|\n| 1 A800 | 3-4s | 23.6s |\n| 8 A800 | 15-20s | 3m46.98s |\n\n**Solution**: Implemented optimizations (detailed in Q3) to address PCIe bottleneck.\n\n## Q3: Host RAM Usage\nWe greatly appreciate the reviewer's valuable suggestions concerning memory efficiency. In response, we have implemented additional architectural optimizations that specifically target these memory concerns, yielding significant improvements:\n\n**Optimizations Implemented**:\n\n1. **Memory-Saving Techniques**:\n   - Layer-wise immediate reduction with inter-GPU all-reduce\n   - Single-copy storage (rank-0 only) in CPU memory\n   - Rank-0 importance computation with broadcast synchronization\n\n2. **Performance Improvements**:\n   - CPU memory: 8× → 1× (16-bit targeted modules' size)\n   - PCIe bandwidth: Reduced via 8→1 offload + broadcast\n   - Initialization time improvements:\n     - 8-GPU: 3m46.98s → 34.52s (4-5s per step)\n     - 8-GPU + early stop: → 20.32s\n\nWe thank the reviewer again for these important suggestions, we will add a section and re-write the algorithm in the revised version of our paper.\n\n## Q4: Hyperparameter Sensitivity\n\nWe greatly appreciate the reviewer's highlighting of HP sensitivity. To minimize manual tuning, we propose these automated strategies: \n\n1. **Rank bounds (r_min/r_max):**\n    - Auto-setting: **`r_max = 4×r_ref`**, **`r_min = 0.5×r_ref`**\n    - Guarantees: Maintains comparable parameters to LoRA (Table 7)\n    - Verification: Best performance across settings we tested (Table 5)\n2. **Initialization steps (N) (new!):**\n    - Dynamic convergence:\n        \n        a) Monitors inter-layer importance variation\n        \n        b) Early stops when <1% change for 2 steps\n        \n    - Outcome: Avg. steps reduced from 8→3 (2.8× faster)\n3. **Scaling factor (γ) (new!):**\n    - Auto-selection:\n        \n        a) Tests γ candidates ∈[1e-1,1e-5] during initialization\n        \n        b) Selects γ minimizing first-step training loss\n        \n    - Overhead: about 30s\n    - best γ searched for CodeFeedback: 0.0549 (5e-2 reported in the paper)\n    - best γ searched for MetaMathQA: 0.0858 (8e-2 reported in the paper)\n\n**Performance Impact**:\n\n| Method | GSM8K | HumanEval |\n|--------|-------|-----------|\n| Original | 72.91±0.76 | 48.98±2.14 |\n| Adaptive $N$ | 72.96±0.19 | 46.85±2.11 |\n| Adaptive γ | 72.5±0.38 | 50±3.23 |\n\n**Key Takeaway**: Automated approaches maintain performance (<±0.5 GSM8K, <±3 HumanEval) while reducing tuning effort.\n\n## Q5: Pseudo-inverse Cost\n**Clarification**: All pseudo-inverse computations are GPU-based with minimal overhead:\n\nBelow are initialization times for an 8B model across different ranks, measured using the following GPU-optimized implementation:\n```python\n    def grad_compress_init(self, \n                      lr: float, \n                      scaling_by_lr: bool = False,):\n        if not hasattr(self.weight, 'grad_stored'):\n            return\n        \n        # Convert weight_a to float32 on correct device\n        origin_weight_dtype = self.weight.dtype\n        weight = self.weight.to(torch.float32)\n        weight_dtype = weight.dtype\n        weight_device = weight.device\n        grad_stored = self.weight.grad_stored.to(weight_dtype).to(weight_device)\n        self.weight_a = nn.Parameter(torch.empty((self.lora_rank, self.in_features), dtype=weight_dtype, device=weight_device))\n\n        AT = self.weight_a.T\n        AAT = torch.matmul(self.weight_a, AT)\n            \n        AAT_inv = torch.linalg.pinv(AAT + 1e-8 * torch.eye(self.lora_rank).to(weight_device)).to(weight_dtype)\n        AAT_inv_AT = torch.matmul(AT, AAT_inv)\n\n        # Compute weight_b using grad_stored (convert to float32 for computation)\n        weight_b_data = torch.matmul(grad_stored, AAT_inv_AT)\n\n        if scaling_by_lr:\n            stable_gemma = (lr / math.sqrt(self.lora_rank/self.in_features)) * (self.scale_rank)\n        weight_b_data *= (stable_gemma / self.scaling_alpha)\n\n        self.weight_b = nn.Parameter(weight_b_data.contiguous())\n\n```\n| Rank | Time (8B model) |\n|------|-----------------|\n| 8 | 1.4s |\n| 32 | 1.56s |\n| 128 | 3.43s |\n\nKey Observations:\n\n- Even at rank 128 (requiring ≈2M FLOPs per inverse), the total initialization adds <4s overhead.\n\n- This represents <0.1% of typical training time, making the cost negligible in practice.\n\nWe will clarify these implementation details and include benchmarking results in the paper. The GPU-based design ensures scalability, and we find no practical bottleneck from pseudo-inverse computations.\n\n## Conclusion\nWe sincerely thank the reviewers for these valuable suggestions. We will:\n1. Add a detailed section on computational optimizations\n2. Rewrite the algorithm description\n3. Clarify pseudo-inverse costs\n4. Include all experimental results in the revised paper"}, {"review_id": "qj4gfE1QK8", "reviewer": "Reviewer_zRT2", "summary": "This work proposed a new parameter-efficient finetuning method, an upgrade to LoRA. It is also a low-rank update method that keeps the bulk of the pretrained model frozen, but introduces two innovations: a) an adapted initialisation strategy that does not start with a zero-initialised matrix (like in LoRA) that tries to minimise the error in the gradient (by comparing it against the full-finetuning gradient), b) it adapts the rank during training using gradient information. On various benchmarks it outperforms baselines LoRA and other popular methods such as PiSSA and DoRA.", "questions": "Q:\nWhat if you rerun the initialisation strategy a couple of times during the training? does the ideal rank allocation change drastically or stay consistent? How far off are the trained matrices from the ones that best compress the full gradients? Does this yield better performances\n\nWhat happens if you initialise not with L1 optimal compression of the gradients but other metrics like L2  (||W * G ||_2)?\n\nCan and how can this be combined with QLoRA type quantisation? Can this scale to 8\n\nsmall things:\n* typo: \"this reduces the their usability\"", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "+ the experiments cover both vision and NLP (and therein includes encoder as well as decoder LMs)\n+ performance gains look solid and are compared against baselines from other papers\n+ experiments include st-deviations\n+ some analysis of the method is provided \n+ initialisation being that important is interesting for the community\n\n\nWeaknesses\n- before training the gradients for all weights are computing in a full-finetuning like fashion to get the gradients. this requires as much memory as full-finetuning so actually makes is quite prohibitive to scale to large LLMs in the 13B+ range.\n\n- the motivation of using non-zero initialisation is states as \"Thus, designing a nonzero initialization method without manipulating pre-trained weights remains an open problem.\" but it's not clear why this is a problem. As the intro also says, substracting AB from the original W seems fine. \n\n- the pseudo-inverse is used to compute the optimal initialisation for compressing the gradient ... in terms of L2 distance, if I understand correctly. It's not clear if this is indeed optimal. one could also find the best initialisation with L1 distance (e.g. by running a few steps of SGD to find the best initialisation). L2 thus seems a bit out of the blue and it would be nice if this motivation is better argued for.\n\n- a lot of work is applied in correctly initialising the matrices including their ranks. But it's not clear if the ideal rank allocation doesn't change during training. \n\n- GoRA, despite its smart rank allocation strategy has quite varying performances depending on the hyperparamter r_ref. E.g. varying from 5.82 on MT-Bench (the worst in the whole table) to 6.34 (second best in the table), or in a reverse of trends, 52 vs 48.9 on HumanEval. This makes it very hard for practioners to just \"use\" the method if there's such crucial and sensitive hyperparamters.", "was_revised": true, "final_justification": "I thank the authors for their rebuttal that addressed most of my concerns.\n\nI do however, disagree on\n\n> Methods like PiSSA rely on randomized SVD to compute, making it impossible to exactly recover the same during inference.\n\nThis is quite feasible by just storing a random seed. \nBut this is minor, i've upgraded to A.", "rebuttal": "## Q1. Gradient Computing Cost \nWe appreciate the reviewer's important observation regarding memory requirements during gradient computation. To clarify our approach:\n\n1. **Memory-Efficient Initialization:**\n    - We employ layer-wise gradient computation with immediate all-reduce and CPU offloading\n    - This eliminates the need to store gradients on GPU\n2. **Optimizer State Advantage:**\n    - Our method requires no optimizer states during initialization\n3. **Scalability:**\n    - The peak memory footprint matches single forward pass requirements\n    - Enables handling of 13B+ models within standard GPU memory constraints (weights can also be offloaded)\n4. **Empirical Evidences**:\n    - For a 7b model trained on 8 * A800 with a batch_size=8 per GPU:\n        - GPU memory (initialization): **56,139MB**\n        - GPU memory (training):  **73,295MB**\n    - For a 32b model trained on 8 * A800 GPUs with a batch_size=1 per GPU:\n        - GPU (initialization): **64,821 MB**\n        - GPU (training): **66,381 MB**\n\n## Q2. Motivation of Our Non-zero Initialisation Method\n\nSubtracting $A_0B_0$ from the pre-trained weights $W$ (as in PiSSA) is indeed feasible, this approach introduces critical practical limitations:\n\n1. **Reproducibility Challenges in Inference:**\n    - Methods like PiSSA rely on randomized SVD to compute $A_0B_0$, making it impossible to exactly recover the same $A_0B_0$ during inference.\n    - Data-dependent methods (e.g., LoRA-GA) require training data to compute $A_0B_0$.\n2. **Storage Overhead:**\n    - Storing either (a) the modified $W - A_0B_0$ or (b) $A_0B_0$ itself would resolve the above issue but at the cost of significant additional storage.\n3. **Theoretical Limitation:**\n\nConsidering there is a optimal downstream transfer $\\Delta W^*$ for a downstream task to be tuned:\n\n$$\\left\\Vert AB - \\Delta W^* \\right\\Vert_F^2 = \\left\\Vert W + (AB - A_0 B_0) - W^* \\right\\Vert_F^2 = \\left\\Vert AB - (A_0 B_0 + \\Delta W^*) \\right\\Vert_F^2$$\n\nThe minimal error of previous non-zero methods is given by $ A_0B_0 + \\Delta W^* $ which is a biased best-r approximation of $\\Delta W^*$. This bias can lead to sub-optimal solution.\n\n## Q3. Optimal Solution for Decomposition\n\nWe appreciate the reviewer’s question regarding our choice of optimization objective. To clarify:\n\n1. **Optimality under L2:**\n    - The **truncated SVD** provides the optimal solution in L2 norm.\n    - we use the **pseudo-inverse** because:\n        - It naturally arises when solving the problem for optimal initialization given an initialized A. Solving this problem can achieve a compression result more aligned with the Eq. (4) in our paper.\n2. **Why Not L1?**\n    - While L1 might offer sparser solutions, it lacks a closed-form solution and requires iterative optimization.\n3. **Empirical Justification:**\n    - We tested a SVD-based initialization (similar to LoRA-GA):\n    \n    | Method | GSM8K | HumanEval |\n    | --- | --- | --- |\n    | LoRA-GA | 71.39±0.90 | 43.29±0.71 |\n    | GoRA(SVD) | 72.68±1.06 | 44.71±0.93 |\n    | GoRA(pseudo-inverse) | 72.91±0.76 | 48.98±2.14 |\n    \nResults demonstrate: Our method outperforms both SVD variants,  justifies our design choice.\n\n## Q4. Dynamics change of rank\n\nThank you for your insightful question. We appreciate this opportunity to clarify our methodology and share additional empirical findings.\n\n**Current Limitations in Dynamic Rank Adjustment**\n\nwe fully acknowledge that the ideal rank distribution may evolve during training. However, implementing dynamic rank adjustment faces two fundamental challenges in modern training frameworks:\n\n1. **Technical Constraints in Data-Parallel Training**: Frameworks like DeepSpeed flatten and distribute gradients/optimizer states across workers, making it *architecturally infeasible* to modify parameter shapes mid-training.\n2. **Training Stability Concerns**: Even if rank adjustment were possible, it would require:\n    - Jagged learning rate scheduling\n    - Complex mechanisms to maintain optimizer state consistency\n    - Risk of destabilizing convergence\n\nGiven these constraints, our current work focuses on ***pretraining-adaptive* rank allocation**\n\n**Empirical Analysis of Layer Importance Dynamics**\n\nWe conducted a detailed study tracking layer importance every 5 steps under:\n\n| **Configuration** | **Details** |\n| --- | --- |\n| Model | Llama-3.1-8b-Base |\n| Dataset | MetaMathQA |\n| Optimizer | Adalomo [1,2] |\n| Batch Size | 8/GPU × 8 GPUs = 64 |\n| Total Steps | 1,024 |\n| Learning Rate | 1e-4 |\n\n**Key Findings:**\n\n1. **Consistent Hierarchy**:\n    - **`wv`** layers maintain *significantly higher importance* throughout training\n    - **`wq`**/**`wk`** layers remain *persistently less important*\n2. **Temporal Stability**:\n    - **Lower Layers**: Slight importance decay after ~200 steps\n    - **Middle Layers**: Moderate importance growth after ~200 steps\n    - **Upper Layers**: Remarkably stable importance trajectories\n    - **Exceptions**: Minority of layers show gradual monotonic changes (sum normalized importance ±<0.02)\n\nThis suggests that while *minor* importance shifts occur, the *relative rank priorities* remain **largely stable** during the fine-tuning process.\n\n**References**\n\n[1] Adalomo (*ACL 2024*)\n\n[2] Full Parameter Fine-tuning for Large Language Models with Limited Resources. (*ACL 2024*)\n\n## Q5. Rerun the initialization strategy\nWe fully agree that periodically re-initializing the low-rank adapters to incorporate full fine-tuning training dynamics could further enhance performance. Below, we clarify the technical feasibility.\n\n### **1. Technical Problem**\n\n**re-compressing gradients into low-rank adapters is possible**, though it requires tight integration with gradient collection pipelines (e.g., DeepSpeed). We are actively engineering this solution.\n\n### **2. GoRA-pro: Integrating Fine-Tuning Dynamics**\n\nMotivated by the reviewer’s suggestion, we propose **GoRA-pro**, which refines initialization by simulating fine-tuning dynamics *before* formal training:\n\n1. **Pre-Training Optimization**:\n    - Use the **AdaLomo** to perform *n*-step **updates** on the *full* model parameters.\n    - Store gradients at each step.\n2. **Reinitialization**:\n    - Reset the model to its *original pre-trained weights*.\n    - Initialize low-rank adapters using the accumulated gradients.\n\n**Advantage over GoRA**:\n\n- **GoRA-pro actively integrates fine-tuning dynamics** into adapter initialization.\n- no prior or concurrent LoRA work explores such a hybrid of full fine-tuning and low-rank initialization.\n\n### **3. Implications and Future Work**\n\n| Method | GSM8K | HumanEval |\n| --- | --- | --- |\n| GoRA | 72.91±0.76 | 48.98±2.14 |\n| GoRA-pro | 72.30±0.77 | 47.36±1.54 |\n\nAlthough currently GoRA-pro exhibits on-par performance compared to GoRA, we believe this direction could open new avenues for improving LoRA’s performance.\n\n## Q6. How far from the best compression\n\nWhile it is challenging to directly compare the trained adapter matrices with the pre-training gradients, our initialization strategy  ensures that GoRA begins training with a low initial loss, implying effective capture of gradient information.\n\nTo provide a rigorous and intuitive comparison, we conducted experiments on the **Llama-3-8B-Base** model trained on the MetaMathQA and Code-Feedback, evaluating the reconstruction error of initialized adapters against the full gradients across all adapted layers:\n\n- Math:\n    - Absolute Error: 0.0382\n    - Relative Error: 0.1147\n- code:\n    - Absolute Error:  0.0322\n    - Relative Error: 0.1152\n\nThis indicates that our compression retains ~90% of the gradient information. Notably, when combined with the *n*-step accumulation strategy (see Q5), the initial loss further drops sharply from 0.68 to 0.44, underscoring the efficacy of our compression.\n\n## Q7. MTBench Score\n\nWe sincerely appreciate the reviewer’s observation regarding performance variations on MTBench. We would like to address this concern through the following clarifications:\n\n1. **Nature of `r_ref` Parameter**:\n    - **`r_ref`** is fundamentally a *configuration parameter* rather than a tunable hyperparameter. Its primary role is to establish parity in total trainable parameters between GoRA and comparable methods (e.g., **`r_ref=32`** in GoRA corresponds to **`r=32`** in LoRA).\n    - We explicitly recommend practitioners use the largest computationally feasible **`r_ref`** value within their resource constraints.\n2. **Understanding MT-Bench Variability**:\n    \n    The observed performance fluctuations are consistent with well-documented phenomena in PEFT:\n    \n    - The non-monotonic relationship between rank size and model performance on MT-Bench has been previously established (e.g., in LoRA-Pro and LoRA-GA).\n    - MT-Bench’s particular characteristics may contribute to this variability:\n        - Limited scale\n        - Possible distributional mismatch between training and evaluation data\n        - Known stochasticity inherent in LLM-as-judge evaluation\n\n3. **Hyperparameter Management**:\n    \n    While GoRA introduces four core parameters, we have additionally developed automated tuning mechanisms to enhance practical usability:\n    \n    - Our response to Reviewer 4NWP’s Q4 details these optimization procedures\n\n## Q8. Combine with QLoRA\n\nWe are pleased to confirm that GoRA can indeed be seamlessly integrated with QLoRA's quantization approach.\n\n**Implementation Details:**\n\n- We quantize the pre-trained weights to **NF4 precision** for storage\n- During computation, we de-quantize these weights back to **bf16** precision\n\n**Empirical Results:**\n\n| **Method** | **GSM8K** | **HumanEval** |\n| --- | --- | --- |\n| QLoRA | 65.1 ± 0.95 | 43.49 ± 0.70 |\n| QGoRA | **70.58 ± 1.33** | **44.1 ± 3.68** |\n\n**Key Observations:**\n\n1. QGoRA consistently outperforms QLoRA\n2. The improvement is particularly notable on GSM8K\n\n## Q9. Typo\n\nThank you for remind us this typo! We have fixed this typo in the revised version of our paper."}, {"review_id": "LRx1cVW6Oi", "reviewer": "Reviewer_stk3", "summary": "The paper introduces GoRA (Gradient-driven Adaptive Low Rank Adaptation), a novel framework designed to enhance the efficiency and effectiveness of fine-tuning large language models (LLMs) using Low-Rank Adaptation (LoRA). GoRA addresses two critical limitations of existing LoRA variants: static rank selection and suboptimal weight initialization. It proposes a unified approach that leverages gradient information during training to dynamically assign optimal ranks to low-rank adapters and initialize their weights adaptively. The authors present extensive experiments across various architectures and modalities, demonstrating that GoRA consistently outperforms existing LoRA-based methods and can even surpass full fine-tuning in certain challenging tasks, all while maintaining the computational efficiency of vanilla LoRA.", "questions": "- Gradient Collection Memory Footprint: The paper states that the initial gradient collection \"do not require extra memory as shown in Table ??\", but the table reference is missing. Could the authors please provide specific figures for the memory footprint (e.g., in GB) required for gradients when offloaded to CPU for a representative model like Llama3.1-8B-Base? Furthermore, can you elaborate on whether this memory usage remains practically insignificant for models orders of magnitude larger (e.g., 70B+ parameters), where the full W matrix and its corresponding gradients could be very substantial?\n- Scalability of Initial Gradient Computation Time: The initial gradient computation takes 4 minutes for the Llama-3.1-8B-Base model . Please provide insights into how this 4-minute overhead scales if the number of gradient accumulation steps in Algorithm 1 were significantly increased, or if the method were applied to models with considerably more parameters than the 8B-parameter Llama models tested. Could this initial phase become a bottleneck in very large-scale fine-tuning?\n- Dynamic Initialization of Matrix A: The paper mentions the non-random initialization of matrix A as a direction for future work . Given that the initialization of matrix B depends directly on A , could the authors discuss the theoretical or practical complexities anticipated in developing a data-driven or non-random initialization for A, especially in maintaining the \"optimal gradient compression\" property and computational efficiency?", "limitations": "The authors have adequately addressed the limitations of their work in Appendix D, titled \"Limitations And Future Works\" . They acknowledge that their evaluation has not yet extended to larger models (like Llama-3.1-70B) and more extensive datasets, and that the primary focus has been on language models and NLP tasks, with potential for generalization to other modalities . They also identify the current restriction of matrix A to random initialization as a future area for improvement  and note theoretical compatibility with other LoRA variants . This transparency about the scope and future work is commendable.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n- Originality and Unified Framework: GoRA represents a novel advancement to unify both adaptive rank selection and adaptive initialization strategies within a single framework. This contrasts with prior approaches that typically address these aspects in isolation, leading to a more holistic and effective solution for LoRA fine-tuning.\n- Novel Initialization Strategy: GoRA's adaptive initialization strategy for matrix B, utilizing pseudo-inverse compressed gradients, is novel and theoretically motivated. This approach crucially avoids manipulating pre-trained weights, thereby eliminating the \"training-inference gap\" that affects many existing non-zero initialization methods.\n- Effective Adaptive Rank Allocation: The proposed gradient-driven, sensitivity-based importance metric for rank allocation is well-justified through ablation studies, demonstrating superior performance compared to alternative metrics like nuclear norms . \n- Superior Performance: The paper provides compelling evidence that GoRA consistently outperforms a wide array of LoRA-based methods across natural language understanding (NLU), natural language generation (NLG), and image classification tasks. Notably, GoRA demonstrates the capability to surpass full fine-tuning performance in challenging tasks such as mathematical reasoning.\n- Computational Efficiency Preservation: A key advantage of GoRA is its ability to achieve performance improvements while preserving the efficiency of vanilla LoRA, both in terms of training time and peak GPU memory usage. This is a critical factor for fine-tuning large models under resource constraints. It also maintains a similar number of trainable parameters compared to standard LoRA.\n- Comprehensive Experimental Validation: The evaluation covers diverse models (T5, Llama, CLIP-VIP) and modalities (NLP, Vision), showcasing GoRA's broad applicability and robust performance across different contexts. The inclusion of ablation studies provides valuable insights into the contribution of individual components .\n- Clarity and Reproducibility: The paper is well-structured and clearly explains the technical contributions, including a detailed algorithm for GoRA's method. The provided experimental settings, hyperparameters, and implementation details enhance the reproducibility of the reported results . Proofs are also included in the appendix. Code is also provided.\n\nWeaknesses:\n- Ambiguous Memory Claim for Gradient Collection: The paper states that the initial gradient computation \"do not require extra memory as shown in Table ??\". However, the table reference is missing, and neither Table 7 (trainable parameters) nor Table 8 (training time, peak GPU memory) explicitly quantifies the memory used for storing the accumulated gradients for all layers, even if offloaded to CPU. While CPU memory might be less constrained than GPU memory, for extremely large models where W matrices can be order of 10s of gigabytes, this unquantified memory footprint for gradient storage could still be a practical concern, making the claim of \"no extra memory\" problematic without direct evidence.\n- Scalability of Initialization Phase for Larger Models/Datasets: Although the initial gradient computation phase takes only 4 minutes for the Llama-3.1-8B-Base experiment , the paper does not extensively discuss how this initial overhead scales with significantly larger models (e.g., Llama-3.1-70B) or much larger datasets, which could potentially make this phase a more substantial part of the total training time.\n- Limited Guidance on $\\alpha$ Hyperparameter Selection: While the paper extensively detail the selection and impact of $\\gamma$ through ablation studies and provide specific values in their experiments, the paper does not explicitly state how $\\alpha$ is selected or what its numerical value is in the experiments conducted for GoRA. Can the authors comment on its selection?", "was_revised": true, "final_justification": "I acknowledge that i have read the authors' rebuttal to all reviewers. The response by the authors and the additional details and experiments provided further solidifies my initial assessment of the work. I will keep my original high rating of 5.", "rebuttal": "We sincerely thank the reviewer for their thoughtful and encouraging assessment of our work, particularly their recognition of GoRA’s originality, computational efficiency, and comprehensive validation. We greatly appreciate the time and care taken to highlight the strengths of our framework. Below, we address the reviewer’s concerns point by point.\n\n## Q1. Memory Claim\nWe sincerely appreciate the reviewer's careful examination of the memory issues. First, we apologize for the citation oversight in our paper - the correct reference should be Table 8. Regarding memory consumption:\n\n1. Our method introduces no additional GPU peak memory:\n    - Initialization peak memory: 16-bit model params + 16-bit activations + single-layer gradients\n    - Training peak memory: 16-bit model params + 32-bit optimizer states + 32-bit param copies + 16-bit activations\n    - Even with ZeRO-2 optimization, initialization memory remains significantly lower than training memory (as shown in Table 8)\n    - Optional: Model can be offloaded to CPU during initialization for further GPU memory reduction\n2. CPU memory clarification:\n    - Storing 16-bit targeted modules’ gradients does incur additional CPU memory compared to LoRA (the gradients are only saved by the local-rank-0 process)\n    - However:\n        \n        a) CPU memory is substantially more affordable than GPU memory\n        \n        b) Our experimental platform features 1.8TB CPU memory (per 8-GPU node)\n        \n        c) For 100B-parameter models (~200GB), this overhead is entirely manageable\n        \n\nWe will include detailed memory analysis in the revised version and correct all relevant citations.\n\n## Q2. Initialization Phase for Larger Models/Datasets\n\nWe appreciate the reviewer's valuable feedback regarding the scalability of our initialization phase. In response, we conduct a additional experiment on larger-scale models with latest optimized algorithms (as detailed in our responses to Reviewer 4NWP's Q3-Q4). Below are the experimental results: \n\n**Experimental Setup:**\n\n1. **Hardware:**\n    - 8 × NVIDIA A800 80GB GPUs\n    - 105 × Intel(R) Xeon(R) Platinum 8336C @ 2.30GHz CPUs\n    - 1.875TB RAM\n2. **Software:**\n    - Model: Qwen2.5-32B-Instruct\n    - Dataset: MetaMathQA-395K\n    - Batch size: 1 per GPU\n    - Gradient accumulation steps: 8\n    - Optimizations: Liger kernel + FlashAttention + Activation Checkpoint\n\n**Results:**\n\n1. **Initialization Phase:**\n    - Completed in **18 steps** (early stopped from 64 planned steps)\n    - Total time: **2 minutes 44.65 seconds** (~8.4s/step)\n    - GPU memory usage: **64,821 MB**\n2. **Training Phase:**\n    - Total time: **~11 hours**\n    - GPU memory usage: **66,381 MB**\n\nThese results demonstrate that our optimized initialization remains highly efficient even for models at the 32B scale, with negligible overhead compared to full training. We will include this analysis in the revised manuscript.\n\n## Q3. How alpha is selected\n\nOur setting for $\\alpha$ is consistent with previous works[1,2,3] (i.e, $\\alpha=16$)\n\n[1] Wang S, Yu L, Li J. Lora-ga: Low-rank adaptation with gradient approximation[J]. Advances in Neural Information Processing Systems, 2024, 37: 54905-54931.\n\n[2] Wang Z, Liang J, He R, et al. Lora-pro: Are low-rank adapters properly optimized? In: Proceedings of the International Conference on Learning Representations, 2025.\n\n[3] Meng F, Wang Z, Zhang M. Pissa: Principal singular values and singular vectors adaptation of large language models[J]. Advances in Neural Information Processing Systems, 2024, 37: 121038-121072.\n\n## Q4. Dynamic Initialization of Matrix A\n\nThank you for your insightful suggestions about initialization strategy of Matrix A.  We conduct follow experiments:\n\n| Method | GSM8K | HumanEval |\n| --- | --- | --- |\n| GoRA (A initialized with gradient principle components) | 72.15±1.29 | 47.15±2.31 |\n| GoRA (A initialized with weight principle components) | 58.73±1.06 | 34.55±3.35 |\n| GoRA (reported in the paper) | 72.91±0.76 | 48.98±2.14 |\n\nThe results demonstrate that initialize A with gradient principle components yield comparable performance compared to standard GoRA while initialize B with weight principle components yield downgraded performance. We hypothesis that the noisy from the weight principle components can lead to sub-optimal gradient compression. We will further investigate and discuss these strategies in the revised version of our paper.\n\n## Q5. Typos\nWe sincerely appreciate the reviewer's careful attention to detail in identifying these typographical errors. We have thoroughly reviewed the manuscript and corrected all instances in our revised version."}, {"review_id": "E0aMpEBY7b", "reviewer": "Reviewer_bxar", "summary": "This paper presents a novel low-rank adaptation technique that is capable of adaptive rank selection as well as informative initialization all within the same theoretical framework. The core idea of the proposed method is based on the importance accumulated gradients from the original backpropagation without LoRA for guiding the architecture and the initialization for LoRA. The authors have shown the empirical merits of the proposed by performing extensive experimentation as well as ablation studies to assess the importance of each component.", "questions": "- Does the memory usage ion Table 8 also include the gradient accumulation step?\n- The authors claimed that the gradient accumulation step doesn't lead to extra memory usage. Why is that the case?\n- The proclaimed \"$n$-step gradient accumulation\" is in fact a 1-step gradient accumulation over a big batch. Why this choice and not the actual $n$-step gradient accumulation which is more intuitive? Any empirical comparisons between the two?", "limitations": "The main limitation IMO that hasn't been discussed in the Limitation section is that the initial gradient accumulation step can be prohibitive in terms of the memory usage for larger scale models. This kind of defeats the purpose of using LoRA in the first place. I'd strongly recommend the authors to discuss this issue not just in the Limitation section but in the main body of the paper.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The theoretical foundations of the proposed methodology is intuitive and well-explained.\n- The experimental study is reasonably extensive with promising results.\n- The paper is generally very well-written.\n\nWeaknesses:\n- The proposed methodology requires to perform $n$-step gradient calculations on the original weights. However, this is prohibitive in many practical situations mainly due to the memory constraints, and hence the use of low-rank adaptation techniques in the first place. The presented experiments are only done for medium size models which are likely to be ok, but there's no indication that'd scale up. The authors have put one sentence in Algorithm 1 claiming that the gradient accumulation step would not lead to extra memory usage. but this claim makes no sense, plus there's no further argument to support this claim. Also, it's not clear the memory usage peaks reported in Table 8 include the  gradient accumulation preprocessing step or just the GORA optimization.\n\n- The proclaimed \"$n$-step gradient accumulation\" is not really the $n$-step gradient accumulation, is it? In Algorithm 1, it's mentioned that the step function of the optimizer is not applied which makes the whole gradient accumulation step equivalent to a 1-step gradients on a larger batch. There's no explanation whatsoever why this choice was made over the actual n$-step gradient accumulation which seems to be more intuitive.\n\nMinor issues:  \n- The reference in Algorithm 1 Line 3 is broken.\n- Confusing notation: $n$ is used for two different things in the paper.\n- The Hadamard product notation is inconsistent between Eq. 5 and Table 6.", "was_revised": true, "final_justification": "I'll maintain my current score as I believe while the paper is above the acceptance threshold, some critical explanations need to be added to the revised draft.", "rebuttal": "We sincerely appreciate the reviewer's insightful and encouraging comments regarding our work. We are grateful for their recognition of the theoretical foundations, extensive experimental validation, and clear presentation of our proposed method. Below we provide detailed responses to address the reviewer's specific concerns.\n\n## Q1. Does the memory usage ion Table 8 also include the gradient accumulation step? Why gradient accumulation step doesn't lead to extra memory usage?\n\nWe appreciate the reviewer's attention to the initialization cost of GoRA. We would like to provide further clarification about our initialization algorithm's memory efficiency and scalability.\n\n**Memory Management During Initialization:**\n\nOur initialization algorithm employs a carefully designed memory management scheme:\n\n1. After computing gradients for a specific layer:\n    - Gradients are immediately averaged across GPUs via all-reduce\n    - Only the local rank-0 process stores the gradients in CPU memory\n    - All other processes discard their gradients\n2. Memory footprint breakdown:\n    - *GPU memory*:\n        - 16-bit model weights\n        - 16-bit activations\n        - 16-bit **single-layer gradients**\n    - *CPU memory*:\n        - One copy of 16-bit gradients for all target modules (e.g., 2-3GB for attention modules in an 8B model)\n\n**Comparison with Training Phase Memory Usage:**\n\n- *Training phase* GPU memory includes:\n    - 16-bit model weights\n    - 16-bit activations\n    - 16-bit full-model gradients\n    - 32-bit trainable parameter copies\n    - 32-bit optimizer states\n- The peak GPU memory usage during initialization remains consistently lower than during training, as demonstrated in Table 8 (which already accounts for initialization memory). Under the experimental conditions reported in this paper, the initialization process consumes **17.35GB** of memory, while training requires **19.75GB**.\n\n**Empirical Validation on Large Models:**\n\nWe conduct an additional experiment on a 32B model under the following conditions:\n\n1. **Hardware Configuration:**\n    - 8 × NVIDIA A800 80GB GPUs\n    - 105 × Intel Xeon Platinum 8336C CPUs @ 2.30GHz\n    - 1.875TB RAM\n2. **Software Configuration:**\n    - Model: Qwen2.5-32B-Instruct\n    - Dataset: MetaMathQA-395K\n    - Batch size: 1 per GPU\n    - Gradient accumulation steps: 8\n    - Optimizations: Liger kernel + FlashAttention + Activation Checkpoint\n\n**Key Results:**\n\n1. *Initialization Phase*:\n    - Completed in 18 steps (early stopped from planned 64 steps)\n    - Total time: **2 minutes 44.65 seconds (~8.4s/step)**\n    - GPU memory: **64,821 MB**\n2. *Training Phase*:\n    - Total time: ~11 hours\n    - GPU memory: **66,381 MB**\n\nThese results demonstrate that GoRA's initialization remains more memory-efficient than training even at 32B scale. For extremely large models (70B+), we can further optimize memory by:\n\n- Offloading model weights to CPU during initialization\n- Reducing GPU memory footprint to just single-layer weights\n\nThis design ensures GoRA's scalability while maintaining its theoretical advantages.\n\n##  Q2. n-step gradient\nWe sincerely appreciate this insightful observation. The reviewer is correct—our original method computes gradients over *n*  micro-batches but applies them in a single aggregated step (equivalent to large-batch training), rather than performing sequential*n*-step updates. We apologize for the unclear terminology and have revised the manuscript to explicitly refer to this as **one-step gradient accumulation** to avoid confusion.\n\n**Introducing GoRA-pro: A Novel *n*-Step Gradient Framework**\n\nMotivated by the reviewer's question, we developed **GoRA-pro**, a memory-efficient algorithm that leverages true*n*-step gradients for rank allocation. Here's how it works:\n\n1.**Lightweight Pre-Training Phase**:\n   - A layer-wise SGD optimizer performs *n* exploratory updates (*η*: learning rate):\n     $$p = p - \\eta \\cdot \\nabla p$$\n   - Gradients are discarded immediately after updates, minimizing memory (only **56,151 MB** vs.**72,971 MB** in formal training).\n\n2.**Delta-Weight Calculation**:\n   - After*n* steps (~1-2s/step), we compute the *n*-step gradient per layer:\n     ```python\n     p.grad_stored += p.grad.detach().clone().cpu()  # Aggregate gradients\n     p.iters += 1\n     ```\n   - The final gradient is normalized:\n     ```\n     grad_stored = p.grad_stored / p.iters\n     ```\n\n3.**Rank Allocation & Reset**:\n   - Layer importance is estimated using the*n*-step gradient and a CPU-stored reference model $W_{\\text{cpu}}$.\n   - The original weights are restored, and low-rank adapters are initialized with the compressed gradients.\n\n**Advantages & Results**\n\nGoRA-pro bridges the gap between theoretical*n*-step gradients and practical constraints:\n\n-**Accuracy**: Simulates full fine-tuning dynamics more faithfully than single-step methods.\n\n-**Efficiency**: No extra memory overhead (exploratory phase uses 23% less memory than formal training).\n\n-**Novelty**: To our knowledge, this is the first work to use*n*-step gradients for low-rank adapter initialization.\n\n| Method | GSM8K | HumanEval |\n| --- | --- | --- |\n| GoRA | 72.91±0.76 | 48.98±2.14 |\n| GoRA-pro | 72.30±0.3 | 47.36±1.54 |\n\nWe will include empirical comparisons (one-step vs.*n*-step) and a broader discussion of this direction in the revised manuscript. Thank you for highlighting this impactful opportunity!\n\n## Minor issues\nThank you for reminding! We will fix these issues in the revised version of our paper!\n\n## Limitations\n\nWe appreciate the reviewer's thoughtful comment regarding memory usage during the initial gradient accumulation step. We agree this is an important consideration for practical applications, particularly when scaling to larger models.\n\nAs we addressed in our response to Q1, our initialization method does not increase peak memory consumption during training. For clarity, we will:\n\n1. Expand the discussion in Section 5.4 (\"Computation and Memory Analysis\") to explicitly address this point\n2. Include a more detailed comparison of memory requirements between our initialization process and standard LoRA training process\n\nThe current brevity of this discussion was due to page constraints rather than oversight. We appreciate the opportunity to make this aspect clearer in our revised manuscript."}]

Author rebuttal:

Meta-review: In this work, the authors propose a new PEFT method, called GoRA, which can be considered an upgrade to LoRA. While similar to LoRA, the method comes with two novel changes: a smarter initialization strategy of the low-rank matrices, and an adaptation of the rank during training based on the gradients. The method is able to outperform several PEFT methods such as rLoRA, LoRA+, AdaLoRA, PiSSA and DoRA (among others) in several standard language and vision benchmarks (GLUE, MT-Bench, GSM8K, HumanEval)  while keeping trainable parameters and GPU memory similar to that of LoRA.

The paper initially got all accept-regime scores, with 3 Borderline Accepts and 1 Accept. Some of the reviewers praised the method's robustness across modalities, ablations, the setting of experiments and evaluation, the performance, and computational efficiency preservation. However, they also requested some clarifications, mostly on regard to experimental section. In summary, the authors provided in the rebuttal the full experimental setup, compute start-up cost, CPU off-load bottleneck, RAM usage, hyperparameter sensitivity, the pseudo-inverse cost, gradient computing cost, optimal solution for decomposition, clarification with regards to dynamics change of rank, combination with quantization (GGoRA). In general, the rebuttal was extremely comprehensive, properly addressing most of the points raised by the reviewers. Saying that, the reviewer 4NWP says that their main concern about compute cost of their method was not fully addressed, but keeps their score, as do reviewers bxar and stk3. Reviewer zRT2 increases their score to Accept saying that the rebuttal addressed all of his concerns except a small one. Thus, the paper got final scores of 5, 5, 4, 4.

Having carefully read all the reviewers, I agree with the reviewers that this is a good paper above the acceptance threshold. I also agree that it is improving in an important and competitive baseline, and its very strong performance will benefit the community. Thus, I recommend the paper to be accepted, while urging the authors to improve their paper by integrating the changes from the rebuttal. Congratulations to the authors!
Decision: Accept (poster)