LoRA-GA: Low-Rank Adaptation with Gradient Approximation
Authors: Shaowen Wang
Linxi Yu
Jian Li
Venue: NeurIPS 2024 poster (2024)
Keywords: Parameter-Efficient Fine-Tuning
Low-Rank Adaptation
Model Initialization
Large Language Model
Primary area: optimization_for_deep_networks

Abstract: Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MTbench, GSM8k, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance.

LoRA-GA: Low-Rank Adaptation with Gradient Approximation
Shaowen Wang wangsw23dmalls. tsinghua. edu. en
Linxi Yu
yulx230nails. htts233mailss.ssnnuaaa edu. cn
Jian Li
1i jian83@mail......mm S.l..ssing tsinghua..... edu. cn Tsinghua University Beijing, China
Abstract
Fine-tuning large-scale pretrained models is prohibitively expensive in terms computational and memory costs. LoRA, Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine- tuning an auxiliary low-rank model that has significantly fewer parameters. Al- though LoRA reduces the computational and memory requirements significantly each iteration, extensive empirical evidence indicates that it converges a con- siderably slower rate compared to full fine- fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improve- ments) while simultaneously attaining comparable even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-hench, GSM8K, and Human-eval, respectively. Additionally, we observe up 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effec- tiveness in accelerating convergence and enhancing model performance. Code available at code
of the most popular Parameter-
Introduction
Fine-tuning large language models (LLMs) is essential for enabling advanced techniques such as instruction fine-tuning [1]. reinforcement learning from human feedback (RLHF) (2), and adapting models to specific downstream applications. However, the computational and storage costs associated with full fine-tuning are prohibitively high, particularly as model sizes continue grow. To address
these challenges, methods of Garameter-Ffficiee Fine- Fine-Tuning (PEFT) (see e.g., 33] ), such as Low- Rank Adaptation (LoRA) 4 have emerged and gained significant attention.
Coorresponding author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Full Fine-Tunling LoRA-GA LoRA
LoRA
LoRA-GA
Parameterization
+ 1
05
Pretrained Weights WE E mIKE
B E R Ruxr
IRT
Adapters
0.3
Steps
U.S.V (d(W
25
50
100
5X SpeedUp
125
350
175
200
Initializzaion
Bus Anni f- Kaiming init
Bur
Uya
Aun
Visi
300 1000
1500
2008
2300
3000
Figure 1: (Left) Training loss curves of Llama 2-7B MetaMathQA to training steps. LoRA-GA converges as quickly as full fine-tuning and outperforms LoRA. (Right) Initialization procedures used in LoRA and LoRA-GA. The key difference is that LoRA-GA initializes adapters using the eigenvectors of the gradient matrix, as opposed to random initialization with a scaling factor.
Instead of updating the parameters of the model directly, LoRA incorporates auxilary low-rank matrices B and A into the linear layers of models (such as the Q, K V, and O matrices in self-attention block (55), while keeping the original layer weights W fixed. The modified layer is represented as scaling factor. This approach significantly reduces the number of parameters that need to be fine-tuned. thereby lowering the computational and memory costs at each step.
(W + BBz,,, where is the input of that layer, is the output, and is
Despite these benefits, extensive empirical evidence (see e.g.. 6 significantly slower compared to full finetune. This slower convergence often increases overall computational costs (measured in Floating Point Operations) and can sometimes lead worse test performance. In our experiments, we typically observe that LoRA requires 5-6x more iterations and FLOPs reach the same performance full tine-tuning under the same learning rate, shown in Figure
shows that LoRA converges
To study the cause of slow convergence, we perform in-depth investigation of the initialization strategy of LoRA's adapter weights. is known that fine-tuning pretrained models using the same objective (c.g., language modeling) often converges faster than re-initializing new parameters (e.g.. a classification head) [10]. This observation leads us to question whether the slow convergence of vanilla LoRA might be attributed the default random initialization of adapter weights (LoRA initializes A using Kaiming initialization [1]] and sets B to zero (4). In our experiments, we find that different initialization strategies for LoRA can significantly impact the results, and its default initialization is suboptimal.
In pursuit of a convergence the update of BAA matches the update of W closely. Previous work suggests that gradient descent operates in a low-dimensional subspace 112 [3] If we can closely approximate the gradients of the full model at the initial step, subsequent steps can also be approximated, potentially accelerating the convergence of LoRA.
comparable full fine-tuning, we aim for initialization so that
To this end, we introduce a novel initialization method, LoRA-GA (Low Rank Gradient Approximation). By initializing Ani and Bma with the eigenvectors of the full gradient matrix, the gradient of the low-rank product BAA aligns with the direction of the gradient of the full weight matrix W. Mathematically, aim to ensure that:
(BAA) ÇAW, for some non-zero positive constant Ç.
Our contributions be summarized as follows:
1. We propose LoRA-GA novel initialization method for LoRA that accelerates convergence by approximating the gradients of the low-rank matrices with ones of the full weight matrix.

---

2. We identify the scaling factor under non-zero initialization, which ensures the variance of adapter outputs invariant to the rank of the adapter and the dimension of the input.
3. We validate LoRA-GA through extensive experiments, demonstrating significant performance improvements and faster convergence compared to vanilla LoRA. Specifically,, LoRA-GA outperforms LoRA by 5.69% on the GLUE [14] subset with T5-Base [15], and by 0.34, 1 11.52%, and 5.05% on MT-bench [16), GSM8K (77. and HumanEval [18] with Llama 2-7B [19] respectively, while achieving up to 2-4 times faster convergence.
2 Related Work
2.1 Initialization
The significance of maintaining variance stability during initialization has been widely acknowledged to prevent the occurrence of diminishing or exploding phenomena. Xavier initialization (20] ensures stability in both the forward and backward passes of network under linear activation function. He initialization [[]] extends this solution to networks using ReLU activation. Distinct from these, LSUV initialization [1 selects a mini-batch of data, performing a forward pass to determine the output variance, and subsequently normalizing it to ensure stability. Tensor program (see e.g., (22)) has emerged as a powerful technique for tuning various hyperparameters, including the initialization, for large models.
2.2 Parameter- EfffiieterEfffiieent Fine-Tuning (PEFT)
To fine-tune increasingly large language models within limited hardware resources, researchers have developed various Erffiiter-fffiient Fine-Tuning (PEFT) methods. Adapter-based meth- ods [23 [2 25 26 incorporate new layers into existing model layers. While fine-tuning only these inserted layers significantly reduces resource consumption and requires much fewer parameters, this approach introduces additional latency during both forward and backward passes. Soft Prompt-based methods [0
prepend learnable soft tokens the model's input to adapt the model to specific tasks. This approach effectively leverages the pre-trained model's capabilities, requiring only appropriate prompts for task adaptation, though it incurs computational overhead during inference. More broadly. GaLore [31] applies low-rank gradients to parameter updates for memory efficiency during training. While this approach is highly expressive and performant, requires storing complete model checkpoints, consuming more storage than other PEFT methods.
27
28
29 30
2.3 LoRA's Variants
LoRA one of the most popular PEFT methods that introduces the product of low-rank matrices alongside existing layers to approximate weight changes during fine-tuning. Several methods have been proposed to improve the structure of LoRA. AdaLoRA (32) dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within fixed parameter budget. DoRA [8] enhances the model's 's expressiveness by adding learnable magnitudes to the direction adjustments made by low-rank matrix products. Additionally, LoHA 33 and LoKr 34 employ Hamiltonian and Kronecker products, respectively.
Despite these advancements, vanilla LoRA remains the most popular method due to its robust library and hardware support. Therefore, improving LoRA without altering its structure and at a low cost crucial. Several recent methods focus on this aspect. ReLoRA 35 suggests periodically merging learned adapters into the weight matrices to enhance LoRA's expressibility. LoRA+ (36) proposes using different learning rates for the two matrices in LoRA to improve convergence. rsLoRA (37] introduces a new scaling factor to make the scale of the output invariant to rank. Although our stable scale approach appears similar to rsLoRA, rsLoRA assumes BA 0 initialization, making r invariant to the update A A.A In contrast, our stable scale ensures that non-zero initialized BA remains invariant to both rank and input dimension from the start.
Recently, PiSSA 38 proposes to initializing A and to approximate the original matrix W, by performing SVD on W. Our method, however, is based on very different idea, that is to approximate the gradient of W. which involves performing SVD on sampled gradients and properly scaling the initialized matrices, as detailed in Section E
3

---

3 Methods
In this section, we analyze the initialization of LoRA and introduce our method, LoRA-GA. LoRA- GA consists of two key components: (i) approximating the direction of the gradient of full finetune and (ii) ensuring rank and scale stability in the initialization process. We examine each component and subsequently present their integration within LoRA-GA.
3.1 Review of Vanilla LoRA
Structure of LoRA Based on the hypothesis that the updates of fine-tuning are low-rank [13]. LoRA [4] proposes to use the product of two low-rank matrices to represent the incremental part of the original matrix W. Here, W is the weight matrix of linear layer in the model. For example, in transformers, it could be the Q, K, V, or O matrices of the self-attention layer or the weight matrix in the MLP layer. Specifically, LoRA has the following mathematical form:
W' Wo AW Wo BA Wo + nBA
where W Wo Rmxn B Rmxr, and A RR"*", with min(m, n). Wo is the pre-trained weight matrix, remains frozen during the fine-tuning process, while A and B are trainable.
Initialization of LoRA Under LoRA's default initialization scheme 4 39 matrix A is initialized using Kaiming uniform [T]. while matrix B is initialized with all zeros. Consequently. BA and W Wo. ensuring that the initial parameters unchanged.
If the additional term AW BA initially non-zero (e.g., 38 the frozen parameter can be adjusted to ensure the initial parameters unchanged. This can be expressed as:
W (Wo- Buit Amnit) + nBA Wfrozen + nBA
where Wfrozen Wo - B Amit is frozen, and B and trainable this case.
3.2 Gradient Approximation
Our goal is to ensure that the first-step update ((nBA) approximate the direction of the weight update W. i.e., A (A) x <AW for some non-zero positive constant Ç. We will discuss how to choose Ç in Section 3.3 and one can treat Ç as a fixed constant for now.
Consider gradient descent step with learning rate X, the updates for A and B are AA = AL (Amn) and AB XV BC (Bmit), respectively. Assuming learning rate is small, the update of 1BBA at the first step can be expressed as:
1)( mmaB BA) 7>|V (Bmit) Ami Binit V VAC (Amit)
To measure its upproximation quality of scaled the update of the weights in full finetune ÇAW we use the Frobenius norm of the difference between these two updates as a criterion:
|nn(ttiiit + Buii AA) ÇAV (Wo)ll => || (Bma) Ami + Bunst C (Amns) - WV (Wo)lF
Lemma 3.1. Suppose the loss function is L and W'x (Wo + BBA)2, where y is the output of a layer and x is the input, the gradients of A and B are linear mappings of the gradient of W':
VAC = W..W. V (V. AT
Remarkably, W. C. in LoRA and V W C in full fine-tuning are equal the beginning of the training.
By substituting the gradients in Lemma 3.1 into Equation] 1 we can rewrite the criterion as follows:
Vlr Afris Ainit + Binit B VW <VwL
(2)
This criterion evaluates how well the adapter's gradient approximates the direction of the gradient of full fine-tuning, and minimizing it brings the gradient of LoRA closer to that of full fine-tuning with scaling factor (:
min Binis
1| VW L A Amnit + T? Binit BiT
VWC - -Vw
(3)

---

Theorem 3.1. For the optimization problem in Equation with given ñ, if the Singular Value Decomposition (SVD) of VWL W USV", the solution is:
Binit SUUA:
v Vi. such that [L = n =. f 20,
Ainit
where LA and Ig index sets.
Theorem 3.1 provides appropriate initialization scheme for Auuit and Binit given a specific Ç. The
selection Ç, which influences the scaling of the update n,A., will be discussed in the following section.
3.3 Scale Stability
Inspired by rsLoRA cickaalajizievsk22222rak and the Kaiming initialization [T] we define the following notions of stability:
Definition 3.1. When dout, din, - oc, an adapter nBA exhibits two distinct rypes of scale stabilities:
Forward stability: If inputs to the adapter are independently and identically distributed (i.i.d) with 2nd moment Or,dvaddion (1), then the 2nd moment of the outputs remains dr (1).
Backward stability: If gradient of the loss with respect to the adapter outputs Or dhre (1),
then the gradient with respect to the inputs remains Dr,daued (1).
Theorem 3.2. Given the initialization proposed in Theorem 3.1 assume that the orthogonal vectors Aini and Brni are randomly selected from the unit spheres in The and TR with the constraint that the vectors are orthogonal to each other, and 1) = -rdoous (1 //)r) as suggested by rsLoRA
37y. Under these conditions, the adapters are forward scale-stable if C = Dr.doneddan (V /oout/
and backward stablestable if = =Gyddout (ddnn/
Similar to the results obtained from Kaiming Initialization we observe that either < = Ordo (Vd (dout/?2 or ( Ondoue (Vden/r2 work well independently. For all mod- els presented in this paper, either form ensures convergence. Consequently, for all subsequent experiments, we adopt Ç Dr.dous (Vdost/r2)
Remark. We would like to remark that the scaling factor proposed in this subsection proves to be beneficial primarily when one adopts the learning rate rypically used in full-finetuning (e.g.. le - 5). since as LoRA-GA attempts to approximate the updates of full finettinn However, recent research (9] suggests that LoRA with default initialization performs much better with larger learning rates. Furthermore, tensor program analysis [40 [2] indicates that higher learning rates should be paired with smaller initialization magnitudes. Therefore, we recommend decreasing or omitting the scaling factor when training using larger learning rates (e.g., le 4).
3.4 LoRA-GA Initialization
Combining the gradient approximation and stable scale components, we propose the LoRA-GA initialization method. First, we initialize Amnit and Bint using the solution from Theorem 3.1 Then, we determine the scaling factor according to Theorem 3.2 to ensure rank and scale stability. Thus,
based on Theorems 3.1 and 3.2
we propose novel initialization method, LoRA-GA.
LoRA-GA We adopt of and (= a² daus,
where y is a hyperparameter. We define the + pr
index sets LA = (i 1 iiiiiii N} and LB (ilr+1<a 1++liss 2r, E N). Denote the singular value decomposition (SVD) of Vw as Vw =UST The initializations are as follows:
Volut DT
Voout
Ainit
Buit
U114122) Wint Wo Bint Anii
[1r]]
5

---

Algorithm LoRA-GA Initialization
Require: Model f(-) with L layers. parameters W, sampled batch B {z, v). LoRA rank LoRA alpha a, loss function L, scale factor y
Ensure: Initialized parameters W, 7/, A, B
1: 1 < f(x, W)
Forward pass
2: ((y.0)
3: 1 - /
4: for oarl=e. L....... do
5:
Compute W.,
D Backward for one layer
6:
dout, din size(Wi)
7:
U,S,V + ww;
8:
A + Vlnr Volut/>
9:
Bi + 11411221 Voout/
Wi Wi- - -IBAA
11:
Clear Ww,
Gradient for this layer is not needed anymore
end for
13:
return W, 1). A, B
To save GPU memory during LoRA-GA initialization, we utilized a technique similar to 41 By hooking into Py'Torch's backward process, compute the gradient for layer time and discard the computed gradients immediately. This ensures that our memory usage remains at (11) instead of OLL), where L is the number of layers. This approach allows the memory consumption during the initialization phase to be less than that during the subsequent LoRA finetuning phase. Our algorithm shown Algorithm [ If the sampled batch size large, we can also use gradient accumulation save memory further, as shown in Algorithm
Experiments
In this section, we evaluate the performance of LoRA-GA on various benchmark datasets, Initially, we assess Natural Language Understanding (NLU) capabilities using subset of the GLUE dataset [14] with the T5-Base model [15). Subsequently, we evaluate dialogue [16] 42). mathematical reasoning
17 43,, and coding abilities (18) 44 using the Llama 2-7B model 19 Finally, we do the ablation
study to prove the effectiveness of our method.
Baselines
We compare LoRA-GA with several baselines to demonstrate its effectiveness:
1. Fall-Finetunce Fine-tuning the model with all parameters. which requires the resources.
2. Vanilla LoRA [4]: Fine-tuning the model by inserting a low-rank matrix product BA into linear
layers. A is initialized using Kaiming initialization, while B is initialized zero.
3. LoRA Variants with Original Structure: This includes several methods that retain the original
LoRA structure:
rsLoRA [37
introduces a new scaling factor to stabilize the scale of LoRA.
-LoRA+ 36
updates the two matrices in LoRA with different learning rates.
proposes performing SVD on the weight matrix W at the beginning of training and initializing and B based on the components with larger singular values.
PPSSA 38
4. LoRA Variants with Modified Structure: This includes methods that modify the original LoRA structure:
DoRA (8] enhances the model's expressiveness by adding Icarnable magnitudes.
AdaLoRA (32 dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within fixed parameter budget.
4.1 Experiments on Natural Language Understanding
We fine-tune the T5-Base model on several datasets from the GLUE bench- mark, including MNLI, SST-2, CoLAA, QNLI, and MRPC. Performance is evaluated on the develop- ment set using accuracy the primary metric.
Models and Datasets

---

Table 1: Results of fine-tuning T5-base using Full-FT and various LoRA variants on a subset of
GLUE
MNLI 393k
SST-2 67k
CoLA 8.5k
QNLI 105k
MRPC 3.7k
Average
Size
Full
86.33.10.00
94.75.0.21
20.70.10.24
23.19_0.22
84.5610.73
87.91 82.08
LoRA
85.3010.04
94.0440.11
69.35
92.96=0.09
68.3810.01
PISSA
85.75
94.07 10.06
74.27 0.339
93.15 0.14
76.31
84.71
rsLoRA
85.73+0.10
94.1940.23
72.32+1.12
93.12 =0.09
52.86+2.22
79.64
LoRA+
85.81
93.85+0.24
77.53
93. 93.14_0.03
74.4341.39
84.95
DoRA
85.67 +0009
94.0440.53
72.04:0.94
93.04=0.06
68.08
82.57
AdaLoRA
85.45+0.
93.69 93.6940.20
69.16-0.24
91.66-0.0s
68. 18..28
81.62
LoRA-GA
85. 85.70+0.00
94.11+0.15
80.57 =0.20
93.18
85. 85.29 40.21
87.77
Implementation Details We utilize prompt tuning to fine-tune the T5-Base model on the GLUE benchmark. This involves converting labels into tokens (e.g.. "positive" "negative") and using the normalized probability of these tokens as the predicted label probability for classification. We provide the hyperparameters in Appendix D.I Each experiment is conducted with different random seeds, and the average performance is reported.
Results As shown in Table D LoRA-GG -AA consistently outperforms the original LoRA and other baseline methods, achieving performance comparable to full fine-tuning, Notably, LoRA-GA excels on smaller datasets such as CoLA and MRPC, demonstrating its ability to converge faster and effectively utilize limited training data.
4.2 Experiment on Large Language Model
Models and Datasets To evaluate the scalability of LoRA-GA, we train Llama 2-7B on three tasks: chat, math, and code.
1. Chat: We train model on 52k subset of WizardLM 42). filtering out responses that begin with "As an AI" or "Sorry". . We test our model on the MT- MT-Bench dataset [16], which consists of 80 multi-turn questions designed assess LLMs multiple aspects. The quality of the responses is judged by GPT-4, and we report the first turn score. 2. Math: We train model
43 dataset bootstrapped from other math instruction tuning datasets like GSM8K and MATH
100k subset of MetaMathQA
with higher complexity and diversity. We select data bootstrapped from the GSM8K training and apply filtering. Accuracy is reported on the GSM8K evaluation set.
45
Code: We train our model on 100k subset Code-Feedback [4), high-quality code instruction dataset, removing explanations after code blocks. The model is tested on HumanEval [8. which consists of 180 Python tasks, and we report the PASS@ I metric.
Implementation Details Our model is trained using standard supervised learning for language modelling. The loss for the input prompt is set to zero. Detailed hyperparameters can be found in Appendix (D22) Each experiment uses 3 different random seeds, and the average performance across these runs is reported.
Result Our results, as summarized in Table indicate that LoRA-GA outperforms or is comparable to other methods, including full full-finttniing Specifically, LoRA-GA achieves superior performance on both the GSM8K and Human-eval datasets, underscoring its effectiveness in handling tasks with higher complexity and diversity. On MT-Bench, LoRA-GA also demonstrates competitive performance, although it slightly trails behind DoRA. Nevertheless, LoRA-Ga achieves this with fewer parameters and approximately 70% of the training time required by DoRA. Additionally, as illustrated in Figure (Left) our method exhibits significantly faster convergence rate compared to Vanilla LoRA, with convergence rates comparable to those of full-finette full-finetunin..
7

---

Effect of Rank We attribute the performance discrepancies on the GSM8K and Human-eval datasets, when compared full-finetunin,, primarily to the representational limitations imposed by the low-rank approximation. To address this, we experimented with higher ranks, specifically rank=32 and rank=128. Our findings reveal that LoRA-GA maintains stability across different rank settings and, in some cases, even surpasses full-finetuning performance. As shown in Figure 2 (Left), higher ranks with our initialization also result in loss curves that closely resemble those of filltiint.nnng..
Table 2: Results of fine-tuning Llama 2-7b using Full-FT and various LoRA variants, tested on MT-Bench, GSM8K, and Human-eval. LoRA-GA significantly outperforms Vanilla LoRA and approaches the performance of Full Finetune. Unless otherwise specified, the LoRA rank is set
MT-Bench
GSM8K
Human-eval
Full
5.56=0.09
54..0
19.87+0.37
LoRA
5.61 10.10
42.08
14.761..11
PiSSA
5.30
44.5440.27
16.02+0.78
LsLoRA
5.25
45.62
16.01
LoRA+
5.71 10008
52. 10062
18.1710.52
DoRA
20.02
53.07 100.75
19.75
AdaLoRA
5.57
50.72
17.80
LoRA-GA
5.
53. 60 =0.30
19.81 81.41..4
LoRA-GA (Rank=32)
5.79
55.12
20. 10.
LoRA-GA (Rank=128)
6.13
55.07 #0.18
23.05
4.3 Ablation Study
We conducted ablation studies to evaluate the contributions of non-zero initialization, stable output, and gradient approximation in LoRA-GA using five distinct experimental settings. Details of each setting are provided in Table
Table 3: Initialization Methods and Corresponding Settings for Ablation Study. The table compares different initialization methods for LoRA and their settings for A, B, and ry. "+SO" denotes stable output, scaling parameters uppropriately to ensure stability. "+GA" refers to gradient approximation, where A and B are initialized using orthogonal matrices derived from singular value decomposition.
Method
A Initialization
B Initialization
LoRA
3 (-vV
3
0 N(O. Jom
a/r a/r
den
din
Gaussian
N(., Joue
+SO
dout/ 1/7 1/NNN0 (O, Jout
dous/ /T/l/V/: N
a/vi
+GA
ViI-r Viarr= ddmat V7
U|++1227]
a/r
LoRA-GA
U[r+1:20] 4 dont/ V7
a/vi
Table 4: Performance of different settings in the ablation study. Results are shown for MT-Bench. GSM8K, and Human-eval on Llama 2 7b, as well as the average performance on a subset of GLUE on T5-Base. Detailed results can be found in Table
MT-Bench
GSMSK
Human-eval
Average of GLUE
Full 5.5640.08
54.20+0.42
19.87
87.91 82.08
LoRA 5.61
42.08
14.7640.17
Gaussian
5.62+0.11
38.2140.06
14.7620.68
81.88
SO
5.72+0.04
42.81+1.14
15.5540.78
82.28
+ GA
5.48+0.02
46.6 46.65+1.17
16.1540.78
82.54
LoRA-GA
5.95+0.16
53.60+0.30
19.81+1.66
87.77

---

For both small and large models, we observe that simply changing LoRA's initialization Gaussian does not yield any performance gains and may result in a slight performance decline. However, when combined with either "+SO" (Stable Outpur) or "+GA" (Gradient Approximation), performance improves upon that of LoRA. LoRA-GA, which integrates both techniques, outperforms other methods. As shown in Figure ( (Left) and Figure 4 +SO and +GA also enhance convergence speed, and when both are combined, the training loss curve is even closer to that of full-finetuning. This indicates that both output stability and gradient approximation contribute to the improvement LoRA, each addressing different aspects of the model'ss 's performance.
Ablation Result The results are presented in Tables 4 and y
0.330
LoRA(Rank-8)
LoRA
0.275
L)RA-GA(Rank=8) LoRA-GA(Rank=32)
Gausslan
+S0
LoRd-GA/Rank=112) Fine-tuning
+GA
LoRA-GA Full Fine-Tuning
0.15
503 1000 2500 2003
23000
3000
300 1000 15:00
2000
2200
2000
Figure 2: (Left) Training loss curves of LoRA-GA with different ranks the MetaMathQA dataset. Higher ranks result in faster loss reduction, approaching the performance of full fine- fine-tuning. (Right) Training loss curves from the ablation study with different settings on the MetaMATHQA dataset. Compared to Vanilla LoRA, both components of LoRA-GA +SO (stable output) and +GA (gradient approximation), improve convergence speed. LoRA-GA achieves the fastest convergence, closely matching that of full fine-tuning.
4.4 Memory Costs and Running Time
We benchmark LoRA-GA on single RTX 3090 24GB GPU, 128-core CPU, and 256GB of RAM. As shown 55 the memory consumption of our new method does not exceed that used for training with LoRA, indicating no extra memory is needed. Additionally, the time cost of this operation is relatively negligible compared to the subsequent fine-tuning process. For instance, in the Fede-Feeddbck task, the training process took approximately 10 hours, while the initialization required only about minute, which is insignificant.
Table 5: Memory and Time Costs for Initialization and Fine-Tuning. "Parameters" indicates the number of parameters in the model, "Time(LoRA-GA) represents the time required for initialization. (iemoor(LoRA-GAN shows the memory usage during initialization, "LoRA" and "Full-FT" display the memory usage during LoRA and full fine- ane-tuning, respectively.
Parameters
Time(LoRA-GA)
Memory(LoRA-GG
LoRA
Full-FT
T5-Base
220M
2.8s
1.69G
2.71G
3.87G
Llama 2-7B
6738M
74.7s
18.77G
23.18G
63.92G
4.5 Performance with Different Index Set Schemas
Theorem 3.1 establishes multiple optimal initialization schemes through different choices of index sets LA and TB. While our primary experiments employed LA .....,, and Ig r 1.... we conducted additional experiments to validate this choice by comparing three schemes:
ArB2r: TA (1....,
r) lB (r 1,..., 2r}
* A2rBr: LA (r 1...., 2r) 1B (1.....r)

---

Random: Random assignment of first 2r indices into two groups
Table 6: Performance comparison of initialization schemes on GSM8k using models trained on MetaMathQA subset.
ArB2r
A2rBr
Random
Performance 52.79
52.38
52.01
proves these schemas are equivalent in the first step, their behaviors diverge afterward. The gradient matrix B (V (V A²) becomes larger than that of A (V B" V W L.,, effectively increasing B's learning rate. This aligns with findings from LoRA+ [36], where larger learning rates for B proved beneficial, potentially explaining ArB2r's superior performance.
As shown in Table ArB2r slightly outperforms the alternatives. While Theorem 3.1
4.6 Impact of Sampled Batch Size
The gradient approximation in LoRA-GA uses sampled batches, with smaller batches resembling Stochastic Gradient Descent (SGD) and larger ones approximating full Gradient Descent (GD). While theoretical work [46] suggests SGD's slower convergence may offer better generalization than GD, we conduct experiments to empirically evaluate different batch sizes.
We assess gradient approximation quality by comparing gradients from various batch sizes against reference batch size of 2048 which serves as a proxy for the full dataset gradient using two metrics:
Sign Similarity: The proportion of parameters sharing the same gradient sign.
Magnitude Similarity: The proportion of parameters within the same order of magnitude (where one's absolute value more than 10 times the other).
Table 7:
Gradient similarity metries (vs. batch size 2048) and model performance on GSM8k using models trained on MetaMathQA subset.
Batch Size
16
64
128
256
Sign Similarity 0.743
0.790
0.838
0.875
0.903
0.925
Magnitude Similarity
0.878
0.908
0.933
0.950
0.962
0.971
Performance
52.79
52.99
52.91
53.56
52.57
53.22
As shown in Table 7 both similarity metrics consistently improve with larger batch sizes, indicating better approximation of the full gradient. The results also demonstrate that while larger batch sizes tend to yield marginally better performance, however, the differences are relatively small. Based on these findings, recommend using moderately large batch size (e.g., 64) when computational resources permit.
5 Conclusions
In this paper, we present a novel initialization scheme for low-rank adaptation (LoRA), with the goal of acelcrating its convergence. By examining the initialization methods and update processes of LoRA, we develop a new initialization method, LoRA-GA, which approximates the gradients of the low-rank matrix product with those of full fine-tuning from the very first step.
Through extensive experiments, we have demonstrated that LoRA-GA achieves a convergence rate comparable that of full fine-tuning while delivering similar even superior performance. Since LoRA-GA solely modifies the initialization of LoRA without altering the architecture or training algorithms, it offers an efficient and effective approach that easy to implement. Furthermore, can also be incorporated with other LoRA variants. For example, ReLoRA 35 periodically merges the adapters into frozen weights W. which may allow LoRA-GA demonstrate advantages over more steps. We leave an interesting future direction.
10

---

Acknowledgments and Disclosure of Funding
The authors are supported in part by the National Natural Science Foundation of China Grant 62161146004.
References
[1] Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang. Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, ct al. Instruction tuning for large language models: survey, arXiv preprint arXiv:2308. 10792, 2023.
[2] Long Ouyang. Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-277 277444, 2022.
[3] Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403. 14608, 2024.
[4] Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.
[5] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances neural information processing systems, 30, 2017.
[6] Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang. Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-eflice fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence, 5(3):220-235, 2023.
[7] Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024.
[8] Shih- Yang Liu, Chien-Y Yi Wang. Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang- Ting Cheng, and Min-Hung Chen. Dora: Weight-decomee decompsee low-rank adaptation, 2024.
[9] Dan Biderman, Jose Gonzalez Ortiz, Jacoh Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint 20X.::2405.06633. 2024.
[10] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing, ACM Computing Surveyx, 55(9):1-35, 2023.
[11] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.
[12] Guy Gur-Ari, Daniel Roberts, and Ethan Dyer. Gradient descent happens a tiny subspace. arXiv preprint / 2018.
[13] Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012. 13255, 2020.
[14] Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv: 1804. 2018.
[15] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 216140))1-67, 2020.
[16] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
[17] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jenry Tworek. Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1 14168, 2021.

---

[18] Mark Chen, Jerry Tworek. Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03372 2021.
[19] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Ilama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[20] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256. JMLR Workshop and Conference Proceedings, 2010.
[21] Dmytro Mishkin and Jiri Matas. All you need is a good init, arXiv preprint arXiv:1511.06422 2015.
[22] Greg Yang. Edward J Hu, Igor Babuschkin. Szymon Sidor. Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv: 22030.2000 2022.
[23] Neil Houlsby, Andrei Ciurgiu, Stanislaw Jastrzebski. Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In International conference on machine learning. pages 2790-2799. PMI.R, 2019.
[24] Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Keng-Kirkpatrick, and Graham Neubig. Towards unified view of parameter-effiee
transfer learning. 2022.
[25] Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: -dduuraof-adappaations for parameter-efficient model tuning. arXiv preprint arXiv:2205. 12410, 2022.
[26] Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.002-4 002447, 2020.
[27] Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. 2021.
[28] Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis. Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization, 2023.
[29] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands. too. Al Open, 2023.
[30] Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXh::2001.00//0 00000000000000000 2021.
[31] Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memorry-fficient IIm training by gradient low-rank projection, 2024.
[32] Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng Hc, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-effiient finc-tuning, 2023.
[33] Nam Hyeon- Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficie federated learning. arXiv preprint arXiv:2108.00000 06098, 2021.
[34] Ali Edalati, Marzich Tahaci, Ivan Kobyzev, Vahid Partovi Nia, James Clark, and Mchdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.1066 2022.
[35] Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates, 2023.
[36] Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lorat: Efficient low rank adaptation of large models, 2024.
[37] Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023.
[38] Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.
[39] Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: Sf parameter-efficien fine-tuning methods. httpps//github. com www/houeeeeeeee.oooeeeeee huggingfacce peft 2022.
12

---

[40] Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 2020.
[41] Kai Lv, Yuqing Yang. Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint wrxi::2306.09722 2023.
[42] Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models follow complex instructions, 2023.
[43] Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li. Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models, 2024.
[44] Tianyu Zheng. Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterppeeer:: Integrating code generation execution and refinement, 2024.
[45] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang. Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with math dataset, 2021.
[46] Idan Amir, Tomer Koren, and Roi Livni. Sgd generalizes hetter than gd (and regularization doesn't help). In Conference on Learning Theory, pages 63-92. PMLR, 2021.
[47] Carl Eckart and Gale Young. The approximation of matrix by another of lower rank. Psychometrika, 13):211-218, 1936.
The quarterly journal of [48] Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. mathematics, 11(1)50055, 1960.
[49] Tlya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
13

---

A Proofs of Theorems
A.1 Proof of Theorem 3.1
Lemma 3.1 Suppose the loss function is L and y W'x (Wo + 1,B,, where is the ouiput layer is the input, the gradients of adapters A and are linear mappings of the gradient of W'
VAC= = B² W VaC= (Vw+L) A
Remarkably, the gradient W' in LoRA and the gradiens of W in full fine-tuning equal at the beginning of the training.
Proof. For the gradients LoRA,
0C
OC dy
0c by x²
V C
O
dy OW
OC
OW"
ac dy
OL x²' VwiL dy
V
B"
0A
0A
Dy OW
0c
ac
dy
OW
ac T A" (V. A
VBC
0B
dy OW
OB
dy
At the beginning of training, both LoR A and full fine-tuning have y' and identical æ, therefore,
2c Oy (y)=
VwL Vw
Theorem 3.1 Consider the following optimization problem:
1 V/V Ala Amit + vj Binit Blss Vw (Vw
min Ainit Pinit
If the Singular Value Decomposition (SVD) of VwL is Vw USV the solution to this optimization problem is:
VL VUA.
VVV s.t. LTA = Tal =, LAU -
Binit
Ainit
where La, 1a are index sets.
Proof. Since that rank( = rank(Buat) = r and 2r < min(m, n), we can assert that the matrix W' V E Binit Bius Vw has rank(WW)) < 2r.
Under this given solution,
W' V LA Amt + n¹² Blus Bins Ww DUnC=CUSV* (VI, V2, + +f
Σ aunt + KETA
Σ auugnn
jelg
By the classic Fekart- Fekart-Young Theorem (see e.g., Frohenius is:
47 48
the optimal low-rank approximation with respect
W arg
min rank(W
W
< -1 =
aquaoo
This is identical what have got. Therefore, this is the optimal solution.
A.2 Proof of Theorem 3.2
Lemma A.I. In R", if we randomly pick a vector x that > x2 = 1. we have:
E (mx)
(x?) 1 and (a?) = Or.doue (2);
E (w:ar))
3. E (x?=3)
Or.doue din (1):
14

---

4. E (1] = 0:
Proof. is equivalent to sampling random point uniformly from unit sphere in It".
For property 1, E (x.) holds obvious by symmetry. Since
> 23 = and uniformly distributed,
each entry has identical expectation, (Ei= x ni (a+) = 1. -.- (x (a? xx)
drdous.den (2) dod
(1) = Or.ooue.dy den (
For property 2, it can also be proved by symmetry: we can always find vector that contains (21, -xzz) also lies on the sphere. Therefore, I (2,22) = 0.
For property 3, (afaz) B (x7 2अ) din (2) dr.dous.da (
Dr.doue.d. din (+:2
property again it can proved by symmetry: we can always find vector that contains (Irs Tizs -2x) also lies on the sphere. Therefore, E (afayaz)
Lemma A.2. For randomly selected orthogonal matrix E IR and we randomly pick two different column vectors and y from it. For these two vectors, we have the following:
1. E (may) = 0;
2. E (x.,y) 0:
Proof. It is equivalent first selecting a random vector from unit sphere in R" uniformly, and then selecting the other one y that orthogonal to x.
For property 1, E.= Zays 0 (E m:i/:) Em= (0 x.yss) = E (a.g)) = 0.
For property consider that TE. (Σ - J.)
R..
(E Di vi) 0, and given
always find -y is also an
orthogonal vector. Therefore, E K= 2s
> U4) 0 B(z,8)) 0.
Theorem 3.2 Given the initialization proposed in Theorem assume that the orthogonal vectors in Anci and
Buii are randomly selected from id and RR/ous, and set Orstvacittnn (+) as suggested by rsLoRA [37].
Under these conditions, the adapters are forward scale-stable if Or,dowsddin (V (daaye and backward
scale-stable ( Ordoouiddn
dap p2
Prouf. LoRA, (W' + B/B.),. since that W' is not considered here, therefore, denote y = 1BAx.. dC
dC
nA n OL Represente
dL
When backward propagation, it's like
v and Th dr
as g. Therefore,
dx
0h
dip
ΣΣ 1=1
Bij Au 1 1si5 dout (Forward)
(4)
dene
=
A 1 1 din
(Backward)
aC
Since that the output of each layer in model always passes softmax function, so that the vector ah. Or,dgus.d,n (1). Further, since that input B(x?) =
is and
i.i.d., without loss of generality, assume that F(
For the adapter, as Equation Ssowss, shows, and by the expectations we have proved in Lemma A. and A.2 calculate the scale of forward and backward process.
we can
15

---

The scale of forward process is:
din
dan
E
(1.)
(B.j, Bijz "
11
dim
B BB,, Biju (Ajiki Ajakz) (3 TT2
(5)
E (Big B(A, = n"
B (B.,) E (A)A)
f1
r²
a² 6²²
11 dont din
done
The scale of the backward process is:
dout dout
E
(
T (AA A3z Bansa Uks M62)
Uki Ukz (AjiiAjai) (Brsis Bk2sa)
by
dout E
dont
1 din
E (B (A))
Uk
17 din dinc α²
j=l km1
(6)
From the results derived by Equation 5 and 6 one can see that we cannot find a proper Ç to make both scales
Or r,dout div (1) unless dumi Or,dout den (1). We also see that the forward scale is stable if adopting
doutt and the backward is stable if Ordouueddin
dán p2
< Orddouedden (pp2
B Additional Experimental Results
B.1 Convergence Speed
As Figure and shown, the convergence of LoRA-GA is significantly faster than vanilla LoRA and other ablation models, almost close to that of full fine-tuning, which support our claim about speed of convergence.
26
Lahh
Lift oo
L.10
1240
025
0.5
0.1
Figure 3: Training Loss curves of Full Fine-tuning, LoRA and LoRA-GA on different datasets.
16

---

CHILI
5312
VRU
COLA
LEGA
LARA
sodefierdback
3.40
0.0
Dan
0.5
LORA-CA
0.4-
E40
3000 2343 2009 2300
1290
3000
2390
3029
1039
1290 1592
Figure 4: Training Loss curves of different LoRA-GA ablations different datasets.
B.2 Evaluating the Rank of the Gradient Matrix
Theorem 3.eoreem..suuggesss. that the closer the rank of the gradient matrix is to 2r, the better the gradient approximated. thereby enhancing the theoretical effectiveness of our initialization. gradient matrices. The left panel depicts grid-like pattern in the gradients of weight matrix, indicating low-rank structure. The middle panel shows steeply declining low-rank nature the gradient matrix. The right panel presents cumulative curve squared singular values, demonstrating that few ranks account for nearly all the singular values of the gradient matrix. Specifically, the coverage in the right panel is defined as
iiluustaats the low-rank nature of
of singular values, reflecting the highly
E T
Coverage
El
where is the LoRA rank used LoRA-GA, indicating how much of the low-rank matrix can be approximated by this rank.
Scale of the s'adennt
Singuier values of the gradlest metri
1.2
16
0.3
3
15
Ohveage
-10
0.5
00
-12
04
05
02
2.2
107
15'
00
Oineraion
14 1000 EX04
Rack
Figure 5: (Left) A gradient matrix of T5-Base during fine-tuning on CoLA. (Middle) The decreasing curve of singular values of the gradient matrix. (Right) The cumulative curve showing the coverage of squared singular values.
We further validate this observation on larger models by analyzing LLaMA 2-7B during MetaMathQA training. Table Tableepreeeents presents the coverage rank achieve mean coverage 92.9% across layers, with the minimum coverage being Increasing the rank to 128 yields an impressive mean coverage of 99,3%, with the minimum coverage reaching 97.5%. These results demonstrate that even for large models with weight matrices of dimension 4096, modest LoRA rank is sufficient capture the majority of the gradient information.
different layers with varying LoRA ranks. Even with relatively small
17

---

Table 8: Coverage of gradient matrix across different layers in LLaMA 2-7B
LoRA Rank
8
32
128
Mean Coverage
0.929
0.974
0.993
Min Coverage
0.851
0.933
0.975
B.3 Detailed Ablation Study Result of GLUE
Table shows the full results of ablation study on the subset of GLUE, where the average scores are briefly reported in Table As dembosteddos,ttteed LoRA-GA outperforms all other ablation models, while both "+SO" and "+GA" methods gain some improvement from vanilla LoRA and simple non-zero initialization "Gaussian". This illustrates that both components LoRA-GA have positive contribution the improvement of performance.
Table 9: Performance comparison of different ablations subset of GLUE dataset. The settings elaborated in Table
MNLI 393k
SST-2 67k
CoLA 8.5k
QNLI 105k
MRPC 3.7k
Average
Trainset
Full 86.3340.00
94.75+0.21
80.70+0.24
93.1940.22
84.5640.73
87.91
LoRA
85.30=0.01
94.0440.11
69.35
92.96
68.3840.01
82.08
Gaussian
85.26
93.85
69.00...22
92.89
68.38+0
81.88
SO
85.47 10.19
94.23+..13
70.63
93.12 10.07
67.97
82.28
+ GA
85.33=0.07
93.88+0.18
74.37 11.12
93.03 93.03+0.06
66.09 -6.09=11.32
82.54
LoRA-GA
85.70 10.09
14.11
80.57 10.20
93. 10.06
85. 25.29.m. 10.24
87.77
B.4 Experimental result with different learning rate
Furthermore, also conduct experiments under learning rates le-5 and 5e-5. As Table TO and [T shown, LoRA-GA maintains strong performance different learning rates, which illustrating its robustness the variation learning rate.
Table 10: Performance comparison of different methods on MT-BBnch, GSM8K, and Human-eval with learning rate le-5
MTTBBech
GSM8K
Human-eval
Full
3..0
43.95
15.9740.42
LoRA
3.53
35.7320.09
14.35+00.00
PiSSA
3.61
38.51
15.37
rsLoRA
5.60
40.56
15.6940.5
LoRA+
3.4840.14
47.06
16.90 16.90+0.89
LoRA-GA
5.82+0.01
51.33
17. 17.6444.13
Table Performance comparison of different methods on MT-Bench, GSM8K, and Human-eval with learning rate 5e-5
MT-Bench
GSM8K
Human-eval
Full
3.33
56.33
25.6710.42
LoRA
5.52
46.8940.05
15.6710.60
PiSSA
3.35
49.70+t.80
17.62+0.60
rsLoRA
5.54+0.00
50.04+0.54
17. 17.38
LoRA+
5.89
55. 23 10.16
19.21
LoRA-GA
5.7640.22
52.79 +..0
20.45
18

---

B.5 Experiments on the Full MetaMathhQA Dataset
Following [ conducted additional experiments training the complete MetaMathQA dataset for
multiple epochs, whereas our main results in previous section were based on fine-tuning for one epoch on 100k subset of MetaMathQA, Due to computational constraints, we limited these extended experiments to three methods:
Lo) A+ (3), and LoRA-GA
Table 12 presents the performance across four epochs, averaged over two random seeds.
Table 12: Performance comparison of different methods full MetaMathQA dataset training for multiple epochs.
Epoch 1
Epoch 2
Epoch 3
Epoch 4
LoRA (Rank=8)
55.19
58.37
59.28
58.90
LoRA+ (Rank=8)
56.37
59 21
59.93
59.97
LoRA-GA (Rank=8)
56.48
58.64
60.16
60.88
The results show that LoRA-GA consistently achieves better performance than vanilla LoRA and outperforms LoRA+ in most cases across multiple epochs of training.
C LoRA-GA Initialization With Gradient Accumulation
Algorithm 2 LoRA-GA Initialization With Gradient Accumulation
Require: Model f(-) with L layers, parameters W. sampled batch B = {z, vt. LoRA rank r with n samples, LoRA alpha a, loss function 2, scale factor 7,, micro-batch size b
Ensure: Initialized parameters W, 7, A, B
f(x W)
Forward pass
2: l < (2y.0)
3: r) + 5
4: for -....... do
5: Vine l 4-
Initialize average gradient for each layer on CPU
6: end for
7: for each micro-batch B, in B do
8:
Ds + f(z,, W)
b Forward pass for micro-bated
9:
L - C(y,, 9:)
10:
for 1= L...., 1 do
Compute W. E
b Backward pass for one layer D Move to CPU Gradient for this layer is not needed anymore
W. VANE
V Vw; ls
13:
Clear V Vw,k. es
14:
end for
15:
end for
16: for
,
17:
dout, din Wi)
18:
U.S.V V
19:
A V/2:1 Voutt/
20:
Bi U1141:20] Vouut/
21:
Wi < Wi W
end for
23: return W. 17,.AAA. A, B
D Hyperparameter
D.I Experiments on Natural Language Understanding
We the following hyperparameters with T5-Base.
19

---

Training Algorithm: Adam [9 with 31 0.9. 32 = 0.999. le 8 and weight decay of O. For full finetuning, LoRA, and its variants, learning rate of 1c a warmup ratio of 0.03, and cosine decay are employed. For DoRA [8] a learning rate 2c used, while Adalora, a learning rate of 5e is applicd, both with the same warmup ratio and cosine decay adhering to their respective papers.
LoRA Hyperparameters: LoRA rank r 8, a 16. LoRA target is linear modules except embedding layer, layer norm and language model head.
LoRA-GA Hyperparameter: 7 16, sampled batch size sbs
Other Hyperparameters: Sequence Length T 128, train batch size bs 32, number of train epochs E 1. Precision FP32
D.2 Experiment on Large Language Model
We the following hyperparameters with Llama 2-7B.
Training Algorithm: AdamW 4 with with 31 0.9, = 0.999, c 1c and weight decay of 0. For full finetuning, LoRA, and its variants, learning rate of 2c [38]. a warmup ratio of 0.03, and cosine decay are employed. For DoRA (8), learning rate of 2c is used, while for Adalora, learning rate of 5e 4 is applied, both with the same warmup ratio and cosine decay adhering their respective papers.
The backbone model uses bf16 precision, while during training, LoRA's B and A matrices use fp32 precision, following the implementation of PEFT 39
Precision:
*
LoRA-GA Hyperparameter: 64, micro sampled batch size sbs = 1 with gradient accumulation of 32.
LoRA Hyperparameters: LoRA rank r = and a = 16 for all experiments.
Generation Hyperparameters: All generation performed with top_p 0.95 and temperature
T 0.8.
Other Hyperparameters: Number of train epochs E 1, train miero batch size màs 1 with gradient accumulation 32. Sequence Length 1024
E
Comparison between LoRA-GA and PiSSA
Both LoRA-GA and PiSSA 38 concentrate on the initialization of LoRA, and utilizing SVD on pre-trained models. While they may appear similar superficially, significant differences exist between them.
Firstly, the motivations behind LoRA-GA and PiSSA are fundamentally different. As discussed in Scction 3.2 LoRA-GA is motivated by the approximation of the LoRA update and full fine-tuning. We employ SVD on gradients solely because the optimal solution to the gradient approximation problem is precisely obtained (as stated in Theorem (.T). Conversely, PiSSA adopts SVD under the assumption that pre-trained weights possess a low intrinsic rank, and thus, the SVD weights can provide an accurate representation of original weights. In essence, LoRA-G GA emphasizes on gradients and decomposes them, whereas PiSSA concentrates on weights and decomposes them.
Secondly, LoRA-GA and PiSSA employ different scales of initialization. In Section 3.3 LoRA-GA derives an appropriate scaling factor by considering the forward and backward stability of our initialization scheme. On the other hand, PiSSA uses the largest singular values as the magnitude of orthogonal matrices directly.
F Limitations
In this paper, we have demonstrated that LoRA-GA can achieve performance comparable to full fine-tuning on the Base (220M) and Llama 2-7B models, while significantly reducing the number of parameters and associated costs. However, due computational resource constraints, we have not validated LoRA-GA on larger pre-trained models (e.g., Llama 2-70B).
In LoRA-GA, proposed that scaling factor is necessary. But in some experiments with large learning rates, we observed potential numerical instability due to the effect of scaling factor and learning rate. This limitation suggests a need for careful tuning of the scaling factor and learning rate to maintain stability.
Another limitation pertains to our evaluation scope. While we provide evaluations on MTBench, GSM8K, and Human-eval we did not assess our method on other datasets. Consequently, we cannot fully guarantee that our findings universally consistent across all benchmarks.
20

---

Additionally, we did not implement our method other LoRA variants that are orthogonal to improvements (e.g.. ReLoRA [3]). Therefore, cannot ascertain whether LoRA-GA would perform equally well with other
LoRA anchhiectures/mnpooveennts
Finally, compared the original LoRA, LoRA-GA requires double the checkpoint storage, as it necessitates storing both the initial adapter checkpoints (Aiust and Besst) and the final adapter checkpoints (A and B).
G Compute Resources
this paper, we utilized two types of GPUs: the RTX 3090 24GB GPU, supported by a 128-core and 256GB of RAM (hereinafter referred to as "the RTX 3090"), and the A100 80GB GPU (hereinafter referred to as "the A100").
For the experiments on T5-Base using the GLUE dataset, reported in Section 4.1 all computations were performed single RTX 3090. For the Llama 2-7B experiments, reported in Scction 4.2 full fine-tuning and DoRA scenarios were conducted on a single A100, while all other LoRA variants and LoRA-GA were executed on a single RTX 3090, Additionally, ablation studies presented in Section 4.3 RTX 3090.
were carried out on a single
H Broader Impacts
this paper, we identify some limitations vanilla LoRA and propose more efficient and effective method for LoRA initialization, LoRA-GA. LoRA-GA converges faster than vanilla LoRA and consistently achieves better evaluation results.
We believe that this work will have positive social impact. The primary reasons are follows: The high cost of training and fine-tuning large models is significant challenge today. LoRA-GA offers way to fine-tune with fewer parameters and lower computational costs while achieving comparable performance. reduce the cost fine-tuning models and, in turn, decrease energy consumption, such as electricity, contributing to the goal of low-carbon environment. Furthermore, as the size of large language models (LLM) continues grow, it becomes increasingly difficult for individuals small organizations develop their LLMs. However, with the help of LoRÁ-GA and open-source large models, the hardware barrier to entry in this area greatly reduced. This will promote democratization in the field large models, preventing monopolies and dictatorships by a few companies.
will
On the other hand, our method could potentially make it casier to train language models that generate fake news misleading information. This underscores the necessity for designing effective detectors to identify content generated by large language models (LLMs). Ensuring the responsible of this technology is crucial mitigating the risks associated with the misuse advanced language models.
21

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We included 3 contributions at the end of introduction. The method LoRA-Ga -A and stable scale are proposed in Section 3 The extensive experiments and results are in Section [
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The last section (Appendix F of this paper is entirely discuss the limitation of our works.
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciflcciiiinnnnn asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions
online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification: There are I lemma (Lemma 3.1 and 2 theorems (Theorem 3.1 and 3.2 proposed in our paper. All them properly proved in Appendix
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and refseeeefereneed.
22

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Hyperparameters are disclosed in Appendix D Other implementation information are disclosed in Section in paragraphs began with "Implementation details".
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example. if the contribution is novel architecture, describing the architecture fully might suffice, if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the case of a large language model), releasing of model checkpoint, other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe architecture clearly and fully.
(c) If the contribution a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct
model (e.g.. large language model), then there should either be
dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors welcome to describe the particular way they provide closed-source source models, it may be that access to the model is limited some way (e.g.. to registered users), hut it should be possible for other rescarchers to have some path to reproducing verifying the results.
reprodueibility. In the case of
5. Open access to data and code
Question: Does the paper provide open access to the data code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer: [Yes]
Justificatin:: All codes of our experiments are uploaded anonymous github. All datasets used in our experiments all open source, which has been declared and cited in Section (Introduction).
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines for more details.
http://niipsccccpplllpp
guides/Cood https///oooouuuaiiiiooooooooo
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
23

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://niss.cc/pppn for more details.
Buidee//DodeSuumiissioooooo
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release amonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g.. data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification: Details that are important to understand evaluate experimental results are shown in our paper or appendix like Table 3 Other details can be found in our codes.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or supplemental material.
Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- lion about the statistical significance of the experiments?
Answer: [Yes]
Justification: We report standard deviations our evaluation results like footnotes Table 2 [n]
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (c.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
is to report I-sigma error bars, one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CL. if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric eтror bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
Question: For each experiment. does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Discussed Appendix G
Guidelines:
24

---

The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Fthies http://wr..p...ccpuubll////taaaa
Answer: [Yes]
Justification:
have read I'm sure that our research conforms NeurlPS Code Ethics.
Guidelines:
The answer means that authors have not reviewed the NeurIPS Code of Fthics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g., if there is a special considcration duc to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed this in Appendix H
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considcrations (c.g., deploy- ment technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
foundational research and not tied to particular
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated relcase models, providing defenses in addition attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g.. pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our research is purely a foundamental reseach about LoRA, which cannot pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
25

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make hest faith effort.
12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected"?
Answer: [Yes]
Justification: All models and datascts used in our paper are all properly cited.
Guidelines:
The answer means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should he provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if it changed) should be provided.
If this information not available online, the authors encouraged reach out the asset's creators.
13. New Assets
Question: Are new assets introduced the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
The way run our codes work attached our codes.
Guidelines:
The answer means that paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: No human subjects participants involved in research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should included the main paper.
According to the NeurIPS Code of Ethics, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals Equivalent for Research with Human Subjects
26

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification: human subjects or participants involved our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such the institution conducting the review.
27


Reviews:
[{"review_id": "R3c1kB7Vyd", "reviewer": "Reviewer_P55u", "summary": "This paper proposes a novel initialization method for LoRA based on detailed theoretical analysis. The experimental results illustrate that the proposed method can achieve a great performance on the most tasks.", "questions": "My question is about the hyperparameter selecetion for the baselines. Whether the hyperparameters the authors used in this paper is a good  selection.", "limitations": "N/A", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Strength: \n1. The paper provide a beautiful theoretical analysis about the initialization method for the gradients of LoRA and full-parameter fine-tuning and then propose their method based on their analysis. \n2. The proposed method is also very clear and easy to follow. \n3. The provided results illustrate that the proposed method can achieve a better performance compared with the most LoRA variants.", "weaknesses": "Weakness:\n1. I think the main weakness is from the experiments. I think the authors should provide more results on some complex tasks. For example, the paper mainly focus on GULU benchmark and metamath-100k. However, I think GULU has been solved by current PEFT methods. Then, for the metamath-100k, it selects 100k data from the vanilla metamath dataset. I recommend the author conduct their experiments on the full metamath dataset because the recent LoRA-based papers usually focus on the full dataset and therefore the readers can obtain a fair comparison and more directly to understand which method is better. I think metamath-100k is not a popular choice although Pissa also uses this dataset. \n2. I am considering whether the authors tune the hyper-parameters of the baseline methods, because I find the results of some baselines are too weak. The best for LoRA-GA is usually not the best hyperparameters for other methods. I hope the paper can provide a fair comparison. That is also the reason why I suggest the author use a more popular dataset. For example, the default learning rate for the experiments on metamath is 2e−5, and the results on GSM8K are: LoRA (42.08), LoRA+(52.11), LoRA-GA(53.60). However, from the table 7 and table 8 in the appendix, we can find that the results of a larger learning rate 5e-5 in table 8: LoRA (46.89), LoRA+(55.23), LoRA-GA(52.79). That means increasing the learning rate from 2e-5 to 5e-5, the baseline LoRA can be improved from 42.08 to 46.89, LoRA+ can be increased from 52.11 to 55.23 and higher than LoRA-GA. So I considering whether we can obtain a better performance for the baselines when we further tune the hyperparameters. I think that is very important since the reader can better know whether each method can really improve the performance. \n\n\n[1] LoRA Learns Less and Forgets Less. \n\n[2] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning. \n\n[3] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your valuable suggestions and have conducted additional experiments in response to your feedback.\n\n### Dataset Selection\n\nRegarding your concern about the complexity of tasks, we acknowledge that GULU may have been effectively addressed by current PEFT methods. Consequently, we expanded our experiments to the full metamath dataset to facilitate fair comparisons and align with recent LoRA-based research.\n\n### Hyperparameter Tuning\n\nInitially, we tuned the hyperparameters using full fine-tuning and applied the same parameters to LoRA, LoRA+, and LoRA-GA. Based on your recommendation, we tuned the hyperparameters specifically for LoRA, LoRA+, and LoRA-GA on the full metamath dataset. This tuning was conducted by evaluating performance after training for one epoch, provided there were no numerical overflow or instability issues. We tested batch sizes of {32, 64, 128} and learning rates of {2e-5, 5e-5, 1e-4, 2e-4}. The optimal hyperparameters identified were:\n\n\\- **Vanilla LoRA:** {128, 1e-4}\n\n\\- **LoRA+:** {128, 5e-5}\n\n\\- **LoRA-GA:** {128, 5e-5}\n\nThe hyperparameter we obtained is similar to the result from hyperparameter search in [3]. For this experiment, we used greedy decoding instead of top-p sampling to better align with previous work.\n\n### Results\n\nWe observed that vanilla LoRA we implemented with rank 8 achieved comparable or superior results to those reported for LoRA with rank 16 in Figure S2 of [2]. The performance across epochs is as follows (average by 2 seeds):\n\n|                                   | Epoch 1   | Epoch 2   | Epoch 3   | Epoch 4   |\n| --------------------------------- | --------- | --------- | --------- | --------- |\n| LoRA (Rank=16, reported in [2])   | ~49       | ~54       | N/A       | ~58       |\n| LoRA (Rank=8, our implementation) | 55.19     | 58.37     | 59.28     | 58.90     |\n| LoRA+ (Rank=8)                    | 56.37     | **59.21** | 59.93     | 59.97     |\n| LoRA-GA (Rank=8)                  | **56.48** | 58.64     | **60.16** | **60.88** |\n\nThese results demonstrate that LoRA-GA consistently outperforms both vanilla LoRA and, most of the time, LoRA+.  The initialization used in LoRA-GA contributes to improved performance across multiple epochs. Additionally, LoRA+ and LoRA-GA are orthogonal methods; combining them might yield even better results with careful design and we leave it as an interesting future direction.\n\nThank you once again for your insightful feedback. We will incorporate these results and update our paper in the next version.\n\n\n[1] Yu, Longhui, et al. \"Metamath: Bootstrap your own mathematical questions for large language models.\" arXiv preprint arXiv:2309.12284 (2023).\n\n[2] Biderman, Dan, et al. \"Lora learns less and forgets less.\" arXiv preprint arXiv:2405.09673 (2024).\n\n[3] Pan, Rui, et al. \"LISA: Layerwise Importance Sampling for Memory-Efficient Large Language Model Fine-Tuning.\" *arXiv preprint arXiv:2403.17919* (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "STVAwC8tlZ", "reviewer": "Reviewer_vjMb", "summary": "LoRA has a slower convergence rate compared to full fine-tuning. This paper proposes a novel initialization method, LoRA-GA (Low-Rank Adaptation with Gradient Approximation), which aligns the gradients of the low-rank matrix product with those of full fine-tuning from the first step. Numerical experiments demonstrate that LoRA-GA achieves faster convergence and better or comparable performance to full fine-tuning, outperforming vanilla LoRA and its variants on several benchmark datasets.", "questions": "See the weakness.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-The paper introduces a novel initialization strategy for LoRA, enhancing its efficiency and performance without altering the architecture or training algorithm.\n-The idea of aligning the gradients of the low-rank product with the full model’s gradients at the initial step is innovative.\n-The combination of gradient approximation and stable scale for initialization is a unique contribution.", "weaknesses": "-The concept of using eigenvectors for initialization might not be entirely new, but its application in this specific context is original.\n-The paper could benefit from a deeper exploration of potential edge cases or limitations of the proposed initialization method.\n-Some sections, particularly those involving complex mathematical derivations, might be challenging for readers without a strong background in the area. Some of the detailed steps may need to be provided in the proof.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable feedback and insightful comments.\n\n### Question 1\n\n**Question1**: The concept of using eigenvectors for initialization might not be entirely new, but its application in this specific context is original. \n\n**Answer1**: Indeed, the idea of utilizing eigenvectors in initialization has been previously employed (e.g., PiSSA[1]). The novelty of our work lies in the approximation of the gradients in LoRA, with the eigenvectors being a result of this optimal low-rank approximation of the gradient.\n\n### Question 2\n\n**Question2**: The paper could benefit from a deeper exploration of potential edge cases or limitations of the proposed initialization method.  \n\n**Answer2**: Thank you for your suggestion. Although we did not encounter any edge cases in our experiments, we briefly discuss the possible edge cases below:\n\n1. The matrix $A$ in our LoRA initialization is $O \\left( \\frac{d_{\\mathrm{out}}^{1/4}}{d_{\\mathrm{in}}^{1/2}} \\right)$. Therefore, for certain extreme layers or MLPs where $d_{\\mathrm{out}}$ is significantly larger than $d_{\\mathrm{in}}$, the matrix $A$ in LoRA may experience numerical overflow. In our experiments, both Llama 2-7B and T5-base do not contain such matrices. We recommend that users of our method exercise caution by trying different stable scaling factor if their models include structures with such matrices to prevent potential numerical issues, or ensure careful management of numerical stability and precision.\n2. The batch size used for sampling to calculate the gradient is crucial. Intuitively, when the batch size is only 1, we initialize LoRA using the gradient of a single sample. If this sample is an outlier, the initialization could degrade to the case of Gaussian initialization (random direction), or even worse, lead to a completely incorrect direction. We recommend using a larger batch size (at least 8) within the available computational power to avoid such edge cases and ensure better optimization.\n\nWe will incorporate the above discussions in the next version of our paper.\n\n### Question 3\n\n**Question3**: Some sections, particularly those involving complex mathematical derivations, might be challenging for readers without a strong background in the area. Some of the detailed steps may need to be provided in the proof.\n\n**Answer3**: We acknowledge that sections with complex mathematical derivations may be challenging for readers without a strong background in this area. We apologize for any possible confusion. In response, we will provide more details and explanations of our mathematical proofs in the revised version.\n\nThank you again for your insightful feedback.\n\n[1] Meng, Fanxu, Zhaohui Wang, and Muhan Zhang. \"Pissa: Principal singular values and singular vectors adaptation of large language models.\" arXiv preprint arXiv:2404.02948 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ox73YXBzO8", "reviewer": "Reviewer_dSFq", "summary": "This paper proposes LoRA-GA, which uses an adapter to approximate the gradient update of weights. This method achieves a 2-4 times improvement in convergence speed compared to vanilla LoRA and offers better accuracy than other LoRA-based methods.", "questions": "1、\tThe accuracy provided in Table 2 is quite different from the numbers in PiSSA [1]. I wonder where the discrepancy comes from.\n2、\tFrom Table 4, the performance improvement brought by SO to Gaussian initialization is about 0.4%, while the improvement brought by SO to GA is 5.0%. Since the SO method appears to be general in Section 3.3, why does SO perform so well with GA?\n3、\tHow does the sampled batch size influence the initialization? \n4、\tWhat would happen if the value of r in A and B were swapped? Or if the index sets I_{A} and I_{B} were randomly chosen from [1,2r]?\n\n[1] Pissa: Principal singular values and singular vectors adaptation of large language models[J]. arXiv.", "limitations": "None", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper provides a novel perspective on the initialization of LoRA from the gradient  updation aspect.\n2. The paper is theoretically comprehensive, well-written, and easy to follow.", "weaknesses": "1. The difference between LoRA-GA and LoRA reparameterization is not shown in Figure 1. \n2. A discussion on the influence of sampled batch size seems necessary.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable questions and suggestion!\n### Weakness\n\n**Weakness**: The difference between LoRA-GA and LoRA reparameterization is not shown in Figure 1. \n\n**Answer**: The parameterization of LoRA and LoRA-GA is identical, with both methods utilizing low-rank matrices $A$ and $B$, as depicted in Figure 1. The key (and only) distinction lies in their initialization approaches, as described in the \"Initialization\" section of Figure 1. We will clarify this explicitly in the next version.\n\n### Question 1\n\n**Question 1**: The accuracy provided in Table 2 is quite different from the numbers in PiSSA [1]. I wonder where the discrepancy comes from. \n\n**Answer 1**: The discrepancy between PiSSA and our results is attributed to different rank settings. PiSSA reports accuracies on various tasks in Table 1 using a rank of 128 (according to **Figure 6(c) and Figure 14(b)** in their paper, only with rank=128 can such high performance be achieved), while our accuracy (**in Table 2**) is based on a rank of 8 for all LoRA variants. When comparing accuracies at a rank of 8, the accuracy for PiSSA that we implemented on GSM8K is 44.54 (**Table 2**), which surpasses the reported accuracy of less than 40 (**Figure 6(c) and Figure 14(b)**) in PiSSA’s paper. Remarkably, when comparing the performance at rank 128 from our paper and PiSSA, LoRA-GA still outperforms PiSSA on GSM8K (**55.07** vs. 53.07), and Human-eval (**23.05** vs. 21.95). Furthermore, even LoRA-GA at rank 8 surpasses PiSSA at rank 128 on GSM8K (**53.60** vs. 53.07).\n\n### Question 2\n\n**Question 2**: From Table 4, the performance improvement brought by SO to Gaussian initialization is about 0.4%, while the improvement brought by SO to GA is 5.0%. Since the SO method appears to be general in Section 3.3, why does SO perform so well with GA?\n\n**Answer 2**: Thanks for noticing this interesting discrepancy. Although we admit that we have not full understood the exact reason for this discrepancy, we suspect the following reason:  The SO method adjusts both the output and update magnitude. The GA method approximates the gradient of full fine-tuning, thereby identifying a good descent direction. Intuitively, with only SO, the model might take a large step in the wrong direction initially (due to random initialization), leading to a suboptimal region and affecting later optimization. However, with SO and GA, the initial steps are more accurate, and a larger step size becomes beneficial. Hence, SO performs much better in combination of GA. Investigating the deeper reason of this fact remains open to further exploration.\n\n### Question 3\n\n**Question 3**: How does the sampled batch size influence the initialization?\n\n**Answer 3**: LoRA-GA approximates the sampled batch gradient. Smaller batches resemble SGD, while larger batches are similar to full GD. According to Amir [1], SGD converges more slowly than GD but has better generalization. Therefore, it's difficult to definitively say which approach is theoretically superior. Hence, we conduct experiments to get empirical results for different sampled batch size:\n\n1. The similarity between sampled batch and full gradient:\n\n   To compare the gradients, we used sampled batch sizes of 8, 16, 32, 64, 128, and 256, and compared them with a large batch size of 2048, simulating the full dataset gradient. We used two metrics:\n\n   - Sign similarity: The proportion of parameters with the same gradient sign, important because Adam's first step is similar to SignGD.\n   - Magnitude similarity: The proportion of parameters within the same magnitude level (one's absolute value not larger than 10 times of the other).\n\n   |                      | 8 | 16 | 32 | 64 | 128 | 256 |\n   | :---: | :---: | :---: | :---: | :---: | :---: | :---: |\n   |   Sign similarity    |  0.743  |  0.790  |  0.838  |  0.875  |  0.903  |  0.925  |\n   | Magnitude similarity |  0.878  |  0.908  |  0.933  |  0.950  |  0.962  |  0.971  |\n\n   Increasing the sampled batch size improves the similarity.\n\n2. The final performance\n\n   We conducted experiments on Metamath-100k with LLaMA-2 7B with learning rate 5e-5 to assess the impact of batch size on final performance.\n\n   |     | 8 | 16 | 32 | 64 | 128 | 256 |\n   | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n   | MetaMath-100k | 52.79 | 52.99  | 52.91  | 53.56  |  52.57  |  53.22  |\n\n   Results indicate that larger batch sizes can lead to slightly better model performance, but generally stay around the same level. We recommend choosing a larger batch size (e.g., 64) to achieve relatively stable performance, if resources permit.\n\n### Question 4\n\n**Question 4**: What would happen if the value of $r$ in $A$ and $B$ were swapped? Or if the index sets $I_{A}$ and $I_{B}$ were randomly chosen from $[1,2r]$?\n\n**Answer 4**: We test it using three schemes:\n\n- ArB2r: choose $I_{A} = [1, r]$ and $I_{B} = [r+1, 2r]$ (The scheme used in the paper).\n- A2rBr: choose $I_{A} = [r+1, 2r]$ and $I_{B} = [1, r]$.\n- Random: choose $I_{A}$ and $I_{B}$ in size r randomly sampled from $[1, 2r]$.\n\n|     | ArB2r | A2rBr | Random |\n| :-: | :--: | :-: | :-: |\n| MetaMathQA-100k | 52.79 | 52.38 | 52.01  |\n\nWe found that the ArB2r scheme yielded the best results. Although equivalent in step 1, from step 2 onwards, the gradient of $B$ ($\\nabla_{B}\\mathcal{L} = (\\nabla_W \\mathcal{L}) A^T$) is larger than that of $A$ ($\\nabla_{A}\\mathcal{L} = B^T \\nabla_W \\mathcal{L}$). This can be seen as increasing the learning rate for matrix $B$. According to LoRA+[2], a larger learning rate for $B$ is beneficial, which might explain why ArB2r performs better.\n\nThank you again for your insightful feedback. We will update our paper in the next version.\n\n[1]Amir, Idan, Tomer Koren, and Roi Livni. \"SGD generalizes better than GD (and regularization doesn’t help).\" Conference on Learning Theory. PMLR, 2021.\n\n[2]Hayou, Soufiane, Nikhil Ghosh, and Bin Yu. \"Lora+: Efficient low rank adaptation of large models.\" arXiv preprint arXiv:2402.12354 (2024).", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces an innovative initialization method for LoRA that enhances convergence speed and performance of fine-tuning LLMs. The reviewers appreciated the novel approach, which aligns gradients of the low-rank adaptation with those of full fine-tuning at the first step, improving the convergence rate and model performance. Despite initial concerns about experimental complexity and hyperparameter tuning, the authors provided additional experiments and clarified their methodology in the rebuttal, addressing the reviewers' feedback. I recommend acceptance.
Decision: Accept (poster)