Title: LoRA-GA: Low-Rank Adaptation with Gradient Approximation
Venue: NeurIPS 2024
Accepted: yes
Keywords: Parameter-Efficient Fine-Tuning

Low-Rank Adaptation

Model Initialization

Large Language Model
Abstract: Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MTbench, GSM8k, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance.

LoRA-GA: Low-Rank Adaptation with Gradient Approximation

Abstract

Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular ParameterEfficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by finetuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA ( Lo w R ank A daptation with G radient A pproximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MT-bench, GSM8K, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance. Code is available at code.

1 Introduction

Fine-tuning large language models (LLMs) is essential for enabling advanced techniques such as instruction fine-tuning [1], reinforcement learning from human feedback (RLHF) [2], and adapting models to specific downstream applications. However, the computational and storage costs associated with full fine-tuning are prohibitively high, particularly as model sizes continue to grow. To address these challenges, methods of Parameter-Efficient Fine-Tuning (PEFT) (see e.g., [3]), such as LowRank Adaptation (LoRA) [4], have emerged and gained significant attention.

Linxi Yu

0.8

Full Fine-Tuning

LoRA

LoRA-GA

LoRA-GA

0.7

0.7

LoRA

Parameterization

0.6

0.6

+n

0.5

0.5

PretrainedWeights

B∈Rmxr

A ∈ Rrxn

0.4

Adapters

0.4

0.3

0.3

0.2

First200Steps

U,S, VT ← svd (VwL)

0

25

50

75

~ 5X SpeedUp

100

125

150

175200

Initialization

Binit

m

U(r+1:2r]

0.2

/m

V(1:r]

7←

0

500

1000

1500

2000

2500

3000

r

Steps

Instead of updating the parameters of the model directly, LoRA incorporates auxilary low-rank matrices B and A into the linear layers of models (such as the Q,K,V , and O matrices in a self-attention block [5]), while keeping the original layer weights W fixed. The modified layer is represented as y = ( W + ηBA ) x , where x is the input of that layer, y is the output, and η is the scaling factor. This approach significantly reduces the number of parameters that need to be fine-tuned, thereby lowering the computational and memory costs at each step.

Despite these benefits, extensive empirical evidence (see e.g., [6, 7, 8, 9]) shows that LoRA converges significantly slower compared to full finetune. This slower convergence often increases overall computational costs (measured in Floating Point Operations) and can sometimes lead to worse test performance. In our experiments, we typically observe that LoRA requires 5-6x more iterations and FLOPs to reach the same performance as full fine-tuning under the same learning rate, as shown in Figure 1.

To study the cause of slow convergence, we perform an in-depth investigation of the initialization strategy of LoRA's adapter weights. It is known that fine-tuning pretrained models using the same objective (e.g., language modeling) often converges faster than re-initializing new parameters (e.g., a classification head) [10]. This observation leads us to question whether the slow convergence of vanilla LoRA might be attributed to the default random initialization of adapter weights (LoRA initializes A using Kaiming initialization [11] and sets B to zero [4]). In our experiments, we find that different initialization strategies for LoRA can significantly impact the results, and its default initialization is suboptimal.

In pursuit of a convergence rate comparable to full fine-tuning, we aim for initialization so that the update of BA matches the update of W closely. Previous work suggests that gradient descent operates in a low-dimensional subspace [12, 13]. If we can closely approximate the gradients of the full model at the initial step, subsequent steps can also be approximated, potentially accelerating the convergence of LoRA.

To this end, we introduce a novel initialization method, LoRA-GA ( Lo w Ra nk G radient A pproximation). By initializing A init and B init with the eigenvectors of the full gradient matrix, the gradient of the low-rank product BA aligns with the direction of the gradient of the full weight matrix W . Mathematically, we aim to ensure that:

∆( BA ) ≈ ζ ∆ W, for some non-zero positive constant ζ.

Our contributions can be summarized as follows:

We propose LoRA-GA , a novel initialization method for LoRA that accelerates convergence by approximating the gradients of the low-rank matrices with ones of the full weight matrix.

2. We identify the scaling factor under non-zero initialization, which ensures the variance of adapter outputs is invariant to the rank of the adapter and the dimension of the input.

3. We validate LoRA-GA through extensive experiments, demonstrating significant performance improvements and faster convergence compared to vanilla LoRA. Specifically, LoRA-GA outperforms LoRA by 5.69% on the GLUE [14] subset with T5-Base [15], and by 0.34, 11.52%, and 5.05% on MT-bench [16], GSM8K [17], and HumanEval [18] with Llama 2-7B [19], respectively, while achieving up to 2-4 times faster convergence.

2 Related Work

2.1 Initialization

The significance of maintaining variance stability during initialization has been widely acknowledged to prevent the occurrence of diminishing or exploding phenomena. Xavier initialization [20] ensures stability in both the forward and backward passes of a network under a linear activation function. He initialization [11] extends this solution to networks using ReLU activation. Distinct from these, LSUV initialization [21] selects a mini-batch of data, performing a forward pass to determine the output variance, and subsequently normalizing it to ensure stability. Tensor program (see e.g., [22]) has emerged as a powerful technique for tuning various hyperparameters, including the initialization, for large models.

2.2 Parameter-Efficient Fine-Tuning (PEFT)

To fine-tune increasingly large language models within limited hardware resources, researchers have developed various Parameter-Efficient Fine-Tuning (PEFT) methods. Adapter-based methods [23, 24, 25, 26] incorporate new layers into existing model layers. While fine-tuning only these inserted layers significantly reduces resource consumption and requires much fewer parameters, this approach introduces additional latency during both forward and backward passes. Soft Prompt-based methods [10, 27, 28, 29, 30] prepend learnable soft tokens to the model's input to adapt the model to specific tasks. This approach effectively leverages the pre-trained model's capabilities, requiring only appropriate prompts for task adaptation, though it incurs computational overhead during inference. More broadly, GaLore [31] applies low-rank gradients to parameter updates for memory efficiency during training. While this approach is highly expressive and performant, it requires storing complete model checkpoints, consuming more storage than other PEFT methods.

2.3 LoRA's Variants

LoRA is one of the most popular PEFT methods that introduces the product of low-rank matrices alongside existing layers to approximate weight changes during fine-tuning. Several methods have been proposed to improve the structure of LoRA. AdaLoRA [32] dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within a fixed parameter budget. DoRA [8] enhances the model's expressiveness by adding learnable magnitudes to the direction adjustments made by low-rank matrix products. Additionally, LoHA [33] and LoKr [34] employ Hamiltonian and Kronecker products, respectively.

Despite these advancements, vanilla LoRA remains the most popular method due to its robust library and hardware support. Therefore, improving LoRA without altering its structure and at a low cost is crucial. Several recent methods focus on this aspect. ReLoRA [35] suggests periodically merging learned adapters into the weight matrices to enhance LoRA's expressibility. LoRA+ [36] proposes using different learning rates for the two matrices in LoRA to improve convergence. rsLoRA [37] introduces a new scaling factor to make the scale of the output invariant to rank. Although our stable scale approach appears similar to rsLoRA, rsLoRA assumes BA = 0 initialization, making r invariant to the update ∆ BA . In contrast, our stable scale ensures that non-zero initialized BA remains invariant to both rank and input dimension from the start.

Recently, PiSSA [38] proposes to initializing A and B to approximate the original matrix W , by performing SVD on W . Our method, however, is based on a very different idea, that is to approximate the gradient of W , which involves performing SVD on sampled gradients and properly scaling the initialized matrices, as detailed in Section E.

3 Methods

In this section, we analyze the initialization of LoRA and introduce our method, LoRA-GA. LoRAGA consists of two key components: (i) approximating the direction of the gradient of full finetune and (ii) ensuring rank and scale stability in the initialization process. We examine each component and subsequently present their integration within LoRA-GA.

3.1 Review of Vanilla LoRA

Structure of LoRA Based on the hypothesis that the updates of fine-tuning are low-rank [13], LoRA [4] proposes to use the product of two low-rank matrices to represent the incremental part of the original matrix W . Here, W is the weight matrix of a linear layer in the model. For example, in transformers, it could be the Q,K,V , or O matrices of the self-attention layer or the weight matrix in the MLP layer. Specifically, LoRA has the following mathematical form:

where W ′ , W 0 ∈ R m × n , B ∈ R m × r , and A ∈ R r × n , with r ≪ min( m,n ) . W 0 is the pre-trained weight matrix, remains frozen during the fine-tuning process, while A and B are trainable.

Initialization of LoRA Under LoRA's default initialization scheme [4, 39], matrix A is initialized using Kaiming uniform [11], while matrix B is initialized with all zeros. Consequently, BA = 0 and W ′ 0 = W 0 , ensuring that the initial parameters are unchanged.

If the additional term ∆ W = ηBA is initially non-zero (e.g., [38]), the frozen parameter can be adjusted to ensure the initial parameters unchanged. This can be expressed as:

where W frozen = W 0 -ηB init A init is frozen, and B and A are trainable in this case.

3.2 Gradient Approximation

Our goal is to ensure that the first-step update ∆( ηBA ) approximate the direction of the weight update ∆ W , i.e., ∆( ηBA ) ≈ ζ ∆ W for some non-zero positive constant ζ . We will discuss how to choose ζ in Section 3.3 and one can treat ζ as a fixed constant for now.

Consider a gradient descent step with learning rate λ , the updates for A and B are ∆ A = λ ∇ A L ( A init ) and ∆ B = λ ∇ B L ( B init ) , respectively. Assuming learning rate λ is small, the update of ηBA at the first step can be expressed as:

To measure its approximation quality of scaled the update of the weights in full finetune ζ ∆ W = ζλ ∇ W L ( W 0 ) , we use the Frobenius norm of the difference between these two updates as a criterion:

Lemma 3.1. Suppose the loss function is L and y = W ′ x = ( W 0 + ηBA ) x , where y is the output of a layer and x is the input, the gradients of A and B are linear mappings of the gradient of W ′ :

Remarkably, ∇ W ′ L in LoRA and ∇ W L in full fine-tuning are equal at the beginning of the training.

By substituting the gradients in Lemma 3.1 into Equation 1, we can rewrite the criterion as follows:

This criterion evaluates how well the adapter's gradient approximates the direction of the gradient of full fine-tuning, and minimizing it brings the gradient of LoRA closer to that of full fine-tuning with a scaling factor ζ :

Theorem 3.1. For the optimization problem in Equation 3 with given ζ , if the Singular Value Decomposition (SVD) of ∇ W L is ∇ W L = USV T , the solution is:

where I A and I B are index sets.

Theorem 3.1 provides an appropriate initialization scheme for A init and B init given a specific ζ . The selection of ζ , which influences the scaling of the update ηBA , will be discussed in the following section.

3.3 Scale Stability

Inspired by rsLoRA citekalajdzievski2023rank and the Kaiming initialization [11], we define the following notions of stability:

Definition 3.1. When d out , d in , r → ∞ , an adapter ηBA exhibits two distinct types of scale stabilities:

Forward stability : If the inputs to the adapter are independently and identically distributed (i.i.d.) with 2nd moment Θ r,d out ,d in (1) , then the 2nd moment of the outputs remains Θ r,d out ,d in (1) .

Backward stability : If the gradient of the loss with respect to the adapter outputs is Θ r,d out ,d in (1) , then the gradient with respect to the inputs remains Θ r,d out ,d in (1) .

Theorem 3.2. Given the initialization proposed in Theorem 3.1, assume that the orthogonal vectors in A init and B init are randomly selected from the unit spheres in R d in and R d out with the constraint that the vectors are orthogonal to each other, and η = Θ r,d out ,d in (1 / √ r ) as suggested by rsLoRA [37]. Under these conditions, the adapters are forward scale-stable if ζ = Θ r,d out ,d in ( √ d out /r 2 )

Similar to the results obtained from Kaiming Initialization [11], we observe that either ζ = Θ r,d out ,d in ( √ d out /r 2 ) or ζ = Θ r,d out ,d in ( √ d in /r 2 ) work well independently. For all models presented in this paper, either form ensures convergence. Consequently, for all subsequent experiments, we adopt ζ = Θ r,d out ,d in ( √ d out /r 2 ) .

Remark. We would like to remark that the scaling factor proposed in this subsection proves to be beneficial primarily when one adopts the learning rate typically used in full-finetuning (e.g., 1 e -5 ), since as LoRA-GA attempts to approximate the updates of full-finetuning. However, recent research [9] suggests that LoRA with default initialization performs much better with larger learning rates. Furthermore, tensor program analysis [40, 22] indicates that higher learning rates should be paired with smaller initialization magnitudes. Therefore, we recommend decreasing or omitting the scaling factor when training using larger learning rates (e.g., > 1 e -4 ).

3.4 LoRA-GA Initialization

Combining the gradient approximation and stable scale components, we propose the LoRA-GA initialization method. First, we initialize A init and B init using the solution from Theorem 3.1. Then, we determine the scaling factor ζ according to Theorem 3.2 to ensure rank and scale stability. Thus, based on Theorems 3.1 and 3.2, we propose a novel initialization method, LoRA-GA.

LoRA-GA : We adopt η = α √ r and ζ = α 2 γ 2 √ d out r 2 , where γ is a hyperparameter. We define the index sets I A = { i | 1 ≤ i ≤ r, i ∈ N } and I B = { i | r +1 ≤ i ≤ 2 r, i ∈ N } . Denote the singular value decomposition (SVD) of ∇ W L as ∇ W L = USV T . The initializations are as follows:

Algorithm 1 LoRA-GA Initialization

To save GPU memory during LoRA-GA initialization, we utilized a technique similar to [41]. By hooking into PyTorch's backward process, we compute the gradient for one layer at a time and discard the computed gradients immediately. This ensures that our memory usage remains at O (1) instead of O ( L ) , where L is the number of layers. This approach allows the memory consumption during the initialization phase to be less than that during the subsequent LoRA finetuning phase. Our algorithm is shown in Algorithm 1. If the sampled batch size is large, we can also use gradient accumulation to save memory further, as shown in Algorithm 2.

4 Experiments

In this section, we evaluate the performance of LoRA-GA on various benchmark datasets. Initially, we assess Natural Language Understanding (NLU) capabilities using a subset of the GLUE dataset [14] with the T5-Base model [15]. Subsequently, we evaluate dialogue [16, 42], mathematical reasoning [17, 43], and coding abilities [18, 44] using the Llama 2-7B model [19]. Finally, we do the ablation study to prove the effectiveness of our method.

Baselines We compare LoRA-GA with several baselines to demonstrate its effectiveness:

Full-Finetune : Fine-tuning the model with all parameters, which requires the most resources.

Vanilla LoRA [4]: Fine-tuning the model by inserting a low-rank matrix product BA into linear layers. A is initialized using Kaiming initialization, while B is initialized to zero.

LoRA Variants with Original Structure : This includes several methods that retain the original LoRA structure:

-rsLoRA [37] introduces a new scaling factor to stabilize the scale of LoRA.

-LoRA+ [36] updates the two matrices in LoRA with different learning rates.

-PiSSA [38] proposes performing SVD on the weight matrix W at the beginning of training and initializing A and B based on the components with larger singular values.

LoRA Variants with Modified Structure : This includes methods that modify the original LoRA structure:

-DoRA [8] enhances the model's expressiveness by adding learnable magnitudes.

-AdaLoRA [32] dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within a fixed parameter budget.

4.1 Experiments on Natural Language Understanding

Models and Datasets We fine-tune the T5-Base model on several datasets from the GLUE benchmark, including MNLI, SST-2, CoLA, QNLI, and MRPC. Performance is evaluated on the development set using accuracy as the primary metric.

Implementation Details We utilize prompt tuning to fine-tune the T5-Base model on the GLUE benchmark. This involves converting labels into tokens (e.g., "positive" or "negative") and using the normalized probability of these tokens as the predicted label probability for classification. We provide the hyperparameters in Appendix D.1. Each experiment is conducted with 3 different random seeds, and the average performance is reported.

Results As shown in Table 1, LoRA-GA consistently outperforms the original LoRA and other baseline methods, achieving performance comparable to full fine-tuning. Notably, LoRA-GA excels on smaller datasets such as CoLA and MRPC, demonstrating its ability to converge faster and effectively utilize limited training data.

4.2 Experiment on Large Language Model

Models and Datasets To evaluate the scalability of LoRA-GA , we train Llama 2-7B on three tasks: chat , math , and code .

1. Chat : We train our model on a 52k subset of WizardLM [42], filtering out responses that begin with "As an AI" or "Sorry". We test our model on the MT-Bench dataset [16], which consists of 80 multi-turn questions designed to assess LLMs on multiple aspects. The quality of the responses is judged by GPT-4, and we report the first turn score.

Math : We train our model on a 100k subset of MetaMathQA [43], a dataset bootstrapped from other math instruction tuning datasets like GSM8K[17] and MATH [45], with higher complexity and diversity. We select data bootstrapped from the GSM8K training set and apply filtering. Accuracy is reported on the GSM8K evaluation set.

3. Code : We train our model on a 100k subset of Code-Feedback [44], a high-quality code instruction dataset, removing explanations after code blocks. The model is tested on HumanEval [18], which consists of 180 Python tasks, and we report the PASS@1 metric.

Implementation Details Our model is trained using standard supervised learning for language modelling. The loss for the input prompt is set to zero. Detailed hyperparameters can be found in Appendix D.2. Each experiment uses 3 different random seeds, and the average performance across these runs is reported.

Result Our results, as summarized in Table 2, indicate that LoRA-GA outperforms or is comparable to other methods, including full-finetuning. Specifically, LoRA-GA achieves superior performance on both the GSM8K and Human-eval datasets, underscoring its effectiveness in handling tasks with higher complexity and diversity. On MT-Bench, LoRA-GA also demonstrates competitive performance, although it slightly trails behind DoRA. Nevertheless, LoRA-GA achieves this with fewer parameters and approximately 70% of the training time required by DoRA. Additionally, as illustrated in Figure 2 (Left), our method exhibits a significantly faster convergence rate compared to Vanilla LoRA, with convergence rates comparable to those of full-finetuning.

Effect of Rank We attribute the performance discrepancies on the GSM8K and Human-eval datasets, when compared to full-finetuning, primarily to the representational limitations imposed by the low-rank approximation. To address this, we experimented with higher ranks, specifically rank=32 and rank=128. Our findings reveal that LoRA-GA maintains stability across different rank settings and, in some cases, even surpasses full-finetuning performance. As shown in Figure 2 (Left), higher ranks with our initialization also result in loss curves that closely resemble those of full-finetuning.

4.3 Ablation Study

We conducted ablation studies to evaluate the contributions of non-zero initialization, stable output, and gradient approximation in LoRA-GA using five distinct experimental settings. Details of each setting are provided in Table 3.

Ablation Result The results are presented in Tables 4 and 9. For both small and large models, we observe that simply changing LoRA's initialization to Gaussian does not yield any performance gains and may result in a slight performance decline. However, when combined with either "+SO" (Stable Output) or "+GA" (Gradient Approximation), performance improves upon that of LoRA. LoRA-GA, which integrates both techniques, outperforms other methods. As shown in Figure 2 (Left) and Figure 4, +SO and +GA also enhance convergence speed, and when both are combined, the training loss curve is even closer to that of full-finetuning. This indicates that both output stability and gradient approximation contribute to the improvement of LoRA, each addressing different aspects of the model's performance.

0.300

0.35

LoRA(Rank=8)

LoRA

0.275

LoRA-GA(Rank=8)

0.250

LoRA-GA(Rank=32)

0.30

+SO

LoRA-GA(Rank=128)

+GA

0.225

FullFine-tuning

LoRA-GA

0.25

FullFine-Tuning

0.200

0.175

0.20

0.150

0.125

0.15

0.100

0

500

1000

1500

2000

2500

3000

0

500

1000

1500

2000

2500

3000

4.4 Memory Costs and Running Time

We benchmark LoRA-GA on a single RTX 3090 24GB GPU, a 128-core CPU, and 256GB of RAM. As shown in Table 5, the memory consumption of our new method does not exceed that used for training with LoRA, indicating no extra memory is needed. Additionally, the time cost of this operation is relatively negligible compared to the subsequent fine-tuning process. For instance, in the Code-Feedback task, the training process took approximately 10 hours, while the initialization required only about 1 minute, which is insignificant.

4.5 Performance with Different Index Set Schemas

Theorem 3.1 establishes multiple optimal initialization schemes through different choices of index sets I A and I B . While our primary experiments employed I A = 1 , . . . , r and I B = r +1 , . . . , 2 r , we conducted additional experiments to validate this choice by comparing three schemes:

ArB2r : I A = { 1 , . . . , r } , I B = { r +1 , . . . , 2 r }

A2rBr : I A = { r +1 , . . . , 2 r } , I B = { 1 , . . . , r }

Random : Random assignment of first 2 r indices into two groups

Table 6: Performance comparison of initialization schemes on GSM8k using models trained on MetaMathQA subset.

As shown in Table 6, ArB2r slightly outperforms the alternatives. While Theorem 3.1 proves these schemas are equivalent in the first step, their behaviors diverge afterward. The gradient of matrix B ( ∇ B L = ( ∇ W L ) A T ) becomes larger than that of A ( ∇ A L = B T ∇ W L ), effectively increasing B 's learning rate. This aligns with findings from LoRA+[36], where larger learning rates for B proved beneficial, potentially explaining ArB2r's superior performance.

4.6 Impact of Sampled Batch Size

The gradient approximation in LoRA-GA uses sampled batches, with smaller batches resembling Stochastic Gradient Descent (SGD) and larger ones approximating full Gradient Descent (GD). While theoretical work [46] suggests SGD's slower convergence may offer better generalization than GD, we conduct experiments to empirically evaluate different batch sizes.

We assess gradient approximation quality by comparing gradients from various batch sizes against a reference batch size of 2048 which serves as a proxy for the full dataset gradient using two metrics:

Sign Similarity : The proportion of parameters sharing the same gradient sign.

Magnitude Similarity : The proportion of parameters within the same order of magnitude (where one's absolute value is not more than 10 times the other).

As shown in Table 7, both similarity metrics consistently improve with larger batch sizes, indicating better approximation of the full gradient. The results also demonstrate that while larger batch sizes tend to yield marginally better performance, however, the differences are relatively small. Based on these findings, we recommend using a moderately large batch size (e.g., 64) when computational resources permit.

5 Conclusions

In this paper, we present a novel initialization scheme for low-rank adaptation (LoRA), with the goal of acelerating its convergence. By examining the initialization methods and update processes of LoRA, we develop a new initialization method, LoRA-GA , which approximates the gradients of the low-rank matrix product with those of full fine-tuning from the very first step.

Through extensive experiments, we have demonstrated that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning while delivering similar or even superior performance. Since LoRA-GA solely modifies the initialization of LoRA without altering the architecture or training algorithms, it offers an efficient and effective approach that is easy to implement. Furthermore, it can also be incorporated with other LoRA variants. For example, ReLoRA [35] periodically merges the adapters into frozen weights W , which may allow LoRA-GA to demonstrate its advantages over more steps. We leave it as an interesting future direction.

Acknowledgments and Disclosure of Funding

The authors are supported in part by the National Natural Science Foundation of China Grant 62161146004.

References

Shengyu Zhang, Linfeng Dong, Xiaoya Li, Sen Zhang, Xiaofei Sun, Shuhe Wang, Jiwei Li, Runyi Hu, Tianwei Zhang, Fei Wu, et al. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792 , 2023.

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems , 35:27730-27744, 2022.

Zeyu Han, Chao Gao, Jinyang Liu, Sai Qian Zhang, et al. Parameter-efficient fine-tuning for large models: A comprehensive survey. arXiv preprint arXiv:2403.14608 , 2024.

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models, 2021.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Parameter-efficient fine-tuning of large-scale pre-trained language models. Nature Machine Intelligence , 5(3):220-235, 2023.

Rui Pan, Xiang Liu, Shizhe Diao, Renjie Pi, Jipeng Zhang, Chi Han, and Tong Zhang. Lisa: Layerwise importance sampling for memory-efficient large language model fine-tuning, 2024.

Shih-Yang Liu, Chien-Yi Wang, Hongxu Yin, Pavlo Molchanov, Yu-Chiang Frank Wang, Kwang-Ting Cheng, and Min-Hung Chen. Dora: Weight-decomposed low-rank adaptation, 2024.

Dan Biderman, Jose Gonzalez Ortiz, Jacob Portes, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, et al. Lora learns less and forgets less. arXiv preprint arXiv:2405.09673 , 2024.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys , 55(9):1-35, 2023.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification, 2015.

Guy Gur-Ari, Daniel A Roberts, and Ethan Dyer. Gradient descent happens in a tiny subspace. arXiv preprint arXiv:1812.04754 , 2018.

Armen Aghajanyan, Luke Zettlemoyer, and Sonal Gupta. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. arXiv preprint arXiv:2012.13255 , 2020.

Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461 , 2018.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research , 21(140):1-67, 2020.

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems , 36, 2024.

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 , 2021.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 , 2023.

Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics , pages 249-256. JMLR Workshop and Conference Proceedings, 2010.

Dmytro Mishkin and Jiri Matas. All you need is a good init. arXiv preprint arXiv:1511.06422 , 2015.

Greg Yang, Edward J Hu, Igor Babuschkin, Szymon Sidor, Xiaodong Liu, David Farhi, Nick Ryder, Jakub Pachocki, Weizhu Chen, and Jianfeng Gao. Tensor programs v: Tuning large neural networks via zero-shot hyperparameter transfer. arXiv preprint arXiv:2203.03466 , 2022.

Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International conference on machine learning , pages 2790-2799. PMLR, 2019.

Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg-Kirkpatrick, and Graham Neubig. Towards a unified view of parameter-efficient transfer learning, 2022.

Yaqing Wang, Sahaj Agarwal, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. Adamix: Mixture-of-adaptations for parameter-efficient model tuning. arXiv preprint arXiv:2205.12410 , 2022.

Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé, Kyunghyun Cho, and Iryna Gurevych. Adapterfusion: Non-destructive task composition for transfer learning. arXiv preprint arXiv:2005.00247 , 2020.

Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning, 2021.

Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Madian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. Residual prompt tuning: Improving prompt tuning with residual reparameterization, 2023.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. AI Open , 2023.

Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint arXiv:2101.00190 , 2021.

Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. Galore: Memory-efficient llm training by gradient low-rank projection, 2024.

Qingru Zhang, Minshuo Chen, Alexander Bukharin, Nikos Karampatziakis, Pengcheng He, Yu Cheng, Weizhu Chen, and Tuo Zhao. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning, 2023.

Nam Hyeon-Woo, Moon Ye-Bin, and Tae-Hyun Oh. Fedpara: Low-rank hadamard product for communication-efficient federated learning. arXiv preprint arXiv:2108.06098 , 2021.

Ali Edalati, Marzieh Tahaei, Ivan Kobyzev, Vahid Partovi Nia, James J Clark, and Mehdi Rezagholizadeh. Krona: Parameter efficient tuning with kronecker adapter. arXiv preprint arXiv:2212.10650 , 2022.

Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. Relora: High-rank training through low-rank updates, 2023.

Soufiane Hayou, Nikhil Ghosh, and Bin Yu. Lora+: Efficient low rank adaptation of large models, 2024.

Damjan Kalajdzievski. A rank stabilization scaling factor for fine-tuning with lora, 2023.

Fanxu Meng, Zhaohui Wang, and Muhan Zhang. Pissa: Principal singular values and singular vectors adaptation of large language models, 2024.

Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. Peft: State-of-the-art parameter-efficient fine-tuning methods. https://github.com/huggingface/ peft , 2022.

Greg Yang and Edward J Hu. Feature learning in infinite-width neural networks. arXiv preprint arXiv:2011.14522 , 2020.

Kai Lv, Yuqing Yang, Tengxiao Liu, Qinghui Gao, Qipeng Guo, and Xipeng Qiu. Full parameter fine-tuning for large language models with limited resources. arXiv preprint arXiv:2306.09782 , 2023.

Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions, 2023.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024.

Tianyu Zheng, Ge Zhang, Tianhao Shen, Xueling Liu, Bill Yuchen Lin, Jie Fu, Wenhu Chen, and Xiang Yue. Opencodeinterpreter: Integrating code generation with execution and refinement, 2024.

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021.

Idan Amir, Tomer Koren, and Roi Livni. Sgd generalizes better than gd (and regularization doesn't help). In Conference on Learning Theory , pages 63-92. PMLR, 2021.

Carl Eckart and Gale Young. The approximation of one matrix by another of lower rank. Psychometrika , 1(3):211-218, 1936.

Leon Mirsky. Symmetric gauge functions and unitarily invariant norms. The quarterly journal of mathematics , 11(1):50-59, 1960.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

A Proofs of Theorems

A.1 Proof of Theorem 3.1

Lemma 3.1. Suppose the loss function is L and y = W ′ x = ( W 0 + ηBA ) x , where y is the output of a layer and x is the input, the gradients of adapters A and B are linear mappings of the gradient of W ′ :

Remarkably, the gradient of W ′ in LoRA and the gradient of W in full fine-tuning are equal at the beginning of the training.

Proof. For the gradients in LoRA,

At the beginning of training, both LoRA and full fine-tuning have y ′ = y and identical x , therefore,

Theorem 3.1. Consider the following optimization problem:

If the Singular Value Decomposition (SVD) of ∇ W L is ∇ W L = USV T , the solution to this optimization problem is:

where I A , I B are index sets.

Proof. Since that rank ( A init ) = rank ( B init ) = r and 2 r < min( m,n ) , we can assert that the matrix W ′ = η 2 ∇ W L A T init A init + η 2 B init B T init ∇ W L has rank ( W ′ ) ≤ 2 r .

Under this given solution,

By the classic Eckart-Young Theorem (see e.g., [47, 48]), the optimal low-rank approximation with respect to Frobenius norm is:

This is identical to what we have got. Therefore, this is the optimal solution.

A.2 Proof of Theorem 3.2

Lemma A.1. In R n , if we randomly pick a vector x that ∑ n i =1 x 2 i = 1 , we have:

E ( x i x j ) = 0 ;

Proof. It is equivalent to sampling a random point uniformly from a unit sphere in R n .

For property 1, E ( x i ) = 0 holds obvious by symmetry. Since ∑ n i =1 x 2 i = 1 and uniformly distributed, each entry has identical expectation, E (∑ n i =1 x 2 i ) = n E ( x 2 i ) = 1 , E ( x 2 i ) = 1 n . E ( x 4 i ) = E ( x 2 i · x 2 i ) = Θ r,d out ,d in ( 1 n ) Θ r,d out ,d in ( 1 n ) = Θ r,d out ,d in ( 1 n 2 ) .

For property 2, it can also be proved by symmetry: we can always find vector that contains ( x i , -x j ) also lies on the sphere. Therefore, E ( x i x j ) = 0 .

For property 4, again it can be proved by symmetry: we can always find vector that contains ( x i , x j , -x k ) also lies on the sphere. Therefore, E ( x 2 i x j x k ) = 0 .

Lemma A.2. For a randomly selected orthogonal matrix A ∈ R n × n , and we randomly pick two different column vectors x and y from it. For these two vectors, we have the following:

E ( x i y i ) = 0 ;

E ( x i y j ) = 0 ;

Proof. It is equivalent to first selecting a random vector x from a unit sphere in R n uniformly, and then selecting the other one y that is orthogonal to x .

For property 2, consider that E (∑ n i =1 x i ) = E (∑ n i =1 y i ) = 0 , and given x , we can always find -y is also an orthogonal vector. Therefore, E (∑ n i =1 x i ∑ n i =1 y i ) = 0 ⇒ E ( x i y i ) = 0 .

Theorem 3.2. Given the initialization proposed in Theorem 3.1, assume that the orthogonal vectors in A init and B init are randomly selected from R d in and R d out , and set η = Θ r,d out ,d in ( 1 √ )

r as suggested by rsLoRA [37]. Under these conditions, the adapters are forward scale-stable if ζ = Θ r,d out ,d in (√ d out r 2 ) and backward scale-stable if ζ = Θ r,d out ,d in (√ d in r 2 ) .

.

Proof. In LoRA, h = ( W ′ + ηBA ) x , since that W ′ is not considered here, therefore, denote y = ηBAx When backward propagation, it's like ∂ L ∂x = ηA T B T ∂ L ∂h . Represente ∂ L ∂h as v and ∂ L ∂x as g . Therefore,

Since that the output of each layer in model always passes a softmax function, so that the vector ∂ L ∂h = v is Θ r,d out ,d in (1) . Further, since that input x i 's are i.i.d. , without loss of generality, assume that E ( x i ) = 0 and E ( x 2 i ) = 1 .

For the adapter, as Equation 4 shows, and by the expectations we have proved in Lemma A.1 and A.2, we can calculate the scale of forward and backward process.

The scale of forward process is:

The scale of the backward process is:

From the results derived by Equation 5 and 6, one can see that we cannot find a proper ζ to make both scales Θ r,d out ,d in (1) unless d out d in = Θ r,d out ,d in (1) . We can also see that the forward scale is stable if adopting ζ = Θ r,d out ,d in ( d out r 2 ) and the backward is stable if ζ = Θ r,d out ,d in ( d in r 2 ) .

B Additional Experimental Results

B.1 Convergence Speed

As Figure 3 and 4 shown, the convergence of LoRA-GA is significantly faster than vanilla LoRA and other ablation models, almost close to that of full fine-tuning, which support our claim about the speed of convergence.

INO

SST2

MNLI

CoLA

LoRA

LoRA

LoRA

LoRA

LoRA-GA

LoRA-GA

12

LoRA-GA

LoRA-GA

Full Fine-Tuning

Full Fine-Tuning

Full Fine-Tuning

Full Fine-Tuning

10

8

4 -

2

D-

F0

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

MRPC

meta-math

codefeedback

wizard-LM

LoRA

LoRA-GA

0.7

LoRA

LoRA-GA

0.70

LoRA

LoRA-GA

0.90

LoRA

LoRA-GA

Full Fine-Tuning

Full Fine-Tuning

Full Fine-Tuning

0.85

Full Fine-Tuning

0.6

0.65

0.80

0.5

09°0

0.75

0.4

0.70

0.55

0.3

0.65

0.2

0.50

0.60

0.1

ANA

0.55

0

20

40

60

80

100

0

500

10001500200025003000

0

500

10001500200025003000

0

250

500

7501000 12501500

QNLI

SST2

MNLI

CoLA

LoRA

LoRA

LoRA

LoRA

+SO

+SO

12

ueissne

+SO

+SO

LoRA-GA

+GA

LoRA-GA

+GA

10

LoRA-GA

+GA

6

M

LoRA-GA

+GA

2

0

D

0

-o

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

0

20

40

60

80

100

MRPC

meta-math

codefeedback

wizard-LM

0.40

0.60

LoRA

LoRA

LoRA

06°0

LoRA

+SO

+GA

0.35 -

+GA

+SO

8S'0

+SO

+GA

0.85

+SO

+GA

LoRA-GA

LoRA-GA

0.56

LoRA-GA

0.80

LoRA-GA

0.54 -

0.75

0.25

0.52

040

0.20

0.50

0.65

0.60

0.15 -

0.48

0.55 -L

0

20

40

60

100

0

500

10001500200025003000

0

500

1000 1500 2000 25003000

0

250

500

7501000 1250 1500

80

B.2 Evaluating the Rank of the Gradient Matrix

Theorem 3.1 suggests that the closer the rank of the gradient matrix is to 2 r , the better the gradient approximated, thereby enhancing the theoretical effectiveness of our initialization. Figure 5 illustrates the low-rank nature of gradient matrices. The left panel depicts a grid-like pattern in the gradients of a weight matrix, indicating a low-rank structure. The middle panel shows a steeply declining curve of singular values, reflecting the highly low-rank nature of the gradient matrix. The right panel presents the cumulative curve of squared singular values, demonstrating that a few ranks account for nearly all the singular values of the gradient matrix. Specifically, the coverage in the right panel is defined as

where r is the LoRA rank used in LoRA-GA , indicating how much of the low-rank matrix can be approximated by this rank.

Scale of the gradients

Singular values of the gradient matrix

1.2

1.2

1.0

1.0

9-

Dimension

8-

Singular value

0.8

Coverage

0.8

-10

0.6

0.6

435

464

-12

522

551

580

493

-14

0.4

0.4

609

89

667

-16

0.2

0.2

696

725

754

-18

0369258

Dimension

147

m

6

6

0

2

7

0.0

100

Singular value index

101

102

103

0.0

1

2

3

4

Rank

5

6

7

8

9

We further validate this observation on larger models by analyzing LLaMA 2-7B during MetaMathQA training. Table 8 presents the coverage across different layers with varying LoRA ranks. Even with a relatively small rank of 8, we achieve a mean coverage of 92.9% across all layers, with the minimum coverage being 85.1%. Increasing the rank to 128 yields an impressive mean coverage of 99.3%, with the minimum coverage reaching 97.5%. These results demonstrate that even for large models with weight matrices of dimension 4096, a modest LoRA rank is sufficient to capture the majority of the gradient information.

B.3 Detailed Ablation Study Result of GLUE

Table 9 shows the full results of ablation study on the subset of GLUE, where the average scores are briefly reported in Table 4. As Table 9 demonstrated, LoRA-GA outperforms all other ablation models, while both "+SO" and "+GA" methods gain some improvement from vanilla LoRA and simple non-zero initialization "Gaussian". This illustrates that both components in LoRA-GA have positive contribution to the improvement of performance.

B.4 Experimental result with different learning rate

Furthermore, we also conduct experiments under learning rates 1e-5 and 5e-5. As Table 10 and 11 shown, LoRA-GA maintains strong performance across different learning rates, which illustrating its robustness to the variation of learning rate.

B.5 Experiments on the Full MetaMathQA Dataset

Following [9], we conducted additional experiments by training on the complete MetaMathQA dataset for multiple epochs, whereas our main results in the previous section were based on fine-tuning for one epoch on the 100k subset of MetaMathQA. Due to computational constraints, we limited these extended experiments to three methods: LoRA[4], LoRA+[36], and LoRA-GA .

Table 12 presents the performance across four epochs, averaged over two random seeds.

The results show that LoRA-GA consistently achieves better performance than vanilla LoRA and outperforms LoRA+ in most cases across multiple epochs of training.

C LoRA-GA Initialization With Gradient Accumulation

Algorithm 2 LoRA-GA Initialization With Gradient Accumulation

D Hyperparameter

D.1 Experiments on Natural Language Understanding

We use the following hyperparameters with T5-Base.

Training Algorithm: AdamW [49] with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 1 e -8 and weight decay of 0. For full finetuning, LoRA, and its variants, a learning rate of 1 e -4 , a warmup ratio of 0.03, and cosine decay are employed. For DoRA [8], a learning rate of 2 e -4 is used, while for Adalora, a learning rate of 5 e -4 is applied, both with the same warmup ratio and cosine decay adhering to their respective papers.

LoRA Hyperparameters: LoRA rank r = 8 , α = 16 . LoRA target is all linear modules except embedding layer, layer norm and language model head.

LoRA-GA Hyperparameter: γ = 16 , sampled batch size sbs = 8

Other Hyperparameters: Sequence Length T = 128 , train batch size bs = 32 , number of train epochs E = 1 . Precision FP32

D.2 Experiment on Large Language Model

We use the following hyperparameters with Llama 2-7B.

Training Algorithm: AdamW [49] with with β 1 = 0 . 9 , β 2 = 0 . 999 , ϵ = 1 e -8 and weight decay of 0. For full finetuning, LoRA, and its variants, a learning rate of 2 e -5 [38], a warmup ratio of 0.03, and cosine decay are employed. For DoRA [8], a learning rate of 2 e -4 is used, while for Adalora, a learning rate of 5 e -4 is applied, both with the same warmup ratio and cosine decay adhering to their respective papers.

Precision: The backbone model uses bf16 precision, while during training, LoRA's B and A matrices use fp32 precision, following the implementation of PEFT [39].

LoRA-GA Hyperparameter: γ = 64 , micro sampled batch size sbs = 1 with gradient accumulation of 32.

LoRA Hyperparameters: LoRA rank r = 8 and α = 16 for all experiments.

Generation Hyperparameters: All generation is performed with top _ p = 0 . 95 and temperature T = 0 . 8 .

Other Hyperparameters: Number of train epochs E = 1 , train micro batch size mbs = 1 with gradient accumulation of 32. Sequence Length T = 1024

E Comparison between LoRA-GA and PiSSA

Both LoRA-GA and PiSSA [38] concentrate on the initialization of LoRA, and utilizing SVD on pre-trained models. While they may appear similar superficially, significant differences exist between them.

Firstly, the motivations behind LoRA-GA and PiSSA are fundamentally different. As discussed in Section 3.2, LoRA-GA is motivated by the approximation of the LoRA update and full fine-tuning. We employ SVD on gradients solely because the optimal solution to the gradient approximation problem is precisely obtained (as stated in Theorem 3.1). Conversely, PiSSA adopts SVD under the assumption that pre-trained weights possess a low intrinsic rank, and thus, the SVD of weights can provide an accurate representation of original weights. In essence, LoRA-GA emphasizes on gradients and decomposes them, whereas PiSSA concentrates on weights and decomposes them.

Secondly, LoRA-GA and PiSSA employ different scales of initialization. In Section 3.3, LoRA-GA derives an appropriate scaling factor by considering the forward and backward stability of our initialization scheme. On the other hand, PiSSA uses the largest r singular values as the magnitude of orthogonal matrices directly.

F Limitations

In this paper, we have demonstrated that LoRA-GA can achieve performance comparable to full fine-tuning on the T5-Base (220M) and Llama 2-7B models, while significantly reducing the number of parameters and associated costs. However, due to computational resource constraints, we have not validated LoRA-GA on larger pre-trained models (e.g., Llama 2-70B).

In LoRA-GA , we proposed that a scaling factor is necessary. But in some experiments with large learning rates, we observed potential numerical instability due to the effect of scaling factor and learning rate. This limitation suggests a need for careful tuning of the scaling factor and learning rate to maintain stability.

Another limitation pertains to our evaluation scope. While we provide evaluations on MTBench, GSM8K, and Human-eval, we did not assess our method on other datasets. Consequently, we cannot fully guarantee that our findings are universally consistent across all benchmarks.

Additionally, we did not implement our method on other LoRA variants that are orthogonal to our improvements (e.g., ReLoRA [35]). Therefore, we cannot ascertain whether LoRA-GA would perform equally well with other LoRA architectures/improvements.

Finally, compared to the original LoRA, LoRA-GA requires double the checkpoint storage, as it necessitates storing both the initial adapter checkpoints ( A init and B init ) and the final adapter checkpoints ( A and B ).

G Compute Resources

In this paper, we utilized two types of GPUs: the RTX 3090 24GB GPU, supported by a 128-core CPU and 256GB of RAM (hereinafter referred to as "the RTX 3090"), and the A100 80GB GPU (hereinafter referred to as "the A100").

For the experiments on T5-Base using the GLUE dataset, reported in Section 4.1, all computations were performed on a single RTX 3090. For the Llama 2-7B experiments, reported in Section 4.2, full fine-tuning and DoRA scenarios were conducted on a single A100, while all other LoRA variants and LoRA-GA were executed on a single RTX 3090. Additionally, all ablation studies presented in Section 4.3 were carried out on a single RTX 3090.

H Broader Impacts

In this paper, we identify some limitations of vanilla LoRA and propose a more efficient and effective method for LoRA initialization, LoRA-GA. LoRA-GA converges faster than vanilla LoRA and consistently achieves better evaluation results.

We believe that this work will have a positive social impact. The primary reasons are as follows: The high cost of training and fine-tuning large models is a significant challenge today. LoRA-GA offers a way to fine-tune with fewer parameters and lower computational costs while still achieving comparable performance. This will reduce the cost of fine-tuning models and, in turn, decrease energy consumption, such as electricity, contributing to the goal of a low-carbon environment. Furthermore, as the size of large language models (LLM) continues to grow, it becomes increasingly difficult for individuals or small organizations to develop their own LLMs. However, with the help of LoRA-GA and open-source large models, the hardware barrier to entry in this area is greatly reduced. This will promote democratization in the field of large models, preventing monopolies and dictatorships by a few companies.

On the other hand, our method could potentially make it easier to train language models that generate fake news or misleading information. This underscores the necessity for designing effective detectors to identify content generated by large language models (LLMs). Ensuring the responsible use of this technology is crucial to mitigating the risks associated with the misuse of advanced language models.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We included 3 contributions at the end of introduction. The method LoRA-GA and stable scale are proposed in Section 3. The extensive experiments and results are in Section 4.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The last section (Appendix F) of this paper is entirely discuss the limitation of our works.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: There are 1 lemma (Lemma 3.1) and 2 theorems (Theorem 3.1 and 3.2) proposed in our paper. All them are properly proved in Appendix A.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Hyperparameters are disclosed in Appendix D. Other implementation information are disclosed in Section 4, in paragraphs began with "Implementation details".

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All codes of our experiments are uploaded to anonymous github. All datasets used in our experiments are all open source, which has been declared and cited in Section 1 (Introduction).

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https://nips.cc/public/ guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Details that are important to understand and evaluate experimental results are shown in our paper or appendix like Table 3. Other details can be found in our codes.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report standard deviations of our evaluation results like footnotes in Table 1, 2 4, 9, 10 and 11.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: Discussed in Appendix G.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: I have read it. I'm sure that our research conforms NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: We have discussed this in Appendix H.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our research is purely a foundamental reseach about LoRA, which cannot pose such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All models and datasets used in our paper are all properly cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The way to run our codes work is attached in our codes.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No human subjects or participants involved in our research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: No human subjects or participants involved in our research.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes LoRA-GA, which uses an adapter to approximate the gradient update of weights. This method achieves a 2-4 times improvement in convergence speed compared to vanilla LoRA and offers better accuracy than other LoRA-based methods.

STRENGTHS:
1. This paper provides a novel perspective on the initialization of LoRA from the gradient  updation aspect.
2. The paper is theoretically comprehensive, well-written, and easy to follow.

WEAKNESSES:
1. The difference between LoRA-GA and LoRA reparameterization is not shown in Figure 1. 
2. A discussion on the influence of sampled batch size seems necessary.

QUESTIONS:
1、	The accuracy provided in Table 2 is quite different from the numbers in PiSSA [1]. I wonder where the discrepancy comes from.
2、	From Table 4, the performance improvement brought by SO to Gaussian initialization is about 0.4%, while the improvement brought by SO to GA is 5.0%. Since the SO method appears to be general in Section 3.3, why does SO perform so well with GA?
3、	How does the sampled batch size influence the initialization? 
4、	What would happen if the value of r in A and B were swapped? Or if the index sets I_{A} and I_{B} were randomly chosen from [1,2r]?

[1] Pissa: Principal singular values and singular vectors adaptation of large language models[J]. arXiv.

LIMITATIONS:
None
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11

comments: SUMMARY:
LoRA has a slower convergence rate compared to full fine-tuning. This paper proposes a novel initialization method, LoRA-GA (Low-Rank Adaptation with Gradient Approximation), which aligns the gradients of the low-rank matrix product with those of full fine-tuning from the first step. Numerical experiments demonstrate that LoRA-GA achieves faster convergence and better or comparable performance to full fine-tuning, outperforming vanilla LoRA and its variants on several benchmark datasets.

STRENGTHS:
-The paper introduces a novel initialization strategy for LoRA, enhancing its efficiency and performance without altering the architecture or training algorithm.
-The idea of aligning the gradients of the low-rank product with the full model’s gradients at the initial step is innovative.
-The combination of gradient approximation and stable scale for initialization is a unique contribution.

WEAKNESSES:
-The concept of using eigenvectors for initialization might not be entirely new, but its application in this specific context is original.
-The paper could benefit from a deeper exploration of potential edge cases or limitations of the proposed initialization method.
-Some sections, particularly those involving complex mathematical derivations, might be challenging for readers without a strong background in the area. Some of the detailed steps may need to be provided in the proof.

QUESTIONS:
See the weakness.

LIMITATIONS:
The authors have adequately addressed the limitations and potential negative societal impact of their work.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel initialization method for LoRA based on detailed theoretical analysis. The experimental results illustrate that the proposed method can achieve a great performance on the most tasks.\n\nSTRENGTHS:\nStrength: \n1. The paper provide a beautiful theoretical analysis about the initialization method for the gradients of LoRA and full-parameter fine-tuning and then propose their method based on their analysis. \n2. The proposed method is also very clear and easy to follow. \n3. The provided results illustrate that the proposed method can achieve a better performance compared with the most LoRA variants.\n\nWEAKNESSES:\nWeakness:\n1. I think the main weakness is from the experiments. I think the authors should provide more results on some complex tasks. For example, the paper mainly focus on GULU benchmark and metamath-100k. However, I think GULU has been solved by current PEFT methods. Then, for the metamath-100k, it selects 100k data from the vanilla metamath dataset. I recommend the author conduct their experiments on the full metamath dataset because the recent LoRA-based papers usually focus on the full dataset and therefore the readers can obtain a fair comparison and more directly to understand which method is better. I think metamath-100k is not a popular choice although Pissa also uses this dataset. \n2. I am considering whether the authors tune the hyper-parameters of the baseline methods, because I find the results of some baselines are too weak. The best for LoRA-GA is usually not the best hyperparameters for other methods. I hope the paper can provide a fair comparison. That is also the reason why I suggest the author use a more popular dataset. For example, the default learning rate for the experiments on metamath is 2e−5, and the results on GSM8K are: LoRA (42.08), LoRA+(52.11), LoRA-GA(53.60). However, from the table 7 and table 8 in the appendix, we can find that the results of a larger learning rate 5e-5 in table 8: LoRA (46.89), LoRA+(55.23), LoRA-GA(52.79). That means increasing the learning rate from 2e-5 to 5e-5, the baseline LoRA can be improved from 42.08 to 46.89, LoRA+ can be increased from 52.11 to 55.23 and higher than LoRA-GA. So I considering whether we can obtain a better performance for the baselines when we further tune the hyperparameters. I think that is very important since the reader can better know whether each method can really improve the performance. \n\n\n[1] LoRA Learns Less and Forgets Less. \n\n[2] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning. \n\n[3] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning\n\nQUESTIONS:\nMy question is about the hyperparameter selecetion for the baselines. Whether the hyperparameters the authors used in this paper is a good  selection.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}