LoRA-GA: Low-Rank Adaptation with Gradient Approximation

Authors: shaowen  wang
linxi  yu
jian  li
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 14:11:51

Abstract: Fine-tuning large-scale pretrained models is prohibitively expensive in terms of computational and memory costs. LoRA, as one of the most popular Parameter-Efficient Fine-Tuning (PEFT) methods, offers a cost-effective alternative by fine-tuning an auxiliary low-rank model that has significantly fewer parameters. Although LoRA reduces the computational and memory requirements significantly at each iteration, extensive empirical evidence indicates that it converges at a considerably slower rate compared to full fine-tuning, ultimately leading to increased overall compute and often worse test performance. In our paper, we perform an in-depth investigation of the initialization method of LoRA and show that careful initialization (without any change of the architecture and the training algorithm) can significantly enhance both efficiency and performance. In particular, we introduce a novel initialization method, LoRA-GA (Low Rank Adaptation with Gradient Approximation), which aligns the gradients of low-rank matrix product with those of full fine-tuning at the first step. Our extensive experiments demonstrate that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning (hence being significantly faster than vanilla LoRA as well as various recent improvements) while simultaneously attaining comparable or even better performance. For example, on the subset of the GLUE dataset with T5-Base, LoRA-GA outperforms LoRA by 5.69% on average. On larger models such as Llama 2-7B, LoRA-GA shows performance improvements of 0.34, 11.52%, and 5.05% on MTbench, GSM8k, and Human-eval, respectively. Additionally, we observe up to 2-4 times convergence speed improvement compared to vanilla LoRA, validating its effectiveness in accelerating convergence and enhancing model performance.

Main paper: 
 introduction: 
 Fine-tuning large language models (LLMs) is essential for enabling advanced techniques such as instruction fine-tuning [1] , reinforcement learning from human feedback (RLHF) [2] , and adapting models to specific downstream applications. However, the computational and storage costs associated with full fine-tuning are prohibitively high, particularly as model sizes continue to grow. To address these challenges, methods of Parameter-Efficient Fine-Tuning (PEFT) (see e.g., [3] ), such as Low-Rank Adaptation (LoRA) [4] , have emerged and gained significant attention. The key difference is that LoRA-GA initializes adapters using the eigenvectors of the gradient matrix, as opposed to random initialization with a scaling factor.Instead of updating the parameters of the model directly, LoRA incorporates auxilary low-rank matrices B and A into the linear layers of models (such as the Q, K, V , and O matrices in a self-attention block [5] ), while keeping the original layer weights W fixed. The modified layer is represented as y = (W + ηBA)x, where x is the input of that layer, y is the output, and η is the scaling factor. This approach significantly reduces the number of parameters that need to be fine-tuned, thereby lowering the computational and memory costs at each step. Despite these benefits, extensive empirical evidence (see e.g., [6, 7, 8, 9] ) shows that LoRA converges significantly slower compared to full finetune. This slower convergence often increases overall computational costs (measured in Floating Point Operations) and can sometimes lead to worse test performance. In our experiments, we typically observe that LoRA requires 5-6x more iterations and FLOPs to reach the same performance as full fine-tuning under the same learning rate, as shown in Figure 1 .To study the cause of slow convergence, we perform an in-depth investigation of the initialization strategy of LoRA's adapter weights. It is known that fine-tuning pretrained models using the same objective (e.g., language modeling) often converges faster than re-initializing new parameters (e.g., a classification head) [10] . This observation leads us to question whether the slow convergence of vanilla LoRA might be attributed to the default random initialization of adapter weights (LoRA initializes A using Kaiming initialization [11] and sets B to zero [4] ). In our experiments, we find that different initialization strategies for LoRA can significantly impact the results, and its default initialization is suboptimal.In pursuit of a convergence rate comparable to full fine-tuning, we aim for initialization so that the update of BA matches the update of W closely. Previous work suggests that gradient descent operates in a low-dimensional subspace [12, 13] . If we can closely approximate the gradients of the full model at the initial step, subsequent steps can also be approximated, potentially accelerating the convergence of LoRA.2. We identify the scaling factor under non-zero initialization, which ensures the variance of adapter outputs is invariant to the rank of the adapter and the dimension of the input.3. We validate LoRA-GA through extensive experiments, demonstrating significant performance improvements and faster convergence compared to vanilla LoRA. Specifically, LoRA-GA outperforms LoRA by 5.69% on the GLUE [14] subset with T5-Base [15] , and by 0.34, 11.52%, and 5.05% on MT-bench [16] , GSM8K [17] , and HumanEval [18] with Llama 2-7B [19] , respectively, while achieving up to 2-4 times faster convergence.2 Related WorkQuestion: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: We included 3 contributions at the end of introduction. The method LoRA-GA and stable scale are proposed in Section 3. The extensive experiments and results are in Section 4.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
initialization: 
 The significance of maintaining variance stability during initialization has been widely acknowledged to prevent the occurrence of diminishing or exploding phenomena. Xavier initialization [20] ensures stability in both the forward and backward passes of a network under a linear activation function.He initialization [11] extends this solution to networks using ReLU activation. Distinct from these, LSUV initialization [21] selects a mini-batch of data, performing a forward pass to determine the output variance, and subsequently normalizing it to ensure stability. Tensor program (see e.g., [22] ) has emerged as a powerful technique for tuning various hyperparameters, including the initialization, for large models.To fine-tune increasingly large language models within limited hardware resources, researchers have developed various Parameter-Efficient Fine-Tuning (PEFT) methods. Adapter-based methods [23, 24, 25, 26] incorporate new layers into existing model layers. While fine-tuning only these inserted layers significantly reduces resource consumption and requires much fewer parameters, this approach introduces additional latency during both forward and backward passes. Soft Prompt-based methods [10, 27, 28, 29, 30] prepend learnable soft tokens to the model's input to adapt the model to specific tasks. This approach effectively leverages the pre-trained model's capabilities, requiring only appropriate prompts for task adaptation, though it incurs computational overhead during inference. More broadly, GaLore [31] applies low-rank gradients to parameter updates for memory efficiency during training. While this approach is highly expressive and performant, it requires storing complete model checkpoints, consuming more storage than other PEFT methods.LoRA is one of the most popular PEFT methods that introduces the product of low-rank matrices alongside existing layers to approximate weight changes during fine-tuning. Several methods have been proposed to improve the structure of LoRA. AdaLoRA [32] dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within a fixed parameter budget. DoRA [8] enhances the model's expressiveness by adding learnable magnitudes to the direction adjustments made by low-rank matrix products. Additionally, LoHA [33] and LoKr [34] employ Hamiltonian and Kronecker products, respectively. Despite these advancements, vanilla LoRA remains the most popular method due to its robust library and hardware support. Therefore, improving LoRA without altering its structure and at a low cost is crucial. Several recent methods focus on this aspect. ReLoRA [35] suggests periodically merging learned adapters into the weight matrices to enhance LoRA's expressibility. LoRA+ [36] proposes using different learning rates for the two matrices in LoRA to improve convergence. rsLoRA [37] introduces a new scaling factor to make the scale of the output invariant to rank. Although our stable scale approach appears similar to rsLoRA, rsLoRA assumes BA = 0 initialization, making r invariant to the update ∆BA. In contrast, our stable scale ensures that non-zero initialized BA remains invariant to both rank and input dimension from the start.Recently, PiSSA [38] proposes to initializing A and B to approximate the original matrix W , by performing SVD on W . Our method, however, is based on a very different idea, that is to approximate the gradient of W , which involves performing SVD on sampled gradients and properly scaling the initialized matrices, as detailed in Section E.We train our model on a 100k subset of MetaMathQA [43] , a dataset bootstrapped from other math instruction tuning datasets like GSM8K [17] and MATH [45] , with higher complexity and diversity. We select data bootstrapped from the GSM8K training set and apply filtering. Accuracy is reported on the GSM8K evaluation set.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: The last section (Appendix F) of this paper is entirely discuss the limitation of our works.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
methods: 
 In this section, we analyze the initialization of LoRA and introduce our method, LoRA-GA. LoRA-GA consists of two key components: (i) approximating the direction of the gradient of full finetune and (ii) ensuring rank and scale stability in the initialization process. We examine each component and subsequently present their integration within LoRA-GA.Structure of LoRA Based on the hypothesis that the updates of fine-tuning are low-rank [13] , LoRA [4] proposes to use the product of two low-rank matrices to represent the incremental part of the original matrix W . Here, W is the weight matrix of a linear layer in the model. For example, in transformers, it could be the Q, K, V , or O matrices of the self-attention layer or the weight matrix in the MLP layer. Specifically, LoRA has the following mathematical form:W ′ = W 0 + ∆W = W 0 + α r BA := W 0 + ηBAwhere W ′ , W 0 ∈ R m×n , B ∈ R m×r , and A ∈ R r×n , with r ≪ min(m, n). W 0 is the pre-trained weight matrix, remains frozen during the fine-tuning process, while A and B are trainable.Initialization of LoRA Under LoRA's default initialization scheme [4, 39] , matrix A is initialized using Kaiming uniform [11] , while matrix B is initialized with all zeros. Consequently, BA = 0 and W ′ 0 = W 0 , ensuring that the initial parameters are unchanged. If the additional term ∆W = ηBA is initially non-zero (e.g., [38] ), the frozen parameter can be adjusted to ensure the initial parameters unchanged. This can be expressed as:W ′ = (W 0 -ηB init A init ) + ηBA := W frozen + ηBAwhere W frozen = W 0 -ηB init A init is frozen, and B and A are trainable in this case.Our goal is to ensure that the first-step update ∆(ηBA) approximate the direction of the weight update ∆W , i.e., ∆(ηBA) ≈ ζ∆W for some non-zero positive constant ζ. We will discuss how to choose ζ in Section 3.3 and one can treat ζ as a fixed constant for now.Consider a gradient descent step with learning rate λ, the updates for A and B are ∆A = λ∇ A L (A init ) and ∆B = λ∇ B L (B init ), respectively. Assuming learning rate λ is small, the update of ηBA at the first step can be expressed as:η(∆BA init + B init ∆A) = ηλ[∇ B L (B init ) A init + B init ∇ A L (A init )]To measure its approximation quality of scaled the update of the weights in full finetune ζ∆W = ζλ∇ W L (W 0 ), we use the Frobenius norm of the difference between these two updates as a criterion:EQUATIONLemma 3.1. Suppose the loss function is L and y = W ′ x = (W 0 + ηBA)x, where y is the output of a layer and x is the input, the gradients of A and B are linear mappings of the gradient of W ′ :∇ A L = B T ∇ W ′ L, ∇ B L = (∇ W ′ L) A TRemarkably, ∇ W ′ L in LoRA and ∇ W L in full fine-tuning are equal at the beginning of the training.By substituting the gradients in Lemma 3.1 into Equation 1, we can rewrite the criterion as follows:EQUATIONThis criterion evaluates how well the adapter's gradient approximates the direction of the gradient of full fine-tuning, and minimizing it brings the gradient of LoRA closer to that of full fine-tuning with a scaling factor ζ:EQUATIONTheorem 3.1. For the optimization problem in Equation 3 with given ζ, if the Singular Value Decomposition (SVD) of ∇ W L is ∇ W L = U SV T , the solution is:B init = √ ζ η U I A , A init = √ ζ η V T I B , such that |I A | = |I B | = r, I A ∪I B = {i | 1 ≤ i ≤ 2r, i ∈ N}where I A and I B are index sets.Theorem 3.1 provides an appropriate initialization scheme for A init and B init given a specific ζ. The selection of ζ, which influences the scaling of the update ηBA, will be discussed in the following section.Inspired by rsLoRA citekalajdzievski2023rank and the Kaiming initialization [11] , we define the following notions of stability: Definition 3. Remark. We would like to remark that the scaling factor proposed in this subsection proves to be beneficial primarily when one adopts the learning rate typically used in full-finetuning (e.g., 1e -5), since as LoRA-GA attempts to approximate the updates of full-finetuning. However, recent research [9] suggests that LoRA with default initialization performs much better with larger learning rates. Furthermore, tensor program analysis [40, 22] indicates that higher learning rates should be paired with smaller initialization magnitudes. Therefore, we recommend decreasing or omitting the scaling factor when training using larger learning rates (e.g., > 1e -4).Combining the gradient approximation and stable scale components, we propose the LoRA-GA initialization method. First, we initialize A init and B init using the solution from Theorem 3.1. Then, we determine the scaling factor ζ according to Theorem 3.2 to ensure rank and scale stability. Thus, based on Theorems 3.1 and 3.2, we propose a novel initialization method, LoRA-GA.LoRA-GA :We adopt η = α √ r and ζ = α 2 γ 2 dout r 2, where γ is a hyperparameter. We define the index setsI A = {i | 1 ≤ i ≤ r, i ∈ N} and I B = {i | r + 1 ≤ i ≤ 2r, i ∈ N}.Denote the singular value decomposition (SVD) of ∇ W L as ∇ W L = U SV T . The initializations are as follows: U, S, V ← svd(∇ W l ℓ)A init = 4 √ d out γ V T [1:r] , B init = 4 √ d out γ U [r+1:2r] , W init = W 0 -ηB init A init8:A l ← V [1:r] • 4 √ d out /γ 9: B l ← U [r+1:2r] • 4 √ d out /γ 10: W l ← W l -ηB l A l 11:Clear ∇ W l ℓ ▷ Gradient for this layer is not needed anymore 12: end for 13: return W , η, A, B To save GPU memory during LoRA-GA initialization, we utilized a technique similar to [41] . By hooking into PyTorch's backward process, we compute the gradient for one layer at a time and discard the computed gradients immediately. This ensures that our memory usage remains at O(1) instead of O(L), where L is the number of layers. This approach allows the memory consumption during the initialization phase to be less than that during the subsequent LoRA finetuning phase. Our algorithm is shown in Algorithm 1. If the sampled batch size is large, we can also use gradient accumulation to save memory further, as shown in Algorithm 2.We train our model on a 100k subset of Code-Feedback [44] , a high-quality code instruction dataset, removing explanations after code blocks. The model is tested on HumanEval [18] , which consists of 180 Python tasks, and we report the PASS@1 metric.Implementation Details Our model is trained using standard supervised learning for language modelling. The loss for the input prompt is set to zero. Detailed hyperparameters can be found in Appendix D.2. Each experiment uses 3 different random seeds, and the average performance across these runs is reported.Result Our results, as summarized in Table 2 , indicate that LoRA-GA outperforms or is comparable to other methods, including full-finetuning. Specifically, LoRA-GA achieves superior performance on both the GSM8K and Human-eval datasets, underscoring its effectiveness in handling tasks with higher complexity and diversity. On MT-Bench, LoRA-GA also demonstrates competitive performance, although it slightly trails behind DoRA. Nevertheless, LoRA-GA achieves this with fewer parameters and approximately 70% of the training time required by DoRA. Additionally, as illustrated in Figure 2 (Left), our method exhibits a significantly faster convergence rate compared to Vanilla LoRA, with convergence rates comparable to those of full-finetuning.We attribute the performance discrepancies on the GSM8K and Human-eval datasets, when compared to full-finetuning, primarily to the representational limitations imposed by the low-rank approximation. To address this, we experimented with higher ranks, specifically rank=32 and rank=128. Our findings reveal that LoRA-GA maintains stability across different rank settings and, in some cases, even surpasses full-finetuning performance. As shown in Figure 2 (Left), higher ranks with our initialization also result in loss curves that closely resemble those of full-finetuning. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? 
experiments: 
 In this section, we evaluate the performance of LoRA-GA on various benchmark datasets. Initially, we assess Natural Language Understanding (NLU) capabilities using a subset of the GLUE dataset [14] with the T5-Base model [15] . Subsequently, we evaluate dialogue [16, 42] , mathematical reasoning [17, 43] , and coding abilities [18, 44] using the Llama 2-7B model [19] . Finally, we do the ablation study to prove the effectiveness of our method.Baselines We compare LoRA-GA with several baselines to demonstrate its effectiveness:1. Full-Finetune: Fine-tuning the model with all parameters, which requires the most resources. 2. Vanilla LoRA [4] : Fine-tuning the model by inserting a low-rank matrix product BA into linear layers. A is initialized using Kaiming initialization, while B is initialized to zero. 3. LoRA Variants with Original Structure: This includes several methods that retain the original LoRA structure:-rsLoRA [37] introduces a new scaling factor to stabilize the scale of LoRA.-LoRA+ [36] updates the two matrices in LoRA with different learning rates.-PiSSA [38] proposes performing SVD on the weight matrix W at the beginning of training and initializing A and B based on the components with larger singular values. 4. LoRA Variants with Modified Structure: This includes methods that modify the original LoRA structure:-DoRA [8] enhances the model's expressiveness by adding learnable magnitudes.-AdaLoRA [32] dynamically prunes insignificant weights during fine-tuning using SVD, allowing more rank allocation to important areas within a fixed parameter budget.Models and Datasets We fine-tune the T5-Base model on several datasets from the GLUE benchmark, including MNLI, SST-2, CoLA, QNLI, and MRPC. Performance is evaluated on the development set using accuracy as the primary metric. Implementation Details We utilize prompt tuning to fine-tune the T5-Base model on the GLUE benchmark. This involves converting labels into tokens (e.g., "positive" or "negative") and using the normalized probability of these tokens as the predicted label probability for classification. We provide the hyperparameters in Appendix D.1. Each experiment is conducted with 3 different random seeds, and the average performance is reported.Results As shown in Table 1 , LoRA-GA consistently outperforms the original LoRA and other baseline methods, achieving performance comparable to full fine-tuning. Notably, LoRA-GA excels on smaller datasets such as CoLA and MRPC, demonstrating its ability to converge faster and effectively utilize limited training data.Models and Datasets To evaluate the scalability of LoRA-GA , we train Llama 2-7B on three tasks: chat, math, and code.1. Chat: We train our model on a 52k subset of WizardLM [42] , filtering out responses that begin with "As an AI" or "Sorry". We test our model on the MT-Bench dataset [16] , which consists of 80 multi-turn questions designed to assess LLMs on multiple aspects. The quality of the responses is judged by GPT-4, and we report the first turn score.We conducted ablation studies to evaluate the contributions of non-zero initialization, stable output, and gradient approximation in LoRA-GA using five distinct experimental settings. Details of each setting are provided in Table 3 . 4 and 9 . For both small and large models, we observe that simply changing LoRA's initialization to Gaussian does not yield any performance gains and may result in a slight performance decline. However, when combined with either "+SO" (Stable Output) or "+GA" (Gradient Approximation), performance improves upon that of LoRA. LoRA-GA, which integrates both techniques, outperforms other methods. As shown in Figure 2 (Left) and Figure 4 , +SO and +GA also enhance convergence speed, and when both are combined, the training loss curve is even closer to that of full-finetuning. This indicates that both output stability and gradient approximation contribute to the improvement of LoRA, each addressing different aspects of the model's performance. Method A Initialization B Initialization η LoRA U - 3 din , 3 din 0 α/r Gaussian N (0, 1 dout ) N (0, 1 din ) α/r +SO 4 √ d out / √ γ • N (0, 1 dout ) 4 √ d out / √ γ • N (0, 1 din ) α/ √ r +GA V [1:r] U [r+1:2r] α/r LoRA-GA V [1:r] • 4 √ d out / √ γ U [r+1:2r] • 4 √ d out / √ γ α/ √ rWe benchmark LoRA-GA on a single RTX 3090 24GB GPU, a 128-core CPU, and 256GB of RAM. As shown in Table 5 , the memory consumption of our new method does not exceed that used for training with LoRA, indicating no extra memory is needed. Additionally, the time cost of this operation is relatively negligible compared to the subsequent fine-tuning process. For instance, in the Code-Feedback task, the training process took approximately 10 hours, while the initialization required only about 1 minute, which is insignificant. As shown in Table 6 , ArB2r slightly outperforms the alternatives. While Theorem 3.1 proves these schemas are equivalent in the first step, their behaviors diverge afterward. The gradient of matrix B (∇ B L = (∇ W L)A T ) becomes larger than that of A (∇ A L = B T ∇ W L), effectively increasing B's learning rate. This aligns with findings from LoRA+ [36] , where larger learning rates for B proved beneficial, potentially explaining ArB2r's superior performance.The gradient approximation in LoRA-GA uses sampled batches, with smaller batches resembling Stochastic Gradient Descent (SGD) and larger ones approximating full Gradient Descent (GD). While theoretical work [46] suggests SGD's slower convergence may offer better generalization than GD, we conduct experiments to empirically evaluate different batch sizes.We assess gradient approximation quality by comparing gradients from various batch sizes against a reference batch size of 2048 which serves as a proxy for the full dataset gradient using two metrics:• Sign Similarity: The proportion of parameters sharing the same gradient sign.• Magnitude Similarity: The proportion of parameters within the same order of magnitude (where one's absolute value is not more than 10 times the other). As shown in Table 7 , both similarity metrics consistently improve with larger batch sizes, indicating better approximation of the full gradient. The results also demonstrate that while larger batch sizes tend to yield marginally better performance, however, the differences are relatively small. Based on these findings, we recommend using a moderately large batch size (e.g., 64) when computational resources permit.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Hyperparameters are disclosed in Appendix D. Other implementation information are disclosed in Section 4, in paragraphs began with "Implementation details".Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully. (c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/ guides/CodeSubmissionPolicy) for more details. • The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc. • The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why. • At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable). • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusions: 
 In this paper, we present a novel initialization scheme for low-rank adaptation (LoRA), with the goal of acelerating its convergence. By examining the initialization methods and update processes of LoRA, we develop a new initialization method, LoRA-GA , which approximates the gradients of the low-rank matrix product with those of full fine-tuning from the very first step.Through extensive experiments, we have demonstrated that LoRA-GA achieves a convergence rate comparable to that of full fine-tuning while delivering similar or even superior performance. Since LoRA-GA solely modifies the initialization of LoRA without altering the architecture or training algorithms, it offers an efficient and effective approach that is easy to implement. Furthermore, it can also be incorporated with other LoRA variants. For example, ReLoRA [35] periodically merges the adapters into frozen weights W , which may allow LoRA-GA to demonstrate its advantages over more steps. We leave it as an interesting future direction. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: Details that are important to understand and evaluate experimental results are shown in our paper or appendix like Table 3 . Other details can be found in our codes.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: We report standard deviations of our evaluation results like footnotes in Table 1 , 2 4, 9, 10 and 11. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Discussed in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All models and datasets used in our paper are all properly cited.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 
new assets: 
 Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?Answer: [Yes] Justification: The way to run our codes work is attached in our codes.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: No human subjects or participants involved in our research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: No human subjects or participants involved in our research.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: A.1 Proof of Theorem 3.1 Lemma 3.1. Suppose the loss function is L and y = W ′ x = (W0 + ηBA)x, where y is the output of a layer and x is the input, the gradients of adapters A and B are linear mappings of the gradient of W ′ :∇AL = B T ∇ W ′ L, ∇BL = (∇ W ′ L) A TRemarkably, the gradient of W ′ in LoRA and the gradient of W in full fine-tuning are equal at the beginning of the training.Proof. For the gradients in LoRA,∇ W ′ L = ∂L ∂W ′ = ∂L ∂y ∂y ∂W ′ = ∂L ∂y x T ∇AL = ∂L ∂A = ∂W ′ ∂A • ∂L ∂y ∂y ∂W ′ = B T • ∂L ∂y x T = B T ∇ W ′ L ∇BL = ∂L ∂B = ∂L ∂y ∂y ∂W ′ • ∂W ′ ∂B = ∂L ∂y x T A T = (∇ W ′ L) A TAt the beginning of training, both LoRA and full fine-tuning have y ′ = y and identical x, therefore,∇W L = ∇ W ′ L = ∂L ∂y (y)x T Theorem 3.1.Consider the following optimization problem:min A init ,B init η 2 ∇W L • A T init Ainit + η 2 BinitB T init • ∇W L -ζ∇W L F If the Singular Value Decomposition (SVD) of ∇W L is ∇W L = U SV T, the solution to this optimization problem is:Binit = √ ζ η UI A , Ainit = √ ζ η V T I B s.t. |IA| = |IB| = r, IA ∪ IB = {i | 1 ≤ i ≤ 2r, i ∈ N}where IA, IB are index sets.Proof. Since that rank(Ainit) = rank(Binit) = r and 2r < min(m, n), we can assert that the matrixW ′ = η 2 ∇W LA T init Ainit + η 2 BinitB T init ∇W L has rank(W ′ ) ≤ 2r. Under this given solution, W ′ = η 2 ∇W LA T init Ainit + η 2 BinitB T init ∇W L = ζU SV T (VI A V T I A ) + ζ(UI B U T I B )U SV T = ζ i∈I A σiuiv T i + ζ j∈I B σjuju T j = ζ 2r i=1 σiuiv T iBy the classic Eckart-Young Theorem (see e.g., [47, 48] ), the optimal low-rank approximation with respect to Frobenius norm is:W ′ * = arg min rank(W ′ * )=2r W ′ * -ζ∇W L F = ζ 2r i=1 σiuiv T iThis is identical to what we have got. Therefore, this is the optimal solution.A.2 Proof of Theorem 3.2Lemma A.1. In R n , if we randomly pick a vector x that n i=1 x 2 i = 1, we have:1. E (xi) = 0, E x 2 i = 1 n and E x 4 i = Θ r,d out ,d in 1 n 2 ; 2. E (xixj) = 0; 3. E x 2 i x 2 j = Θ r,d out ,d in 1 n 2 ; 4. E x 2 i xjx k = 0;Proof. It is equivalent to sampling a random point uniformly from a unit sphere in R n .For property 1, E (xi) = 0 holds obvious by symmetry. Since n i=1 x 2 i = 1 and uniformly distributed, each entry has identical expectation,E n i=1 x 2 i = nE x 2 i = 1, E x 2 i = 1 n . E x 4 i = E x 2 i • x 2 i = Θ r,d out ,d in 1 n Θ r,d out ,d in 1 n = Θ r,d out ,d in 1 n 2 .For property 2, it can also be proved by symmetry: we can always find vector that contains (xi, -xj) also lies on the sphere. Therefore, E (xixj) = 0.For property 3, E x 2 i x 2 j = E x 2 i • x 2 j = Θ r,d out ,d in 1 n Θ r,d out ,d in 1 n = Θ r,d out ,d in 1 n 2 .For property 4, again it can be proved by symmetry: we can always find vector that contains (xi, xj, -x k ) also lies on the sphere. Therefore, E x 2 i xjx k = 0.Lemma A.2. For a randomly selected orthogonal matrix A ∈ R n×n , and we randomly pick two different column vectors x and y from it. For these two vectors, we have the following:1. E (xiyi) = 0;2. E (xiyj) = 0;Proof. It is equivalent to first selecting a random vector x from a unit sphere in R n uniformly, and then selecting the other one y that is orthogonal to x.For property 1, .n i=1 xiyi = 0 ⇒ E n i=1 xiyi = n i=1 E (() xiyi) = 0 ⇒ E (xiyi) = 0.Proof. In LoRA, h = (W ′ + ηBA)x, since that W ′ is not considered here, therefore, denote y = ηBAx.When backward propagation, it's like ∂L ∂x = ηA T B T ∂L ∂h . Represente ∂L ∂h as v and ∂L ∂x as g. Therefore,EQUATIONSince that the output of each layer in model always passes a softmax function, so that the vector (1) . Further, since that input xi's are i.i.d., without loss of generality, assume that E(xi) = 0 and E(x 2 i ) = 1. For the adapter, as Equation 4shows, and by the expectations we have proved in Lemma A.1 and A.2, we can calculate the scale of forward and backward process.∂L ∂h = v is Θ r,d out ,d inThe scale of forward process is:E y 2 i = η 2 r j 1 =1 r j 2 =1 d in k 1 =1 d in k 2 =1 E (Bij 1 A j 1 k 1 Bij 2 A j 2 k 2 x k 1 x k 2 ) = η 2 r j 1 =1 r j 2 =1 d in k 1 =1 d in k 2 =1 E (Bij 1 Bij 2 ) E (A j 1 k 1 A j 2 k 2 ) E (x k 1 x k 2 ) = η 2 r j 1 =1 r j 2 =1 d in k=1 E (Bij 1 Bij 2 ) E (A j 1 k A j 2 k ) = η 2 r j=1 d in k=1 E B 2 ij E A 2 jk = η 2 r j=1 d in k=1 ζ 2 η 4 1 dout 1 din = 1 α 2 • ζ 2 • r 2 dout (5)The scale of the backward process is:E g 2 i = η 2 r j 1 =1 r j 2 =1 d out k 1 =1 d out k 2 =1 E (Aj 1 iBk 1 j 1 Aj 2 iBk 2 j 2 v k 1 v k 2 ) = η 2 r j 1 =1 r j 2 =1 d out k 1 =1 m k 2 =1 v k 1 v k 2 E (Aj 1 iAj 2 i) E (B k 1 j 1 B k 2 j 2 ) = η 2 r j=1 d out k=1 v 2 k E A 2 ji E B 2 kj = η 2 r j=1 d out k=1 v 2 k ζ 2 η 41 dinEQUATIONFrom the results derived by Equation 5 As Figure 3 and 4 shown, the convergence of LoRA-GA is significantly faster than vanilla LoRA and other ablation models, almost close to that of full fine-tuning, which support our claim about the speed of convergence. Theorem 3.1 suggests that the closer the rank of the gradient matrix is to 2r, the better the gradient approximated, thereby enhancing the theoretical effectiveness of our initialization. We further validate this observation on larger models by analyzing LLaMA 2-7B during MetaMathQA training. Table 8 presents the coverage across different layers with varying LoRA ranks. Even with a relatively small rank of 8, we achieve a mean coverage of 92.9% across all layers, with the minimum coverage being 85.1%. Increasing the rank to 128 yields an impressive mean coverage of 99.3%, with the minimum coverage reaching 97.5%. These results demonstrate that even for large models with weight matrices of dimension 4096, a modest LoRA rank is sufficient to capture the majority of the gradient information. 9 shows the full results of ablation study on the subset of GLUE, where the average scores are briefly reported in Table 4 . As Table 9 demonstrated, LoRA-GA outperforms all other ablation models, while both "+SO" and "+GA" methods gain some improvement from vanilla LoRA and simple non-zero initialization "Gaussian". This illustrates that both components in LoRA-GA have positive contribution to the improvement of performance. Following [9] , we conducted additional experiments by training on the complete MetaMathQA dataset for multiple epochs, whereas our main results in the previous section were based on fine-tuning for one epoch on the 100k subset of MetaMathQA. Due to computational constraints, we limited these extended experiments to three methods: LoRA [4] , LoRA+ [36] , and LoRA-GA .Table 12 presents the performance across four epochs, averaged over two random seeds. ∇ avg W l ℓ ← 0 ▷ Initialize average gradient for each layer on CPU 6: end for 7: for each micro-batch B i in B do8: ŷi ← f (x i , W )▷ Forward pass for micro-batch 9:ℓ i ← L(y i , ŷi ) 10:for l = L, . . . , 1 doCompute ∇ W l ℓ i ▷ Backward pass for one layer 12:∇ avg W l ℓ ← ∇ avg W l ℓ + ∇ W l ℓ i • b n ▷ Move to CPU 13:Clear ∇ W l ℓ i ▷ Gradient for this layer is not needed anymore 14:end for 15: end for 16: for l = L, . . . , 1 do 17:d out , d in ← size(W l ) 18: U, S, V ← svd(∇ avg W l ℓ) 19: A l ← V [1:r] • 4 √ d out /γ 20: B l ← U [r+1:2r] • 4 √ d out /γ 21:W l ← W l -ηB l A l 22: end for 23: return W , η, A, BWe use the following hyperparameters with T5-Base.• Training Algorithm: AdamW [49] with β1 = 0.9, β2 = 0.999, ϵ = 1e -8 and weight decay of 0.For full finetuning, LoRA, and its variants, a learning rate of 1e -4 , a warmup ratio of 0.03, and cosine decay are employed. For DoRA [8] , a learning rate of 2e -4 is used, while for Adalora, a learning rate of 5e -4 is applied, both with the same warmup ratio and cosine decay adhering to their respective papers.• LoRA Hyperparameters: LoRA rank r = 8, α = 16. LoRA target is all linear modules except embedding layer, layer norm and language model head.• LoRA-GA Hyperparameter: γ = 16, sampled batch size sbs = 8• Other Hyperparameters: Sequence Length T = 128, train batch size bs = 32, number of train epochs E = 1. Precision FP32We use the following hyperparameters with Llama 2-7B.• Training Algorithm: AdamW [49] with with β1 = 0.9, β2 = 0.999, ϵ = 1e -8 and weight decay of 0. For full finetuning, LoRA, and its variants, a learning rate of 2e -5 [38] , a warmup ratio of 0.03, and cosine decay are employed. For DoRA [8] , a learning rate of 2e -4 is used, while for Adalora, a learning rate of 5e -4 is applied, both with the same warmup ratio and cosine decay adhering to their respective papers.• Precision: The backbone model uses bf16 precision, while during training, LoRA's B and A matrices use fp32 precision, following the implementation of PEFT [39] .• LoRA-GA Hyperparameter: γ = 64, micro sampled batch size sbs = 1 with gradient accumulation of 32.• LoRA Hyperparameters: LoRA rank r = 8 and α = 16 for all experiments.• Generation Hyperparameters: All generation is performed with top_p = 0.95 and temperature T = 0.8.• Other Hyperparameters: Number of train epochs E = 1, train micro batch size mbs = 1 with gradient accumulation of 32. Sequence Length T = 1024Both LoRA-GA and PiSSA [38] concentrate on the initialization of LoRA, and utilizing SVD on pre-trained models. While they may appear similar superficially, significant differences exist between them.Firstly, the motivations behind LoRA-GA and PiSSA are fundamentally different. As discussed in Section 3.2, LoRA-GA is motivated by the approximation of the LoRA update and full fine-tuning. We employ SVD on gradients solely because the optimal solution to the gradient approximation problem is precisely obtained (as stated in Theorem 3.1). Conversely, PiSSA adopts SVD under the assumption that pre-trained weights possess a low intrinsic rank, and thus, the SVD of weights can provide an accurate representation of original weights. In essence, LoRA-GA emphasizes on gradients and decomposes them, whereas PiSSA concentrates on weights and decomposes them.Secondly, LoRA-GA and PiSSA employ different scales of initialization. In Section 3.3, LoRA-GA derives an appropriate scaling factor by considering the forward and backward stability of our initialization scheme. On the other hand, PiSSA uses the largest r singular values as the magnitude of orthogonal matrices directly.In this paper, we have demonstrated that LoRA-GA can achieve performance comparable to full fine-tuning on the T5-Base (220M) and Llama 2-7B models, while significantly reducing the number of parameters and associated costs. However, due to computational resource constraints, we have not validated LoRA-GA on larger pre-trained models (e.g., Llama 2-70B).In LoRA-GA , we proposed that a scaling factor is necessary. But in some experiments with large learning rates, we observed potential numerical instability due to the effect of scaling factor and learning rate. This limitation suggests a need for careful tuning of the scaling factor and learning rate to maintain stability.Another limitation pertains to our evaluation scope. While we provide evaluations on MTBench, GSM8K, and Human-eval, we did not assess our method on other datasets. Consequently, we cannot fully guarantee that our findings are universally consistent across all benchmarks.Additionally, we did not implement our method on other LoRA variants that are orthogonal to our improvements (e.g., ReLoRA [35] ). Therefore, we cannot ascertain whether LoRA-GA would perform equally well with other LoRA architectures/improvements.Finally, compared to the original LoRA, LoRA-GA requires double the checkpoint storage, as it necessitates storing both the initial adapter checkpoints (Ainit and Binit) and the final adapter checkpoints (A and B).In this paper, we utilized two types of GPUs: the RTX 3090 24GB GPU, supported by a 128-core CPU and 256GB of RAM (hereinafter referred to as "the RTX 3090"), and the A100 80GB GPU (hereinafter referred to as "the A100").For the experiments on T5-Base using the GLUE dataset, reported in Section 4.1, all computations were performed on a single RTX 3090. For the Llama 2-7B experiments, reported in Section 4.2, full fine-tuning and DoRA scenarios were conducted on a single A100, while all other LoRA variants and LoRA-GA were executed on a single RTX 3090. Additionally, all ablation studies presented in Section 4.3 were carried out on a single RTX 3090.In this paper, we identify some limitations of vanilla LoRA and propose a more efficient and effective method for LoRA initialization, LoRA-GA. LoRA-GA converges faster than vanilla LoRA and consistently achieves better evaluation results.We believe that this work will have a positive social impact. The primary reasons are as follows: The high cost of training and fine-tuning large models is a significant challenge today. LoRA-GA offers a way to fine-tune with fewer parameters and lower computational costs while still achieving comparable performance. This will reduce the cost of fine-tuning models and, in turn, decrease energy consumption, such as electricity, contributing to the goal of a low-carbon environment. Furthermore, as the size of large language models (LLM) continues to grow, it becomes increasingly difficult for individuals or small organizations to develop their own LLMs. However, with the help of LoRA-GA and open-source large models, the hardware barrier to entry in this area is greatly reduced. This will promote democratization in the field of large models, preventing monopolies and dictatorships by a few companies.On the other hand, our method could potentially make it easier to train language models that generate fake news or misleading information. This underscores the necessity for designing effective detectors to identify content generated by large language models (LLMs). Ensuring the responsible use of this technology is crucial to mitigating the risks associated with the misuse of advanced language models.NeurIPS Paper ChecklistJustification: There are 1 lemma (Lemma 3.1) and 2 theorems (Theorem 3.1 and 3.2) proposed in our paper. All them are properly proved in Appendix A.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced.• The answer NA means that the paper does not include experiments. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 


Decision: The paper introduces an innovative initialization method for LoRA that enhances convergence speed and performance of fine-tuning LLMs. The reviewers appreciated the novel approach, which aligns gradients of the low-rank adaptation with those of full fine-tuning at the first step, improving the convergence rate and model performance. Despite initial concerns about experimental complexity and hyperparameter tuning, the authors provided additional experiments and clarified their methodology in the rebuttal, addressing the reviewers' feedback. I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel initialization method for LoRA based on detailed theoretical analysis. The experimental results illustrate that the proposed method can achieve a great performance on the most tasks.
strength_weakness: strength_weakness: Weakness:
1. I think the main weakness is from the experiments. I think the authors should provide more results on some complex tasks. For example, the paper mainly focus on GULU benchmark and metamath-100k. However, I think GULU has been solved by current PEFT methods. Then, for the metamath-100k, it selects 100k data from the vanilla metamath dataset. I recommend the author conduct their experiments on the full metamath dataset because the recent LoRA-based papers usually focus on the full dataset and therefore the readers can obtain a fair comparison and more directly to understand which method is better. I think metamath-100k is not a popular choice although Pissa also uses this dataset. 
2. I am considering whether the authors tune the hyper-parameters of the baseline methods, because I find the results of some baselines are too weak. The best for LoRA-GA is usually not the best hyperparameters for other methods. I hope the paper can provide a fair comparison. That is also the reason why I suggest the author use a more popular dataset. For example, the default learning rate for the experiments on metamath is 2e−5, and the results on GSM8K are: LoRA (42.08), LoRA+(52.11), LoRA-GA(53.60). However, from the table 7 and table 8 in the appendix, we can find that the results of a larger learning rate 5e-5 in table 8: LoRA (46.89), LoRA+(55.23), LoRA-GA(52.79). That means increasing the learning rate from 2e-5 to 5e-5, the baseline LoRA can be improved from 42.08 to 46.89, LoRA+ can be increased from 52.11 to 55.23 and higher than LoRA-GA. So I considering whether we can obtain a better performance for the baselines when we further tune the hyperparameters. I think that is very important since the reader can better know whether each method can really improve the performance. 


[1] LoRA Learns Less and Forgets Less. 

[2] MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning. 

[3] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Finetuning
questions: questions: My question is about the hyperparameter selecetion for the baselines. Whether the hyperparameters the authors used in this paper is a good  selection.
limitations: limitations: N/A
paper_summary: paper_summary: LoRA has a slower convergence rate compared to full fine-tuning. This paper proposes a novel initialization method, LoRA-GA (Low-Rank Adaptation with Gradient Approximation), which aligns the gradients of the low-rank matrix product with those of full fine-tuning from the first step. Numerical experiments demonstrate that LoRA-GA achieves faster convergence and better or comparable performance to full fine-tuning, outperforming vanilla LoRA and its variants on several benchmark datasets.
strength_weakness: strength_weakness: -The concept of using eigenvectors for initialization might not be entirely new, but its application in this specific context is original.
-The paper could benefit from a deeper exploration of potential edge cases or limitations of the proposed initialization method.
-Some sections, particularly those involving complex mathematical derivations, might be challenging for readers without a strong background in the area. Some of the detailed steps may need to be provided in the proof.
questions: questions: See the weakness.
limitations: limitations: The authors have adequately addressed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper proposes LoRA-GA, which uses an adapter to approximate the gradient update of weights. This method achieves a 2-4 times improvement in convergence speed compared to vanilla LoRA and offers better accuracy than other LoRA-based methods.
strength_weakness: strength_weakness: 1. The difference between LoRA-GA and LoRA reparameterization is not shown in Figure 1. 
2. A discussion on the influence of sampled batch size seems necessary.
questions: questions: 1、	The accuracy provided in Table 2 is quite different from the numbers in PiSSA [1]. I wonder where the discrepancy comes from.
2、	From Table 4, the performance improvement brought by SO to Gaussian initialization is about 0.4%, while the improvement brought by SO to GA is 5.0%. Since the SO method appears to be general in Section 3.3, why does SO perform so well with GA?
3、	How does the sampled batch size influence the initialization? 
4、	What would happen if the value of r in A and B were swapped? Or if the index sets I_{A} and I_{B} were randomly chosen from [1,2r]?

[1] Pissa: Principal singular values and singular vectors adaptation of large language models[J]. arXiv.
limitations: limitations: None

Comments: title: Thanks for your response
Thanks for your response.
comment: Thank you for your response. We appreciate that most of your concerns have been addressed, and we are grateful for your willingness to raise the score. If you have any further feedback, please let us know!
Thank you for your response. We appreciate that most of your concerns have been addressed, and we would welcome any additional feedback. Please let us know if you have any further questions.
Thanks for your response and new results. 

The authors have solved most of my concerns and I will raise my score to 6.
Most of my concerns are addressed. So I'm willing to raise the score.