Title: GoRA: Gradient-driven Adaptive Low Rank Adaptation
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

LoRA

PEFT
Abstract: Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework—**GoRA** (**G**radient-driven Adaptive L**o**w **R**ank **A**daptation)—that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches—which often focus on either rank selection or initialization in isolation—but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel low-rank adaptation technique that is capable of adaptive rank selection as well as informative initialization all within the same theoretical framework. The core idea of the proposed method is based on the importance accumulated gradients from the original backpropagation without LoRA for guiding the architecture and the initialization for LoRA. The authors have shown the empirical merits of the proposed by performing extensive experimentation as well as ablation studies to assess the importance of each component.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The theoretical foundations of the proposed methodology is intuitive and well-explained.\n- The experimental study is reasonably extensive with promising results.\n- The paper is generally very well-written.\n\nWeaknesses:\n- The proposed methodology requires to perform $n$-step gradient calculations on the original weights. However, this is prohibitive in many practical situations mainly due to the memory constraints, and hence the use of low-rank adaptation techniques in the first place. The presented experiments are only done for medium size models which are likely to be ok, but there's no indication that'd scale up. The authors have put one sentence in Algorithm 1 claiming that the gradient accumulation step would not lead to extra memory usage. but this claim makes no sense, plus there's no further argument to support this claim. Also, it's not clear the memory usage peaks reported in Table 8 include the  gradient accumulation preprocessing step or just the GORA optimization.\n\n- The proclaimed \"$n$-step gradient accumulation\" is not really the $n$-step gradient accumulation, is it? In Algorithm 1, it's mentioned that the step function of the optimizer is not applied which makes the whole gradient accumulation step equivalent to a 1-step gradients on a larger batch. There's no explanation whatsoever why this choice was made over the actual n$-step gradient accumulation which seems to be more intuitive.\n\nMinor issues:  \n- The reference in Algorithm 1 Line 3 is broken.\n- Confusing notation: $n$ is used for two different things in the paper.\n- The Hadamard product notation is inconsistent between Eq. 5 and Table 6.\n\nQUESTIONS:\n- Does the memory usage ion Table 8 also include the gradient accumulation step?\n- The authors claimed that the gradient accumulation step doesn't lead to extra memory usage. Why is that the case?\n- The proclaimed \"$n$-step gradient accumulation\" is in fact a 1-step gradient accumulation over a big batch. Why this choice and not the actual $n$-step gradient accumulation which is more intuitive? Any empirical comparisons between the two?\n\nLIMITATIONS:\nThe main limitation IMO that hasn't been discussed in the Limitation section is that the initial gradient accumulation step can be prohibitive in terms of the memory usage for larger scale models. This kind of defeats the purpose of using LoRA in the first place. I'd strongly recommend the authors to discuss this issue not just in the Limitation section but in the main body of the paper.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces GoRA (Gradient-driven Adaptive Low Rank Adaptation), a novel framework designed to enhance the efficiency and effectiveness of fine-tuning large language models (LLMs) using Low-Rank Adaptation (LoRA). GoRA addresses two critical limitations of existing LoRA variants: static rank selection and suboptimal weight initialization. It proposes a unified approach that leverages gradient information during training to dynamically assign optimal ranks to low-rank adapters and initialize their weights adaptively. The authors present extensive experiments across various architectures and modalities, demonstrating that GoRA consistently outperforms existing LoRA-based methods and can even surpass full fine-tuning in certain challenging tasks, all while maintaining the computational efficiency of vanilla LoRA.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Originality and Unified Framework: GoRA represents a novel advancement to unify both adaptive rank selection and adaptive initialization strategies within a single framework. This contrasts with prior approaches that typically address these aspects in isolation, leading to a more holistic and effective solution for LoRA fine-tuning.\n- Novel Initialization Strategy: GoRA's adaptive initialization strategy for matrix B, utilizing pseudo-inverse compressed gradients, is novel and theoretically motivated. This approach crucially avoids manipulating pre-trained weights, thereby eliminating the \"training-inference gap\" that affects many existing non-zero initialization methods.\n- Effective Adaptive Rank Allocation: The proposed gradient-driven, sensitivity-based importance metric for rank allocation is well-justified through ablation studies, demonstrating superior performance compared to alternative metrics like nuclear norms . \n- Superior Performance: The paper provides compelling evidence that GoRA consistently outperforms a wide array of LoRA-based methods across natural language understanding (NLU), natural language generation (NLG), and image classification tasks. Notably, GoRA demonstrates the capability to surpass full fine-tuning performance in challenging tasks such as mathematical reasoning.\n- Computational Efficiency Preservation: A key advantage of GoRA is its ability to achieve performance improvements while preserving the efficiency of vanilla LoRA, both in terms of training time and peak GPU memory usage. This is a critical factor for fine-tuning large models under resource constraints. It also maintains a similar number of trainable parameters compared to standard LoRA.\n- Comprehensive Experimental Validation: The evaluation covers diverse models (T5, Llama, CLIP-VIP) and modalities (NLP, Vision), showcasing GoRA's broad applicability and robust performance across different contexts. The inclusion of ablation studies provides valuable insights into the contribution of individual components .\n- Clarity and Reproducibility: The paper is well-structured and clearly explains the technical contributions, including a detailed algorithm for GoRA's method. The provided experimental settings, hyperparameters, and implementation details enhance the reproducibility of the reported results . Proofs are also included in the appendix. Code is also provided.\n\nWeaknesses:\n- Ambiguous Memory Claim for Gradient Collection: The paper states that the initial gradient computation \"do not require extra memory as shown in Table ??\". However, the table reference is missing, and neither Table 7 (trainable parameters) nor Table 8 (training time, peak GPU memory) explicitly quantifies the memory used for storing the accumulated gradients for all layers, even if offloaded to CPU. While CPU memory might be less constrained than GPU memory, for extremely large models where W matrices can be order of 10s of gigabytes, this unquantified memory footprint for gradient storage could still be a practical concern, making the claim of \"no extra memory\" problematic without direct evidence.\n- Scalability of Initialization Phase for Larger Models/Datasets: Although the initial gradient computation phase takes only 4 minutes for the Llama-3.1-8B-Base experiment , the paper does not extensively discuss how this initial overhead scales with significantly larger models (e.g., Llama-3.1-70B) or much larger datasets, which could potentially make this phase a more substantial part of the total training time.\n- Limited Guidance on $\\alpha$ Hyperparameter Selection: While the paper extensively detail the selection and impact of $\\gamma$ through ablation studies and provide specific values in their experiments, the paper does not explicitly state how $\\alpha$ is selected or what its numerical value is in the experiments conducted for GoRA. Can the authors comment on its selection?\n\nQUESTIONS:\n- Gradient Collection Memory Footprint: The paper states that the initial gradient collection \"do not require extra memory as shown in Table ??\", but the table reference is missing. Could the authors please provide specific figures for the memory footprint (e.g., in GB) required for gradients when offloaded to CPU for a representative model like Llama3.1-8B-Base? Furthermore, can you elaborate on whether this memory usage remains practically insignificant for models orders of magnitude larger (e.g., 70B+ parameters), where the full W matrix and its corresponding gradients could be very substantial?\n- Scalability of Initial Gradient Computation Time: The initial gradient computation takes 4 minutes for the Llama-3.1-8B-Base model . Please provide insights into how this 4-minute overhead scales if the number of gradient accumulation steps in Algorithm 1 were significantly increased, or if the method were applied to models with considerably more parameters than the 8B-parameter Llama models tested. Could this initial phase become a bottleneck in very large-scale fine-tuning?\n- Dynamic Initialization of Matrix A: The paper mentions the non-random initialization of matrix A as a direction for future work . Given that the initialization of matrix B depends directly on A , could the authors discuss the theoretical or practical complexities anticipated in developing a data-driven or non-random initialization for A, especially in maintaining the \"optimal gradient compression\" property and computational efficiency?\n\nLIMITATIONS:\nThe authors have adequately addressed the limitations of their work in Appendix D, titled \"Limitations And Future Works\" . They acknowledge that their evaluation has not yet extended to larger models (like Llama-3.1-70B) and more extensive datasets, and that the primary focus has been on language models and NLP tasks, with potential for generalization to other modalities . They also identify the current restriction of matrix A to random initialization as a future area for improvement  and note theoretical compatibility with other LoRA variants . This transparency about the scope and future work is commendable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposed a new parameter-efficient finetuning method, an upgrade to LoRA. It is also a low-rank update method that keeps the bulk of the pretrained model frozen, but introduces two innovations: a) an adapted initialisation strategy that does not start with a zero-initialised matrix (like in LoRA) that tries to minimise the error in the gradient (by comparing it against the full-finetuning gradient), b) it adapts the rank during training using gradient information. On various benchmarks it outperforms baselines LoRA and other popular methods such as PiSSA and DoRA.\n\nSTRENGTHS AND WEAKNESSES:\n+ the experiments cover both vision and NLP (and therein includes encoder as well as decoder LMs)\n+ performance gains look solid and are compared against baselines from other papers\n+ experiments include st-deviations\n+ some analysis of the method is provided \n+ initialisation being that important is interesting for the community\n\n\nWeaknesses\n- before training the gradients for all weights are computing in a full-finetuning like fashion to get the gradients. this requires as much memory as full-finetuning so actually makes is quite prohibitive to scale to large LLMs in the 13B+ range.\n\n- the motivation of using non-zero initialisation is states as \"Thus, designing a nonzero initialization method without manipulating pre-trained weights remains an open problem.\" but it's not clear why this is a problem. As the intro also says, substracting AB from the original W seems fine. \n\n- the pseudo-inverse is used to compute the optimal initialisation for compressing the gradient ... in terms of L2 distance, if I understand correctly. It's not clear if this is indeed optimal. one could also find the best initialisation with L1 distance (e.g. by running a few steps of SGD to find the best initialisation). L2 thus seems a bit out of the blue and it would be nice if this motivation is better argued for.\n\n- a lot of work is applied in correctly initialising the matrices including their ranks. But it's not clear if the ideal rank allocation doesn't change during training. \n\n- GoRA, despite its smart rank allocation strategy has quite varying performances depending on the hyperparamter r_ref. E.g. varying from 5.82 on MT-Bench (the worst in the whole table) to 6.34 (second best in the table), or in a reverse of trends, 52 vs 48.9 on HumanEval. This makes it very hard for practioners to just \"use\" the method if there's such crucial and sensitive hyperparamters.\n\nQUESTIONS:\nQ:\nWhat if you rerun the initialisation strategy a couple of times during the training? does the ideal rank allocation change drastically or stay consistent? How far off are the trained matrices from the ones that best compress the full gradients? Does this yield better performances\n\nWhat happens if you initialise not with L1 optimal compression of the gradients but other metrics like L2  (||W * G ||_2)?\n\nCan and how can this be combined with QLoRA type quantisation? Can this scale to 8\n\nsmall things:\n* typo: \"this reduces the their usability\"\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes **GoRA**, a new method for setting Initialization and Rank in LoRA finetuning. GoRA allocates LoRA ranks before training by scoring each weight matrix with a gradient-based importance metric, and derives a data-driven, non-zero initialization for the B matrix. Empirically, GoRA seems to improve over LoRA, LoRA+, AdaLoRA, and several other baselines on language (GLUE, MT-Bench, GSM8K, HumanEval) and vision while keeping trainable parameters and GPU memory similar to that of LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n### I. Strengths\n\n- **Unified treatment of rank and initialization** : GoRA re-uses the same gradient statistics to decide both how many LoRA ranks each layer receives and how the adapters are initialized. This avoids the masking overhead of AdaLoRA-style methods.\n\n- **Robust gains across modalities** The reported result show that GoRA wins or competes with the strongest baseline on most tasks tested. With \\(r_{\\max}=128\\) it even surpasses full fine-tuning on GSM8K and HumanEval, demonstrating that gradient-driven rank allocation scales to both language understanding and generation as well as vision classification.\n\n- **Ablations**:   The paper provides ablations results on the effect of rank bounds, alternative importance metrics, etc. These ablations clarify why the chosen design works and how sensitive it is to hyper-parameters.\n\n\n### II. Weaknesses\n\n- **Compute start-up cost**: GoRA always requires N full forward-and-backward passes *before* training starts. While negligible for multi-hour jobs, this warm-up can dominate short experiments or hyper-parameter sweeps. Could N be selected adaptively—e.g., stop once the layer-wise importance estimates converge?\n\n- **CPU off-load bottleneck**: Copying every gradient tensor to host memory each step may saturate PCIe on multi-GPU or FSDP setups. The paper shows single-GPU numbers only; bandwidth and wall-clock analyses for 2- and 8-GPU nodes would clarify scalability.\n\n- **Host RAM usage**: Off-loading stores one fp32 copy of all targeted weights (about 20 GB for a 70 B model). Techniques such as bf16 accumulation or on-the-fly reduction could cap memory, but these are not discussed.\n\n- **Hyperparameter sensitivity.**  GoRA introduces at least 4 other HPs $r_{min}, r_{max}, \\gamma, N$. Performance can shift by ~3 accuracy points when the scaling factor $\\gamma$ moves from 0 to 0.08. It is unclear how to set $\\gamma\\$ without an exhaustive grid search. \n\n-  **Pseudo-inverse cost**: Unless I didn't see it, the authors do not discuss the compute cost of pseudo inverse. The initialization step requires computing $\\bigl(A^{\\mathsf T}A\\bigr)^{-1}$ for *every* adapter. Although the matrix to invert is only $r \\times r$, the cost scales cubically with rank ($\\mathcal O(r^{3})$) *and linearly with the number of insertion points* (six per transformer block).  *For example*: on an 8 B model with 32 layers and six LoRA adapters per layer, moving from $r=32$ (≈ 32 k flops per inverse) to $r=128$ (≈ 2 M flops) increases total inverse cost by a factor of 64 -- non-trivial if performed on CPU. \n\n### III. Suggestions for improvement\n\nThere are a couple of things that could help improve the paper.\n\n1. **Adaptive or progressive probing.** Stop the warm-up when layer-wise importance estimates stabilize, or continue refining ranks asynchronously during early training.  \n2. **Guidelines for $\\gamma$ and $N$.** Provide practical heuristics to avoid grid search.\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}