Title: RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Low-rank adaptation

parameter-efficient fine-tuning

optimization
Abstract: Low-Rank Adaptation (LoRA) lowers the computational and memory overhead of fine-tuning large models by updating a low-dimensional subspace of the pre-trained weight matrix. Albeit efficient, LoRA exhibits suboptimal convergence and noticeable performance degradation, due to inconsistent and imbalanced weight updates induced by its nonunique low-rank factorizations. To overcome these limitations, this article identifies the optimal low-rank factorization per step that minimizes an upper bound on the loss. The resultant refactored low-rank adaptation (RefLoRA) method promotes a flatter loss landscape, along with consistent and balanced weight updates, thus speeding up stable convergence. Extensive experiments evaluate RefLoRA on natural language understanding, and commonsense reasoning tasks with popular large language models including DeBERTaV3, LLaMA-7B, LLaMA2-7B and LLaMA3-8B. The numerical tests corroborate that RefLoRA converges faster, outperforms various benchmarks, and enjoys negligible computational overhead compared to state-of-the-art LoRA variants.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper “RefLoRA: Refactored Low-Rank Adaptation for Efficient Fine-Tuning of Large Models” improves upon standard LoRA by dynamically re‐factorizing its low‐rank updates to minimize a provable upper bound on the loss, yielding more balanced and consistent weight adjustments. Empirical results show that RefLoRA and its lightweight variant converge faster and more stably on low‐rank matrix problems and standard PEFT benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n\n-  The method derivation is sold - I did not spot any logic flaws. \n- Pseudocode is well readable. \n- I appreciate the matrix completion benchmark - as it gives good insight in the method without the bloat of a full LLM architecture\n\nWeakness: \n\n- It is a bit unclear how the simplified method (Algorithm 2) is derived. \n- See questions below\n\nQUESTIONS:\nQuestions: \n1. Can the authors provide a convergence theorem, similar to [1] (Theorem 2)? I think the preconditioning of the gradients are somewhat related to the manifold optimization problem introduced in [1]. I.e. they combat the problem of high curvature of the manifold consisting of rank r matrices. Can the authors comment on this?\n2. What happens if the Gram matrices A^TA and B^TB are near singular? Computation of $\\tilde{S}$ can become hard? \n3. Have you evaluated performing the refactor step every k>1 iterations (instead of every step)? How does this affect convergence speed and runtime?\n\nComments:\n1. Assumption 1 can also be enforced by using the methods of [1,2]. \n2. In [1], the authors also discuss the shortcomings of the LoRA learning dynamics and how to circumvent them. It is relevant work that should be added to the manuscript. \n\n\n\n\n[1] https://arxiv.org/abs/2410.18720\n[2] https://arxiv.org/abs/2205.13571\n\nLIMITATIONS:\n- Ill conditioning of the Gram matrices needs to be addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces RefLoRA, a novel method for efficient fine-tuning of large language models (LLMs) by addressing the limitations of Low-Rank Adaptation (LoRA). The authors propose dynamically selecting the optimal low-rank factorization per step to minimize an upper bound on the loss, leading to balanced updates and faster convergence. The method is evaluated on matrix factorization, natural language understanding, and commonsense reasoning tasks, demonstrating improved performance over existing LoRA variants. The theoretical analysis is rigorous, and the experimental results are comprehensive, covering multiple LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides a solid theoretical foundation for RefLoRA, including proofs for optimal factorization and balanced updates (Section 3). The derivation of the closed-form solution for the optimal S_t (Theorem 3) is particularly compelling.\n2. The experiments are extensive, covering diverse tasks and models. The results consistently show RefLoRA’s superiority over LoRA and other variants.\n3. The paper is well-written, with clear motivations, methods, and results. The visualization of loss landscapes (Figure 1) effectively supports the theoretical claims.\nWeaknesses:\n1. While RefLoRA-S is introduced as a simplified variant, its trade-offs (e.g., performance vs. complexity) are not thoroughly explored. For instance, how much performance is sacrificed compared to full RefLoRA, especially in low-rank settings (r=16)?\n2. The GLUE results for LoRA-Pro and LoRA-RITE are omitted due to runtime or convergence issues. This raises questions about the fairness of comparisons.\n\nQUESTIONS:\n1. The largest model tested is LLaMA3-8B. The authors mention scaling to 30B models as future work, but can RefLoRA’s overhead become prohibitive for even larger models (e.g., 100B+ parameters)?\n2. RefLoRA closes the gap to full fine-tuning, but the gap persists in some tasks. Is this due to inherent limitations of low-rank updates, or could further refinements help?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper introduces RefLoRA, a novel approach to improve Low-Rank Adaptation (LoRA) for parameter-efficient fine-tuning (PEFT) of large language models (LLMs). It addresses LoRA’s limitations, such as slow convergence and unbalanced weight updates due to nonunique low-rank factorizations, by optimizing the factorization process to minimize a loss upper bound, resulting in faster and more stable convergence.

STRENGTHS AND WEAKNESSES:
The strengths of this paper are as follows.
Rigorous Theory: RefLoRA’s optimization of the SPD matrix $ \mathbf{S}_t $ is supported by detailed proofs (Appendix B, Theorem 3), ensuring mathematical rigor.
Strong Experiments: Outperforms LoRA, DoRA, and AdaLoRA on GLUE (88.50 vs. 88.15) and reasoning tasks (78.85 vs. 76.19) using DeBERTaV3 and LLaMA models (Tables 2–3).

There are some weaknesses in this paper:
Limited Model Scope: Tested only on transformer-based LLMs, limiting generalizability (not addressed in limitations).
Incomplete Baselines: Omits LoRA-Pro and LoRA-RITE due to runtime/convergence issues (Section D.6), weakening claims.

QUESTIONS:
RefLoRA is tested only on transformer-based LLMs (DeBERTaV3, LLaMA series) for NLU and reasoning tasks. Can the authors provide evidence or analysis of RefLoRA’s applicability to other architectures (e.g., vision transformers, diffusion models) or tasks (e.g., text generation, RLHF)?
The paper omits LoRA-Pro and LoRA-RITE due to runtime and convergence issues (Section D.6). Can the authors provide a partial comparison, alternative evaluation, or explanation of why these methods failed, and how RefLoRA compares to other recent LoRA variants?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-04