Title: Uni-LoRA: One Vector is All You Need
Venue: NeurIPS 2025
Accepted: yes
Keywords: Extreme parameter-efficient fine-tuning

unified framework

intrinsic dimension
Abstract: Low-Rank Adaptation (LoRA) has become the de facto parameter-efficient fine-tuning (PEFT) method for large language models (LLMs) by constraining weight updates to low-rank matrices. Recent works such as Tied-LoRA, VeRA, and VB-LoRA push efficiency further by introducing additional constraints to reduce the trainable parameter space. In this paper, we show that the parameter space reduction strategies employed by these LoRA variants can be formulated within a unified framework, Uni-LoRA, where the LoRA parameter space, flattened as a high-dimensional vector space R^D, can be reconstructed through a projection from a subspace R^d, with d << D. We demonstrate that the fundamental difference among various LoRA methods lies in the choice of the projection matrix, P ∈ R^{D×d}. 
Most existing LoRA variants rely on layer-wise or structure-specific projections that limit cross-layer parameter sharing, thereby compromising parameter efficiency. In light of this, we introduce an efficient and theoretically grounded projection matrix that is isometric, enabling global parameter sharing and reducing computation overhead. Furthermore, under the unified view of Uni-LoRA, this design requires only a single trainable vector to reconstruct LoRA parameters for the entire LLM -- making Uni-LoRA both a unified framework and a “one-vector-only” solution. Extensive experiments on GLUE, mathematical reasoning, and instruction tuning benchmarks demonstrate that Uni-LoRA achieves state-of-the-art parameter efficiency while outperforming or matching prior approaches in predictive performance.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors present Uni-LoRA, a unified framework for understanding and improving parameter-efficient fine-tuning (PEFT) methods for large language models (LLMs), particularly those based on Low-Rank Adaptation (LoRA). The key insight is that various LoRA variants, including Tied-LoRA, VeRA, and VB-LoRA, can be reinterpreted through a common lens: all implicitly perform a projection from a low-dimensional subspace \\( \\mathbb{R}^d \\) to the full LoRA parameter space \\( \\mathbb{R}^D \\), where \\( d \\ll D \\). Under this formulation, the primary difference among methods lies in the choice of the projection matrix \\( P \\in \\mathbb{R}^{D \\times d} \\). The authors further propose an isometric projection matrix that enables global parameter sharing across layers and modules, leading to a more efficient use of trainable parameters. The authors conduct experiments to demonstrate the efficiency of the Uni-LoRA.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n1. Presents a novel and unified perspective that subsumes a wide range of LoRA-based extreme PEFT methods under a common framework.\n\n2. Provides both theoretical justification and empirical validation for the proposed projection matrix design.\n\n3. The method is lightweight and easily integrable into existing mainstream fine-tuning frameworks.\n\n**Weaknesses**:\n\n1. The comparisons in Table 3 and Table 4 are limited to only two or three baselines. It would strengthen the evaluation to include methods such as LoRA-XS and FourierFT under comparable trainable parameter budgets.\n\n2. While the proposed projection matrix improves parameter efficiency, it may introduce additional inference latency compared to standard LoRA. An analysis of inference speed would provide a more complete picture of the trade-offs.\n\t\n3. Although Uni-LoRA significantly reduces the number of trainable parameters, the gains in training time and memory usage are marginal, particularly on larger models (e.g., LLaMA2-7B and LLaMA2-13B, as shown in Table 10). Additional discussion would be beneficial.\n\nQUESTIONS:\n1. How do the authors ensure training stability, particularly given the random initialization of the projection matrix? Does this introduce higher variance or increase the standard error across runs?\n\n2. Can the authors provide results under varying values of r and d to better illustrate the trade-off between parameter efficiency and performance?\n\n3. It would be helpful to include an analysis of GPU memory consumption trends as the model size increases, to understand the scalability of Uni-LoRA in terms of memory footprint.\n\n4. I am curious about the performance of Uni-LoRA on reasoning-focused models. Could the authors provide preliminary results or insights in this area?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a general framework that expresses mappings between the trained parameter subspace and the LoRA parameters (usually matrices $A_l$ and $B_l$) of different more efficient LoRA variants (LoRA, Tied-LoRA, VeRA, VB-LoRA, LoRA-XS, FourierFT). The paper expresses the concatenation of all trained and fixed LoRA parameters $\\theta_D\\in \\mathbb R^{D}$ as a product of a projection matrix $P\\in\\mathbb R^{D\\times d}$ (trained or fixed) and a trained lower-dimensional vector $\\theta_d\\in \\mathbb R^d$:\n$$\n\\theta_D = P \\theta_d \\text.\n$$\n\nBased on this expression, where each trained parameter can influence parameters in multiple LoRA modules, the paper proposes Uni-LoRA for greater parameter efficiency. In Uni-LoRA, $P$ is fixed and each of its rows has a single randomly chosen non-zero element. The sparsity enables $O(D)$ time complexity. The value of non-zero elements is set to $P[i, j] = 1/\\sqrt{[\\![\\sum_{j=1}^D P[i, j]\\neq 0]\\!]}$ to make $P$ isometric (L2 distance preserving).\n\n**Results**\n\nThe method is compared with other PEFT methods (LoRA, Tied-LoRA, VeRA, LoRA-XS, and FourierFT) on the GLUE language understanding benchmark (6 tasks) with 2 RoBERTa-base and RoBERTa-large. Uni-LoRA optimizes about 23k (~8% of LoRA) parameters (less than all the other methods) and scores probably among the best 2 PEFT methods.\n\nUni-LoRA is also very competitive on mathematical reasoning (Mistral-7B and Gemma-7B on GSM8K and MATH) and instruction tuning (LLama2 7B and 13B on MT-Bench) while optimizing the least amount of parameters (0.52M - 1M, 0.26%-0.4% of LoRA).\n\nAblation experiments on 4 GLUE tasks show that the uniform projection of UniLoRA is more parameter-efficient than non-uniform (twice as much parameters allocated to $A$ matrices than to $B$ matrices) and (maybe) local (per-layer) projections.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The presented framework is interesting.\n2. The proposed method outperforms or competes very well with other PEFT methods.\n3. The paper is very well written.\n4. The source code for reproducing the experiments will be published (If I understand correctly.).\n\n**Weaknesses**\n\n1. L202, 233-237: \"isometry [...] can substantially enhance adaptation performance\". This sounds intuitive, but I don't find it clear how important isometry is. I suggest supporting the claims better.\n\nMinor issues:  \n- I am unable to find the code for reproducing some experiments (Fastfood).\n- LoRA-XS could be described more clearly: SVD is not mentioned.\n- L57: Consider clarifying in which way $P$ is theoretically grounded.\n- Figure 1: \"top-k of learnable logits\" is mentioned in the legend, but I don't see it being used.\n- L167: I don't find it clear why \"Apparently\" is used.\n\nQUESTIONS:\n**Questions:**\n\n1. Please provide better support regarding weakness 1.\n2. Table 3: Do you have any intuition why LoRA outperforms other methods on MATH, but not GSM8k?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Uni-LoRA, a unified framework for Low-Rank Adaptation (LoRA) that generalizes and improves upon existing parameter-efficient fine-tuning (PEFT) methods. The authors claim that their Uni-LoRA achieves state-of-the-art parameter efficiency while\noutperforming or matching prior approaches in predictive performance.\n\nSTRENGTHS AND WEAKNESSES:\n> **Strengths:**\n>\n> Uni-LoRA advances PEFT by unifying LoRA variants into a single framework, this can help us better understand LoRA-based fine-tuning methods and further reduce the number of parameters required by LoRA.\n\n> **Weaknesses:**\n> \n> (1) Line 213-214 (Theorem 1) says \"P is a projection matrix where each row selects exactly one entry uniformly at random to be nonzero, and sets all other entries to zero\". Lacks stability analysis (sampling has randomness).\n>\n> (2) Theorem 1 is incorrect because it fails to account for cases where all rows of matrix P are sampled identically. For example, when D=3, P = ((1/√3, 1/√3, 1/√3)ᵀ, (0, 0, 0)ᵀ). In this case, clearly PᵀP ≠ I. Additional analysis on the impact of P's rank should be included.\n>\n> (3) For fine-tuning, LoRA is not shown to reach a comparable performance as full-rank fine-tuning [1]. Further investigation should be conducted on the impact of reducing the number of LoRA parameters.\n>\n> [1] GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection. ICML 2024 Oral.\n>\n> (4) Lines 329-334: Why does \"local projection perform worse than global projection\"? Under sufficient training (with other hyperparameters being the same), each local part should learn parameters corresponding to its global counterpart.\n>\n> (5) Line 304: When comparing LoRA, only the case of rank=4 was evaluated. Different ranks should be compared to better demonstrate the effectiveness of the proposed method.\n>\n> (6) Lines 319-320: Could this be a formatting issue?\n\nQUESTIONS:\nCheck the weaknesses part, I'm mainly concerned about 1-5.\n\nLIMITATIONS:\nyes, some limitations are discussed in Conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 1, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nUni-LoRA proposes a unified framework that maps various LoRA variants into a shared low-dimensional subspace via a global projection matrix, enabling parameter sharing and compression. Its core innovation lies in leveraging a single trainable vector to achieve parameter sharing while preserving the isometric structure of the parameter space. This approach significantly reduces memory and computational overhead, while achieving performance comparable to traditional fine-tuning methods on tasks such as GLUE, GSM8K, and MATH.\n\nSTRENGTHS AND WEAKNESSES:\nAdvantages:\n1. Uni-LoRA demonstrates high computational and memory efficiency in the application of large-scale pre-trained models. The experiments are sufficiently conducted and are backed by a solid theoretical foundation.\n2. The structure of the paper is clear, effectively conveying the core ideas of the research, with a well-defined experimental design. The paper provides reproducible experimental procedures and code support.\n3. Uni-LoRA reduces the number of parameters while maintaining good performance, making it highly significant for practical applications, particularly in resource-constrained environments.\n4. The paper introduces a unified framework that integrates various LoRA variants through a global projection matrix, allowing all variants to be optimized via a shared low-dimensional subspace, thus reducing computational complexity. Additionally, the proposed projection matrix P ensures that the geometric structure of the optimization space remains preserved.\n\nDisadvantages:\n1. The details of the projection matrix selection are not fully addressed. While the projection matrix P is described as globally shared and optimized via backpropagation, the paper lacks sufficient discussion on how to dynamically adjust and select the optimal dimension d in practical applications.\n2. While the unified framework proposed in the paper improves both parameter and computational efficiency, the simplification of the method may limit its adaptability in certain tasks, especially in more complex scenarios. The effectiveness of a single projection matrix in maintaining good performance for complex tasks warrants further investigation.\n\nQUESTIONS:\n1. The distance maintained by the projection matrix P is Euclidean, while most data typically resides in non-Euclidean spaces. Could the adoption of geodesic distance on manifolds be considered to optimize the projection matrix P?\n2. The projection matrix P is relatively simple, and it is unclear whether it can maintain good performance on more complex data. Could the use of multiple projection matrices be considered to achieve better performance?\n3. Could a discussion on the dimension d of the low-dimensional subspace be included in the parameter experiments to explore the impact of different dimension choices on performance?\n\nLIMITATIONS:\n1. Despite the outstanding performance of Uni-LoRA on NLP tasks, its effectiveness in cross-domain applications or complex tasks (such as image processing and cross-lingual transfer) has yet to be fully validated. The authors could further discuss the method's adaptability in other domains, such as computer vision or reinforcement learning, and highlight potential challenges related to its adaptability.\n2. The experimental validation in the paper primarily focuses on medium- to large-scale models (e.g., Gemma-7B, Mistral-7B), Uni-LoRA: One Vector is All You Need with limited testing on extremely large models (such as GPT-3). There is insufficient discussion on whether the method can effectively operate on ultra-large models, particularly in distributed training environments.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}