Title: Accurate and Efficient Low-Rank Model Merging in Core Space
Venue: NeurIPS 2025
Accepted: yes
Keywords: Core Space

LoRA

Model Merging

Low-Rank

Efficiency
Abstract: In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel approach for merging LoRA-finetuned instances of a same model on various tasks within Core space as opposed to the full space. The authors have shown the proposed approach preserves information while significantly improving the computational complexity as well as the ultimate accuracy. The empirical study on both language and vision tasks have shown the merits of the proposed framework in practice.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The paper proposed an interesting low-rank merging methodology via the core space that is theoretically sound.\n- The experimental study is well-done and convincing.\n- The paper is generally very well-written with a good flow.\n\nWeaknesses:\n\n- While the authors have claimed in the Limitation section that the proposed approach can be also applied in scenarios with heterogenous ranks, the extension is not straightforward unless the corresponding LoRA updates still have the same rank, even though the rank might be different across different layers. In fact, this points to an important limitation of the proposed framework which hasn't been discussed in the paper; that is, the proposed method is not applicable to models that are finetuned via the rank adaptive version of LoRA (e.g. AdaLoRA) where the rank assigned to each weight matric may be different across the tasks. However, the full space and KnOTS do not suffer from this issue in general.\n\n- The argument provided for explaining the superior accuracy of the proposed framework is somewhat weak. In particular, the authors have implied that the better alignment between the core spaces of different tasks is the main reason for the better accuracy, but this is merely a correlation at best. A theoretical analysis or ablation studies are needed to back this \"implicit\" claim and establish the connection between subspace alignment and the generalization error.\n\nQUESTIONS:\n- Are you claiming that the proposed methodology can be applied even when the LoRA rank is different among different tasks? If so, how?\n\n- How does higher subspace alignment explain better generalization of the model after the merge?\n\nLIMITATIONS:\nAs I mentioned above, dealing with different ranks for the same weight matrix is not straightforward and seems to be an important limitation of the proposed framework which hasn't been discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of merging multiple models that have been fine-tuned using Low-Rank Adaptation (LoRA). The authors identify a key challenge: existing merging methods either operate on the full-size weight matrices, sacrificing the efficiency of LoRA, or are suboptimal when applied directly to the low-rank factors. To address this, they propose \"Core Space Merging,\" a novel framework that operates entirely in a low-dimensional space. The core idea is to first project the individual LoRA adaptors into a shared, common basis called \"reference bases.\" This projection creates small \"core matrices\" for each task. The paper proves that this projection is lossless, meaning no information is lost in the transformation. Merging is then performed efficiently on these small core matrices. The final merged model is reconstructed by projecting the merged core matrix back into the original parameter space. The authors provide theoretical analysis of the method's correctness and computational complexity, and demonstrate through experiments on both language (Llama 3 8B) and vision (ViT) models that their approach is faster and also achieves superior state-of-the-art performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nLoRA merging a compelling research area following the democratization of large model finetuning. Improving model merging methods allows to practitioners to potential leverage existing pre-trained checkpoints to improve performance at a faction of the finetuning cost. \n\nAligning the SVD bases is a novel idea that proves effective in practice.\n\nTheorem 10 to justifies that the projection into the core space is lossless as long as the a zero reconstruction error with regards to the reference bases is achieved.\n\nThe complexity analysis (Section 4.3) demonstrates the efficiency gains over prior work.\n\nGiven these strengths, I believe that the proposed algorithm merging is likely to be adopted by both researchers and practitioners.\n\nWeaknesses:\n\nA dominant advantage of model merging which is not studied here is how existing knowledge can be combined to generalize to new tasks. Prior works such as aTLAS [1] have studied this task (including with LoRAs) and should be compared with in future work or cited as a limitation of the current paper.\n\nIt is not clear that the reconstruction loss will be lossless if the total rank of LoRA to combine exceeds the maximum rank of the target weight matrix \n\n[1] Zhang, et al. \"Knowledge composition using task vectors with learned anisotropic scaling.\" Advances in Neural Information Processing Systems (2024)\n\nQUESTIONS:\nHave the authors studied edge cases where the total rank of the LoRA exceeds the maximum rank of the target weight matrix. Is the reconstruction error still 0 in this case ?\n\nLIMITATIONS:\nIt is not clear that the reconstruction will still be lossless if the total rank of the LoRAs exceed the maximum rank of the weight matrix\n\nNo experiments studying generalization to a unseen task", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Core Space Merging, a novel framework for efficiently and effectively merging LoRA modules. The core idea is to project the task-specific low-rank matrices into a referenced \"Core Space\" before applying existing model merging algorithms. The authors provide theoretical guarantees that this strategy is lossless and conduct extensive experiments on vision and language tasks to demonstrate that their method improves the performance of various merging techniques with computation efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method is simple, yet remarkably effective and efficient. It significantly improves upon existing merging techniques while drastically reducing computational overhead.\n- The paper provides a solid theoretical foundation for the Core Space framework. \n- The comprehensive experiments show consistent performance improvements when integrating Core Space with a wide range of existing merging methods.\n\n- The paper is well-written and provide complete and effective experimental details, along with sufficient analysis and ablation studies to ensure the validity and reproducibility of their findings. \n\nWeaknesses：\n\n- The notation of $\\tilde M^{(t)}$ in Eq.(7) and Algorithm 1 is inconsistency and the dimensions in the latter formulation seem incorrect. If the formulation in Eq.(7) is correct, it raises a question about the necessity of performing SVD on each individual task's LoRA matrices since $\\tilde M^{(t)}$ will be ${U_{B}^{ref}}^\\top B^{(t)} A^{(t)} V_{A}^{ref}$ directly.\n- As the authors claim, a current limitation is that the method's applicability has only been demonstrated for LoRA. It would be valuable for future work to explore its compatibility with other PEFT methods to test the generality of the Core Space concept. Examples can include:\n  - VeRA, which uses trainable vectors and frozen random matrices.\n  - HydraLoRA, which uses an asymmetric structure with a shared A matrix.\n\n​\t[r1] VeRA: Vector-based Random Matrix Adaptation\n\n​\t[r2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning\n\nQUESTIONS:\n- In your proposed method, the final merged update has a rank of up to $Tr$.  In contrast, methods like Full Space or KnOTS merging might result in merged updates of rank $r$. For a fair comparison, do you consider providing the merging results under a same target rank? \n\n- The construction of reference bases by stacking task-specific matrices can be viewed as creating a basis for a mixture of LoRA experts. This bears some conceptual similarity to findings in CopRA (specifically their Section 3.1). It introduces a learnable invertible matrix `P` to minimize the alignment difference, which seems related to Eq.(5) in this paper. Could you please elaborate on the relationship and key differences between Core Space Merging and the strategy proposed in CopRA?\n\n[r3] CopRA: A Progressive LoRA Training Strategy, UniReps Workshop.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of efficiently merging multiple models that have been adapted using Low-Rank Adaptation (LoRA). The authors observe that existing merging techniques often require reconstructing the full weight matrices from the low-rank components, before merging, which results in a computational overhead. To solve this, they authors propose to identify a shared, low-dimensional basis for all LoRA updates by performing an SVD on the concatenated LoRA matrices. Then, any merging algorithm can be applied to this compact space. The authors provide a theoretical framework to show that the projection is lossless alongside a complexity analysis showing significant efficiency gains. The authors show extensive experiments on both vision (ViT) and language (Llama 3 8B) models and show that the proposed method not only has significant computational advantage but also results in improved performance\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper is well-written, with a clear motivation.   \n2. The proposed method is theoretically grounded and straightforward.   \n3. The primary contribution is that the proposed method is not only dramatically more efficient compared to KnOTS but also improves the final performance of merging algorithms.   \n4. The experimental validation includes many model scales, domains and model merging techniques as baselines. The authors show consistent improvements across all combinations. \n\nWeaknesses:\n\n1. The paper focuses on the efficiency of merging which I am not sure is really an issue. Even in weight space, the transformation from low rank matrices to full will include a few matrix multiplications which is minor compared to fine-tuning. In the meantime, there exist new sota methods (in full space) that work much better than TA (perhaps with stronger assumptions). The paper focuses on optimizing merging efficiency rather than full final performance.   \n2. The framework's ability to merge models assumes that the different task-specific LoRA updates can be aligned within a common reference basis. This assumption may not hold for highly dissimilar tasks, which could limit the method's applicability. This can also be problematic beyond classification benchmarks where the fine-tuning dynamics become more complex.  \n3. Minor: Figure 2 needs more context to become standalone (as much as possible of course)\n\nQUESTIONS:\n1. Why are the numbers so low in Table 3? For TA: MNIST 53.12, SVHN 41.25 for an avg of 63.78 normalized. In comparison, TSV paper reports 76.55 for TA and 92.31 for TSV (for full-fine-tuning), while the paper reports 66.66 (for LoRA transformed to full). WHy such a huge discrepancy? Similar comments apply to the ViT-L/14 table in the appendix. \n2. What are the single-task accuracies here? Most papers report both absolute and normalized, or just absolute, but the authors here choose only normalized.  \n3. “in Sec. 5.1 we show that merging in Core Space yields better performance than merging in alternative spaces, when the merging function is non-linear.” Can you explain your intuition as to why this happens?  \n4. “Our approach is found to yield compact representations of each task and improve alignment between subspaces of each task.” Isnt this at odds with weight interference from ties \\[32\\] and task interference from \\[a\\]?   \n5. Figure 1: isn;t the bulk of the computation for e.g. ties the validation to compute the optimal scaling? This applies to knots as well as the proposed method.\n\n\\[a\\] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., & Frossard, P. (2024). Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-05"}