Accurate and Efficient Low-Rank Model Merging in Core Space
Authors: name: Aniello Panariello; name: Daniel Marczak; name: Simone Magistri; name: Angelo Porrello; name: Bartłomiej Twardowski; name: Andrew D. Bagdanov; name: Simone Calderara; name: Joost van de Weijer
Published: 2025-09-22 13:48:15

Abstract: In this paper, we address the challenges associated with merging low-rank
adaptations of large neural networks. With the rise of parameter-efficient
adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning
has become more accessible. While fine-tuning models with LoRA is highly
efficient, existing merging methods often sacrifice this efficiency by merging
fully-sized weight matrices. We propose the Core Space merging framework, which
enables the merging of LoRA-adapted models within a common alignment basis,
thereby preserving the efficiency of low-rank adaptation while substantially
improving accuracy across tasks. We further provide a formal proof that
projection into Core Space ensures no loss of information and provide a
complexity analysis showing the efficiency gains. Extensive empirical results
demonstrate that Core Space significantly improves existing merging techniques
and achieves state-of-the-art results on both vision and language tasks while
utilizing a fraction of the computational resources. Codebase is available at
https://github.com/apanariello4/core-space-merging.