Accurate and Efficient Low-Rank Model Merging in Core Space
Authors: Aniello Panariello
Daniel Marczak
Simone Magistri
Angelo Porrello
Bartłomiej Twardowski
Andrew D. Bagdanov
Simone Calderara
Joost van de Weijer
Venue: NeurIPS 2025 poster ()
Keywords: Core Space
LoRA
Model Merging
Low-Rank
Efficiency
Primary area: deep_learning
TL;DR: The paper proposes Core Space Merging, a method to efficiently merge LoRA-adapted models by aligning them in a shared low-rank subspace, achieving higher accuracy and major speedups over prior merging techniques.

Abstract: In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

Accurate and Efficient Low-Rank Model Merging in Core Space
Aniello Panariello'*
Daniel Marczak
Simone Magistri'
Angelo Porrello'
Bartlomiej Twardowski
Andrew D. Bagdanov Simone Calderara'
Joost van de Weijer
AlmageLab, University of Modena and Reggio Emilia, Italy
Warsaw University of Technology, Poland IDEAS NCBR, Warsaw, Poland Media Integration and Communication Center (MICC), University of Florence, Italy
IDEAS Research Institute, Warsaw, Poland Computer Vision Center, Universitat Autônoma de Barcelona, Spain
Abstract
In this paper, we address the challenges associated with merging low-rank adap- tations of large neural networks. With the rise of parameter-efficien adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight ma- trices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide formal proof that projection into Core Space ensures no loss of information and provide complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of- the-art art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at httth::///// // github........ comhuu..ooomaaoemommmmma..o.eee. com/ http://aaaaiell///rreeee http///ggoto..oo.ppapaaaao/////e -pw.----ea----rrr-iin
1 Introduction
In recent years, the size of neural networks has grown substantially 2 4. 9 13 51, increasing the economic and computational costs associated with training from scratch and fine-tuning. As consequence, efficient low-rank adaptation techniques have emerged, which enable broader access these powerful models [6 21 26 (56). Techniques like Low-Rank Adaptation (LoRA) [I6] reparameterize model updates to significantly reduce the number of trainable parameters. This makes feasible for a broader range of users to fine-tune large architectures on their specific tasks.
At the same time, the advent of model hubs such as Hugging Face 17 has simplified the diffusion of pre-trained and fine-tuned models, opening new opportunities for collaborative and multi-task learning by allowing users to acquire and build upon existing models easily. In this context, model merging, which aims to combine multiple specialized models into one capable of handling various 48 27 39). However, most prior works focus on fully 38 While this is practical for smaller architectures. fully
tasks, has been gaining interest 32 37 fine-tuned models fine-tuned versions of larger models are rare due to their high memory and compute costs. As model
'Equal Contribution. aniello. panariello@unimore it, daniel. marczak marczak.dokt@pp eekktppw.edu.p.....
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

sizes grow, new strategies are needed to efficiently merge fine-tuned adaptations without incurring the prohibitive overhead of merging full models.
In [42] the authors observe that directly applying existing merging techniques 18
50
55
updates derived by multiplying low-rank components leads to suboptimal results. To ad- dress this, they introduce an alignment space that improves update compatibility. However, merging in this alignment space requires abandoning the low-rank representation and perform- ing a singular value decomposition (SVD) on the horizontally concatenated full space up- dates. This suffers from two significant drawbacks: it eliminates the efficiency benefits of low-rank adaptation, and becomes prohibitively expensive as the base model size increases.
To overcome such limitations, we propose Core Space, novel parameter-efficient sub- space that supports arbitrary merging tech- niques while retaining the benefits of low-rank adaptation. Core Space provides common alignment basis for all task-specific low-rank components without loss of information. No- tably, its dimensionality depends solely on the number of tasks and the LoRA rank, remain- ing tractable regardless of the base model size. Beyond its advantages in terms of efficiency, merging in Core Space also consistently im- proves the performance of existing merging strategies. We evaluate three setups: (1) first multiplying low-rank matrices and then applying merging techniques, (2) merging
Best
perforaance cost
Merging method:
Space:
TIES DARE-TIES
Full
95
KnOTS
TSV
Core
94
280x speed-up
Accuracy
+1.6% acc
92-
91
90
10'
10a
10a
Merging Time (s)
in the KnOTS space 422, and (3) merging in our proposed Core Space. Across both vi- sion and language domains, merging in Core Space achieves the best results demonstrated on ViT-B/32, ViT-L/14, and Ilama 3 8B back- bones highlighting that our approach not only preserves parameter efficiency but also leads to improved generalization and task performance (see Fig. D
Figure 1: Merging in full space is fast but subopti- mal (bottom center). Merging KnOTS space or using strong merging methods (e.g.. TSV) improves performance but increases cost by orders of magni- tude (right). Core Space merging is effective and efficient (top left). Results on Llama 8B.
The main contributions of this paper are the following:
We introduce Core Space Merging, framework to merge LoRA-adapted models in shared low-rank basis, avoiding costly full space operations, while improving accuracy. Our approach can be easily integrated with existing merging methods.
We prove that projection into Core Space ensures loss of information, and provide complexity analysis demonstrating the efficiency gain of merging in the proposed space.
We present an extensive empirical evaluation showing state-of-the-art results achieved at a fraction of the computational cost of competing methods by merging in Core Space for vision and language tasks, including experiments on ViT-B/32, VIT-L/14, and Llama 8B.
2 Related Work
Parfmneterefffcient fine-tuning (PEFT). Pre-trained models serve as a starting point for training
experts specialized in various downstream tasks [10 36]. As the size of frontier models grows, the cost
of fully fine-tuning such models increases accordingly. Therefore, several parameter-efficien fine- tuning (PEFT). updating a small fraction of parameters, have been proposed, including adapters [15] prefix tuning (25), and prompt tuning (24]. Nowadays, LoRA (16) and its variants [2] 26], which rely on low-rank updates, have emerged as of the most popular PEFT techniques.

---

Model merging. The abundance of available expert models inspired the fundamental question behind model merging [32]: how can we integrate knowledge from multiple expert models into single mulfi-task model? Task Arithmetic [88] proposed to construct task vectors (i.e., the parameter difference between expert and base model) and aggregate them via scaled addition, creating multi- task expert. Since a significant performance gap between the single-task and the merged models remains, many approaches were proposed to address this issue I 29
31 43 52
54 35]. TIES-
Merging (50] focuses on reducing sign conflicts between the parameters of expert models. Model Breadcrumbs [8] removes outliers from the task vectors, while Consensus Merging 46] eliminates
catastrophic and selfish weights. Most recent methods, like TSV [12] and Iso-C [28 rely on singular value decomposition (SVD) of weight update matrices to reduce task interference when merging models. However, most methods are designed for merging fully fine-tuned models.
Merging LoRA-adapted models. Methods designed to merge fully fine-ture fine-tuned models do not necessarily transfer well to merging LoRA- models [44). The authors of [44] proposed and improved a method to merge LoRAs. However, their approach relies on altering the fine-tuning procedure. KnOTS 42 proposes merge LoRA updates in the shared subspace, achieving significant improvement. However, KnOTS performs SVD on the concatenation of full-size matrices instead of leveraging their decomposed update representations, making it costly, especially for large weight matrices. Therefore, finding a method that effectively and efficiently merges LoRA-adapted models remains an open challenge.
Preliminaries
LoRA fine-tuning. Low-Rank Adaptation (LoRA) [16] is a technique for efficient fine-tuning of large pre-trained models. Instead of updating the full model weights W E: R=*x LoRA introduces two learnable matrices A E R=*x" and B E mx (where r < (mn,m, n)), and modifies the weight update as W Wo AW Wo B A, where Wn is the original weight matrix. This significantly reduces the number of parameters that need to be updated during fine-tuning.
Model merging. Given a set of T parameters (W1,-- Wr), for common architecture trained on different tasks, a basic model merging approach calculates task vectors AW, = W; W. and computes a weighted sum W merged = Wo + a Σ AW, When dealing with LoRA- models we obtain common Wo and set of decomposed, low-rank updates { {A A,A,) J However, merging such weight matrices obtained from LoRA leads to suboptimal performance as shown in 42). since LoRA- -dRated models are less aligned w.r.t. to their fully fine-tuned counterparts.
4 The Core Space Merging Framework
In this section, we introduce Core Space Merging (see Fig. 2). a framework designed to identify an effective and efficient subspace referred to the Core Space in which model merging for LoRA-adapted models can be performed while remaining in the low-rank regime. Core Space is designed to be reversible it ensures no loss of information when projecting into Core Space and back to the original space while being as compact as possible. Compactness allows for the use of state-of-the-an merging methods relying on Singular Value Decomposition (SVD) of weight matrices, which are highly costly to perform in the original space for large models.
4.1 Model Merging in Core Space
Let A E Brxx and B(O) E _R denote the low-rank matrices for task i, derived from a shared pre-trained base model Wo. Each task update (W ((())(( can be reconstructed from the SVD of the matrices:
1 U (t)T
B(t) = U I V
(I)
(t) (8) U
(t) (t)
> (()) (()
B
where the shapes of the matrices in the decomposition are: (ttt) Rexr E(t) Rexr, v(()) E Raxr,
(1(0) E mxxr 2010 E grxr, and v(( E grxr.
3

---

Full Space Merging
KnoTs Merging
Core Space Merging
Compute reference bases
A
A
A
SVD
B
B
B
B
----
B
B
B
SVD
+
AWO
Aw
AW
AWO
AW
AWM
Horizonta Concal
2 Project into Core Space
Concat
SVD
Merging
Task
UB
A
Core Matrices
U
Σ
AW
Merging
3 Merge Reconstruct
Merged
SVD
Decomposition
Singular Value
! AW
Merging
AW
Full Space Merging (left) firstly reconstructs full space matrices AW B()) and then performs merging in the full space to obtain AW. KnOTS Merging concatenates the ( matrices, and performs a costly SVD on this high- oig-dimmnsiooal matrix. Then, the v(() matrices are merged and used to obtain the final AW. The proposed Core Space Merging (right) performs SVD on a concatenation of low-dimensional ( and (()) matrices obtain reference bases (Vyef Utef).
Figure 2:
Afterwards, it projects each update into the Core Space obtain the matrices {M It=1 It then performs merging in the Core Space and reconstructs to obtain the final AW.
Under the hypothesis that all tasks share approximately the same common bases (UB, VA) such that VA. E (1...., T) UB s U and VA Re V/A, we have:
Motivation.
T
AM UB AW Σ AW Σ
Σ
D (0) T
U VT.
(2)
where ((!) T E Rx encodes the directional transformation applied by the low-rank update of task t. This suggests that, under aligned bases, the of low-rank updates (i.e., Task Arithmetic (18) can be reduced to merging operations much smaller r x space.
Projecting into the Core Space. In practice, task-specific bases are not aligned, making direct merging of matrices as Eq. (2) infeasible. Therefore, aim find shared basis that can represent all tasks without loss information and that enables merging to be performed in a reduced space compared to the full space (w("), Intuitively, such a shared basis should span the subspace formed by the union of the individual task subspaces.
Definition (Reference Bases), Given a collection of low-rank matrices (A(1), B(O) as reference bases the orthonormal matrices Une E mRx and Vxel E RRxxTr obtained by performing SVD over the vertically stacked (+) and horizontally stacked ((1) low-rank matrices across tasks:
1 - we define
A(1)
B [T)] = U EBV
= UAAA (V
A
These bases span a shared latent subspace into which all task-specific updates are projected.
Although the reference bases (UTef' VXof) span all task-specific directions, each task is originally expressed its own local bases (Uff) v(1)). To express each update in the common coordinate system for merging, we solve the following least frast-squares problems:
R
arg min neR
ree R U(tt
Q
arg min QER Q P-rx
Viell Q
| p²
F

---

where V Rax and Vi E R=xI- (and similarly for U and U These problems are convex, and since Ugf and Vi are orthonormal, setting the gradients to zero yields the global minimizers (see Sec. for the full derivation):
R = U U
Q4 = V
(5)
As we will show Sec. Ug R and similarly VA
4.2
1 12 O, which allows us to substitute U with
with Vr Q in Eq.
1 yielding:
AW = UB RB M
V 0 V
(6)
task-t update in reference coordinates
By substituting the least-squares solutions from Eq. (5) and using the definitions of B(() and A(O) from Eq. ( we can equivalently write:
(w Ug (UBr B A(v Viet"
This reformulation expresses each AW()) in the reference hasis, enabling all updates to be compared or merged within a shared coordinate system.
Definition (Core Matrix). We define the core matrix M(1) as:
M = (U T B()) (A Viret E Trxxy
(8)
This formulation generalizes the middle expression in Eq. (2). where aligned task-specific bases were implicitly assumed. contrast, the core matrix (t) is expressed in the reference bases (U'ef Virf) and thus does not rely any alignment assumption. It encodes the directional transformation applied by the low-rank update of task f, providing a compact and lossless representation of each task update in the shared reference space and enabling efficient merging in reduced Tr x Tr space.
Reparametrized Model Merging in Core Space. Once task-specific updates (W have been repa- rameterized into their corresponding core matrices M()) in the shared reference bases (Uref Viel), Core Space Merging enables model merging to be performed entirely within a compact, aligned, low-rank subspace. Specifically, the merged update is computed by applying merging operator M over the set of core matrices:
Mmerged M({M(0) T Pe
where M may be any merging function, ranging from simple arithmetic averaging 18 to more advanced, non-linear or geometry-aware techniques [50, (5..55) The final update the original model space, as described also in Algorithm is recovered by projecting M Mmerged back through the reference bases:
AW = U M V/ref" A
(10)
Because Core Space a lossless representation of the original updates for each individual task (see Eq. 7)), merging in this space preserves all relevant task information. Furthermore, when M is linear, such as Task Arithmetic, the merge operation in Core Space is exactly equivalent to applying the same merge in the full model space:
M((Aw()) T' = M((UTE M T
} = Un! M({M(9) T )
(11)
Core Space merging offers key benefits over full space merging:
Efficiency. Core matrices (t E RT are significantly smaller than their full space counterparts (W E RR This reduction enables high-cost merging algorithms to run at fraction of the time and memory footprint (see Sec. 4.3)
Efficacy. As shown in Sec. 5.1 Core Space merging improves performance over full space merging when non-linear methods are used. In Sec.5... we show that this improvement stems from better alignment and more compact representation of task-specific directions.
5

---

Algorithm Core Matrix Alignment and Merging
Require: Low-rank updates {(A'9, B merging function M(-).
1: Stack A vertically, B() horizontally
2: Compute V stack(A(()) = UAEAVA
D reference bases
3: Compute U stack(Bi() U VBV V
4: for 1 to T do
5: Compute: M (Ugel) B(O)
Eq. (
6: Merge aligned core matrices: Mmerged = M((MM(TZ
7: return AW = LB Mmerged VrefT
> reconstructed merged model
4.2 No Information Loss in Core Space Representation
Replacing U and v(t) with U R and vx Q4 to obtain Eqs. (6) and (7.. which define the final form of the core matrix, requires that the solutions to the least-squares problems in Eq. (4) incur zero alignment error. That is.
UH R 1 0.
| 12 =0 =
(12)
In this section, we show that the reference bases Uteef and Vreff obtained via the SVD of the stacked matrices B(t) and A (see Eq. (3), minimize the total alignment error across all T tasks, achieving an error of exactly zero. To illustrate this, we first analyze the alignment error for a single task t, focusing on Uge Analogous results hold symmetrically for V For clarity, we assume in the following derivations that . S m and T r S so that the total LoRA rank does not exceed the maximum possible rank of the target weight matrix, In Sec. A.4] we provide a more general analysis that removes this assumption and demonstrate that the zero alignment error result continues to hold.
Lemma. Let (1) E Ruxr and U E RmxxT-r be matrices with orthonormal columns, and let R Un U E TT be the optimal solution minimizing the error of the least-square problem. Then, the optimal alignment error is given by:
EU
U
(0)
1 = r- 100
(13)
The proof, provided Sec. A.2 leverages the properties Frobenius norm and the orthonormality of U and Uneff To formally demonstrate that our chosen reference basis Une minimizes the alignment error across all T' tasks (or cquivalently maximize || Urref for each task t), we first formulate the following constrained optimization problem for single task, and then extend it to the multi-task scenario:
(()
U
max (U (UTUUUT (100)((0) ((O U
UES
UES
(14)
where
S = -U R - = Ir-}
and ur(-) denotes the trace operator. The optimization domain is restricted to the Stiefel manifold S
(i.e., the of matrices with orthonormal columns). The following lemma characterizes the solution to the following optimization problem:
Lemma. A solution U to the quadratic program in Eq. 14 is given by a basis whose columns
include the eigenvectors corresponding nonzero eigenvalues of B(1) B(t) mmxx or, equiv- alently, by the r left singular vectors of the matrix B(0) Moreover, at the optimum, the objective attains its maximum value r, resulting zero alignment error in Eq. 13
We refer the reader to Sec. A.3 for a detailed proof. Briefly, the result follows by applying the method of Lagrange multipliers to augment the optimization objective with the Stiefel manifold constraint and then enforcing stationarity by setting the gradient of the Lagrangian to zero.

---

Varying (I' = 8, -= 16)
Varying r (1 768, =
Varying 7 (n = 708, r 16)
Operations (log scale)
10-4
10
10P
10a
1010
10y-
Iso-C
Full
10"
10
10"
ko--C KnOTS
10
107
1000
2000
3000
4000
50 100
30
R
Figure 3: Core Space merging is more efficient than the previous state-of-the-art KnOTS. The cost is similar to full space merging, which results in much lower performance. We visualize the number of operations performed merge 7 rank LoRA modules of final shape n n.
Extension multiple tasks. Achieving zero reconstruction error for single model does not guarantee optimality for any other model t' # t. Therefore, we aim to identify a reference basis U that jointly optimizes Eq. (14p across all T models. We formulate this global problem as:
Σ tr(U UB U
U) max tr( UBULU),
(15)
1U U/ The equality Eq. (15 follows directly from the linearity of the trace operator and the distributivity
where UB = =B=1U*
U denotes the horizontal concatenation of all U matrices.
of matrix multiplication concerning matrix addition: M A, + M A2M = M (Ai + A2)
By considerations analogous to the single task-case, global solution U is given by the top T r left singular vectors of the matrix B, obtained by vertically stacking each matrix B(t), i.e., U = Ure This choice ensures zero alignment error simultaneously across all T tasks, consistent with the procedure described in Sec. 4.1
4.3 Computational Complexity Analysis
We summarize the time complexities of TA, Iso-C. and TSV merged in all three spaces (Full, KnOTS, and our Core) in Tab. T and Fig. 3 Details on the derivation of the complexities are in Appendix B Our approach exhibits time complexity comparable that of Task Arithmetic in full space. Our method's additional terms are negligible unless the product T becomes significantly large. A key advantage of our method lies in its scalability compared to KnOTS, whose time complexity is super-cubic, driven by factor that scales cubically with the weight matrix size n. Finally, we emphasize the minimal additional overhead incurred when combining our method with Iso-C TSV in the Core Space; it introduces cost substantially lower than its counterpart in full space or KnOTS space.
Table 1: O()) time complexities. The cheap- est method is highlighted in bold (I', < n).
Space
TA
Iso-C
TSV
Full
n²Tr
n
n
KnOTS
n²I2
n + nTr
T 19,2 n²Tr T
Core
n²Tr
n²Tr T
Experimental Results
Experimental setup. We follow the experimental setup KnOTS and use the LoRA checkpoints For the vision experiments, we use two variants of CLIP (36] with [1 as vision encoders fine-tume tne-tuned on standard set of tasks. We employ Llama 3 8B 13 fine-tuned on 6 NLI tasks for the language experiments. All models are fine-tuned with LoRA [66 with rank 16 applied on matrices (keys, queries, values, and outputs) across all attention layers. Following 42 we report normalized accurucy as a ratio of the accuracy of the merged model on a given task to the accuracy the model fine-tuned on this task. We also report absolute accuracy for the joint-task setting (additional experimental details in Sec. D
provided by the authors 42
ViT-B/32 and ViT-L/14
7

---

Table 2: Normalized accuracies of merged models on NLI tasks for Llama 3 8B.
Method
Space
SNLI
MNLI
SICK
QNLI
RTE
SCITAIL
Avg (AAcc)
Time [s]
Rel. Time
Abs. Accuracy
92.50
90.31
91.58
94.49
89.86
96.52
TA
Full
93.57
95.28
87.96
68.71
100.0
96.73
90.38 (4000)
Full
95.17
96.19
84.18
74,18
100.0
96.78
91.08 (40.00
72
TIES
KnOTS
78.57
100.0
(+145)
3000
Core
92.07
93.51
93.63
83.72
99.19
97.66
93.30 (42.2
8
Full
94.76
96.8
78.39
72.08
98.39
96.20
89.44 (10.00)
108
13
DARE-TIES
KaOTS
91.62
96.72
74.90
84.75
99.48
99.13
91.10 (41.66)
3180
397
Core
92.10
93.58
93.70
83.68
99.19
97.66
93.32
Full
95.38
95.12
88.83
76.80
101.61
97.56
92.55
3360
280
TSV
KnOTS
92.53
95.83
82.77
77.01
100.0
97.08
90.87 (3168)
4800
400
Core
95.70
(41,01)
Full
55.00
39.04
76.54
55,90
46.77
69.25
57.08
540
67
Iso-C
KnOTS
85.28
52.86
89.43
54,90
75.00
77.73
72.53 (+15.45)
4860
607
Core
91.54
90.10
87.87
75.85
99.19
97.42
90.33 (433.25)
Baseline merging spaces. We compare our proposed Core Space with two alternative merging spaces. Full Space operates in space of full reconstructed weight matrices (W B(9 Rmxn. KnOTS Space [42] operates in the space of the right singular vectors of the concatenated
reconstructed weight matrices (AW())) E 1Rmx"T"
Baseline merging methods. We evaluate each merging space using the following merging meth- ods. Task Arithmetic (TA) [18] performs a scaled summation each task matrix W Wmergedd Wo + a Σ AW; As this is linear operation, the results of merging in each space are the same (see Eq. (II) for Core and [22] for KnOTS). TIES [50] trims low-magnitude parameters and
averages parameters with dominating sign, while DARE [55] preprocesses task vectors by randomly
dropping fraction of parameters and rescaling the remaining ones. TSV [12] concatenates low-rank approximations of task matrices and orthogonalizes them across tasks. CART [6] calculates centered task vectors difference of fine-tuned weights from the average of all fine-tuned weights and performs task arithmetic on the low-rank approximation of these centered task vectors. Iso-C [28 flattens the spectrum of singular values for a model merged with task arithmetic. As the spectrum flattening can be performed on weights merged with any merging technique, we combine Iso with other merging techniques, denoting it with +Iso-C.
5.1 Results
LLMs merging. We present Llama 3 8B results in natural language inference in Tab. 2 In line with our complexity analysis, merging in Core Space is much more efficient than merging in Full or KnOTS space, bringing up to 600 merging speed-up. Moreover, merging in Core Space improves the performance of all tested merging methods. In particular, elevates TSV to 94.16% average normalized accuracy, achieving matee-of-the-ars results.
Per-task evaluation in vision setting. We present per-task vision results for VIT-B/31 Tab. 3] We observe that 8 out of 9 merging methods achieve their highest highest average accuracy when performed in our proposed Core Space. The best combination TSV + Iso-C merged in Core Space achieves state- of-the-art average normalized accuracy of 76.3%. It significantly outperforms the previously reported SoTA of TIES in KnOTS space, achieving 68.0% [2]. Similar conclusions hold for experiments on ViT-L/14 presented in Sec. E.I
Heterogeneous ranks. While handling LoRA modules with heterogencous ranks might seem non-trivial, our method supports it seamlessly without modification. Even with different ranks, the modules can be concatenated across tasks to form an aggregate basis spanning the combined subspaces, after which projection and alignment are applied to each local task core matrix. Since SVD makes no assumptions about input ranks, it yields valid orthonormal bases in all cases, enabling our framework to merge variable-rank LoRA modules naturally. We evaluate this setting by assigning rank 16 to half the tasks and rank 64 the rest; the results reported in Sec. B.2 show that our method still outperforms other approaches.
8

---

Table 3: Normalized accuracies of merged models on the vision tasks with ViT-B/32.
Method
Space
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC
SUN397
SVHN
Avg (A Acc)
Abz. Accurecy
74,00
58.30
99.00
92.70
99.30
88.40
64.50
96.20
TA
Full
81.97
73.72
48.97
42.24
53.12
71.50
97.46
41,25
63.78
Full
82.37
72.72
49.91
36.62
57.16
69.38
96.92
44.56
63.70 (40.00
KrOTS
83.75
74.45
67.01
(44.00)
Core
84.74
76.46
52.19
50.41
67.36
71.21
96.45
50.18
68.63 (-U95)
Full
82.14
73.72
49.35
31.78
56.63
70.14
97.35
42.12
63.65
(10.00)
DARE TIES
KnOTS
82.01
72.90
44,
45.54
60.59
70.89
95.56
47.64
64.91
Core
84.57
76.09
57.09
51.01
66.64
71.39
96.16
52.14
69.39 (1874)
Full
83.44
75.55
50.99
45.03
59.31
73.33
96.40
49.23
66.66
ISV
KaOTS
81.86
74.91
51.25
41.64
53.93
71.64
y7.95
40.36
64.19 (2:47)
Core
83.86
75.09
52.64
45.39
58.53
72.95
97.63
45,21
66.41
Full
83.04
81.93
50.39
70.17
59.
79.11
99.26
49.1
71.52 (-0.00)
CART
KnOTS
83,94
18.18
52.23
54.48
64.78
74.48
95.88
55.73
69.50
Core
80.83
83.94
54.99
73.28
66.25
80.95
98.69
48.57
73.44 (+1.92)
74.45
39.02
66.65
67.03 (40.00)
TIES Ibs.CC
KnOTS
78.46
80.38
58.81
64.97
72.10
76.89
98.33
49.78
72.47 (15.4)
Core
84.76
78.79
(48.40
Full
78.71
75.54
50.84
42.86
65.03
71.88
98.
48.08
66.48 (-0.00
DARE-TIES +be-C
KnOTS
82.93
74.18
49.31
46.73
66.64
71.82
96.72
50.57
67.36 (10.8))
Core
83,27
83.12
54,55
79.04
71.83
82.08
99.36
52,37
75.70 (-9.22)
Full
79.38
80.38
57.99
65.64
64.22
79.74
98.59
46.49
71.55 (10.00)
TSV
KnOTS
80.81
83.03
58.25
74.34
67.66
79.69
98.54
49.86
74.02
Core
82,98
85.12
50.95
84.25
71.14
84.39
99.06
53.53
76.43 (14.8)
Full
80.33
82,11
57.31
77.38
71.17
81.57
98.72
51.91
75.06
CART +
KnOTS
82.05
80.47
56.12
64.58
62.40
78.81
99.22
45.05
71.09 (:397)
Core
82.93
84.21
51.14
81.32
72.12
82.83
99.33
55,32
76.15
Full
80.16
83.03
51.44
74.76
70.72
79.89
98 66
50.20
73.60 (10.00)
Iso-C
KnOTS
79.29
57.50
(1.0)
Core
83.35
84,30
50.13
81.97
71.07
83:46
99.17
53.90
75.92 (12.32)
Table 4: Joint-task setting absolute accuracy of merged models on the vision tasks with ViT-B/32.
Space
TA
TIES
DARE-TIES
TSV
CART
TIES +Iso-C
DARE-TIES +Is--C
TSV +ls--C
CART
Iso-C
Full
43.5
43.6
44.0
45.4
44.8
43.5
44.3
48.3
44.8
52.1
KnOTS
43.5
46.8
45.2
44.6
44.7
40.5
44.8
51.4
52.6
52.9
Core
43.5
47.4
47.6
44.5
49.6
54.1
54.0
55.7
55.6
55.9
Additional PEFT methods Our method can also be applied to other PEFT methods, such as VeRA 21) VeRA, AW A,,A, where Ad E Rexi, Unlike LoRA, in VeRA the A and B matrices are randomly chosen, frozen, and shared across the network, while only the two scaling vectors A are learned for each layer. To adapt VeRA to our Core Space merging, we absorb the scaling vectors into the matrices, i.e., B = A, and Ā A, and then treat Á and as the LoRA A and B matrices. confirm that our method also works with VeRA, we report additional experiments in Sec. E.3 which show that our method outperforms other baselines also in this setting.
Rrxx, B E Rmxr, A, Rixm, and
Joint-task evaluation in vision setting, We also evaluate vision models in the challenging joint-task setting introduced in 42 in which the task ID unknown during inference. Instead of performing a multi-task evaluation, it evaluates the merged model on the union of all classes, requiring the model distinguish between classes from all tasks. We present the results in Tab. 4 Core Space facilitates merging with almost all methods, achieving state-of-the-art results when combined with Iso-C.
5.2 Analysis
Truncation. We herein compare the utilization of full space and Core Space for models merged with TA. Firstly, we calculate the SVD of the merged matrices: AW AWmerge
merged for full space and Meerged for Core Space. Then, we truncate fraction of p least significant values, i.e., a; = for i (1 p) dim(E), and observe a drop in the accuracy of the merged model after truncation. As shown in Fig. 4 full space we can truncate a fraction up to p = 0.8 values without performance loss, while in Core Space, truncation of any component results in performance drop. It shows that

---

0.5
we show Full Space
Core Space
0.4
Accuracy (%)
63
Average SAR
0.3
62-
0.2
61
TA Full space
TA Core space
0.0
0.0
0.2
0.4
0.6
0.8
1.0
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC
SUN397
SVHN
Fraction of truncated components
Figure Most components full space are irrelevant when doing Arithmmiiit (TA). Re- moving any components from the Core Space results in a performance drop, showing that it is an information-dense space. We report the results on vision tasks with ViT-B/32.
Figure 5: Subspace Alignment Ratio (SAR) 28 Each bar shows the average SAR between LoRA task matrices, Full, and Core Space. In Core Space, task matrices exhibit higher SAR. The as- sociated performance gains suggest that better alignment facilitates more effective merging.
the Core Space is dense while the full space contains many unused redundant components. We hypothesize that the compactness of Core Space facilitates model merging as it extracts only the relevant components.
Core Space improves subspace alignment. In this Section, we evaluate the Subspace Aligmment Ratio (SAR) (28) between each pair of LoRA updates fine-tuned on different tasks. The SAR measures how much of the subspace of one task is contained in another and correlates with post- merge performance. We compute SAR in full and Core Space. Fig. 5 shows that Core Space yields consistently higher alignment. We argue that this result is because the Core Space enforces a shared basis across tasks, which filters out task-specific noise and promotes alignment. In Sec. C.1 that higher SAR correlates with lower merging interference.
Choice of the reference basis. To evaluate the reference bases choice, assess the performance when adopting different reference bases, and compute the alignment error sv defined in Eq. (13) (averaged over all layers and tasks). We present the re- sults in Tab. 5 In row (1), we evaluate using the basis of the first task reference basis. In row (2), we set the reference basis to random orthonor- mal basis of the same dimensionality. These two bases perform much less than our reference basis in row (3). Moreover, we confirm that the optimal reference basis from row (3) achieves alignment emror. Additionally, verified experimentally that even in the extreme case where T >> min(m, n) (e.g., Tr 2048 > 768 for merging ViT-B/32 LoRA models), the reconstruction error defined in Eq. (13) remains exactly zero, consistent with the generalized theoretical result Sec. A.4
Table 5: Ablations on the choice of reference basis. Our basis (3) achieves higher results than the single-task basis (1) and the random orthonormal basis of the same dimensionality
(2). We proceed with Vief analogously Unef We report
the TIES-Core results on vision tasks with ViT-B/32.
Reference Basis Upe
Shape
Avg. Acc.
Avg. EU
U-() (first task) Random orthonormal
60.4
13.4
Tr
61.6
13.3
Concatenation (Eq.
Tr
68.6
0.0
Conclusion
We propose Core Space, efficient and effective method for merging LoRA modules. By projecting task-specific LoRA updates into a common subspace, Core Space reduces alignment error, leading consistent accuracy improvements and SOTA results in both vision and language settings, while remaining computationally efficient. Our evaluations across vision and language domains confirm its scalability and strong performance in practical settings. We believe that Core Space can contribute to more efficient and accessible model adaptation in multi-task settings, particularly for large models.
10

---

Acknowledgments
We acknowledge the Spanish project PID2022-143257NBB CCIN/AAE//1.11333//// 1033 and ERDI/EU, and Funded by the European Union ELLIOT project. This work was supported by the Fortissimo Plus (FFplus) project Grant Agreement No. 101 163317), under the European HighPerformance Computing Joint Undertaking (EuroHPC JU) and the Digital Europe Programme. The authors gratefully acknowledge access to compute resources enabled by FFplus. Funded by the European Union. Aniello Panariello acknowledges travel support from ELIAS (GA no 101120221 120237). Daniel Marczak is supported by National Centre of Science (NCN, Poland) Grant No. 2021/43/0/ST6/02488. Barthomiej Twardowski acknowledges the grant RYC2021-03277S-I and National Centre of Science (NCN, Poland) Grant No. 2023/5//D/SST6022466
financed by
References
[1] T. Akiba, M. Shing, Y. Tang, Q. Sun, and D. Ha. Evolutionary optimization of model merging recipes. Nature Machine Intelligence, 2025.
[2] E. Almazrouei, H. Alobeidli, A. Alshamsi, A. Cappelli, R. Cojocaru, M. Alhammadi, M. Daniele, D. Heslow, Launay, Q. Malartic, B. Noune, B. Pannier, and G. Penedo. The falcon series of language models: Towards open frontier models. arXiv preprint arXiv: 2311.16867, 2023.
[3] S. R. Bowman, G. Angeli, C. Potts, and C. D. Manning. A large annotated corpus for learning natural language inference. Empirical Methods Natural Language Processing. 2015.
[4] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, Agarwal, A. Herbert-Voss G. Krueger, Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei. Lan- guage models are few-shot learners. Advances in Neural Information Processing Systems, 2020.
[5] G. Cheng, J. Han, and X. Lu. Remote Sensing Image Scene Classification: Benchmark and State of the Art. Proceedings of the IEEE, 2017.
[6] J. Choi, D. Kim, C. Lee, and Hong. Revisiting weight averaging for model merging, arXiv preprint arXiv:2412.12153, 2024.
[7] M. Cimpoi, S. Maji, Kokkinos, S. Mohamed, and A. Vedaldi. Describing Textures in the Wild. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, 2014.
[8] M.-J. Davari and F. Belilovsky. Model breadcrumbs: Scaling multi-task model merging with sparse masks. Proceedings of the European Conference on Computer Vision, 2024.
[9] DeIpSeck-AL. Deepseck-v2 v3 technical report. arXiv preprint arXiv: 2412. 2024.
[10] Devlin, M.-W. Chang, K. Lee, and Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technolo- gies, 2019.
[11] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[12] A. A. Gargiulo. D. Crisostomi, M. S. Bucarelli, S. Scardapane, F. Silvestri, and E. Rodolà. Task singular vectors: Reducing task interference in model merging. Proceedings of the IEEE conference Computer Vision and Pattern Recognition, 2025.
[13] A. Grattafiori, A. Dubey, A. Jauhri, and A. et al. The laaa herd of models, arXiv preprint arXiv: 2407.21783, 2024.
11

---

[14] P. Helber, B. Bischke, A. Dengel, and D. Borth. Introducing eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE International Geoscience and Remote Sensing Symposium, 2018
[15] N. Houlsby, A. Giurgiu, Jastrzebski, B. Morrone, Q. De Laroussilhe, A. Gesmundo, M. At- tariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning. 2019.
[16] J. E. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. International Conference on Learning Representations, 2021.
[17] Hugging Face. httpss//huggingffcce.oo 2024.
[18] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In International Conference Learning Representations, 2023.
[19] X. Jin, X. Ren, D. Preotiuc-Piese Pietro, and Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations, 2023.
[20] Khot, A. Sabharwal, and Clark. SciTail: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, 2018.
[21] J. Kopiczko, T. Blankevoort, and Y. M. Asano. Vera: Vector-based random matrix adaptation. International Conference on Learning Representations, 2023.
[22] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-graines categorization. IEEE International Conference on Computer Vision and Pattern Recognition Workshops, 2013,
[23] Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online].. 2010.
[24] B. Lester, R. Al-Rfou, and N. Constant. The power of scale for parameter--fffiient prompt tuning. arXiv preprint arXiv:2104.0869) 2021.
[25] X. L. Li and P. Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1rXiv:2/01.00/99 2021.
[26] S. Liu, C. Wang, H. Yin, P. Molchanov, F. Wang, K. Cheng, and M. Chen. Dora: Weight- decomposed low-rank adaptation. In International Conference on Machine Learning, 2024.
[27] G. Mancusi, M. Bernardi, A. Panariello, A. Porrello, S. Calderara, and R. Cucchiara. Is multiple object tracking a matter of specialization? Advances in Neural Information Processing Systems, 2024.
[28] D. Marezak, S. Magistri, S. Cygert, Twardowski, A. Bagdanov, and J. van de Weijer. No task left behind: Isotropic model merging with common and task-specific subspaces. In International Conference on Machine Learning, 2025.
[29] D. Marezak, B. Twardowski, T. Trzci nski, and S. Cygert. Magmax: Leveraging model merging for seamless continual learning. Proceedings of the European Conference on Computer Vision, 2024.
[30] M. Marelli, S. Menini, M. Baroni, L.. Bentivogli, R. Bernardi, and R. Zamparelli. A SICK cure for the evaluation of compositional distributional semantic models. In Proceedings of the Ninth International Conference on Language Resources and Evaluation. 2014.
[31] I. E. Marouf, S. Roy, E. Tartaglione, and S. Lathuilière. Weighted ensemble models are strong continual learners. arXiv preprint arXiv: 23 23/2.08977,200 2023.
[32] M. Matena and C. Raffel. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems, 2021.
12

---

[33] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural im- ages with unsupervised feature learning, In Neural Information Processing Systems Workshops, 2011.
[34] G. Ortiz-Jiménez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Advances in Neural Information Processing Systems, 2023.
[35] A. Panariello, E. Frascaroli, P. Buzzega. L. Bonicelli, A. Porrello, and S. Calderara. Modular embedding recomposition for incremental learning. Brit. Mach. Vis. Conf., 2025.
[36] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, J. Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
[37] A. Rame, M. Kirchmeyer, T. Rahier, A. Rakotomamonjy, P. Gallinari, and M. Cord. Di- verse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems. 35, 2022.
[38] F. Rinaldi, G. Capitani, L. Bonicelli, D. Crisostomi, F. Bolelli, E. Ficarra, E. Rodolà, S. Calder- ara, and A. Porrello. Update your transformer the latest release: Re-basin of task vectors. International Conference on Machine Learning, 2025.
[39] A. Soutif, S. Magistri, J. v. d. Weijer, and A. D. Bagdanov. An empirical analysis of forgetting in pre- pre-trained models with incremental low-rank updates. In Proceedings The 3rd Conference on Lifelong Learning Agents, Proceedings of Machine Learning Research, 2025.
[40] C. C. Staff. Singular value decompositions, 2020. Course Notes, CS 357: Numerical Methods I, Fall 2020.
[41] Stallkamp, M. Schlipsing. J. Salmen, and lgel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332, 2012.
[42] G. Stoica, P. Ramesh, B. Ecsedi, L. Choshen, and J. Hoffman. Model merging with svd to tie the knots. International Conference on Learning Representations, 2025.
[43] D. Tam, M. Bansal, and C. Raffel. Merging by matching models in task subspaces. arXiv preprint arXiv: 2312.04339, 2023.
[44] A. Tang, L. Shen, Y. Luo, Y. Zhan, H. Hu, B. Du, Y. Chen, and D. Tao. Parameter efficient multi-task model fusion with partial linearization. In International Conference on Learning Representations, 2023.
[45] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. GLUE: A multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2018
[46] K. Wang, N. Dimitriadis, G. Ortiz- Jimééz,iménez, F. Fleuret, and P. Frossard. Localizing task information for improved model merging and compression. In International Conference Machine Learning, 2024.
[47] A. Williams, N. Nangia, and S. R. Bowman. A broad-coverage challenge corpus for sentence understanding through inference. Naacl, 2018.
[48] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
[49] J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 2016.
[50] Yadav, D. Tam, I. Choshen, C. Raffel, and M. Bansal. Terging Resolving interference when merging models. In Advances in Neural Information Processing Systems, 2023.
13

---

[51] A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang. H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang. Yang. J. Zhou, Lin, K. Dang, K. Lu, K. Bao, K. Yang. I.. Yu, M. Li, M. Xue, P. Zhang, Q 7hu, R. Men, R. Lin, T. Li, T. Xia, X. Ren, X. Ren, Y. Fan, Su, Y. Zhang. Y. Wan, Y. Liu, Z. Cui, Z. Zhang, and Qiu. Qwen2.5 technical report. arXiv preprint arXiv:2412. /51 15, 2024.
[52] E. Yang, L. Shen, G. Guo, X. Wang, X. Cao. J. Zhang, and D. Tao. Model merging in llms, milms,, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv: 2408.07666 2024.
[53] E. Yang. L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, and D. Tao. Representation surgery for multi-task model merging. International Conference on Machine Learning, 2024.
[54] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations, 2024.
[55] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: from homologous models as a free lunch. In International Conference on Machine Learning, 2024.
Absorbing abilities
[56] Q. Zhang, M. Chen, A. Bukharin, N. Karampatziakis, P. He, Y. Cheng, W. Chen, and T. Zhao Adalora: Adaptive budget allocation for parameter-efficien fine-tuning. International Conference Learning Representations, 2023.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We clearly state main contributions in both the abstract and the introduction, specifically proposing the Core Space merging framework for LoRA-adapted models. These claims supported by both theoretical analysis and extensive empirical validation in the main body of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We include discussion of limitations the conclusion, noting that while Core Space can handle varying LoRA ranks, we do not empirically explore this capability. We also highlight the restriction same-architecture merging and leave extensions to heterogeneous backbones and alternative PEFT methods (e.g.. VeRA) as future work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model vell-specificattin, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
15

---

3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: We present full derivations and assumptions for our theoretical results, includ- ing proofs of zero alignment error and properties of the Core Space in the main text and supplementary material.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We detail the datasets, models (ViT-B/32, ViT-L/14, Llama 3 8B), LoRA rank configuration, and baselines used. The experimental settings closely follow those established in KnOTS, and we clarify this alignment.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g., in the of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
16

---

(d)
reproducible, they We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We do not produce any new data. The checkpoints used are from Model merging with SVD to tie the Knots. The code used to generate results will be shared in the supplementary materials and released upon acceptance, along with detailed instructions for reproducing our experiments.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We specify all essential settings such as LoRA rank, number of tasks, model types, and the exact layers modified. Supplementary material provides additional configura- tion details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
17

---

Justification: Due to the computational cost of running multiple repetitions across dozens of tasks and large models (e.g.. Llama 3 8B), we do not include error bars statistical significance tests. Instead, we focus on reporting consistent trends across a wide range of datasets and provide detailed per-task results. Given the stable performance across runs and the deterministic nature of the merging process (given fixed seeds and checkpoints), we believe the reported results reliably support our main claims.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.: Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (c.g. negative eтror rates).
show in tables
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We describe our experimental setup in supplementary materials, specifying the exact GPU models used (NVIDIA L40S and RTX 4080), memory capacity, and the con- sistency of hardware across timing benchmarks. Merging times and relative computational costs are reported in Table 2, allowing for a fair comparison.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: Our research complies with the NeurIPS Code of Ethics. We use only publicly available datasets and models, and our contributions do not involve sensitive private data.
Guidelines:
18

---

The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We address broader impacts in the conclusion, noting that Core Space can contribute to more efficient and accessible model adaptation multi-task settings.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work does not involve the release of models or data that pose high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
19

---

Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We list all datasets used along with their licenses in the supplementary material. We also ensure any third-party checkpoints (e.g., from KnOTS) properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g.. website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the assct's ss creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new datasets models as part of this work.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Our paper does not involve crowdsoureing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our research does not involve human subjects or require IRB approval.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Large language models were not used in the methodology, experiments, or theoretical analysis. Any LLM usage was limited to minor editing assistance and did not affect the scientific content.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy battps://neur// CwwperipsssccCooffeeee ccc/Cofferencees/20000 /LMM)LL for what should or should be described.
21

---

Appendix
Contents of the Appendix
A No Information Loss in Core Space Representation
A.1
Least Squares Solutions
A.2
Alignment Error Quantification
A.3
Optimal Reference Bases
A.4
Generalization Beyond the . S m, Assumption
B
Computational Complexity Analysis
C
Additional Analysis
-C.1
High subspace alignment leads to lower interference
C.2
Rank of the merged update matrices
D
Additional Experiment Details
D.1
Experimental Environment
D.2
Hyperparameter Search
D.3
Code Availability and Pseudocode
E
Additional Results
E.1
Per-task evaluation in vision setting for ViT-L/14 Experiments with Heterogeneous Ranks
E.2
E.3
Extension to VeRA
A No Information Loss in Core Space Representation
A.1 Least Squares Solutions
femx,,_$ Let (:(() E g'xr, 20tt E Rrxr, vith) Raxr, U(() E 1Rxr >B E Rrxr and vi(0) E gr be the components of two low-rank matrices A(1) and B(O) represented in SVD form. Let Up! E RmxI- and Vi E TRaxT-r be the shared reference bases, obtained by taking the left and the right singular vectors from the SVD of the horizontally and vertically stacked low-rank matrices B(t) and A(), respectively (see Eq. in the main paper). We assume T m and T < n, where T is the number of tasks and r LoRA rank, as both are typically small relative to the feature dimension. Then, the solutions to the least square problems:
R
arg min Unet R U( 7
Q arg min QERT-xP
Vio 1/0
(16)
are given by:
R = Une - Ul!)
Q4 = V ((
(17)
Proof. Since v(() and ((() come from the SVDs of A and (tt), they orthonormal:
((t) =
U B Ir.
where J, is the r x r identity matrix. Similarly, the reference bases Upel and Viel also have orthonormal columns:
Vree Vref IT,,, Unef Unet IT
where Jr. is the T - T and T x r identity matrix.
Consider the problems in Eq. (6)). Each objective is convex and admits unique global minimizer, as Ureef and V have full column rank due to their orthonormality. To solve the first problem, compute the gradient of the objective function with respect to Rt
Une R -R 11
2
(UTel R U(!)).
ØR
22

---

Setting the gradient to zero and solving the resulting equation gives:
U
Usel
UneeRR = Une
R = U
U/
since Ug UB ITr. Similarly, for the second problem in Eq. T6
041 Var V(1)
A.2 Alignment Error Quantification
Lemma. Let U Rrxr and U RRxX.r be matrices with orthonormal columns where T rank([B(),.... ..... S and let R = UUg U(t) E TT be the optimal solution minimizing the error of the least-square problem. Then, the optimal alignment error given by:
EU = U R(G - 1 -
| UB
(18)
Proof. derive the alignment error, we simplify the notation by temporarily omitting the dependency on the model index t. Substituting the definition Rg Une UB from Eq. (77) write:
EU U UB U
U [1(UB) Up Um)B
tr ([UTY' Un! Lm)UB [UUT - Lm)UB]
ir (UT(UE'UB" Im) Un
tr (UZ(UE UB UNI -UUFF uoo Lm)BB)
tr UUT 2 + {mm)Un)
(by using Und Unet I)
tr (U U Un)
tr (UB - tr (UTUR UB)
(by linearity of trace)
| |
where tr( |I r, since UB has orthonormal columns, and tr(U B Utee Une UB) | Uref | by the cyclic property the trace and definition of the Frobenius norm.
A.3 Optimal Reference Bases
Lemma. A solution U to the quadratic program:
(()
U
maxti tr UES
(UTUHUHO U)) S S= SUR |
- IT,),
is given by any orthonormal basis whose columns include the top r eigenvectors corresponding to nonzero eigenvalues of B(0) B(O mmmx or, equivalently, by the top left singular vectors of B(), At the optimum, the objective achieves the maximum value of r, yielding zero alignment error Ey as defined in Eq. (8)
Proof. The proof proceeds in two steps. First, we show that any orthonormal basis U. containing the top r eigenvectors of (+)) (y(A) solves the optimization problem. Second, we establish that the cigenvectors of (
( (t)T are the same those of ((t) B())T_
23

---

Step 1: Solving the constrained optimization. By leveraging the method of Lagrange multipliers, we write the augmented objective function:
((U, tr(U U T (A IT+)))
(19)
where A E RT is a matrix of Lagrange multipliers. Taking gradients with respect to U and A, we obtain:
VoC(U,A) au YOUTUGUU
tr((U U U U) au tr(A(U -Ir-))
(20)
UU AUA
a tT - = U IT.
VAC(U,A) DA
(21)
Setting the gradients to zero gives the two necessary optimality conditions:
U U U. U.A,,
(22)
U_ U Ir-r-
(23)
The second condition, Eq. (33 holds by construction: we explicitly choose U, E RmxT-r to be an orthonormal matrix. Specifically, we define U. = [ui...., Uy Pi........... Prr 11r-- where {u))=1 are orthonormal eigenvectors U associated with its nonzero eigenvalues ......, X,, and (p;)){-1") (r-1))r are additional orthonormal vectors chosen to complete the basis.
Next, to verify the first condition reported in Eq. 22 we define the diagonal matrix A, = diag(Ai,..... Ar, O,.... RTTxxTr where the trailing zeros correspond the eigenvalues associated with the orthogonal complement. Since each U, an eigenvector of U with eigenvalue >; and each pj lies in the nullspace of that matrix, have:
U (0) 10) 0s Apv, for i I,..., and Ul
10
Pj =0 for j I...., (T (T-1) 1).
Therefore:
T U. -
A, O.
0] = U.A.:
which confirms that the first-order condition in Eq. (22 is satisfied.
Next, substitute this result into the original expression obtain the optimal value:
L- tr T U
TT U.) = ANUTU.AA) = tr(A.) = ~ ( =r,,
i=1
proving that the maximum value of the quadratic problem is r, and the corresponding alignment error EU for U, is zero.
Step 2: Equivalence of eigenspaces. Now we show that, given the matrix (()) Raxr from a LoRA-based adaptation, if v is an eigenvector of B(OBa then v is also, by construction, an eigenvector of U (t) T(() where B(t) (t) ( D( v(O (()
e is an eigenvector of B(t) B(t)
B(t) B(t) ==
B Xu
U (20))(1)))) U (() v= xv
(B()) B(O ( 1 (() == (((0 )Au
220 yle (0) == (1(1) ((1)T
From the second and fourth rows, obtain:
(B T Xv
(24)
24

---

If À >> O, we obtain that is an eigenvector of B(t) B and its corresponding eigenvalue is = 1. If
0, then we have:
is an eigenvector of ((0) B(U) =0 = B w=
B 1 -=0 TU
x U v=
Since the matrix B has rank it has exactly strictly positive singular values, while all other singular values beyond rank r are zero. Thus, the matrix 22(()) is positive definite. Hence, it must be:
v J D 2 TTT
(() -= U
U
(O) 0,
(25)
which confirms that if
0, then e is an eigenvector of U with eigenvalue O.
L
Generalization to Multiple Tasks. This result generalizes directly to the multi-task setting, as defined in the main paper, by applying it to the matrix B E R=xI'r obtained by horizontally stacking the LoRA updates B from all tasks T. Then the optimal reference basis U, U (as done in our approach) is given by the left singular vectors of B, obtained via SVD. This matrix satisfies both orthonormality and the optimality conditions derived above. Defining A, diag(A),- >T where X the eigenvalues of BB guarantees perfect reconstruction and zero alignment error for
all tasks.
A.4 Generalization Beyond the T r S m, Assumption
The derivations in Secs. A. I to A.3 assumed that the total LoRA rank T -r is less than both m and We now show that this assumption is not necessary, and that the reconstruction error remains zero even when T r > m or T r >> n.
Intrinsic rank. When T > (for B) or T (for A), stacking the LoRA matrices still yields reference bases with intrinsic dimensions:
du = rankk[ (B,
B(T)) S m, dy = rankk[A
(T) n.
since the number of linearly independent directions cannot exceed the number of rows or columns We therefore replace the reference bases Uno mx and V.ol RaxT< with truncated orthonormal bases:
U
1 mexd
Vidl guxdy
whose columns span the full LoRA update space.
Least-Squares Solution. Rewriting the least-squares problem (Eq. (16 in terms of the truncated Unef gives:
arg min |UE R 12 Re
Following the same derivation in Sec. A.
the optimal solution is
RlO (U
((U))
with an analogous expression for Q(4)
Aligument Error Substituting this solution into the error expression of Sec. A.2 shows that the optimal alignment error remains
EU r-=--l ( Unef 1|
which achieves zero when Use spans the column space of all B(O) Hence, the theoretical guarantees extend unchanged to the case T
m, n.
25

---

B Computational Complexity Analysis
Iso-C Complexity. To assess the time complexity of our approach, we begin by analyzing that of Iso-C (28), to establish a baseline for comparison. For each layer, the Iso-C procedure can be broken down into the following steps:
LoRA + full space: Compute (4) A() for t = I,..., T, given the matrices B(t) and A The resulting complexity is O(T n -)).
Summation: Applying task arithmetic in the full space, ATA Σx (4), involves cost of O n m.).
SVD computation: Computing the decomposition ΑTA UEV for an m x n matrix has complexity of (m? + n²)) 40]
Isotropization: The final step, A Iso-C = U'avg V is dominated by O(m² n).
Overall, the total cost Iso-C is dominated by O(T - m + m² nn -n+ x2)) Assuming that m = n, then the time complexity of Is-CCin is approximately cubic: O(n T r ..n))) with respect to the number features.
KnOTS Complexity. Secondly, we analyze the computational cost of the KnOTS method 42
LoRA - full spuce: As in Iso-C, this step has complexity O r m).
Concatenation:
Stacking all weight matrices AW [AW()) AW as block columns of a global matrix has time complexity of O -n- m).
SVD computation: This is performed of Onn² 1inn"7" 7
matrix of size (n-T), resulting in complexity m + m²), by making use of the transpose trick
Merge: Assuming simple task arithmetic is performed in the VT space, T blocks of columns each are summed, yielding a complexity of O -r.n) n) to compute Vinerge
Reconstruction: The final step AKNOTS UEVT merge has complexity of O(m T T nn).
Overall, the total cost of KNOTS is dominated by O(m T (2r m + Assuming m = n. the time complexity simplifies to 37)) Compared to Iso-C, the time complexity of KNOTS remains cubic with respect to the number of features. However, includes an additional 2 factor that scales quadratically with the number of tasks being merged.
T r)).
TSV Complexity. Then we analyze the cost of TSV 12
LoRA full space: As previously, this step has complexity O(T r rr -m).
SVD Computation: this step SVD is performed on matrices size m x n resulting in n))).
complexity of O(T (m²
Concatenation: Stacking the first k components of left and right singular vectors for all tasks results in O(T k(n + m)).
Global SVD Computation: The SVD performed on the stacked matrices n x m requires O(2 (m² n + n²)).
Obtaining orthogonal matrices: This step requires O m²- nn)
Merge: The final merge has a complexity of O(m . T...T.r..
The overall cost of TSV is thus -r mm Assuming that and T,r,k 1e < the computational cost is dominated by O(T nn).
n 1--n++T-- 1T-k-m+7 1s k-- +
Core Space Complexity. Finally, we analyze the cost of our approach in Core Space.
"SVD(PT)= U'''v' + P (''''''' v''e 'U"" thus, if & n. this will reduce the number of operations. We will apply the transpose trick throughout.
26

---

Stacking A and B(;; Stacking two sequences of T' matrices one with each matrix shape n and the other with each matrix of shape m x r results in cost of O(T m)).
SVD computation: The stacked global A matrix has shape (T . r) x nt, hence, the cost of its SVD is O (T r) + (T r))). by using the transpose trick. The stacked B matrix has shape m x (T. r). with a cost of (( (T. (T r))). The overall cost of this step is O(mm² + n²) (T r) (T (T r))).
Low-rank loop: In the optimized version of the low-rank loop, we only compute the matrix multiplication obtain the aligned matrices. In this case the total cost is O (Tmr² Tr²n+ )) (using the optimal matrix multiplication order (AVV )). Assuming that T,r << n, m, the cost is dominated by ( + n)).
Merge: Assuming simple task arithmetic is performed in the aligned Core Space, the cost is O(T5,,2
Isotropization: Optionally, Iso-C can be applied in the Core Space: since the Core Space is defined within a square matrix of dimension T T this step adds an additional time complexity of (T3))
Reconstruction: The final step requires (m T x² + --
To sum up, our approach involves:
O (Tr(n + m) + Tr(m² + n²) + 2(Tr)²
Stacking
SVD refs.
2 (m + n)) + Tr + Trm + mn))
Low-rank
Merge & Rec.
O(Tr(m + n 2m 2n + mn + +²) + T + + + T2)))
If assume that n, the total time complexity simplifies (Tr((nn + 5n )²) T2 +T 13)),,
(26)
which, if we assume T,r < n, is dominated by:
O
(27)
Additional Analysis
C.1 High subspace alignment leads to lower interference
In this Section, we experimentally show that merging in Core Space reduces interference when merging models. We follow [53 28] and measure the interference as the LI distance between the final embeddings of task-specific models and the merged with TSV Iso-C in Full Space versus Core Space. For each dataset, we collect the activations from the final layer (i.e., the projection a common vision-language space) of both the task-specific model and the merged model. We present the average distance across all the samples in the test set. We observe lower interference when merging Core Space, highlighting its effectiveness. Note that Full Space merging, which causes higher interference, also exhibits higher SAR in Sec. 5.2
We compare the interference when merging
C.2 Rank of the merged update matrices
Consider T' = 8 -B32 models fine-tuned with LoRA of rank 16. The merged update matrices AW resulting from different merging methods and spaces can have different effective ranks raw. repprts the average rank of AW across all layers.
In most cases, the target rank of AW is equal Tr 128. The only exception is merging with TIES in Full space, where for weight matrices W E IR m the effective rank approaches the dimensionality of the matrices d = min(m, n1) = 768. This phenomenon arises because TIES performs trimming on the reconstructed weight matrices AW, BA, which destroys the low-rank structure. In contrast, both Core and KnOTS operate directly in constrained Tr-dimensional space, ensuring that the merged AW maintains the intended rank.
27

---

Final Activation Distances: Models merged with TSV
Iso in Full vs Core Space
0.356
0.30
Core
0.278
0.25
0.234
0.23
0.224
Mean L1 Distance
0.20
0.184
0.175
0.154
0.15
0.133
0.10
0.05
0.00
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC45
SUN397
SVHN
Figure 6: Mean LI distance between the final embeddings of task-specific models and the merged one using TSV Iso in Full and Core Space. We used ViT-B/16 model.
Table 6: Average rank raw of merged update matrices AW obtained by merging 8 ViT-B/32 LoRA models with 16 (so Tr 128).
Merging Space
TA
TSV
TIES
Full
128.00
128.00
766.25
KnOTS
128.00
128.00
128.00
Core 128.00
128.00
128.00
D Additional Experiment Details
Licenses of Used Datasets and Models
In our research, we employed publicly available datasets and models, each governed by specific licenses. Below, we outline sources and associated licenses for each:
KnOTS LoRA Checkpoints 42] The KnOTS repository, which provides LoRA-adapted
model checkpoints and training scripts, is licensed under the MIT License. This permissive license allows for reuse and modification with proper attribution.
The Cars 196 dataset is available for non-commercial research purposes. Specific licensing details are not explicitly provided.
Cars196 222
Describable Textures Dataset (DTD) 77]: The DTD is made available to the computer vi- sion community for research purposes. The dataset is licensed under the Creative Commons Attribution 4.0 License (CC BY 4.0).
EuroSAT 14
The FuroSAT dataset licensed under the MIT License.
German Traffic Sign Recognition Benchmark (GTSRB) 41: The GTSRB dataset is licensed under the Creative Commons Zero (CCO) Public Domain Dedication.
The MNIST dataset is publicly available for research purposes. Specific licensing details are not explicitly provided; users advised to consult the dataset's source for more information.
MNIST [23]:
NWPU-RESISC45 (5): The NWPU-RESISCA5 dataset licensed under the Creative Commons Attribution 4.0 License (CC BY 4.0).
28

---

The SUN397 dataset is available for research purposes only. Specific licensing details are not explicitly provided; users are advised to consult the dataset's source for more information.
SUN397 49
Street View House Numbers (SVHN) commercial use only.
33 The SVHN dataset is available for non-
Stanford Natural Language Inference (SNLI) 3] The SNLI dataset licensed under the
Creative Commons Attribution-haare.l 4.0 International License (CC BY-SA 4.0).
Multi-Gent Natural Language Inference (MNLI) 47]: The MNLI dataset is released under the Open American National Corpus (OANC) license, which permits free use, modifi- cation, and sharing under permissive terms.
Sentences Involving Compositional Knowledge (SICK) 30 : The SICK dataset is dis-
tributed under Creative Commons Tuanbuton-NooCCCmmrriiiaSSSSiii
license.
Question Natural Language Inference (QNLI) [45] The QNLI dataset is part of the GLUE benchmark. Specific licensing details are not explicitly provided; users are advised to consult the dataset's source for more information.
The RTE dataset is part of the GLUE benchmark, Specific licensing details are not explicitly provided; users are advised to consult the dataset's s source for more information.
Recognizing Textual Entailment (RTE) 45]:
SciTail 20 The SciTail dataset is licensed under the Apache License 2.0.
D.1 Experimental Environment
The language experiments with Llama 8B were performed with a single 48G NVIDIA L40S. In contrast, the more affordable vision experiments were executed using a single 16G NVIDIA RTX 4080. To keep things fair, the reported times for the language experiments all refer to experiments performed on the same machine.
Our implementation builds directly on the KnOTS codebase [42 and uses the exact LoRA checkpoints they released. For full details on the original training and adaptation procedures, please refer to 42
D.2 Hyperparameter Search
To find optimal hyperparameters for each model, we adopt the widely used validation holdout strategy [42] 28 [0). Specifically, we perform a linear search for hyperparameters on the validation set, starting from defined minimum value and incrementally increasing it until performance declines, indicating the optimal range. The identified optimal hyperparameters are then applied to the test set. We the following search settings:
Scaling factor starts 0.1, increasing in increments of 0.1. This is used for every approach.
The top-K parameter for TIES and DARE-TIES begins at 10 and increases in increments of 10.
The pruning factor for DARE-TIEE starts at 0.1 and increases in increments of 0.1.
For CART, the pruning rank is searched over the {0.04, 0.08, 0.16, 0.32), following the methodology of the original paper. Additionally. CART includes an extra scaling factor À in its merging formulation. Specifically,, the merged weights are computed as Wo + a(Ovg A ET Fr), where Oavg denotes the average the updates and F. represents the centered task vector for task f. For further details, we refer the reader to [6]
Wmerged =
In Tab. 7 we report the parameters used for the various merging methods in Core Space across all backbones. Note that we search for the optimal parameters for all methods across all spaces to maintain fairness. For the natural-languaa inference experiments, we omit CART as its hyperparam- eter tuning proved prohibitively expensive and exclude +Iso-C variants, as they consistently degraded performance.
29

---

Table 7: Optimal hyperparameters for Core-Space merging on each backbone and merging strategy.
including +Is--C variants.
Backbone
Merging Method
a
Top-K
Pruning p
CART rank
X
TIES-Core
0.6
TIES-Core+Isod
2.0
30
DARE-TIE TIES-Core
0.6
0.1
DARE-TIES SEB-TIE--Cor-eeee
2.0
30
0.1
ViT-B/32
TSV-Core
0.2
TSV-Core+Lso-C
0,9
CART-Core
0.4
0.32
5.8
CART-Core
0.7
0.04
2.6
Iso-C-Core
0.9
TIES-Cor
0.4
10
TIES Core+Iso-C
2.4
20
DARE-TIES-COE
0.4
10
0.1
DARE-TIES-Core-Iaee
2.4
0.2
VIT-L/14
TSV-Core
0.2
TSV-Core+Iso-C
0,9
CART-Core
0.1
0.04
6.5
CART-Core-Is--
1,0
0.08
2.0
Iso-C-Core
0.9
TIES-Core
1.1
80
DARE-TIES-CN
1.1
80
0.1
Llama 8B
TSV-Core
0.5
Iso-C-Core
2.8
D.3
Code Availability and Pseudocode
Our Core Space merging implementation is released at htttps://githuu.ccom////
core- eete-space-merppe
Listing gives Py Torch--style pseudocode illustrating how to apply any merging strategy within the Core Space framework.
E Additional Results
E.1
Per-task evaluation in vision setting for ViT-L/14
We provide in Tab. 8 per-task results vision model merging using the ViT-L/14 backbone. Similarly to what we observed for other backbones, in this case, performing the merging in Core Space yields consistent improvements across all methods, resulting in state-of-the-art results.
E.2 Experiments with Heterogeneous Ranks
the main paper, we discussed that Core Space merging naturally extends to the heterogeneous rank setting (Tab. 99. Here provide additional details.
When tasks are fine-tuned with different LoRA ranks, we horizontally and vertically stack the (()) and (t) matrices across tasks as usual. The resulting aggregate matrices have rank equal to the dimension of the union of all task subspaces. Performing SVD on these aggregates yields orthonormal reference bases Uref and V that span the combined subspaces, regardless of how individual task ranks vary.
Projection into these reference bases followed by task-specific alignment (see Sec. A. guarantees that reconstruction is lossless. This explains why the results in Tab. show that heterogeneous ranks incur no additional degradation performance under our framework, whereas baselines that lack such alignment struggle with mismatched subspace dimensions.
30

---

Listing Basic PyTorch pseudocode for model merging in Core Space.
from torch. .inaalg import svd
2
3 A A_list, B_list
r, A_list[0]] shape
m, B_list[0] shape
A_stack torch. carch.cat(4 (A cat(A_list, _isit, dim=0) (T*r, n) B_ stack = torch. cat list, dim=1) # (m, T+r)
Calculate reference bases lg_A_eff svack_stack, full mal_ [2] ref_ svd (B stack, full_matrices=Fal [o]
(T*r, n) # (m, T+r)
M_list = 0
16
for A, B in (A_list, B_list):
M_aligned (U___ref.T a (A Vh_A_ref.T)
M_list. _.list.append (M ( alignedd
if pargesstrategy = 'TA':
21
M_merged torch. stack (M_ (M_list) sum (dim=0)
elif merge.
merge_stratess strategy 'ties':
m_merged = ties_meit deesmmeggngg (a_lgggM_llls list)
elif berge_strated == '...':
M_merged
26
Reconstruct delta
delta_W W U_B_ret ref @ M merree @ th_A_res
E.3 Extension to VeRA
We also evaluated the applicabiliiy of Core Space merging beyond LoRA, specifically on VeRA (1 (Tab. (0). In VeRA, the decomposition AW A, BA,A differs structurally from LoRA since A and B are fixed random matrices, and only the scaling vectors As, Aa are trainable.
To apply our method, we absorb the scaling vectors into the low-rank matrices:
ArB,
A=A4A.
and then treat (Á, B) as if they were standard LoRA components. Since the subsequent steps (stacking, SVD, projection, and alignment) are agnostic how and were obtained, the derivations in Sec. appply without modification.
The empirical results in Tab. conffim this reasoning: Core Space merging consistently outperforms other approaches even in the VeRA setting, validating that the framework is general to low-rank adaptation methods beyond LoRA.
31

---

Table Accuracies of merged models normalized against fine-tuned models on the vision datasets with ViT-L/14.
Method
Space
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC
SUN397
SVHN
Arg Acc)
Abs. Accursey
99.70
70.00
98.50
97.20
99.50
95.70
79.60
97.7
TA
Full
80.01
79.50
65.59
59.98
82.20
79.55
86.71
64,74
74.79 (+000))
Full
83.82
87.45
75.51 (+0000
TIES
KnOTS
82.47
80.26
64.65
68.85
88.48
82.37
88. 18
76.63
78.99 (+348)
Core
81.94
80:03
65.78
87.44
81.85
88.48
73.75
78.5 (1302)
Full
79.70
78.82
64,99
60.63
83.92
79.32
87.07
84
75.53 (1000)
DARE-TIES
KaOTS
80.41
79.65
64.84
62.95
85.33
79.73
87.55
71.00
76.43 (+0.0))
Core
82.97
87.75
82.48
88.80
75.88
78.99 (1346)
Full
82.38
80.11
66.12
68.18
85.46
83.02
87.89
70.76
77.99 (1000)
KaOTS
80.17
79.95
66.94
60.24
83.76
79.35
86.85
65.59
75.36 (22.8)
Core
82.10
79.80
67.05
66.92
87.50
82.32
87.71
71.89
78.16
Full
91.88
88.53
75.51
80.88
68,99
92.77
88.17
64.81
81 .44 (+000)
KnOTS
86.52
63.24
(-7.00
Core
86.02
86.33
72,39
82.82
91.03
85.93
88.46
72.67
83.21 (+1.77)
Full
84.11
83.07
74.94
76.66
92.
87.88
88.58
74.61
82. (+0:00)
TIES +lu-C
KnOTS
86.44
88.23
78.74
78.94
94.27
87.94
88.62
72.24
84.43
Core
91.13
90.58
79.53
87.67
86.47
90.46
90.05
72.28
86.02 (+5,80
Full
77.25
86.81
88.24
74.44
(+0000)
DARE-TIES +loot
KnOTS
87.32
87.78
74.12
81.50
94,21
89.48
88.88
70.87
84.27 (+2.52)
Core
90.84
91.12
79.15
88.14
90.19
90.73
89.92
72.05
86.52 (15.7)
Full
86.44
89.07
82.49
68
90.76
90.03
87.99
67.99 98
84.93 (1000)
TSV +lo-C
KnOTS
88.47
90.58
77.69
83.91
87.48
90.00
88.76
67. .44
84.30 (4.65))
Core
91.54
91.34
80.24
86.79
87.39
91.51
89.59
71.30
86.21
Full
88.78
90.43
78.59
87.04
91.96
90.96
89.32
75. 18
86.53 (+0000)
CART Boc
KaOTS
88.72
89.60
80.77
79.84
87.18
89.50
88.38
68.07
84.01 (:2.2)
Core
92.08
92.48
81,22
88.96
89.80
91.97
89.81
73.56
87.49
Full
86.83
86.94
80.65
77.99
92.09
87.88
88.50
68.
83. (1000)
Iso-C
KnOTS
90.93
70.97
85 1.3)
Core
91.23
90.28
80.28
85.29
89.71
90.96
89.58
70.66
86.00 (12.30)
Table 9: Normalized accuracies of merged models on the vision tasks with ViT-B/32 with LoRA mixed ranks 16 (Cars, EuroSAT, MNIST, SVHN) and 64 (DTD, GTSRB, RESISC, SUN397).
Method
Space
EuroSAT
GTSRB
RESISC
SUN397
SVHN
Avg Acc)
Abs. Accunccy
74.00
68.03
99.00
98.00
99.30
93.85
70.85
96.20
TA
81.97
65.83
47.06
51.66
57.18
72.91
89.75
48.37
64.34 (e000)
Full
80.94
64.74
42.01
53.91
55.41
72. 12.16
89.76
09
63.50
(+000)
TIES
KaOTS
82.70
67.08
49.01
55.00
57.39
73.74
90.14
45.97
65.13
Core
79.74
67.71
41.38
71.86
69.59
75.02
91.43
67.97
70.59 (+7.09
Full
56.
72.45
89.74
(+000)
DARE-TIBS
KnOTS
82.98
67.08
45.57
56.34
62.34
75.65
90.06
53.53
66 66.69 (+2.55)
Core
80.98
68.80
44.26
68.07
64.43
74.45
91.44
62.99
69.43 (1542)
Full
80.98
67.94
50.95
59.10
64.65
75.82
90.76
53.37
67.95 (18,00
TSV
KnOTS
82.51
65.91
43.66
48. 98
60.52
71.47
89.23
51 51.29
64.20 (3.75)
Core
80.46
69.51
51.44
58.33
61.03
76.70
89.37
52.43
67.41
Full
78.94
69.90
54.02
53.13
66.83
75. 11
90.64
46.63
66. (+0.0)
TIES do.c
KaOTS
81.42
68.49
60.19
45.09
56.70
70.83
90.43
40.11
64.16 (2.74)
Core
79.32
75.84
58.66
64.09
80.14
76.76
90.52
55.13
72.56
Full
80.33
70.76
60,98
$1.48
61.58
74.16
90.41
47.26
67. 12 (+000)
DARE <la-C
KoOTS
$7.92
69.69
90.37
38.79
(35.5)
Core
78.75
75.37
58.47
64.98
80.83
77.00
90.49
55.53
72.68 (+5.56)
Full
76.46
73.65
61.02
56.41
69.02
74.04
90.01
49.12
68. 68.72 (10.0)
TSV Jla-C
KoOTS
78.88
72.79
66.59
$4.61
78.83
72.21
89.03
50.28
70.40
Core
79.30
78.26
63.94
58.11
77.28
74.40
89.85
50.97
71.51 (+276)
Full
63.05
77.14
77.10
90.22
$1.50
72.06 (+000)
Iso-C
KnOTS
78.92
76.31
56. 19
63.50
75. 11
77.08
90.53
$2.44
71.26 (0.5m
Core
81.17
79.20
59.71
70.86
81.90
78.93
90.81
56.60
74.90 (2.54)
32

---

Table 10: Normalized accuracies of merged models on the vision tasks with ViT-B/32 with VeRA
rank 16.
Method
Space
Cars
DTD
EuroSAT
GTSRB
MNIST
RESISC
SUN397
SVHN
Asg Acc)
Abs. Accuracy
62.79
57.07
96.55
90.85
98.60
88.50
62.79
93.10
IA
77.73
47.26
39.77
49.10
70.61
100.74
37.29
64.78 (1000)
Full
95.53
77.73
44.76
39.07
48.25
69.92
100.58
35.62
63.9 (1000)
TIES
KaOTS
96.64
77.91
50.13
39.61
50.46
70.52
100.85
35.93
(41.33)
Core
97.06
77.73
44.30
41.98
50.35
71.63
100.44
38.95
65.31
Full
94.44
77.73
47.26
42.09
49.82
69.37
100.05
38.28
64.88 (e000)
DARE
KaOTS
(+825)
Core
97.09
77.82
44.27
42.02
50.41
71.75
100.48
38.97
65 35 (4047)
Full
93.35
77.54
45.11
37.4 .4
$3.50
69.40 .04
99.70
33.85
63.74 (e000)
TSV
KaOTS
92.73
78.19
54.85
38.77
56.20
69.10
98.82
34.02
65 65.33
Core
93.75
76.51
60.15
37.94
54.32
38
98.77
34.63
65.56 (+1,3)
Full
34.03
63.35 (e0000)
TIBS +loiC
KnOTS
94.81
77.26
47.33
35.49
48.67
68. 10
100.67
33.74
63.26 (0000)
Core
77.63
50.79
37.14
50.61
69.06
100.62
33.77
64.38 (11.8))
Full
93.82
76.98
46.34
36.80 86.80
49.47
68.26
100.44
34.69
63
DARE-TIES +lose
KnOTS
94.88
77.17
47.37
35.52
48.64
100.66
33.75
63.27 (4:00
Core
77.26
53.01
37.45
52.90
68.13
100.09
31.80
64.35
Full
93.37
76.70
48.29
35.92
51.47
68. 11
100.30
34.35
63 56 (+0.0)
d
KaOTS
92.58
75.30
52.51
36.47
58.16
67.36
99.83
35.43
64.71 (11.5)
Core
93.57
76.89
63,33
39.27
57.53
68.63
99.18
34,07
66.56
Full
93.50
77.26
$2.05
37.42
49.49
68.06
100.62
34.22
64.08 (10.0)
Iso-C
KnOTS
33.68
(0078)
Core
91.77
75.77
63.60
38.53
58.53
67.81
97.67
36.30
66.27 (.2.8)
33


Reviews:
[{"review_id": "AiomdPf1zW", "reviewer": "Reviewer_XA7i", "summary": "This paper addresses the problem of efficiently merging multiple models that have been adapted using Low-Rank Adaptation (LoRA). The authors observe that existing merging techniques often require reconstructing the full weight matrices from the low-rank components, before merging, which results in a computational overhead. To solve this, they authors propose to identify a shared, low-dimensional basis for all LoRA updates by performing an SVD on the concatenated LoRA matrices. Then, any merging algorithm can be applied to this compact space. The authors provide a theoretical framework to show that the projection is lossless alongside a complexity analysis showing significant efficiency gains. The authors show extensive experiments on both vision (ViT) and language (Llama 3 8B) models and show that the proposed method not only has significant computational advantage but also results in improved performance", "questions": "1. Why are the numbers so low in Table 3? For TA: MNIST 53.12, SVHN 41.25 for an avg of 63.78 normalized. In comparison, TSV paper reports 76.55 for TA and 92.31 for TSV (for full-fine-tuning), while the paper reports 66.66 (for LoRA transformed to full). WHy such a huge discrepancy? Similar comments apply to the ViT-L/14 table in the appendix. \n2. What are the single-task accuracies here? Most papers report both absolute and normalized, or just absolute, but the authors here choose only normalized.  \n3. “in Sec. 5.1 we show that merging in Core Space yields better performance than merging in alternative spaces, when the merging function is non-linear.” Can you explain your intuition as to why this happens?  \n4. “Our approach is found to yield compact representations of each task and improve alignment between subspaces of each task.” Isnt this at odds with weight interference from ties \\[32\\] and task interference from \\[a\\]?   \n5. Figure 1: isn;t the bulk of the computation for e.g. ties the validation to compute the optimal scaling? This applies to knots as well as the proposed method.\n\n\\[a\\] Wang, K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., & Frossard, P. (2024). Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813.", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written, with a clear motivation.   \n2. The proposed method is theoretically grounded and straightforward.   \n3. The primary contribution is that the proposed method is not only dramatically more efficient compared to KnOTS but also improves the final performance of merging algorithms.   \n4. The experimental validation includes many model scales, domains and model merging techniques as baselines. The authors show consistent improvements across all combinations. \n\nWeaknesses:\n\n1. The paper focuses on the efficiency of merging which I am not sure is really an issue. Even in weight space, the transformation from low rank matrices to full will include a few matrix multiplications which is minor compared to fine-tuning. In the meantime, there exist new sota methods (in full space) that work much better than TA (perhaps with stronger assumptions). The paper focuses on optimizing merging efficiency rather than full final performance.   \n2. The framework's ability to merge models assumes that the different task-specific LoRA updates can be aligned within a common reference basis. This assumption may not hold for highly dissimilar tasks, which could limit the method's applicability. This can also be problematic beyond classification benchmarks where the fine-tuning dynamics become more complex.  \n3. Minor: Figure 2 needs more context to become standalone (as much as possible of course)", "was_revised": true, "final_justification": "Given the authors' rebuttal and the comments from other reviewers, I have updated my score.", "rebuttal": "We are pleased that the Reviewer appreciates the clarity of the paper, its theoretical soundness, the effectiveness and efficacy of the proposed method, and the thorough empirical evaluation. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.\n\n### W1 On the focus on efficiency\n\n> The paper focuses on the efficiency of merging which I am not sure is really an issue. Even in weight space, the transformation from low rank matrices to full will include a few matrix multiplications which is minor compared to fine-tuning. In the meantime, there exist new sota methods (in full space) that work much better than TA (perhaps with stronger assumptions). The paper focuses on optimizing merging efficiency rather than full final performance. \n\nWe acknowledge that training LoRA adapters for single-task weights is more expensive than most merging strategies, and improving final performance remains essential. Our method contributes to this by showing that **merging in the Core Space significantly boosts the performance of several state-of-the-art approaches**.  \n\nThat said, we believe that *efficiency is also an important and orthogonal goal*. While merging is cheaper than fine-tuning, it can still be costly for large models, such as Llama or ViT, especially when done at inference time. For instance, very recent methods, such as MASS [A], EMR-Merging [B], and Twin Merging [C], perform multiple merges during inference, where lightweight operations directly impact speed and scalability. Even when merging is done offline, reducing its cost enables efficient tuning of the scaling factor, especially in large models such as Llama, where merging is more expensive than running multiple validation inferences (as we discuss in Q5).\n\nRegarding the point *“there exist new SOTA methods (in full space) that work much better than TA”*, our work assumes access to LoRA fine-tuned weights, as in KnOTS. The LoRA setting is increasingly relevant due to the prohibitive cost of *full fine-tuning* of large models like Llama 8B, which requires large-scale computational resources. Additionally, as deeply described in the KnOTS paper, **merging LoRA models is not as straightforward as the fully fine-tuned models**, leading to much poorer performance.\n\nWe comprehensively evaluate recent SOTA methods in this setting -- including TSV, CART, and Iso-C -- which were originally tested on fully fine-tuned models. When LoRA checkpoints are available, this corresponds to merging in the full space matrix $\\Delta W$. We also compare against SOTA LoRA-specific methods such as the KnOTS variants. Our results show that merging LoRA modules with TSV, CART, and Iso-C is significantly improved when performed in the Core Space, which we find to be both **more effective and more efficient** than KnOTS.\n\n\n### W2 Common reference basis and task diversity\n\n>The framework’s ability to merge models assumes that the different task-specific LoRA updates can be aligned within a common reference basis. This assumption may not hold for highly dissimilar tasks, which could limit the method’s applicability. This can also be problematic beyond classification benchmarks where the fine-tuning dynamics become more complex.\n\nWe do not assume a shared basis a priori. Instead, we construct the *“reference basis”* as defined in Eq. 4, using the left singular vectors of the horizontally concatenated B matrices and the right singular vectors of the vertically concatenated A matrices. These **singular vectors always exist and are obtained via SVD**. Regarding task diversity, in our experiments **we have already merged a wide range of diverse tasks** (as suggested by the Reviewer), including fine-grained datasets (Cars and CUB), satellite imagery (EuroSAT and RESISC), and digits (MNIST and SVHN). Exploring merging beyond classification is indeed an open and interesting direction, which we plan to explore in future work.\n\n### W3 Figure 2 context\n\nWe thank the Reviewer for the suggestion and will revise Figure 2 to include more standalone context and improve clarity in the final version. Moreover, we propose an improved caption as follows:\n\nFigure 2. **Full Space Merging** (left) firstly reconstructs full space matrices $\\Delta W^{(t)} = B^{(t)} A^{(t)}$, and then performs merging in the full space to obtain $\\Delta W$. **KnOTS Merging** concatenates reconstructed $\\Delta W^{(t)}$ matrices, and performs a costly Singular Value Decomposition (SVD) on the high-dimensional concatenated matrix. Afterwards, it merges the $V^{(t)}$ matrices and performs reconstruction to obtain the final $\\Delta W$. The proposed **Core Space Merging** (right) performs SVD on a concatenation of low-dimensional $A^{(t)}$ and $B^{(t)}$ matrices to obtain reference bases. Afterwards, it performs SVD on the individual $A^{(t)}$ and $B^{(t)}$ matrices and calculates the core matrices. It then performs merging in the Core Apace and reconstructs to obtain the final $\\Delta W$. Core Space Merging performs merging in a low-dimensional space, allowing for efficient merging methods that would be extremely expensive when performed in a high-dimensional space, such as Full Space or KnOTS.\n\n### Q1 Low numbers with respect to the TSV paper\n\n>Why are the numbers so low in Table 3? [...]\n\nAs discussed above, converting LoRA to full matrices does not yield the same result as full-finetuning. Indeed, we are using the same checkpoints as the KnOTS paper, which differ from those used in TSV; thus, the results differ significantly. The low normalized scores, on the other hand, highlight how underexplored and difficult LoRA merging still is (as discussed in the KnOTS paper), as most prior work focuses on merging fully fine-tuned models. We hope that our work will spark further attention to this important setting.\n\n### Q2 Missing absolute accuracies\n\nWe follow the KnOTS setting and thus obtain the same fine-tuned accuracies, which are as follows:\n\n|Backbone|Cars|DTD|EuroSAT|GTSRB|MNIST|RESISC|SUN397|SVHN|Avg|\n|-|-|-|-|-|-|-|-|-|-|\n|ViT-B/32 r=16|74.0|58.3|99.0|92.7|99.3|88.4|64.5|96.2|84.1|\n|ViT-L/14 r=16|99.7|70.0|98.5|97.2|99.5|95.7|79.6|97.7|92.7|\n\nDue to limited space, we decided to follow KnOTS and report only the normalized accuracies. However, we agree that it would be helpful to include the absolute accuracies, and we will add them in the final version of the manuscript. It is possible to obtain the absolute accuracies through $\\text{Absolute Accuracy} = \\frac{\\text{Normalized Accuracy}}{100} \\times \\text{Fine-tuned Accuracy}$, where the fine-tuned accuracy is reported above.\n\n\n### Q3/Q4 Better performance and weight interference\n\n> “in Sec. 5.1 we show that merging in Core Space yields better performance than merging in alternative spaces, when the merging function is non-linear.” Can you explain your intuition as to why this happens?\n\n> “Our approach is found to yield compact representations of each task and improve alignment between subspaces of each task.” Isnt this at odds with weight interference from ties [32] and task interference from [a]?\n\nWe appreciate the Reviewer’s question about potential conflicts with prior work on *weight interference* (TIES [32]) and *task interference* (TALL [a]). \nOur approach, however, is fundamentally different: rather than merging in original weight space, we operate in a shared Core Space, constructed via SVD over all task-specific LoRA updates.\n\nIn our paper (L271), we demonstrate that merging in the Core Space yields dense representations, whereas the full space contains many unused or redundant components (as illustrated in Figure 4). This dense space has an important property: as shown in Figure 5, the Subspace Alignment Ratio (SAR) across tasks increases when working in the Core Space. In the Iso-C paper ([27], Appendix A.3), the authors demonstrate that a higher SAR correlates with reduced task interference, which can be measured as the L1 distance between the activations of the merged model and those of the single task fine-tuned model. We follow the evaluation protocol from [27] and measure the L1 distance between the activation of the merged model (in Core Space) and the original task-specific models. We found a reduction in interference compared to merging in full space (see our response to Reviewer DB4t, W2, for a detailed description of these results).\n\nThe performance improvement occurs only with non-linear merging functions due to the design of our Core Space. When combined with a linear merging function, merging in Core Space becomes equivalent to merging in the Full Space (as shown in L200).\n\n\n### Q5 Cost of searching optimal scaling factor\n\n>Figure 1: isn;t the bulk of the computation for e.g. ties the validation to compute the optimal scaling? This applies to knots as well as the proposed method.\n\nAs noted by the Reviewer, inference is costly when tuning the scaling factors. This is a common problem among all model merging approaches. However, in large models such as Llama 8B, the cost of the merging becomes dominant. For example, in the KnOTS space, merging can be over five times more expensive than inference on the validation set due to its cubic complexity. In contrast, inference cost grows linearly with the model size. We report, in the following, the costs of a merging and single validation inference of a Llama 8B.\n\n|Space|Val. Inference (s)|TIES merging (s)|TSV merging (s)|Iso-C merging (s)|\n|-|-|-|-|-|\n|Full|900|72|3360|540|\n|KnOTS|900|3000|4800|4860|\n|Core|900|8|12|8|\n\n---\n\n[A] Crisostomi et al. MASS: MoErging through Adaptive Subspace Selection, arxiv preprint 2025\n\n[B] Huang et al. EMR-Merging: Tuning-Free High-Performance Model Merging, NeurIPS 2024\n\n[C] Lu et al. Twin-Merging: Dynamic Integration of Modular Expertise in Model Merging, NeurIPS 2024"}, {"review_id": "1PF27AWzga", "reviewer": "Reviewer_57zV", "summary": "This paper introduces Core Space Merging, a novel framework for efficiently and effectively merging LoRA modules. The core idea is to project the task-specific low-rank matrices into a referenced \"Core Space\" before applying existing model merging algorithms. The authors provide theoretical guarantees that this strategy is lossless and conduct extensive experiments on vision and language tasks to demonstrate that their method improves the performance of various merging techniques with computation efficiency.", "questions": "- In your proposed method, the final merged update has a rank of up to $Tr$.  In contrast, methods like Full Space or KnOTS merging might result in merged updates of rank $r$. For a fair comparison, do you consider providing the merging results under a same target rank? \n\n- The construction of reference bases by stacking task-specific matrices can be viewed as creating a basis for a mixture of LoRA experts. This bears some conceptual similarity to findings in CopRA (specifically their Section 3.1). It introduces a learnable invertible matrix `P` to minimize the alignment difference, which seems related to Eq.(5) in this paper. Could you please elaborate on the relationship and key differences between Core Space Merging and the strategy proposed in CopRA?\n\n[r3] CopRA: A Progressive LoRA Training Strategy, UniReps Workshop.", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is simple, yet remarkably effective and efficient. It significantly improves upon existing merging techniques while drastically reducing computational overhead.\n- The paper provides a solid theoretical foundation for the Core Space framework. \n- The comprehensive experiments show consistent performance improvements when integrating Core Space with a wide range of existing merging methods.\n\n- The paper is well-written and provide complete and effective experimental details, along with sufficient analysis and ablation studies to ensure the validity and reproducibility of their findings. \n\nWeaknesses：\n\n- The notation of $\\tilde M^{(t)}$ in Eq.(7) and Algorithm 1 is inconsistency and the dimensions in the latter formulation seem incorrect. If the formulation in Eq.(7) is correct, it raises a question about the necessity of performing SVD on each individual task's LoRA matrices since $\\tilde M^{(t)}$ will be ${U_{B}^{ref}}^\\top B^{(t)} A^{(t)} V_{A}^{ref}$ directly.\n- As the authors claim, a current limitation is that the method's applicability has only been demonstrated for LoRA. It would be valuable for future work to explore its compatibility with other PEFT methods to test the generality of the Core Space concept. Examples can include:\n  - VeRA, which uses trainable vectors and frozen random matrices.\n  - HydraLoRA, which uses an asymmetric structure with a shared A matrix.\n\n​\t[r1] VeRA: Vector-based Random Matrix Adaptation\n\n​\t[r2] HydraLoRA: An Asymmetric LoRA Architecture for Efficient Fine-Tuning", "was_revised": true, "final_justification": "Most of my concerns have now been addressed, and I will maintain my 'Accept' rating.", "rebuttal": "We are pleased that the Reviewer recommends full acceptance, acknowledging the simplicity, effectiveness, and efficacy of the proposed method, as well as its solid theoretical foundation, comprehensive empirical evaluation, and the clarity of the paper. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.\n\n### W1 SVD redundancy\n\n> The notation of $\\tilde{M}^{(t)}$ in Eq.(7) and Algorithm 1 is inconsistent and the dimensions in the latter formulation seem incorrect. If the formulation in Eq.(7) is correct, it raises a question about the necessity of performing SVD on each individual task's LoRA matrices since $\\tilde{M}^{(t)}$ will be $U_B^{\\text{ref} \\top} B^{(t)} A^{(t)} V_A^{\\text{ref}}$ directly.\n\nWe thank the Reviewer for identifying the inconsistency between Eq. (7) and Algorithm 1, and for raising the concern about the necessity of computing per-task SVDs.\n\nAs correctly observed, the core matrix $\\tilde{M}^{(t)}$ can be computed directly via:\n\n$\\tilde{M}^{(t)} = U^{\\text{ref}^\\top}\\_B B^{(t)} A^{(t)} V^{\\text{ref}}\\_A$,\n\nwithout the need to decompose each LoRA matrix individually.\n\nWe have tested the simplified formulation and confirm that it yields identical results, while being marginally more computationally efficient. We will revise both Eq. (7) and Algorithm 1 accordingly to reflect the corrected and streamlined implementation.\n\nThat said, the per-task SVDs are needed from a theoretical standpoint: they help formalize the projection process and support the derivations in Appendix A, including orthogonality, rank preservation, and the zero-error reconstruction proofs. For this reason, we may retain the SVD-based formulation in the theoretical exposition while aligning the algorithmic description with the more efficient practical implementation.\n\nWe thank the Reviewer again for this helpful clarification, which improves both the accuracy and clarity of our work.\n\n### W2 Core Space for other PEFT methods\n\n> As the authors claim, a current limitation is that the method's applicability has only been demonstrated for LoRA. It would be valuable for future work to explore its compatibility with other PEFT methods to test the generality of the Core Space concept.\n\nWe present below the average normalized accuracy of merged **VeRA** models with rank 16. Our method consistently outperforms KnOTS, showing that the Core Space framework applies to various types of PEFT methods. However, we also notice that the normalized accuracies are generally lower than those obtained with LoRA, suggesting that further research in this direction is needed. We will provide the detailed results and absolute accuracies in the final version of the manuscript and release the checkpoints used.\n\n| Space  | TA         | TIES            | DARE-TIES       | TSV             | Iso-C           | TIES + Iso-C        | DARE-TIES + Iso-C   | TSV + Iso-C         |\n|--------|------------|------------------|------------------|------------------|------------------|------------------|------------------|------------------|\n| Full   | 64.78 (-)| 63.93 (0.00)     | 64.88 (0.00)     | 63.74 (0.00)     | 64.08 (0.00)     | 63.35 (0.00)     | 63.35 (0.00)     | 63.56 (0.00)     |\n| KnOTS  | 64.78 (-)| 65.26 (+1.33) | 65.63 (**+0.75**) | 65.33 (+1.59) | 63.30 (-0.78)    | 63.26 (-0.09)    | 63.27 (-0.08)    | 64.71 (+1.15)    |\n| Core   | 64.78 (-)| 65.31 (**+1.38**) | 65.35 (+0.47)    | 65.56 (**+1.82**) | 66.27 (**+2.19**) | 64.38 (**+1.03**) | 64.35 (**+1.00**) | 66.56 (**+3.00**) |\n\n### Q1 Effective rank of Full and KnOTS spaces\n\n> In your proposed method, the final merged update has a rank of up to $Tr$. In contrast, methods like Full Space or KnOTS merging might result in merged updates of rank $r$. For a fair comparison, do you consider providing the merging results under a same target rank?\n\nThe final rank $r_{\\Delta W}$ of the merged update matrix $\\Delta W$ depends on both merging space and merging method. We report in the table below the rank of the $\\Delta W$ matrices averaged across all layers obtained by merging 8 ViT-B/32 models fine-tuned with LoRA rank $r=16$.\n\n| Merging Space   | TA     | TSV    | TIES   |\n|-----------------|--------|--------|--------|\n| Full            | 128.00 | 128.00 | 766.25 |\n| KnOTS           | 128.00 | 128.00 | 128.00 |\n| Core            | 128.00 | 128.00 | 128.00 |\n\nIn most cases, the target rank of $\\Delta W$ is equal to $Tr = 128$. The only exception is merging with TIES in Full space, where for weight matrices $W \\in \\mathbb{R}^{m \\times n}$ the target rank approaches the dimensionality of the matrices $d = \\text{min}(m, n) = 768$. This is because when TIES performs trimming on reconstructed weight matrices $\\Delta W_t = BA$ it does not preserve the low-rank structure of the matrix. On the other hand, both Core and KnOTS operate entirely in a constrained $Tr$-dimensional space.\n\n### Q2 Comparison with CopRA\n\n> The construction of reference bases by stacking task-specific matrices can be viewed as creating a basis for a mixture of LoRA experts. This bears some conceptual similarity to findings in CopRA (specifically their Section 3.1). It introduces a learnable invertible matrix $P$ to minimize the alignment difference, which seems related to Eq.(5) in this paper. Could you please elaborate on the relationship and key differences between Core Space Merging and the strategy proposed in CopRA?\n\nWe thank the Reviewer for bringing this work to our attention. We agree there is some connection between our approach and CopRA regarding the alignment of low-rank updates. However, there are key differences in motivation, scope, and technical implementation.\n\nIn CopRA (Sec. 3.1), a learnable invertible matrix $P$ is introduced to reparameterize a set of LoRA matrices (_i.e._, $B_2P, P^{-1}A_2$) to align them better with those of another task. In contrast, Core Space aligns low-rank updates across tasks by projecting them into a shared latent subspace.\n\n**Alignment Mechanism**: CopRA uses a single learnable invertible matrix to minimize the difference between two LoRA modules. In Core Space, we use reference bases (via SVD) and task-specific **closed-form** projections $R_B^{(t)}, Q_A^{(t)}$ to align all task updates into a common subspace.\n\n**Learning Objective**: While CopRA minimizes the alignment error between two specific models through gradient descent, we construct a globally optimal shared basis across all tasks, ensuring zero reconstruction error without additional learning (see Eq. 9, Sec. 4.2).\n\n**Scalability**: CopRA is limited to pairwise alignment, whereas Core Space is designed for merging multi-task models with efficient subspace operations. Specifically, as shown in Eq. 5 of CopRA, one of the two LoRA matrices needs to be used as an anchor. It is unclear, in a multiple-task scenario, how CopRA would select the anchor point.\n\n**Information Preservation**: There is no guarantee that reparameterization with $P$ is lossless, while in Core Space, the transformation is lossless.\n\nFinally, while CopRA introduces a valuable strategy for pairwise alignment using a learnable transformation, Core Space Merging generalizes this concept to the multi-task setting, offering a non-parametric, efficient, and provably lossless merging framework."}, {"review_id": "SVbi0os5Gm", "reviewer": "Reviewer_nLge", "summary": "This paper tackles the problem of merging multiple models that have been fine-tuned using Low-Rank Adaptation (LoRA). The authors identify a key challenge: existing merging methods either operate on the full-size weight matrices, sacrificing the efficiency of LoRA, or are suboptimal when applied directly to the low-rank factors. To address this, they propose \"Core Space Merging,\" a novel framework that operates entirely in a low-dimensional space. The core idea is to first project the individual LoRA adaptors into a shared, common basis called \"reference bases.\" This projection creates small \"core matrices\" for each task. The paper proves that this projection is lossless, meaning no information is lost in the transformation. Merging is then performed efficiently on these small core matrices. The final merged model is reconstructed by projecting the merged core matrix back into the original parameter space. The authors provide theoretical analysis of the method's correctness and computational complexity, and demonstrate through experiments on both language (Llama 3 8B) and vision (ViT) models that their approach is faster and also achieves superior state-of-the-art performance.", "questions": "Have the authors studied edge cases where the total rank of the LoRA exceeds the maximum rank of the target weight matrix. Is the reconstruction error still 0 in this case ?", "limitations": "It is not clear that the reconstruction will still be lossless if the total rank of the LoRAs exceed the maximum rank of the weight matrix\n\nNo experiments studying generalization to a unseen task", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nLoRA merging a compelling research area following the democratization of large model finetuning. Improving model merging methods allows to practitioners to potential leverage existing pre-trained checkpoints to improve performance at a faction of the finetuning cost. \n\nAligning the SVD bases is a novel idea that proves effective in practice.\n\nTheorem 10 to justifies that the projection into the core space is lossless as long as the a zero reconstruction error with regards to the reference bases is achieved.\n\nThe complexity analysis (Section 4.3) demonstrates the efficiency gains over prior work.\n\nGiven these strengths, I believe that the proposed algorithm merging is likely to be adopted by both researchers and practitioners.\n\nWeaknesses:\n\nA dominant advantage of model merging which is not studied here is how existing knowledge can be combined to generalize to new tasks. Prior works such as aTLAS [1] have studied this task (including with LoRAs) and should be compared with in future work or cited as a limitation of the current paper.\n\nIt is not clear that the reconstruction loss will be lossless if the total rank of LoRA to combine exceeds the maximum rank of the target weight matrix \n\n[1] Zhang, et al. \"Knowledge composition using task vectors with learned anisotropic scaling.\" Advances in Neural Information Processing Systems (2024)", "was_revised": true, "final_justification": "In light of the other reviews and of the author's answer, I maintain my original score", "rebuttal": "We are pleased that the Reviewer leans towards acceptance, acknowledging the significance of the research topic, the theoretical soundness, as well as novelty, efficiency, and efficacy of the proposed approach. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.\n\n### W1/L2 Generalization to unseen tasks\n\n> A dominant advantage of model merging which is not studied here is how existing knowledge can be combined to generalize to new tasks. Prior works such as aTLAS [1] have studied this task (including with LoRAs) and should be compared with in future work or cited as a limitation of the current paper.\n\nWe would like to clarify that our method is not designed for out-of-distribution generalization. Instead, our approach focuses on efficient and accurate merging of already fine-tuned LoRA modules where all task-specific components are available at merge time. Additionally, we note that aTLAS operates in a fundamentally different setting: it learns trainable parameters to guide the merging process based on the target task data, requiring an additional training stage. In contrast, our method performs training-free merging, eliminating the need for labeled data or task supervision. This makes our approach truly zero-shot at the merging stage and suitable in situations where retraining is undesirable or infeasible.\n\nExploring generalization to unseen tasks is an interesting future direction. However, it falls outside the current scope, where our focus is on designing simple, efficient, and high-performing merging strategies for practical multi-task use.\n\nWe thank the Reviewer for the suggestion. We will consider studying this setting and include the corresponding discussion, mentioning aTLAS, in the Future Work section.\n\n### W2/Q1/L1 Reconstruction error when total LoRA rank exceeds maximum rank of target weight matrix\n\n> Have the authors studied edge cases where the total rank of the LoRA exceeds the maximum rank of the target weight matrix. Is the reconstruction error still 0 in this case ? It is not clear that the reconstruction will still be lossless if the total rank of the LoRAs exceed the maximum rank of the weight matrix \n\nWe thank the Reviewer for the insightful question and revisit our analysis under the relaxed assumption that the total LoRA rank may exceed the maximum rank of the weight matrix.\n\nFrom an experimental perspective, **we confirm that the reconstruction error remains zero** in this case. Specifically, consider merging 8 ViT-B/32 models fine-tuned with LoRA $r=256$. In this case $T r = 8 * 256 = 2048$ while all the matrices fine-tuned with LoRA $W_q, W_k, W_v, W_o \\in \\mathbb{R}^{768 \\times 768}$, so $T r > 768$. We computed the value of $\\varepsilon$ (Eq. 10) for both $U$ and $V$ in this scenario when merging in Core Space, and found it to be exactly zero.\n\nFrom a theoretical point of view, although the derivations in Appendix A assume $T r \\leq m, n$, we find that this assumption is not strictly necessary.\n\nWhen $T r > m$ (for $B$) and $T r > n$ (for A), stacking the matrices $A^{(t)}$, $B^{(t)}$ and taking their SVD, produces reference bases $U_{B}^{ref}$ and $V_{A}^{ref}$ with intrinsic:\n$d_U = rank([B^{(1)}, .., B^{(T)}] \\leq m, \\quad d_V = rank([A^{(1)}, .., A^{(T)}]^T) \\leq n$, since the number of linearly independent directions cannot exceed the number of rows or columns. We now focus on $U_{B}^{ref}$; analogous reasoning applies on $V_{A}^{ref}$. \n\nWe use the truncated orthonormal basis $U^{\\text{ref}}_B \\in \\mathbb{R}^{m \\times d_U}, \\quad (U^{\\text{ref}}_B)^\\top U^{\\text{ref}}_B = I\\_{d_U}$, \nwhere $d_U$ is the intrinsic rank of the stacked LoRA matrices $B^{(t)}$. These $d_U$ directions span the entire LoRa update space.\n\nReferring back to Eq. (A) in Appendix A.1, we can restate the least-squares problem for $R_B^{(t)}$ as:\n$R_B^{(t)} = \\text{argmin}_{R \\in \\mathbb{R}^{d_U \\times r}} || U^{ref}_B R -  U_B^{(t)}||_F^2$\n\nFollowing the steps as in the original derivation, we recover the same solution: $R_B^{(t)} = {U_B^{ref}}^T U_B^{(t)}$. The same reasoning applies to $Q_A^{(t)}$.\n\nSimilarly, the alignment error defined in Eq. ( C ) of Appendix A.2 remains unchanged when using $R_B^{(t)} \\in \\mathbb{R}^{d_U \\times r}$. In Appendix A.3, the search space for orthonormal matrices becomes $\\mathcal{S} =  \\{ U \\in \\mathbb{R}^{m \\times d_U} \\;\\ |\\; U^\\top U = I_{d_U} \\}$ and the resulting alignment error is still zero. \n\nTo summarize, the problems behave as if the total LoRA rank $T \\cdot r$ were reduced to intrinsic ranks $d_U$ and $d_V$, since directions beyond $m$ or $n$ are linearly dependent and do not affect the reconstruction. \n\nWe will update the appendix with a generalized formulation that incorporates $d_U$ and $d_V$, removing the need for the assumption $T r \\leq m, n$."}, {"review_id": "6Pjr1n1q0W", "reviewer": "Reviewer_DB4t", "summary": "This paper proposes a novel approach for merging LoRA-finetuned instances of a same model on various tasks within Core space as opposed to the full space. The authors have shown the proposed approach preserves information while significantly improving the computational complexity as well as the ultimate accuracy. The empirical study on both language and vision tasks have shown the merits of the proposed framework in practice.", "questions": "- Are you claiming that the proposed methodology can be applied even when the LoRA rank is different among different tasks? If so, how?\n\n- How does higher subspace alignment explain better generalization of the model after the merge?", "limitations": "As I mentioned above, dealing with different ranks for the same weight matrix is not straightforward and seems to be an important limitation of the proposed framework which hasn't been discussed in the paper.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- The paper proposed an interesting low-rank merging methodology via the core space that is theoretically sound.\n- The experimental study is well-done and convincing.\n- The paper is generally very well-written with a good flow.\n\nWeaknesses:\n\n- While the authors have claimed in the Limitation section that the proposed approach can be also applied in scenarios with heterogenous ranks, the extension is not straightforward unless the corresponding LoRA updates still have the same rank, even though the rank might be different across different layers. In fact, this points to an important limitation of the proposed framework which hasn't been discussed in the paper; that is, the proposed method is not applicable to models that are finetuned via the rank adaptive version of LoRA (e.g. AdaLoRA) where the rank assigned to each weight matric may be different across the tasks. However, the full space and KnOTS do not suffer from this issue in general.\n\n- The argument provided for explaining the superior accuracy of the proposed framework is somewhat weak. In particular, the authors have implied that the better alignment between the core spaces of different tasks is the main reason for the better accuracy, but this is merely a correlation at best. A theoretical analysis or ablation studies are needed to back this \"implicit\" claim and establish the connection between subspace alignment and the generalization error.", "was_revised": true, "final_justification": "The authors have made a convincing argument in their response to my concerns about the proposed methodology. As a results, I'd like to increase my score to 5.", "rebuttal": "We are pleased that the Reviewer leans towards acceptance, acknowledging the paper’s clarity, novelty, efficiency, and effectiveness of the proposed approach, as well as the convincing theoretical and experimental study. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.\n\n### W1/Q1/L1 Application to heterogeneous rank of LoRAs setting\n\n> While the authors have claimed in the Limitation section that the proposed approach can be also applied in scenarios with heterogenous ranks, the extension is not straightforward unless the corresponding LoRA updates still have the same rank, even though the rank might be different across different layers. In fact, this points to an important limitation of the proposed framework which hasn’t been discussed in the paper; that is, the proposed method is not applicable to models that are finetuned via the rank adaptive version of LoRA (e.g. AdaLoRA) where the rank assigned to each weight matric may be different across the tasks. However, the full space and KnOTS do not suffer from this issue in general.\n> Are you claiming that the proposed methodology can be applied even when the LoRA rank is different among different tasks? If so, how?\n\nWhile at first glance, handling LoRA modules with heterogeneous ranks might seem non-trivial, in practice, our method naturally supports it without requiring any modification.\n\nAs shown in our implementation (see Listing A in the Appendix or the `get_core_matrices` function in `task_merger.py` in the provided code at L214), we compute the reference bases via SVD on the stacked A and B matrices across all tasks. **Even when individual LoRA modules have different ranks $r^{(t)}$, these matrices can still be concatenated across tasks**, resulting in an aggregate basis that spans the combined subspaces. The projection and alignment operations are then applied accordingly for each $r^{(t)} \\times r^{(t)}$ local task core matrix. \n\nTo illustrate this, consider two tasks: one with a LoRA module of rank 4 and another with a rank of 8. Suppose:\n\nTask 1 has $A^{(1)} \\in \\mathbb{R}^{4 \\times 768}, B^{(1)} \\in \\mathbb{R}^{768 \\times 4}$\n\nTask 2 has $A^{(2)} \\in \\mathbb{R}^{8 \\times 768}, B^{(2)} \\in \\mathbb{R}^{768 \\times 8}$\n\nWe concatenate these as:\n\n$A_{\\text{stack}} \\in \\mathbb{R}^{(4+8) \\times 768} = \\mathbb{R}^{12 \\times 768}$\n\n$B_{\\text{stack}} \\in \\mathbb{R}^{768 \\times (4+8)} = \\mathbb{R}^{768 \\times 12}$\n\nApplying SVD to these stacked matrices yields shared reference bases $V^{\\text{ref}}_A \\in \\mathbb{R}^{768 \\times d}$ and $U^{\\text{ref}}_B \\in \\mathbb{R}^{768 \\times d}$, where $d=12$. Each task’s $A^{(t)}$ and $B^{(t)}$ are then individually aligned to these reference bases, regardless of their original rank. **This setup enables us to handle heterogeneous-rank LoRA modules naturally**, without the need to pad, truncate, or enforce uniformity.\n\nThis flexibility arises because **SVD makes no assumptions about input ranks**; it always produces valid orthonormal bases for the column and row spaces. Our merging framework leverages this to support variable-rank LoRA modules seamlessly.\n\nTo further support the generality of our method, **we evaluate performance in a heterogeneous-rank setting**, where LoRA modules have varying capacities across tasks. Specifically, we randomly assign rank 16 to half of the tasks and rank 64 to the other half, maintaining a 50/50 split. The assigned ranks per dataset are shown below:\n\n| LoRA Rank 16       | LoRA Rank 64      |\n|--------------------|-------------------|\n| Cars               | DTD               |\n| EuroSAT            | GTSRB             |\n| MNIST              | RESISC            |\n| SVHN               | SUN397            |\n\nDespite this rank mismatch, **our method handles the merging seamlessly**, without requiring any changes to the implementation. This is possible because we compute the Core Space by stacking the A and B matrices across all tasks—regardless of their individual ranks—and perform SVD to obtain shared reference bases. Each task’s local core matrix is then aligned accordingly, based on its native $r^{(t)} \\times r^{(t)}$ dimensionality.\n\nThe results below show that **our method continues to outperform other merging baselines** even in this more general setting:\n\n\n| Space  | TA         | TIES              | DARE-TIES         | TSV               |TIES + Iso-C          | DARE-TIES + Iso-C     | TSV + Iso-C           | Iso-C             |\n|--------|------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|-------------------|\n| Full   | 64.34 (-)         | 63.50 (0.00)      | 63.81 (0.00)      | 67.95 **(0.00)**  | 66.90 (0.00)      | 67.12 (0.00)      | 68.72 (0.00)      | 72.06 (0.00)      |\n| KnOTS  | 64.34 (-)         | 65.13 (+1.63) | 66.69 (+2.88) | 64.20 (-3.75)     | 64.16 (-2.74)     | 63.37 (-3.75)     | 70.40 (+1.68)     | 71.26 (-0.80)     |\n| Core   | 64.34 (-)         | 70.59 (**+7.09**) | 69.43 (**+5.62**) | 67.41 (-0.54)     | 72.56 (**+5.66**) | 72.68 (**+5.56**) | 71.51 (**+2.79**) | 74.90 (**+2.84**) |\n\nWe will add this discussion to the final version of the manuscript.\n\n\n### W2/Q2 Relationship between subspace alignment and the generalization error\n\n> The argument provided for explaining the superior accuracy of the proposed framework is somewhat weak. In particular, the authors have implied that the better alignment between the core spaces of different tasks is the main reason for the better accuracy, but this is merely a correlation at best. A theoretical analysis or ablation studies are needed to back this “implicit” claim and establish the connection between subspace alignment and the generalization error.\n\nThe authors of the Subspace Alignment Ratio (SAR) metric provide a theoretical analysis explaining why higher SAR leads to better performance (see [27], Appendix A.3). They demonstrate that when SAR is low, the merging interference is high. The interference is defined as the L1 distance between the final activations of the task-specific model and the activations of the merged model. Therefore, we follow the experimental protocol from [27] to compare the interference when merging with TSV + Iso-C in Full Space versus Core Space. For each dataset, we collect the activations from the final layer (*i.e.*, the projection to a common vision-language space) of both the task-specific model and the merged model. We present the average distance across all the samples in the test set in the table below. We observe lower interference when merging in Core Space, highlighting its effectiveness.\n\n| Dataset | L1(Full, Task-specific) | L1(Core, Task-specific) |\n|---|---|---|\n| Cars | 0.255 | **0.234** |\n| DTD | 0.224 | **0.183** |\n| EuroSAT | 0.178 | **0.164** |\n| GTSRB | 0.184 | **0.103** |\n| MNIST | 0.175 | **0.132** |\n| RESISC | 0.245 | **0.211** |\n| SUN397 | 0.308 | **0.278** |\n| SVHN | 0.239 | **0.197** |\n\nWe will report this analysis in the final version of the paper."}]

Author rebuttal:

Meta-review: This paper proposes Core Space Merging, a framework for efficiently and effectively merging LoRA-adapted models. The key idea is to project low-rank updates into a shared Core Space before merging, ensuring information preservation and computational efficiency. The approach is supported by theoretical analysis and validated with extensive experiments on both vision and language tasks, showing clear gains in efficiency and accuracy over existing methods.

Pros:
* Clear, well-written paper with strong motivation and theoretical grounding.
* Demonstrates both efficiency (avoiding costly full-matrix reconstructions) and improved performance.
* Extensive empirical validation across domains, scales, and baselines, with consistent improvements.
* Rebuttal convincingly addresses reviewer concerns (e.g., heterogeneous ranks, subspace alignment, comparison with alternative methods).
* High potential for adoption in both research and practice due to simplicity and effectiveness.

Cons:
* Focuses mainly on efficiency; broader questions of merging for generalization to unseen tasks remain unexplored.
* Assumptions about shared alignment basis may limit applicability to highly dissimilar tasks.
* Current scope limited to LoRA (though preliminary results on other PEFT methods are promising).

The paper makes a solid, well-justified contribution to efficient model merging with LoRA. Despite some limitations, the method is novel, theoretically sound, and empirically strong, with clear impact for both the deep learning and parameter-efficient fine-tuning communities.
Decision: Accept (poster)