No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Authors: Daniel Marczak
Simone Magistri
Sebastian Cygert
Bartłomiej Twardowski
Andrew D. Bagdanov
Joost van de Weijer
Venue: ICML 2025 poster (2025)
Keywords: Model merging
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: We achieve state-of-the-art model merging results by proposing isotropic merging in common and task-specific subspaces of weight update matrices across vision and language, for fully fine-tuned and LoRA-adapted models

Abstract: Model merging integrates the weights of multiple task-specific models into a single multi-task model. Despite recent interest in the problem, a significant performance gap between the combined and single-task models remains.  In this paper, we investigate the key characteristics of task matrices -- weight update matrices applied to a pre-trained model -- that enable effective merging. We show that alignment between singular components of task-specific and merged matrices strongly correlates with performance improvement over the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task matrices, enhances alignment, and reduces the performance gap. Additionally, we incorporate both common and task-specific subspaces to further improve alignment and performance. Our proposed approach achieves state-of-the-art performance on vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, offering an effective methodology to merge models without requiring additional training.

No Task Left Behind:
Isotropic Model Merging with Common and Task-Specific Subspaces
Daniel Marczak 12 Simone Magistri
Sebastian Cygert* 45
Bartlomiej Twardowski 678 678
Andrew D. Bagdanov
Joost van de Weijer 78
Abstract
Task Arithmetic
Normalized Accuracy Improvement
0.1
Coumon subspacc
Model merging integrates the weights of multi- ple task-specific models into a single multi-task model. Despite recent interest in the problem, significant performance gap between the com- bined and single-task models remains. In this paper, we investigate the key characteristics of task matrices weight update matrices applied to a pre-trained model - that enable effective merg- ing. We show that alignment between singular components of task-specific and merged matri- ces strongly correlates with performance improve- ment the pre-trained model. Based on this, we propose an isotropic merging framework that flattens the singular value spectrum of task ma- trices, enhances alignment, and reduces the per- formance gap. Additionally, we incorporate both common and task-specific subspaces to further im- prove alignment and performance. Our proposed approach achieves state-of-the-art performance vision and language tasks across various sets of tasks and model scales. This work advances the understanding of model merging dynamics, of- fering an effective methodology to merge models without requiring additional training.
0.0
0.8
Iso-C
0.1
Common subspare
0.6
0.4
0.0
Iso-CTS
0.2
0.1
Common subspace
Tesk-specitic subspacces
0.0
8
14
0.0
Number of tasks
Figure I. Spectrum of singular values for a single layer weight update matrix obtained by merging using Task Arithmetic (top) compared to our approaches: Iso-C (middle) and Iso-CTS (bottom). Task Arithmetic sums the task-specific matrices, which result in a spectrum with a few dominant components. Iso-C instead replaces this spectrum with uniform one, which results in significant performance improvement. Iso-CTS en- hances the common subspace with task-specific subspaces and yields state-of-the-are model merging performance.
2021; Caron et al., 2021; Zhai et al., 2023). In practice, they are typically fine-tuned for specialization on specific tasks (Wortsman et al., 2022b; Ilharco et al., 2022). Recently, growing body of research has focused on model merg- ing (Li et al., 2023), which combines multiple task-specific experts into single multi-task model. Many methods have been proposed to improve the effectiveness of model merg- ing by reducing sign conflicts (Yadav et al., 2023), by align- ing gradients (Daheim et al., 2024), or through magnitude- based selection (Marczak et al., 2024). However, a signifi- cant performance gap between the combined and single-task models remains.
1. Introduction
Pre-trained models are the foundation of modern ma- chine learning systems (Carion et al., 2020; Radford et al.,
'Warsaw University of Technology, Poland VDEAS NCBR, Warsaw, Poland Deparrmentt of Information Engineering, Media Integration and Communication Center (MICC), University of Florence, Italy *NASK PIB, National Research Institute, War- saw, Poland 'Gdańsk University of Technology, Poland "IDEAS Research Center. Warsaw, Poland 'Computer Vision Center, Barcelona, Spain 'Department of Computer Science, Universi- tat Autonoma de Barcelona, Spain. Correspondence to: Daniel Marezak www.hll..aaazakkkkkoooooo www.kktww....u.......
A key insight from Ilharco et al. (2023) is that task vec- tors, defined as the offset between the flattened fine-tuned weights and the pre-trained checkpoint, from different tasks are typically close to orthogonal. This orthogonality has been seen as a fundamental property enabling effective merg- ing with reduced interference and has inspired works that en-
Proceedings of the 42nd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
force the orthogonality by modifying the fine-tuning proce- dure (Po et al., 2024). Most recently, Stoica et al. (2025) and Gargiulo et al. (2025) have shown that accounting for the structure of the weight update matrix, dubbed task matrix, is a more effective strategy for improving the performance of model merging. In this paper, we investigate precisely what the characteristics of task matrices are that favor effective model merging. Different from previous works, we propose to analyze the alignment between task-specific and merged subspaces.
nique for model merging that we call Iso-C, which outperforms most baselines.
We further enhance our approach by incorporating task- specific directions into the merged matrix resulting in Iso-CTS, a merging method that achieves state-of- the-art results, in particular for large number of tasks.
Our methods demonstrate versatility, achieving state- of-the-art on vision and language merging benchmarks for both fully and LoRA fine-tuned models'.
Specifically, to capture the similarity between task matrices, we propose to investigate the Subspace Alignment Ratio. Through the lens of Singular Value Decomposition, our metric quantifies the similarity between subspaces spanned by the top singular vectors of task matrices. When applied to compare matrices of the merged model to the task-specific ones, this metric strongly correlates with the performance of the merged model on given task. This allows us to identify the directions amplified by multiple tasks as well as the underrepresented directions that lead to poor performance on corresponding tasks.
2. Related Work
Model merging. Pre-trained models serve as a founda- tion for expert models specialized in specific downstream tasks (Radford et al., 2021). Recently, model merging has emerged as a promising technique to combine multiple ex- pert models into a single multi-task model. One of the pioneering works in the field, Task Arithmetic (TA) (Iharco et al., 2023), proposed to compute a task vector as dif- ference between the expert and the pre-trained model and to then aggregate task vectors via scaled addition to create an expert in multiple tasks. The significant performance gap between individual experts and the combined model sparked an abundance of works with the aim of reducing in- terference when merging models. TIES (Yadav et al., 2023) proposed novel way to reduce sign conflicts between the parameters of expert models, Model Breadcrumbs (Davari & Belilovsky, 2024) removed outliers from the task vec- tors, and Consensus Merging (Wang et al., 2024b) removed catastrophic and selfish weights. These methods focused on per- per-parameter techniques to mitigate the interference, treating each parameter independently.
Our goal is design model merging technique that bal- ances directions in the weight space across different tasks. We achieve this by flattening the singular values spectrum of the merged matrix, making it more uniform. Enfore- ing a uniform (isotropic) spectrum significantly improves the alignment and performance of the merged model. This simple yet effective adjustment, which requires no changes to the fine-tuning procedure, leads to substantial gains in merging performance (see method Iso-C in Figure 1).
However, tasks with dominant directions of smaller intensity compared to the majority of tasks and whose directions are orthogonal to the common directions may still remain under- represented, especially when the number of tasks increases. To address this, we enhance isotropic model merging by introducing task-specific subspaces that retain unique task features while preserving shared knowledge. Our approach begins with the top singular values of the common subspace and iteratively replaces the least significant singular vectors with aask-specific directions. This strategy allows us to in- crease the scalability of our merging approach to more tasks (see method Tso-CTS in Figure 1).
The aforementioned static merging methods output single set of multi-task weights which can be used as a drop-in replacement for the pre-trained model. However, number of recent methods, dubbed dynamic merging, alter the infer- ence procedure to improve the results. Twin-Merging (Lu et al., 2024) composes task-specific components at test-time and alters the inference algorithm requiring two forward passes. EMR- Merging (Huang et al., 2024) uses additional per-task parameter masks and rescalers to perform inference. In this paper, we consider static merging exclusively.
The main contributions of this paper are:
Singular Value Decomposition of model weights. While SVD of weight matrices has been primarily used for model compression (Denton et al., 2014; Kim et al., 2016), recently its effectiveness was also identified for fine-tuning of large models. LoRA (Hu et al., 2021) uses SVD to identify the similarities of weight updates between low-rank and full- rank fine-tuning. MiLORA (Wang et al., 2024a) identifies
We show that the aligmment between the subspace spanned by the principal directions of the task- task-specific matrices and that of the merged matrix positively cor- relates with the performance of the merged model.
We demonstrate that applying an isotropic scaling to singular directions of merged task matrices improves the alignment between merged and task-specific matri- ces. This results in a simple yet highly effective tech-
'The code is available at https: https:///ithubbooom danielml 0001mm140///sss ttts//ssmmerrrg -m--ggingg.
2

---

No Task Left Behind: Isotropic Model Merging with Common and Task- Task-Specific Subspaces
that the bottom singular components correspond to noisy or long-tail information, while the top singular vectors contain important knowledge. Therefore, they propose a fine-tuning approach that updates only the minor singular components of the weight matrix while keeping the top singular compo- nents frozen. SVFT (Lingam et al., 2024) computes outer products of its singular vectors and, during fine-tuning up- dates, only sparse coefficients of these combinations.
Codte crclecitine of Sade mertoes
Car
ADNIT
RUN
1.12
DTT
013165
1.00
6:C4
3:14
5.3L
SUT
KISS
1.5
1:12
082
6.44
1A0
*
112
082
TM
00G
SVD for model merging. The structure imposed by SVD was used for model merging in KnOTS (Stoica et al., 2025), which proposes to concatenate the task-specific low-rank adaptation matrices (LoRA) and average the right-singular vectors before SVD reconstruction to obtain the merged weights. The most similar work to us is the parallel work Task Singular Vectors (TSV) (Gargiulo al., 2025), which measures task interference based on the interaction of sin- gular vectors from different tasks and uses it to increase merging effectiveness. We share the motivation to improve model merging through SVD decomposition. However, while they focus on the orthogonalization of task-specific subspaces to reduce interference, we show that making sin- gular values uniform in a common subspace is a surprisingly powerful method. Further, we show how to combine shared and task-specific subspaces for improved performance.
0356
(a) Cosine similarity between (b) NAI vs cosine similarity be- pairs of task vectors.
tween task and merged vectors.
Figure 2. (a) Tasks vectors are typically close to orthogonal to each other. (b) Models with very different normalized accuracy improvements (NAT) exhibit very close cosine similarities, and the correlation between cosine similarity and NAI is low.
of the model 0, and the pre-trained model Og for layer f:
( o 06
(2)
In the rest of the paper, the superscript is omitted when not relevant to the discussion, and all definitions refer to an arbitrary layer. The authors of Task Arithmetic propose to solve the problem of model merging by defining merging function that sums all task matrices to the pre-trained model weights:
3. Background and Motivation
In this Section, we first describe the general framework of model merging and provide the notation used throughout the rest of the paper. We then motivate our approach via an analysis of the correlation between task similarity and performance improvement of the merged model.
0 + as
(E) TA
(3)
where is a scaling factor determined on held-out valida- tion dataset and 1 ΣT= A,²) The advantage of this merging strategy is that allows for the reuse and transfer of knowledge from many fine-tuned models to the pre- pre-trained model without requiring additional training or access to the original training data (Ilharco et al., 2023).
3.1. Model Merging
Model merging integrates multiple deep neural network models, each individually trained (i.e. fine-tuned) distinct tasks starting from the same pre-trained model, into a single merged model. Let On denote the weights of the pre-trained network, and O, denote the fine-tuned weights for task t, with 1 = I, I,...,T: ,, where T is the total number of tasks. We will use the notation 0 to identify the weights of layer I for task t and L to denote the total number of layers in a network. The objective of model merging is to find a merging function f. such that the model:
3.2. Cosine Similarity and Performance Improvement are Uncorrelated
Starting from the definition of Task Arithmetic (TA) in Eq. (3), we aim to explore the possible reasons for the im- provement achieved by TA merging over the pre-trained (or zero-shot) model across multiple tasks. To empirically quan- tify performance gain, we propose the Normalized Accuracy Improvement (NAI) metric, defined as:
NAI(OM, 0o) = Acc(Om) - Acc(Oo) Acc(O.) Acc(Oo)
PM 1(0b", (T)) =1=1....
(4)
(1)
which quantifies the improvement of the merged model fM relative that achieved by the task-specific model Or, both measured with respect to the zero-shot baseline 0g.² 2
is able to perform all tasks on which the individual models 0, are trained.
Building upon Task Arithmetic (TA), we define the layer- wise task matrix A as the difference between the weights
NAT differs from Normalized Accuracy (Ortiz-Jiménez al.,
2023) which does not account for zero-shot performance.
3

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Ihaarco et al. (2023) hypothesize that minimal inter-task interference - captured by near-zero cosine similarity be- tween the vectorized representation of the task matrices, i.e.. ()), vec(A,)) ~ 0 for i * j (see Figure 2a) explains the effectiveness of Task Arithmetic. To investi- gate this further, we examine whether the cosine similarity between each task vector and the merged Task Arithmetic vector, (vee( (vec(ArA), vee (A))), serves as an indicator of per- formance improvement, as quantified by NAI 0e: 0o)- However, we observe no clear correlation (see Figure 2b), suggesting that cosine similarity alone does not fully explain the observed performance gains. This indicates that the im- provement achieved by Task Arithmetic likely originates from other factors, which we unveil below through spectral analysis of the Task Arithmetic and task-specific matrices.
In Figure 3a (left, represented by stars), we plot the Nor- malized Accuracy Improvement achieved by TA on each task, given by NAI(OTA, 0;; 0o), against the Average Sub- space Alignment Ratio of each task matrix As with the merged task matrix ATA i.e. Sag (A,, ATA; kTT). First, we note that the alignment between task and merged matri- ces are notably high (ranging from 0.75 to 0.87), but vary significantly across datasets. This suggests that task vec- tors are well represented in the subspace identified by the task-arithmetic matrix but with different degrees of align- ment and consistency depending on dataset characteristics. Furthermore, we highlight strong correlation (Pearson cor- relation coefficient PIA = 0.94) between the performance improvement on individual tasks achieved by 0Tx and the degree of alignment of A, with ATA-
Analogous to the pairwise cosine similarity analysis be- tween task vectors performed by Ilharco et al. (2023), in Fig- ure 3b we measure the SAR between pairs of task matrices, SARavg (A,, Ay; kTA)., using the İTw dominant components of the merged Task Arithmetic model. Some groups of tasks exhibit higher alignment which is due to their seman- tic similarity, e.g. MNIST, SVHN, and GTSRB are digit recognition datasets, while EuroSAT and RESISC45 are satellite image datasets. On the other hand, datasets such as Cars, DTD or SUN397 are less aligned to other tasks. Most importantly, tasks belonging to highly aligned groups are also highly aligned with the TA model and achieve the highest accuracy improvements (see Figure 3a). The tasks that are not aligned are underrepresented in the dominant subspace of ATA, and the performance on them is low.
3.3. Performance Correlates with Subspace Alignment
We argue that the improvement in Task Arithmetic perfor- mance derives from the relationship between the top singular vectors of ATA and those of each A, Specifically, we hy- pothesize that the subspace of A approximates the union of the subspaces of each A., and that the overlap of this overall subspace with each task matrix correlates with the performance improvement of the merged model.
In order to empirically quantify the overlap between sub- spaces, we propose the Subspace Alignment Ratio (SAR) metric. We define SAR between a task matrix A, and generic merged task matrix AM as:
Malle ||
SAR(Ar: AM: kM)
(5)
Based on the observed correlation between performance and alignment ratio, we hypothesize that a merging method that aims to achieve high alignment will also achieve strong performance. Therefore, in the next Section, we propose an approach called Isotropic Merging that improves align- ment and, most importantly, the performance of the merged models.
where II..M UkM, T M M is the projection matrix onto the subspace spanned by the top km left-singular vectors of AM- The columns of UkM M are obtained from the SVD decomposition of AM. and the number of singular vectors used (kx) is determined from the merged task matrix AM by minimizing the approximation error with e = 0.05:
4. Isotropic Merging in Common and Task-specific Subspaces
km min {k: AM AM Se Se||Am]
Ei of El of <e2
min k:
(6)
In this Section, we propose novel model merging method we call Isotropic Merging in Common and Task-Specific Subspaces (Tso-CTS). First, we introduce Isotropic Merg- ing in Common Subspace (Iso-C), which is able to en- hance the normalized accuracy improvement and the align- ment of each task matrix using common directions identified by Task Arithmetic. Then, we show how further enhance the performance of merged models by introducing task- specific directions to improve merging performance on sets of many diverse tasks.
where Σ- = (i.,, contains the singular values of AM, and the equivalence follows from the definition of the Frobenius norm (see Appendix A.
SAR quantifies the alignment between the subspaces of two task matrices as a function of the number of dominant singular vectors of the merged matrix. To provide a single score measuring the overlap between two models, we denote with SARay the Average Subspace Alignment Ratio across all layers.

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
0.99
0.75
0.68
0.58
0.57
0.58
0.58
0.56
MNIST
1.0
0.773
0.99
0.68
0.59
0.57
0.58
0.57
0.56
SVHN
0.8
0.64
0.66
0.90
0.60
0.58
0.59
0.58
0.58
GTSRR
0.6
0.57
0.58
0.50
0.991
0.67
0.62
0.57
0.62
EurSAT
NAI
0.56
0.57
0.58
0.6-1
0.97
0.58
0.63
RESISC45
0.4
0.58
0.58
DTD
TA
Iso-C
Cars
MNIST
0.2
0.35
0.55
0.56
0.56
0.57
0.57
0.96
0.58
Cazs
DTD
RESISC45
EuroSAT
SUN397
0.54
0.55
0.56
0.58
0.60
0.50
0.58
0.95
SUN397
0.0
GTSRB
SVHN
MNIST
SVHN
CTSRD
EAmSAT
RESISCAS
DTD
Cars
SUNAI7
0.75
0.80
0.85
0.90
0.95
SARag
(a) Normalized Accuracy Improvement (NAI) Average Subspace Alignment Ratio (SARivg).
(b) Average Subspace Alignment Ratios (SARwg) between pairs of task matrices.
Figure 3. (a) NAI strongly correlates with SARay (Pearson correlation coefficient PTA = 0.94). (b) Note the groups of highly aligned tasks such as (MNIST. SVHN, GTSRB) and (EuroSAT, RESISC45). By comparing (b) and (a), mutually aligned datasets exhibit higher alignment with the merged model and consequently achieve good performance. On the other hand, tasks with low mutual alignment, such as DTD, Cars, and SUN397, are less aligned with the merged model achieve poor performance.
4.1. Isotropic Merging in Common Subspace
Algorithm 1 Iso-C: Isotropic Merging in Common Sub- space
In Section 3.3, we demonstrated the high alignment of each task matrix with the matrix obtained by Task Arithmetic. This alignment indicates that the span of dominant singular vectors of the merged matrix effectively covers the sub- space of each task and provides good approximation of the common subspace. However, significant variability in the average alignment ratio across the dataset leads to lower accuracy improvement for less aligned tasks compared to the tasks belonging to groups with high alignment. This variability stems from the skewness of the task arithmetic spectrum (Figure I and 12), which is concentrated in the first few singular values (which we call top or dominant), favoring the tasks from the highly aligned groups. Our pro- posed methodology, which we call Isotrpicc Merging in Common Subspace (Iso-C), C), aims to equalize the spectrum of the task arithmetic matrix in order to enhance the aver- age subspace alignment ratio and ensure a more balanced representation across tasks in the merged model.
Require: Task matrices A A.,..., Ar with A, mu I: Sum task matrices: ATA > A,
2: Compute the SVD of ATX: ATA = UEV with U e Rmxr S RY V E Bnxr = diag EE R'
3: Calculate isotropic factor: δ = o,
(Eq.7) (Fq.8)
Reconstruct the matrix: Arso-c = oUV
return A T
To reduce the skewness towards the dominant singular vec- tors of ATA, we propose scaling all directions of the trans- formation applied by the right-singular vectors V to fixed value rather than using their corresponding singular values. This ensures that the final transformation is isotropic, with the scaling factor set to the average singular value:
Consider the sum of task matrices ATA = =, A., where A, Rm Via Singular Value Decomposition (SVD) on ATA we obtain ΑTA = U.VT, where U E Rmxr and V E [Rax represent, respectively, the left and right singular vectors of ATA. and Σ E x2 is the diagonal matrix con- taining the singular values. We denote the vector of singular values by a diag(E) E R
and merged matrix is computed using the reconstruction:
A TUV
(8)
We apply this operation all network layers, and the final merged model is defined as:
0(0) 0 = o6( + (9 100-s =e= 1...., .,. L.
(9)
5

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Algorithm 2 Iso-CTS: Isotropic Merging in Common and Task-Specific Subspaces (green shared with Iso-C)
negligible information. At the same time, tasks with domi- nant directions of smaller intensity compared to the majority of tasks and whose directions are orthogonal to the com- mon directions remain underrepresented. This limitation becomes more pronounced as the number of tasks increases and the tasks become more diverse (see Appendix A.4 for an extended discussion).
Require: Task matrices
Ar with A, E Ik 1: Sum task matrices ATA = >i=1 A
2: Compute the SVD of ATA: ATA = UEVT, with U E
Rmxr, Σ E R V E R"xr, o diag(E) R"
3: Retain top-k singular vectors and values from common subspace: U [uil... mull....uer [um]
vis
[oi]... |ux]
To address this limitation, we propose enhancing the range of directions used by Iso-C ensure that the task-specific directions, which are orthogonal to those of the common subspace, are incorporated into the singular basis of the final merged matrix. We call this methodology as Isotropic Merg- ing in Common and Taec-Specifiic Subspaces (Iso-CTS).
0=m = diag(E) l:k
4: Accumulate task-specific directions via projection:
5: for = 1 to T do
6: A; - A. - 12: (U):k)TT A
(0q
7: Compute SVD: A. = U.E.V
8: Retain first = r-k components of U, and Ve
Our approach starts with the top singular values of the com- mon subspace and iteratively replaces the singular vectors associated with the lowest singular values with task-specific directions. The final goal is to find two orthonormal matri- ces U. E RR and V. E _R whose columns contain both common and specific directions. Afterward, the final matrix is reconstructed. and isotropic merging is ap- plied. In the following, we provide detailed explanation of our proposed algorithm.
T!is = Tc,1 I... /...
Vils [U, [r.ss]
of = diag [,,,..
9: end for
10: Combine common and task specific spaces:
U. = T 1:s
UT
1:
E Ruxr
V. =- ===Vv>> V
V R
11: Orthogonalize U. and V. via whitening
(Eq.11)
12: Calculate isotropic factor T.
Retaining components from the common subspace. We retain the top-k singular vectors associated with the sub- space identified by ATA:
L.
(Eq.13)
13: Reconstruct the matrix A Aao-CTS = TU.V.
(Bq.12)
14: return A 150-CTS
Ulk
[u1l hall.../uu /usl
V [ur]... Iugl.
where U lik V are the top-k left- and fightssinglaa vec- tors from the SVD of ATA. We analyze the impact of select- ing & in Section 5.4.
where a is chosen on a held-out validation set.
A ccumulating task-specifi directions. We project each task-specific matrix A, onto the subspace orthogonal to the common subspace, i.e. the space spanned by top left- singular directions of the common subspace Ul*k;
Applying isotropic merging results in an enhancement of the normalized accuracy improvement and subspace alignment ratio (SAR) compared to Task Arithmetic (see Figure 3a). The increase in SAR is due to higher number of dominant components kisc-e in A Aso-c (see Equation (6)), derived from the singular vectors of TTA, which are aligned with the subspaces of individual tasks (see Appendix A.2 for details). In Appendix A.3, we show that increased SAR is associated with reduced inter-task interference, measured by changes in internal activations induced by merging. In Algorithm 1, we present the Iso algorithm for a single layer.
A = A, - U (U
(10)
We then compute the SVD of A. = U, E,V, and retain the top r- directions for each task t:
1:s TT. s. V. I:s = [u. Ur..). Vt .,.
The orthogonal projection Eq. (10) guarantees that both the left- and right-singular vectors of A,, representing task- specific directions, are orthogonal to the subspace spanned by the common directions (given by 111.)).
4.2. Isotropic Merging in Common and Task-Specific Subspaces
The effectiveness of Iso-C depends on how well the com- mon subspace- identified by the dominant singular vectors of ATA approximates the subspaces of the individual tasks. The approximation егтor arises from how these tasks in- teract when summed. The top singular directions of ATA capture only the dominant common variations, while singu- lar vectors associated with near-zero singular values provide
Combining common and task-specific matrices. After identifying the & principal vectors for the common sub- space and s = r principal vectors for each task, we now combine the common and task-specific directions by con- 1 I...IU. E guxr and [ Raxr.
catenating them: U_
V. | 1:
6

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Table Iso-CTS achieves state-of-the-ari performance for all backhones on all evaluated scenarios. We present average absolute accuracy and average normalized accuracy (in subscript) in %. The best method in bold and the second- best underlined.
VIT-B/32
VIT-B/16
VIT-1/14
Method
tasks
14 tasks
20 tasks
8 tasks
tasks
20 tasks
8 tasks
14 tasks
20 tasks
Zero-shot
48.3
57.2
56.1
55.3
61.3
59.7
64.7
68.2
65.2
Fine-tuned
92.8
90.9
91.3
94.6
92.8
93.2
95.8
94.3
94.7
Weight Averaging
(72.1
(4.3
(1.0(
(76.6)
(7.5)
(5.3
79.6(83.2)
(6.7(81.1)
71.6[75.6)
Task Arithmetic
70.8(re.s)
(5.3
60.5
(1.4()))))
70.5(75.9)
65.8(70.8)
84.9(88.7)
79.4(64.0)
T4.0(78.1)
TIES
75.1(81.0) 0z
68. (8..8
63.4(60.9)
(84.3))
(16.)
68.2(1a.3)
$6.9(00.7)
(9.5(
(5.7(79.8)
Consensus TA TSV-M
75. (5.0
70.4(77.4)
65.4(72.0)
(9349)339)
74.4(79.9)
69.8(14.9)
(6.3
82 (e6.))
79.0rs3.2)
(5.9
80.1(87.9)
77. (7.1
(930
84.6(91.0)
80.6
93. (3.0
89.2 (9.2(04.4)
87.7(12.5)
Iso-C (Ours)
86.3(92.9) 9)
80.3
(5.5
90.6(05.6)
84 84.8
79.6 4)
(9.2()
80.3(04.5)
87.6 87.6(02.2)
Iso-CTS (Ours)
86 86.2:02.8:
81. (197)
78.1
91.1(96.1) 1)
86.4(92.8)
82.4(68.4)
94. 7,9..9888 8)
91.0
90.1 90.1(94.9:
There no guarantee that the left- and task- lask-specific vectors are orthogonal to each other, as we are only projecting each task matrix onto the common subspace. To reconstruct the final merged matrix, we must orthogonalize U= and V.. Following Gargiulo et al. (2025), we compute the SVD of U. = Pv. Σv. QU, and V. = Pv. Ev, QV. and whiten (Schönemann, 1966):
Orthogonalizatio.
variants of CLIP (Radford et al., 2021) with ViT-B/32, ViT- B/16 and ViT-L/14 as visual encoders (Dosovitskiy et al., 2021). We use the checkpoints fine-tuned on the tasks above, provided in Wang et al. (2024b) (see Appendix D.I for re- sults using TA checkpoints). If not stated otherwise, present the results using the ViT-B/16 visual encoder.
We compare our approaches with the following model merg- ing methods: weight averaging (Wortsman et al., 2022a), Task Arithmetic (Ilharco et al., 2023), HES-Merging (Ya- dav al., 2023), Consensus TA (Wang et al., 2024b) and TSV-M (Gargiulo et al., 2025). We include the results of the zero-shot model and fine-tuned models serving as lower- and upper-bound, respectively. We compare the results based on absolute and normalized accuracy following standard practice (Wang et al., 2024b; Gargiulo et al., 2025).
U. = Pu.QU. V. = Vv=Vv.QV PV.
(11)
Isotropic scaling and reconstruction. Finally, we recon- struet the final merged matrix and apply isotropic merging:
AIso CTs -..V."
(12)
where is obtained by averaging the singular values as- sociated with the vectors selected for both common and task-specific subspaces. Specifically, defining oem diag(E) 1:k E R², the vector of singular values associated with the common subspace identified by UL and Vi:k. and of = diag diag(E,) E R2, with s = - the vector singu- lar values associated with each task-specific subspace U, 1:s and V define the scaling factor
Table presents our main results for multi-task model merg- ing. Isc-CTS achieves sate--ff-the-rr results in all of the settings. Iso-C achieves very similar results to Tso-CTS in the 8 task scenario. However, Iso-CI: significantly outperforms Iso-C when merging 14 and 20 models, with improvements of up to 2.8% in absolute accuracy. This suggests that it is possible to faithfully represent a small number of tasks in the common subspace. However, when the number of tasks increases, it becomes crucial to retain important directions from the task-specific subspaces in order to maximize model merging effectiveness.
(13)
Finally, similar to ISO-C, C, the merged model is defined as:
5.2. LoRA Aadapted vision models
0
T8o-CTS = 0 + a (E)
= 1..,...,,4 L (14)
To evaluate our approaches in low-rank adaptation scenario, we follow the evaluation protocol of KnOTS (Stoica et al., 2025), recent state-of-the--at method for merging LoRA fine-tuned models. We use codebase and checkpoints pro- vided by KnOTS: ViT-B/32 and ViT-L/14 fine-tuned with rank 16 LoRA (Hu et al., 2021) on 8 vision tasks. To adapt our methodologies to low-rank regime, we simply operate on reconstructed task matrices, i.c. AW, BrA, where A,, B, are LoRA matrices for task t. We compare Isc-C and Tso-CTS with TIES and DARE-TIES et 2024)
where a is chosen on held-out validation set.
5. Experimental Results
5.1. Fully fine-tuned vision models
We evaluate our approaches over sets of 8, 14, and 20 datasets, following Wang et al. (2024b). We provide the details of the datasets in Appendix C. We consider three
7

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
0.1
1,0
2-00
A-0.5
3-07
0.00
0.5
Singule Villen
0.6
3=110 (1
0.25
6.4
Cass
MNNST
0.80
PTD
RESISC
0.2
EumSAT
SUNII
0.70
GTSRE
SVEN
0.0
100
200
2000
.000
500
600
700
0.0
0.2
644
0.6
0.8
0.0
0.2
0.4
0.6
3.8
1.0
Singuler Value Indle
8
(a) Spectra of singular values for different (b) Average Subspace Alignment Ratio
(c) Normalized Accuracy Improvement
values of interpolation coefficient (9).
(SARwg) vs. interpolation coefficient (3). (NAI) vs. interpolation coefficient (9).
Figure 4. (a) Interpolating from ATA = 0) towards Atso-c increases the number of preserved components ky (Eq. (6)) denoted by dashed lines. (b) This results in an increased alignment between each task-specific model and merged model measured by SARig. (c) As alignment increases, the performance also improves as predicted based on the strong correlation between these two properties investigated in Section 3.3.
1) makes the spectrum of singular values of AM uniform and
Table 2. Normalized per-task average accuracy. We merge 8 mod- els fine-tuned with LoRA following (Stoica et al., 2025).
Table 3. NLP results using "S-Large-LM-AAaap fine-tuned on tasks from TO mixture. We present average absolute accuracy.
Method
VIT-L/14 VIT-B/32
Method
8 tasks
7 tasks (Yudav ot al., 2023)
(Zhou et al., 2022)
63.7
74.4
TIES
63.7
75.2
Fine-tuned
80.7
85.9
DARE-TIES
63.7
74.7
Weight Averaging
56.4
60.5
KnOTS-TIES -IESS
68.0
78.2
Task Arithmetic
63.8
69.2
KnOTS- CESS-DARR-TEES
63.9
75.6
TIES
62.8
71.9
Fisher Merging
57.7
61.0
Iso-C (Ours)
73.6
83.7
RegMean
69.1
74.3
Iso-CTS (Ours)
73.7
85.3
MaTS
72.5
81.5
Iso-C (Ours)
75.6
83,3
combined with KnOTS or not - and TA.
Iso-CTS (Ours)
75.2
82.8
We present the results in Table 2. Our methods, which are general purpose merging techniques, significantly outper- form KnOTS, which are specifically designed for the LoRA merging. This highlights the versatility of Iso methods.
computational complexity analysis of our approaches.
From Task Arithmetic to Isotropic Merging, We analyze what happens when interpolating between the singular val- ues obtained by Task Arithmetic (TA) and those obtained by Iso-C, C, i.c. the model with the following spectra:
5.3. Language models
We present NLP results following the experimental setup from MaTS (Tam et al., 2023). We use T5-Large-LMs Adapt (Lester et al., 2021) base model (a variant of T5- Large (Raffel et al., 2020)) fine-tuned on subsets of 8 and 7 NLP tasks from TO mixture (Sanh et al., 2022). We compare our approaches with weight averaging, TA, TIES, Fisher Merging (Matena & Raffel, 2021), RegMean (Jin et al., 2023), and MaTS (Tam al., 2023).
Σg = -1- + 3E C+
(15)
where 3 is an interpolation coefficient. Firstly, Figure 4a presents the change in singular values spectrum as we in- terpolate towards Aiso-c (8 - 1). The skewed spectrum achieved by Task Arithmetic becomes isotropic, i.e. the scaling factor is equal along all of the singular directions. In Figure 4b we observe a steady increase in alignment between task- aasciscccific and merged models as measured by SARau (Eq. (5)). and Figure 4c shows that as alignment in- creases (with 3 + 1), the performance of the merged model improves across all tasks. These results are consistent with our findings from Section 3.3 that show strong correlation between alignment and the performance of the final model.
We present the results in Table 3. Both Iso-C and Iso-CTS significantly outperform the competing ap- proaches, which highlights the versatility of our proposed methods. We observe that Iso-C CTS achieves very similar results to Iso-C C suggesting that the common space cap- tures all the directions necessary to reliably represent these 7 and 8 NLP tasks.
The impact of singular directions on performance. We analyze which singular directions contribute to the improve- ment of individual tasks. We truncate the flattened spectrum of Iso-C, CC keeping the & directions associated with the left- most singular values, i.e. a; a for S k and o;
5.4. Analysis and Ablations
All the experiments in this Section are conducted on fully fine-tuned ViT-B/16 models. In Appendix we provide the
8

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
1.0
1.0
0.8
0.9
Accaracy
NAI
0.2
RESISC45
TA
EuroSAT GTSRB
SUN397
Ico-C
SVHN
1co-CTS
0.0
0.4
0.8
20
Fraction of retained componeute k/r from Iso
Number of tasks
Number of tasks
(a) Normalized Accuracy Improvement (NAI) of a model created by retaining & components of Iso-C (associated with top-k singular vectors from Aux).
(b) Average Subspace Aligument Ra-
(c) Distribution of accuracies of the tios (SARwg)) between merged and task- merged models for varying sets of specific models for varying sets of tasks. tasks.
Figure (a) The directions associated with the least significant singular values ÂTA have a minor contribution to the performance so-C model. (h) Task-specific directions introduced in lso-CTS improve the Average Subspace Alignment Ratio (SARay) between task-specific models and the merged model compared to Iso- C. which uses only a common subspace. (c) Higher alignment translates to higher accuracy of Iso-CTS with respect Iso-C.
racy and the fraction of subspace assigned for the com- mon subspace ($) when merging tasks. When & 1 Iso-CTS is equivalent Iso-C and suffers 2.8% drop in accuracy from the maximum. The optimal fraction of common subspace
Tso-CT3
0.82
Tsa-C
Acounry
0.81
0.8, and we this default value for Tso-CTS across all settings. Moreover, note that Iso-CTS is quite robust to the selection of this hyperparam- eter - atyy-anyll E E (0.0, 1.0) offers performance improvement over Iso-C while the performance for & E [0.5, 0.9) varies by less than 0.5% from the optimal one.
0.80
0.2
0.8
Relative size the common subepace
Figure 6. Iso-CTS robust the selected size of the common subspace as any value leads to improvement over Iso-C. C. These results for the 20-task scenario.
6. Conclusion
for k. Note that the leftmost & directions are the ones associated with the highest singular values of ÂTA- We plot the task-wise Normalized Accuracy Improvement (NAI, Eq. (4)) for varying k in Figure 5a. We observe that the first few directions are responsible for rapid improvement on several tasks. Notably, these tasks belong to the aligned groups identified Section 3.3 such as (MNIST, SVHN, GTSRB) and AA, RESISC45). Moreover, the directions asso- ciated with the least significant singular values of ÂTa have negligible contribution to the performance. This supports intuition for replacing less significant common direc- tions with task-specific components in Iso-CTS (see Sec- I Iso-CTS achieves higher Average Subspace Alignment Ratio (SARavg Eq. (5)) than Iso-C. Most importantly, Figure 5c shows that thanks to the addition of task-specific directions, ISO-CTS achieves better performance across tasks.
In this work, we introduced an isotropic model merging framework that enhances alignment between task-specific and merged model subspaces significantly improve the multi-tass performance of the final merged model. We pro- posed Tso-C, which leverages Singular Value Decomposi- tion to equalize singular values and create a more balanced representation across tasks, and ESo-CTS, which further incorporates task-specific directions retain unique task features while preserving shared knowledge. Iso-CTS achieves state-of-the-ard results across multiple model scales and task sets, demonstrating that subspace alignment is critical factor in effective model merging. These findings provide new insights into model merging and pave the way for the future development of more effective techniques to combine the knowledge of multiple models.
tion 4.2). Figure 5b shows that
Limitations. The common subspace is determined by Task Arithmetic, which can be suboptimal, and better meth- ods could be developed. Although the proposed methods achieve state-of-the-a results in the LoRA merging sce- nario, they could be adapted to leverage the low-rank struc- ture of task matrices further improve the performance and efficiency.
Size of the common subspace for Iso-CTS. While Tso-C operates only in the common subspace, Tsc-CTS enhances it with task-specific subspaces. Therefore, we must select the size of the common subspace & (and con- sequently the size of each task-specific subspace given by )))) Figure plots the relationship between accu-

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Acknowledgements
Vedaldi, A. Describing textures in the wild. In CVPR, 2014.
Daniel Marezak supported by National Centre of Science (NCN, Poland) Grant No. 2021/43///ST6/02448. This work was supported by Horizon Europe Programme under GA no. 101 101127, project "ELIAS: Buropean Lighthouse of AI for Sustainability". Simone Magistri acknowledges travel support from ELIAS (GA no 101120)37). We acknowl- financed
Clanuwat, T., Bober-Irizar, M., Kitamoto, A., Lamb, A.,
Yamamoto, K., and Ha, D. Deep Learning for Classical Japanese Literature. arXiv preprint arXiv: 1607.06450 2018.
Coates, A., Ng. A., and Lee, H. An Analysis of Single- Layer Networks Unsupervised Feature Learning. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics. JMLR Workshop and Conference Proceedings, 2011.
edge the Spanish project by MCIN/AEE/11. 13039/501 10001 1033 and FEDER, and Funded by the European Union ELLIOT project. Bartlomiej Twardowski acknowledges the grant RYC2021-0327654 and National Centre of Science (NCN, Poland) Grant No. 2023/5 1/22STT/D//6//02246.. Andrew D. Bagdanov acknowl- edges funding support from the Italian national project "Collaborative Explainable neuro-symbolic AI for Decision Support Assistant", CAI4DSA, CUP B B13C23005640006
Cohen, G., Afshar, S., Tapson, J., and van Schaik, A. EMNIST: Extending MNIST to handwritten letters. In IJCNN, 2017.
Daheim, N., Möllenhoff, T., Ponti, E. M., Gurevych, I., and Khan, M. E. Model merging by uncertainty- based gradient matching. In ICIR, 2024.
Impact Statement
This paper aims to advance the field of Machine Learning, specifically the subfield focused merging models fine- tuned on different tasks to create more effective multi- task model. With the growing popularity of deep learning, increasingly powerful open-source models are becoming widely available and are being adopted in both research and industry. Advances model merging could enhance the flexibility of utilizing these models by providing efficient way to combine their specialized capabilities. Beyond this, our paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
Davari, M.-. and Belilovsky, E. Model breadcrumbs: Scal- ing multi-task model merging with sparse masks. ECCV, 2024.
Denton, E. L., Zaremba, W., Bruna, J., LeCun, Y., and Fer- gus, R. Exploiting linear structure within convolutional networks for efficient evaluation. In NeurIPS, 2014.
Dosovitskiy, A., Beyer, L.,, Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In ICI.R, 2021.
Du, G., Lee, J., Li, J., Jiang, R., Guo, Y., Yu, S., Liu, H., Goh, S. K., Tang, H.-K., He, D., and Zhang, M. Parameter competition balancing for model merging. In NeurIPS, 2024.
References
Ba, I.,, Kiros, R., and Hinton, E. Layer normalization. arXiv preprint arXiv: 1607.06450, 2016.
Bossard, L., Guillaumin, M., and Van Gool, L. Food- 101 Mining Discriminative Components with Random Forests. In ECCV, 2014.
Gargiulo, A. A., Crisostomi, D., Bucarelli, M. S., Scar- dapane, S., Silvestri, F., and Rodolà, E. Task singular vectors: Reducing task interference in model merging. In CVPR, 2025.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. ECCV, 2020.
Goodfellow, I. J., Erhan, D.,, Carrier, P. L., Courville, A., Mirza, M., Hamner, B., Cukierski, W., Tang, Y., Thaler, D., Lee, D.-H., Zhou, Y., Ramaiah, C., Feng, F., Li, R., Wang, X., Athanasakis, D., Shawe-Taylor, J., Mi- lakov, M., Park, J., Ionescu, R., Popescu, M., Grozea, C., Bergstra, J., Xie, J., Romaszko, I.., Xu, B., Chuang, Z.., and Bengio, Y. Challenges Representation Learning: A Report Three Machine Learning Contests. Neural Networks, 2013.
Caron, M., Touvron, H., Misra, I., Jegou, H., Mairal, J., Bojanowski, P., and Joulin, A. Emerging properties in self-supervised vision transformers. ICCV, 2021.
Cheng, G., Han, J.. and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proceed- ings of the IEEE, 2017.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land
Cimpoi, M., Maji, S., Kokkinos, I,, Mohamed, S., and
10

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
use and land cover classification. Journal of Selected Top- ics in Applied Earth Observations and Remote Sensing, 2019.
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin-merging: Dynamic integration of modular expertise in model merging. NeurlPS, 2024.
Hu, J. E.,, Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., and Chen, W. Lora: Low-rank adaptation of large language models. ICLR, 2021.
Marezak, D., Twardowski, B., Trzcinski, T., and Cygert, S. MagMax: Leveraging Model Merging for Seamless Continual Learning. In ECCV, 2024.
Huang, C., Ye, P., Chen, T., He, T., Yue, X., and Ouyang, W. Emr-merging: Tuning-free free reeh-performacce model merging. NeurlPS, 2024.
Matena, M. and Raffel, C. Merging models with fisher- weighted averaging. In NeurlPS, 2021.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., and Ng, A. Y. Reading digits natural images with unsupervised feature learning. In NeurlPS Workshops, 2011.
Ilharco, G., Wortsman, M., Gadre, S. Y. Song. S., Hajishirzi, H., Kornblith, S., Farhadi, A., and Schmidt, L.. Patching open-vocabulary models by interpolating weights. In NeurIPS, 2022.
Nilsback, M.-E. and Zisserman, A. Automated Flower Classification over a Large Number of Classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, 2008
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In ICLR, 2023.
Ortiz-Jiménez, G., Favero, A., and Frossard, P. Task arith- metic in the tangent space: Improved editing of pre- trained models. In NeurlPS, 2023.
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In ICLR, 2023.
Parkhi, O. M., Vedaldi, A., Zisserman, A., and Jawahar, V. Cats and dogs. In CVPR, 2012.
Kim, Y., Park, E., Yoo, S., Choi, T., Yang, L., and Shin, D. Compression of deep convolutional neural networks for fast and low power mobile applications. In ICLR, 2016.
Po, R., Yang, G., Aberman, K., and Wetzstein, G. Orthog- onal adaptation for modular customization of diffusion models. In CVPR, 2024.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3D Object representations for lme-graines categorization. In ICCV Workshops, 2013.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language supervision. In ICML, 2021.
Krizhevsky, A. and Hinton, G. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009. URL https:://// /www. cs www.oto.oooooooooooo "xriz/leernine features-eeeeeeeeeee 2009-0000000 TR.pdf.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of machine learning research, 2020.
Lecun, Y., Bottou, I.., Bengio, Y., and Haffner, P. Gradient- based learning applied to document recognition. Proceed- ings of the IEEE, 1998.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L., Alyafeai, Z.,, Chaffin, A., Stiegler, A., Scao, T. L.., Raja, A., et al. Multitask prompted training enables zero-shot task generalization. ICLR, 2022.
Lee, C., Choi, J., Lee, C., Kim, D., and Hong, S. Adarank: Adaptive rank pruning for enhanced model merging. arXiv preprint arXiv: 2503.22178, 2025.
Lester, B., Al-Rfou, R., and Constant, N. The power of scale for pframeter-efficient prompt tuning. EMNLP, 2021.
Schönemann, P. H. A generalized solution of the orthogonal procrustes problem. Psychometrika, 1966.
Li, W., Peng, Y., Zhang, M., Ding, L., Hu, H., and Shen, L. Deep model fusion: A survey, arXiv preprint arXiv: 2309. /599, 2023.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
Lingam, V., Tejaswi, A., Vavre, A., Shetty, A., Gudur, G. K., Ghosh, J., Dimakis, A., Choi, E., Bojchevski, A., and Sanghavi, S. SVFT: parameter-efffiient fine-tuning with singular vectors. CORR, abs/2405.1 19597, 2024.
Stallkamp, J., Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: a multi-class classification competition. IJCNN 2011.

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Stoica, G., Ramesh, P., Ecsedi, B., Choshen, L.., and Hoff- man, J. Model merging with SVD tie the Knots. In ICLR, 2025.
Zhou, J., Lin, Z., Zheng, Y., Li, J., and Yang. Z. Not all tasks are born equal: Understanding zero-shot generalization. In ICLR, 2022.
Tam, D., Bansal, M., and Raffel, C. Merging by matching models in task subspaces. TMLR, 2023.
Vasudevan, V. and Ramakrishna, M. A hierarchical singular value decomposition algorithm for low rank matrices. arXiv preprint arXiv: 1710.02812, 2017.
Veeling, B. S., Linmans, J., Winkens, J., Cohen, T., and Welling, M. Rotation Equivariant CNNs for Digital Pathology. In MICCAI, 2018.
Wang, H.. Xiao, Z., Li, Y., Wang, S., Chen, G., and Chen, Y. Milora: Harnessing minor singular compo- nents for parameter-efficien LLM finetuning. CoRR, abs/2406. 09044, 2024a.
Wang, K., Dimitriadis, N., Ortiz-Jiméned: G., Fleuret, F., and Frossard, P. Localizing task information for improved model merging and compression. In ICML, 2024b.
Wortsman, M., Ilharco, G., Gadre, S. Y., Roelofs, R., Gontij-Lopes, R., Morcos, A. S., Namkoong, H., Farhadi, A., Carmon, Y., Kornblith, S., al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022a.
Wortsman, M., Ilharco, G., Kim, W., Li, M., Kornblith, S., Roclofs, R., Lopes, R. G., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. In CVPR, 2022b.
Xiao, H., Rasul, K., and Vollgraf, R. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv: 1708.07 077477 2017.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring large collection of scene categories, IJCV, 2016.
Yadav, P., Tam, D., Choshen, I.., Raffel, C., and Bansal, M. TIES-merging Resolving interference when merging models. In NeurIPS, 2023.
Yang, E., Shen, L., Wang, Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. ICML, 2024.
Yu, L., Yu, B., Yu, H., Huang, F., and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as a free lunch. In ICML, 2024.
Zhai, X., Mustafa, B., Kolesnikov, A., and Beyer, L. Sig- moid loss for language image pre-training. ICCV, 2023.
12

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
A. Theoretical properties of Iso-C
In this Appendix, we discuss the theoretical properties of Iso CC by explicitly showing the connection between spectral skewness and the increased subspace dimensionality hy the merged model achieved by Iso-C, which leads to a higher Subspace Alignment Ratio (SAR). Moreover, we explain why the increased SAR reduces inter-task interference. Finally, we highlight the limitations of so--C that lead to the development of Iso-CTS.
A.1. Spectral skewness and the definition of ky
In this Section, we show that the number of dominant components ky of the merged model AM (see Equation (6)) is directly influenced by the skewness of its singular value spectrum. Using the singular value decomposition (SVD), let Am UEVT where Σ diag(a1....... Ør). 0r)) By the definition of Frobenius norm:
A
Σ 1=1
of
A - HLM
Σ o
i=k+1
Hence, the relative approximation error becomes:
Am II A
Σ 2i=k+1 of Ei o7
AM
Accordingly, AM can be defined in terms of singular values:
Ei of Ei=0 of s
lM k:
min
This formulation explicitly shows how the skewness of the spectrum {a, controls kv. When AM has skewed spectrum (e.g. of >> 2 oi), a small AM is sufficient to meet the error bound. This explains why Task Arithmetic ATA (3 = 0 in Figure 4a) - which has a skewed spectrum yields smaller Âдд than Iso-C, whose flatter spectrum leads to larger kis0-2- Therefore, expressing KM directly in terms of singular values highlights the link between the spectral skewness and subspace dimensionality.
A.2. Iso-C C increases Subspace Alignment Ratio (SAR)
In this Section, we formally show how Tso-C increases Subspace Alignment Ratio (SAR) by expanding the effective subspace dimensionality of the merged model - from ÂTΑ in Task Arithmetic to kiso-c in Iso-C.
The rank Am defines the effective rank of the subspace identified by the merged model and is directly determined directly by its spectrum (as discussed Appendix A.1). Let NTA be the effective rank of ATA, and define
(u1, .., MKr
as the orthonormal basis formed by those ÂYTA singular vectors. Flattening the spectrum of ATA (Figure 4a), yields A A with effective rank Krao-c > KTA (as discussed in Appendix A. This flattening modifies only the singular values of TA, leaving the singular vectors unchanged. Therefore, the original subspace T is contained within the larger subspace spanned by the top singular vectors of A Aiso-c, defined as:
(u1,.. Ukno W.
Thus, by construction, we have C
For simplicity, let I1 IIMAA,TA TA and II, -R c. Tso-c denote the projection operators onto the subspaces spanned by T and I, respectively. Since T I. for any matrix A, it holds that:
A.lle
I. SAR(A,, Arso krso-c),
SAR(A, ATASkTA)
<
(16)
|
13

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
This inequality holds because by definition:
I |
Σ ATA
Lim1 E, 1
KOD
(1g 1 + E
Misoie
|, 1
|
1
|
where ( denotes the j-th column of A,-
The equality Equation (16) holds only if the additional vectors added to the basis T that is (Ukra+ Miisonc - are orthogonal to each A or, equivalently, if they do not intersect the column space of A, (i.e. its left singular vectors).
Hence, in general a lower ky yields smaller or equal SAR than a larger KM-- However, our empirical findings show that enriching the basis 7 with singular vectors corresponding to smaller singular values original task arithmetic spectrum (i.e. {uerx+1,.... (kzasee) consistently increases the alignment ratio (Figure 4b), implying that these vectors are relevant for representing each task matrix A and not orthogonal to its left singular vectors.
This analysis formally supports the claim that increasing the effective rank kM of the merged matrix achieved by spectrum flattening in Iso-C- leads to higher Subspace Alignment Ratio.
A.3. Iso-C mitigates inter-task interference
Iso-C increases the Subspace Alignment Ratio (SAR), which quantifies how well the principal directions of task matrix align with the principal directions of the merged model. In this Section, we demonstrate how higher SAR contributes to mitigate inter-task interference by analyzing the relationship between subspace alignment and changes in internal activations following merging. Specifically, we define the interference as the degradation in a task's internal representation due to merging - that is, the deviation between the activations of the merged model and those of the corresponding single-task fine-tuned model. Intuitively, we can minimize the task interference by ensuring that the internal representations of task j remain stable after merging.
Let Og be the pre-trained weights for a layer 1. Define the task matrix Aj 0, 0o and the merged task matrix AM for the layer| Then, for an input " we desire that the post-merging activation n.))) = (0g + adm)) with a chosen on a validation set, be close to the task- speciiicciiic activation h. = (Ou + و Hence, we can quantify the interference as:
11 h 11 |I(A, adM) !1) A aAmll 11
(17)
To show that the interference is lower when the Subspace Alignment Ratio (SAR) between D, and AM is higher, decompose Aj into components aligned with and orthogonal to Am:
A, -!. + + where
A!- = Ie
A = IT
(18)
and M is the projection matrix onto the subspace spanned by the top Ay left-singular vectors of Am (see Equation (6) for the definition of kM). The Subspace Alignment Ratio is then:
||
[
SAR(AJ, AMi KM)
(19)
|Ail|
||A) + +
Similarly, decomposing AM into |!ل and A and substituting Equation (18) in Equation (17), the interference becomes:
||A, - aAmll إإا| aلll + D ال ||ر!ا aAll Attl,
(20)
since kM minimizes the approximation error of AM. leading to
A || x 0.
If the SAR defined in Equation (19) is close to 1, then || is small, so the interference in Equation (20) mainly depends on ||ا| -ل Conversely, SAR is near zero, the large orthogonal component A increases the overall interference, regardless of the choice of a. Even with an optimal a chosen via validation, interference cannot reduced below the norm of the orthogonal component.
14

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Iso-C increases the SAR of A with the merged model bringing it close I, as shown in the paper - by flattening the singular values. Thus, the optimal can adjust the merged model such that interference is minimized. In contrast, Task Arithmetic (TA), with SAR varying across tasks, exhibits interference that cannot be reduced below the norm of the orthogonal component. We experimentally evaluate that the interference is lower for Iso-C than TA in Appendix D.2.
A.4. Limitations of Iso-C that motivate Iso-CTS
This Section details the limitations of Iso-C that motivate the development of Iso-CTS. Specifically, Iso-C relies the singular vectors obtained through Task Arithmetic to perform model merging. As a result, it tends to underrepresent tasks whose dominant directions have lower intensity compared to the majority, particularly when those directions are orthogonal to the shared (common) directions. This limitation becomes increasingly pronounced as the number and diversity of tasks increase (see Section 4.2).
To make this limitation explicit, we formalize the computation via SVD of the first left singular vector in Task Arithmetic, used by Iso-C, as the variance maximization problem:
T || u EAAT u + T
T Σ D,AD) 1,=1,14
U1 = arg max ||l||=|
If a particular task A has dominant directions with significantly lower intensity compared to the other tasks (i.e. lower Frobenius Norm), then its individual contributions ر to the total variance becomes smaller. Similarly, cross terms involving A, will also be comparatively small. Therefore, task j explicitly contributes less to the maximized variance captured by the first principal singular direction.
Moreover, the directions of A are orthogonal or nearly orthogonal to u1, (i.e. uI TTA,, or not at all along this principal direction. Similar considerations apply to subsequent singular vectors ug.... Wg defining the common subspace. Finally, the number of tasks T increases and tasks become more diverse, it becomes increasingly likely that tasks with distinct but smaller-magnitude directions will be underrepresented or absent in the dominant singular directions identified by the task arithmetic decomposition.
A3 = 0), task j contributes minimally
The goal of Iso-CTS to address this limitation by incorporating orthogonal directions that are overlooked by the Task Arithmetic spectrum. This strategy yields the greatest improvements settings with a large number of diverse tasks, as shown in our experimental results.
B. Computational complexity analysis
In this Section, we analyze the computational complexity of Iso-C and Iso-CTS and compare it with that of our main competitor, TSV-M (Gargiulo et al., 2025).
Let A, E _ and let T' and L be the number of tasks and network layers, respectively. For simplicity, assume that each layer consists of a single squared n x matrix.
In our analysis, we focus on the number of SVD performed by each algorithm, as this is by far the most costly component of each algorithm. The complexity of single SVD on A, E _g*x is equal to O (Vasudevan & Ramakrishna, 2017). Below, we detail the total computational complexity for each merging method:
Iso-C performs a single SVD on ATa per layer, with total complexity:
( = (TT3))
Iso I performs:
One SVD on ATA per layer (lines 2-3, Algorithm 2) with complexity O(L)²)
One SVD on each A., for all I tasks and each of the L layers (line 5, Algorithm 2), with complexity O(
- Two SVDs on two matrices U., V, E Raxn per layer (line 11, Algorithm 2), with complexity of O(2mn²).
Therefore, the total complexity equals:
(IsO-CTS) OIn + TIn² + 2Tm + (3T++))L O
15

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
TSV-M (Gargiulo al., 2025) performs:
'' SVDs per layer each task matrix (line 1, Alg. from Gargiulo et al. (2025)): O(TLn))
Two additional SVDs per layer (lines 10-11, Alg.1 from Gargiulo al. (2025)): ((2mn²)
Yielding the total complexity:
O(TSV) = O(TLn O(T -(())))))==) O
While Iso-CTS and TSV-M share the same asymptotic complexity, Iso-CTS incurs slightly more overhead due to the SVD on ΑTA (lines 2-3, Algorithm 2). Both methods can be further optimized by computing Truncated SVDs for Tso-CTS and TSV-M, since only a few components are retained. This reduces the complexity for both approaches. Iso-C the most computationally efficient algorithm its complexity constant with respect number of task T.
C. Experimental details
In this Appendix, we provide the dataset and implementation details used to carry out the experiments presented in the paper.
C.1. Datasets
The 8-dataset benchmark consists of: Cars (Krause al., 2013), DTD (Cimpoi et al., 2014), EuroSAT (Helber et al., 2019), GTSRB (Stallkamp et al., 2011] 1), MNIST (Lecun et al., 1998), RESISC45 (Cheng et al., 2017), SUN397 (Xiao et al., 2016), and SVHN (Netzer et al., 2011).
The 14-dataset benchmark builds on the preceding one, incorporating six additional datasets: CIFARI00 (Krizhevsky & Hinton, 2009), STL10 (Coates et al., 2011), 1), Flowers 102 (Nilsback & Zisserman, 2008), OxfordIIITPet (Parkhi et al., 2012), PCAM (Veeling et al., 2018), and FER2013 (Goodfellow et al., 2013).
Finally, the 20-dataset benchmark includes the preceding 14 plus the following six: EMNIST (Cohen al., 2017), CIFAR1O (Krizhevsky & Hinton, 2009), Food101 (Bossard et al., 2014), FashionMNIST (Xiao et al., 2017), Ren- deredSST2 (Socher et al., 2013), and KMNIST (Clanuwat et al., 2018).
C.2. Implementation details
Our method relies on SVD, which is defined for two-dimensional matrices A E R However, some weights of the neural networks are represented by vectors 8 E R", e.g. bias vectors and parameters of layer normalization (Ba et al., 2016), Therefore, following Gargiulo et al. (2025), we apply simple averaging to combine these parameters.
D. Additional experiments
In this Appendix, present additional experiments that complement the main paper, including comparisons with new vision baselines using Task Arithmetic model checkpoints (Ilharco et al., 2023) for evaluation. We empirically assess the reduced interference of Iso-C compared to Task Arithmetic and analyze the impact of the scaling factor a our approaches. Finally, we present an ablation study showing what happens when spectrum flattening is applied to each task model individually.
D.I. Additional vision baselines
In this Section we provide results with additional methods: Fisher Merging (Matena & Raffel, 2021), RegMean (Jin et al., 2023), PCB (Du et al., 2024), MaTS (Tam et al., 2023) and CART (Lee et al., 2025). These methods were originally evaluated on checkpoints from Task Arithmetic (Ilharco et al., 2023) provided for 8 tasks on ViT-B/32 and ViT-L/14. We follow this experimental protocol with Iso-C and Iso-CTS and present the results in Table 4. Iso-CTS sill achieves state-of-the-an performance followed by Iso-C. Note that the results differ from Table I where used checkpoints from Consensus Merging (Wang al.. 2024b).
16

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Table Additional baselines for merging ViT-B/32 and ViT-I/14 tasks. We report absolute accuracy.
Method
VIT-B/32
VIT-L/14
Zero-shot Fine-tuned
48.3
64.7
90.5
94.2
Task Arithmetic
70.5
84.6
Fisher Merging
68.3
83.7
RegMean
71.8
82.2
PCB
76.3
87.5
MaTS CART
82.6
90.2
83.0
90.8
Iso-C (Ours)
84.1
92.5
Iso-CTS (Ours)
84.3
93.0
Final Aclivation Distances: Izo
Task Arilhmetic Merged Modals
0.30
1
0.25
0.20
Mean L1 Dietanco
0.15
0.10
0.05
0.00
Car
DTD
GTSBB
MNIST
RRSISCH
SVHN
SUN
Datanets
Figure 7. Mean LI distance between the final embeddings task-specific models and the merged one for Iso-C and TA. We used VIT-B/16 model.
D.2. Interference quantification
In this Section, experimentally show that merging interference (defined in Appendix A.3) is lower when merging is performed with Iso-C than with TA. Following Yang al. (2024), measure the interference LI distance between the final embeddings of task-specific models and merged one. In Figure 7 we present the results for merging 8 tasks on ViT-B/16. We observe that the interference is lower for Iso-C C than for TA highlighting the effectiveness of Iso-C C in reducing interference when merging models.
D.3. Selection of scaling coefficient
In Figure 10, we present the relationship between the validation accuracy and scaling factor a. We observe that TA is very sensitive to the selection of a, which potentially may require a more fine-grained search. On the other hand, both Iso-C and Iso-CTS are more robust to a selection, resembling the task-specific models. For reproducibilityy In Table 5, we provide the optimal value chosen on the held-out validation for each model and number of tasks.
D.4. Importance of isotropic scaling Iso-CTS
In this Section we ablate the need for isotropic scaling in Iso-CTS. We present the comparison of the performance of Iso-CTS with and without isotropic scaling (Equation 13) in Figure 8. We observe that isotropic scaling is indeed
17

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Table Optimal value chosen held-out validation for different model types and numbers of tasks for 1s0-C and 1so-CTS.
Method
Model
8 tasks
14 tasks
20 tasks
ViT/32-B
1.30
1.00
0.90
Iso-C
ViT/16-B
1.40
1.00
0.80
ViT/14-L
1.50
1.30
1.00
ViT/32-B
1.50
1.20
1.10
1so-CT3 CTS
ViT/16-B
1.60
1.20
1.10
ViT/14-L
1.90
1.50
1.20
Performance of Iso-CTS with and without isotropic scaling
0.80
Accuracy
0.75
0.70
Iso-CTS (varying k)
Iso-CTS (optimal k))
Iso-CTS (varying &) w/o line 12
Iso-C (optimal k)
0.0
0.2
0.4
0.6
0.8
1.0
Relative size of the common subspace
Figure Performance of Iso-CTS with and without isotropic scaling (Eq.13). Isotropic scaling crucial component of Isc-CTS. Results for merging 20 tasks with ViT-B/16.
crucial component of Tso-CTS S as long as common subspace exists. When only task-specific subspaces are in use ( 0), isotropic scaling does make a significant difference. However, the design in Algorithm also plays an important role, especially when the number of merged models increases, leading to up to 2.8% improvement over Iso-C on 20 tasks (see Table 1).
D.5. Applying Iso to individual task matrices
Flattening the skewed spectrum of singular values significantly improves the performance of the merged model, as demonstrated in Section 5.4. One may wonder if this operation might also be effective strategy for improving single-task models. Figure I1 presents the performance of task-specific models in their original form along with their modified versions with singular value spectra of their task matrices flattened (which is equivalent to performing Iso-C C for a single model). We observe a 3.3% drop average performance across tasks. Therefore, the reason for the success of Iso-C lies in its ability mitigate the negative effects of summing task matrices, not in inadvertently improving the original individual task matrices.
E. Additional visualizations
In this Appendix, we provide additional visualizations that could not be included in the main paper due to space constraints. These include the spectra of task matrices, the Subspace Alignment Ratio per layer, and the correlation between Normalized Accuracy Improvement and Subspace Alignment Ratio when using the larger ViT-L/14 model.
E.I. Visualization of task matrix spectra
When visualizing spectra of singular values of task matrices (Figure 1 and Figure 4a), we selected an output projection matrix 0O from layer 4 of ViT/B-16 as an illustrative example. In Figure 12, we present spectra across a variety of layers of VIT/B-16 for the task matrices of task task-specific models, TA, Iso-C and Tso-CTS.
18

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
NAI vs. SARgeg for ViT-L/14
1.0
0.9
*
0.8
0.7
NAI
0.6
TA
Iso-C
0.5
Cars
MNIST
DTD
RESISC45
0.4
EuroSAT GTSRB
SUN397
SVHN
0.3
0.75
0.80
0.85
0.90
0.95
SARaag
Figure 9. Normalized Accuracy Improvement (NAI) vs. Average Subspace Alignment Ratio (SARwg) VIT-L/14.
E.2. Visualization of per layer Subspace Alignment Ratio
In Figure 3a and Figure 4b in the main paper, we presented SARavg - Subspace Alignment Ratio averaged across all the task matrices. In this Section, we present SAR at different depths of the model. Specifically, we calculate SAR between fine-tuned and merged weight matrices and an average of all the matrices for given layer of the ViT-B/16 model. We present the results in Figure 13. We observe that the alignment is higher for sso-C across all layers of the vision transformer. One may expect early layers to be more aligned but find that for both approaches the alignment is similar across the layers.
E.3. Visualization of Normalized Accuracy Improvement versus Subspace Alignment Ratio for VIT-L/14
In Figure 9 we replicate the experiment from Figure 3a (conducted on VÝT-B/16) on ViT-L/14. The observations from the main paper h Normalized Accuracy Improvement strongly correlates with average Subspace Alignment Ratio, and increasing SARavg via merging with Iso-C C leads to better performance.
19

---

No Task Left Behind: Isotropic Model Merging with Common and Tpsk-Specific Subspaces
Cars
DTD
EuroSAT
Validation Accuracy (%)
90
100
100
80
70
60
0.0
0.5
0.0
2.0
0.0
2.0
GTSRB
MNIST
RESISC45
Validation Accuracy (%)
100
100
100
90
80
Individual task vectors
80
TA
60
60
Iso-C
Iso-CTS
70-
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
SUN397
SVIIN
Average
100
Validlation Accuracy (%)
75
80
70
70
60
65
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Figure TA sensitive to the selection a. while both Iso-C Iso-CTS are more robust to selection, resembling the task-specific models. The is chosen based on the best average performance on the validation set across tasks. The bottom right subplot denotes the optimal a for each method (Eq. (3), Eq. (9) and Eq. (14)). The model is ViT-B/16.
20

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Original individual task matriccs A
Iso applied individual task matrices A
Cars
DTD
Validation Accuracy (90
80
75-
70-
60
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
EuroSAT
GTSRB
100
100
Validation Acmiary (%6
70
60
60
40
0.5
1.5
2.0
2.5
3.0
0.5
1.5
2.0
2.5
3.0
MNIST
RESISC45
100
Valddttin Accutacy (%)
70
60
70
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
SUN397
SVHN
80
Validation Accuracy (%)
90
75
80
70
70
60
05-
50
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0.0
0.5
1.0
1.5
2.0
2.5
3.0
0
Figure 11. Validation Accuracy while scaling task matrices with coefficient (Fq. (3) applicd for single task). We observe a performance gap between accuracy original and modified models optimal values a (denoted by square).
21

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Cars
MNIST
TA
DTD
RESISC45
Iso-C
EuroATT
SUN397
Iso-CTS
GTSRB
SVHN
positional_embeeding
proj
rosblocks. (waplocks...ml.... 0.mlp.c.fc
0.15
0.4
0.15
Singolar Value
0.10
0.3
0.10
0.2
0.05
0.05
0.1
0.00
0.00
0.0
50
100
150
200
200
400
600
800
0
200
400
600
800
weight
resblookss.3attmmoout.ppo
resblocks.3.mlp.c...
0.25
0.10
0.25
0.20
0.08
Singllar Valne
0.15
0.06
0.15
0.10
0.04
0.10
0.05
0.02
0.05
0.00
0.00
0.00
200
400
600
800
200
800
200
400
600
800
resblocks.3.mg bebblocs.3.mlp...prr))
www.blocsss..attt..oo jurjpo
resblocks. Tabblocks.7.mll.. 7.l.p.c...lffff
0.10
0.25
0.08
Singular Value
0.20
0.2
0.06
0.15
0.04
0.10
0.1
0.02
0.05
0.0
0.00
0.00
200
400
600
800
200
400
600
800
0
200
400
600
800
resblocks. 11.attn.ahtt in proj weight
resblocks. 11 11.mp.cc c.roj
proj
0.4
0.4
1.00
0.3
Singular Value
0.3
0.75
0.2
0.50
0.1
0.25
0.1
0.0
0.00
0.0
0
200
400
600
800
200
400
600
800
0
200
400
Singular Value Index
Singular Value Index
Singular Value Index
Figure 12. Visualization of singular value spectra of different task matrices for different types of layers in VIT/B-1e 16.
22

---

No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspaces
Subspace Alignment Ratios by Layer for All Datasets
Cars
DTD
1.0
0.8
0.8
0.6
SMR
SAR
0.4
0.2
Alignment with Iso-C
A Alignment with TA
0.0
0.0
Logaz
Lape
Layer
Lapos
Lapre
Layer
Lagor
Layer
Layer
Lapor
Layer in
Lave 11
Laprr
Layer
Laper
Lapoe
Laper
Lapor
Lapor
Layer
Lagor
Lape
Lave to
Layor 12
EuroAAT
GTSRB
1.0
1.0
0.8
0.3
0.6
0.6
SAR
SAR
0.4
0.4
0.2
0.2
0.0
0.0
Lager
Lapor
Layer
Latre
Lapre
Layer
Lagor
Laysr
Layer
Layvy in
Lave 11
Lapor
Eayer
Latoe
Lapoy
Layer
Lapor
Lapor
Layer
Lapo
Lawe to
Lave 12
MNIST
RESISCAB
0.8
0.8
0.6
SAR
SAR
0.4
0.2
0.2
0.0
0.0
Layor 0
Lapor/
Layer
Lapor
Lanor
Layer
Layer
Laysr
Layer
Layor
Layer to
Layer 17
Lanor 0
Layer
Lapor
Lanor
Lawer
Laver
Layar
Laver
Lays
Layer to
Later 12
SVHN
SUN397
0.8
0.8
0.6
SAR
SAR
0.1
0.2
0.2
0.0
0.0
Laww 0
Lany
Layer 2
Laww g
Lany
Lawn
Laver
Lani
laver
Lawe
Loves
Lawa 1f
Lanor 0
Loyrr
Law 2
Lane
Lavr
Lawer
Lawe
Laver
Lawe
Lays
Love to
Late 11
Figure 13. Per layer Subspace Alignment Ratio between fine-tuned and merged weight matrices for ViT-B/16.
23


Reviews:
[{"review_id": "QbEC6yKlNH", "reviewer": "Reviewer_mnUL", "summary": "The paper studies how to improve model merging methods by leveraging the singular value decomposition (SVD) of task matrices, defined as the differences between fine-tuned models' weight matrices and the pre-trained model. The authors first show that merging performance correlates with the alignment between the top eigenspace of task-specific and merged matrices. Building on this insight, they propose *isotropic merging* (ISO-C), which replaces the singular values of merged matrices with a uniform spectrum.  Additionally, they refine the merged matrices by substituting directions associated with small singular values with task-specific eigenspaces orthogonal to the top eigenspace of the merged matrices before flattening the spectrum. These approaches achieve state-of-the-art performance on standard computer vision model merging benchmarks.\n\n## Update after rebuttal\n\nI maintain my positive assessment.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the Reviewer acknowledges the novelty of the proposed method, the contribution towards understanding model merging and clear writing. We thank the Reviewer for the constructive feedback, and below we respond to specific points raised.\n\n> [Minor weakness 1 (MW1)]: *Sec. 3, along with Fig. 2 and 3, lacks details of the experimental setting and the models considered.*\n\nWe used ViT-B/16 and 8 tasks (see Sec. 5.1 for details). We will clarify this in the revised manuscript.\n\n> [MW2]: *$\\rho$ (L195, right) is undefined in the main text.*\n\n$\\rho$ is a Pearson correlation coefficient – defined in the caption of Fig. 3. We will unify the notation to $\\rho_{\\text{TA}}$ and add the definition to the main text.\n\n> [MW3]: *L240-246 repeatedly mention more/less correlated tasks. I think this terminology is vague and should be clarified in terms of alignment.*\n\nWe will clarify this part in the revised manuscript to avoid vagueness:\n\n*However, significant variability in the average alignment ratio across the dataset leads to a lower accuracy improvement for **less aligned tasks** compared to **the tasks belonging to groups of high alignment**. This variability stems from the skewness of the task arithmetic spectrum (Fig. 1), which is concentrated in the first few singular values (which we call top or dominant), favoring **the tasks from the highly aligned groups**.*\n\n> [MW4]: *The claim in L256-258 (right) would benefit from explicit justification.*\n\nWe can formalize the SVD problem for the first left principal singular vector as the variance maximization problem:\n\n$u_1=\\arg\\max_{||u|\\vert=1}||\\Delta_{TA}^Tu||^2=u^T\\left(\\sum_{t=1}^T\\Delta_t\\Delta_t^T\\right)u+u^T(\\sum_{{t,s=1,t\\neq s}}^T\\Delta_t\\Delta_s^T)u$\n\nIf a particular task $\\Delta_j$ has dominant directions with significantly lower intensity compared to the other tasks (i.e. lower Frobenius Norm), then its individual contributions $\\Delta_j \\Delta_j^T$ to the total variance becomes smaller. Similarly, cross terms involving $\\Delta_j$ will also be comparatively small.  Therefore, task $j$ explicitly contributes less to the maximized variance captured by the first principal singular direction.\n\nMoreover, if the directions of $\\Delta_j$ are orthogonal or nearly orthogonal to $u_1$, (i.e. $u_1^T\\Delta_j=0$), task $j$ contributes minimally or not at all along this principal direction. Similar considerations apply to subsequent singular vectors $u_2, \\ldots u_k$, defining the common subspace. Finally, as the number of tasks $T$ increases and tasks become more diverse, it becomes increasingly likely that tasks with distinct but smaller-magnitude directions will be underrepresented or absent in the dominant singular directions identified by the task arithmetic decomposition. This is empirically supported by our results. Iso-CTS provides the most improvement when the number of tasks increases. \n\n> [Comment 1 (C1)]: *Have you analyzed SAR at different depths of the models?*\n\nWe analyze SAR at different depths. For the ViT-B/16 model, we calculate SAR between fine-tuned and merged weight matrices and an average of all the matrices from a given layer. We present the results here: https://imgur.com/a/tLnEoAi. We observe that the alignment is higher for Iso across all layers of the vision transformer. One may expect early layers to be more aligned but we find that for both approaches the alignment is similar across the layers.\n\n> [C2]: *I assume Fig. 3(a) is obtained for a ViT base. How would it look for the larger model?*\n\nYes, we obtain Fig. 3(a) for ViT-B/16. See the Figure for ViT-L/14 here: https://imgur.com/a/3V6xv7T. It closely resembles Fig. 3(a) for ViT-B/16 from the paper.\n\n> [C3]: *I think the paper would be much stronger presenting some results for NLP...*\n\nWe present NLP results following the experimental setup from [1]. We use T5-Large-LM-Adapt base model fine-tuned on tasks from T0 mixture. We consider subsets of 8 and 7 NLP tasks adhering to the setup from Table 1 from [1] and compute an average accuracy of Iso-C and Iso-CTS in these settings:\n\n|**Method**|**8 tasks (Zhou et al., 2022)**|**7 tasks (Yadav et al., 2023)**|\n|-|-|-|\n|Weight Avg.|56.4|60.5|\n|TA|63.8|69.2|\n|TIES|62.8|71.9|\n|Fisher|57.7|61.0|\n|RegMean|69.1|74.3|\n|MaTS|72.5|81.5|\n|**Iso-C**|**75.6**|**83.3**|\n|**Iso-CTS**|_75.2_|_82.8_|\n\nBoth Iso-C and Iso-CTS significantly outperform the competing approaches, which highlights the versatility of our proposed methods. We observe that Iso-CTS achieves slightly worse results than Iso-C. This is consistent with our vision results in which both approaches performed very similarly when merging 8 models. We argue that the common space captures all the directions necessary to reliably represent these 7 and 8 NLP tasks, while task-specific subspaces may became more effective when merging more models.\n\n[1] Tam et al. Merging by Matching Models in Task Parameter Subspaces, TMLR 2024", "claims_and_evidence": "The claims are well supported.", "methods_and_evaluation_criteria": "The paper includes 3 standard model merging benchmarks with 8, 14, and 20 tasks and evaluates 3 CLIP models with VIT base and large encoders, as it is standard in this research area.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design is sound.", "relation_to_broader_scientific_literature": "The paper contributes to the ongoing research on improving model merging and mitigating task interference. Similarly to concurrent studies, it considers SVD decomposition of the weight matrices. The proposed techniques – uniform singular value scaling and selective incorporation of task-specific subspaces – are novel and improve performance over previous techniques across model sizes and the number of merged tasks.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "Other strengths:\n- The proposed method is original and contributes novel insights into model merging.\n- Model merging via weight interpolation is a relatively recent but impactful area. Improving it by reducing task interference is a significant contribution.\n- The paper is clearly written and well-structured.\n\nMinor weaknesses:\n- Sec. 3, along with Fig. 2 and 3, lacks details of the experimental setting and the models considered.\n- $\\rho$ (L195, right) is undefined in the main text.\n- L240-246 repeatedly mention *more/less correlated tasks*. I think this terminology is vague and should be clarified in terms of alignment.\n- The claim in L256-258 (right) would benefit from explicit justification.", "other_comments_or_suggestions": "- Have you analyzed SAR at different depths of the models?\n- I assume Fig. 3(a) is obtained for a ViT base. How would it look for the larger model?\n- Finally, I think the paper would be much stronger presenting some results for NLP as well, as it is standard in the field since model merging is widely relevant beyond computer vision.", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YPRAgQYqmk", "reviewer": "Reviewer_ff9S", "summary": "This paper focuses on bridging the performance gap between the merged and task-specific models. They first show that the subspace alignment of merged and task-specific models correlates with performance improvement. Then, they propose an isotropic merging method to improve the merging performance via flattening the singular values. An extension is proposed to further improve the alignment and performance by considering the task-specific subspaces. Empirical results show that their method consistently outperforms the baselines.\n\n## After rebuttal:\n\nI think most concerns are well addressed after the rebuttal. I particularly like the interesting findings of the reasons that leads to performance gap in TA. I believe those insights are valuable and critical to the community. Theoretical analysis is provided to enhance the understanding. Experimental design and results are also improved.\n\nI have no questions now. Just suggest that the authors summarize all the reviews and appropriately integrate them in the paper, no matter in the main paper or the appendix. I am very glad to raise my score to 4.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the Reviewer appreciates the novelty of the proposed method, the significance of our contribution, and the clear writing. We thank the Reviewer for their constructive feedback, and below we respond to specific points raised.\n\n> [Question 1 (Q1)]: *In Fig.2, the author tries to show...*\n\nProviding Fig. 2(a) along with only the Normalized Accuracy Improvement (NAI) for each task would show that all task vectors $\\Delta_t$ exhibit similar cosine similarities close to zero, yet the NAIs vary significantly (e.g. the DTD task vector is orthogonal to all others but has the lowest NAI). We agree that this alone suggests that mere orthogonality among task vectors does not explain differences in performance.\n\nFig. 2(b) however takes this analysis one step further by examining whether the cosine similarity between each $\\Delta_t$ and the task arithmetic vector $\\Delta_{TA}$ correlates with NAI. Intuitively, one might expect that a higher similarity between $\\Delta_t$ and $\\Delta_{TA}$ would result in a higher NAI, but no clear correlation is found. This reinforces the observation that cosine similarity is not a good predictor of performance improvement. We conducted this analysis because we believe that a key factor in understanding the effectiveness of task arithmetic model merging is to directly compare each task matrix with the *merged* model matrix. Since cosine similarity between task vectors and task arithmetic vector alone does not explain performance gains, we propose analyzing the Subspace Alignment Ratio (SAR) between each individual and merged task matrices, which indeed shows a positive correlation with performance improvement (Fig. 3). In the revised manuscript we will include elements of this discussion to more clearly link the results in Figs. 2 and 3.\n\n> [Q2]: *I may be missing or misunderstanding something, but I have a question regarding the motivation of the proposed method...*\n\nThe subspace alignment ratio $SAR_\\text{avg}(\\Delta_t, \\Delta_{TA})$ quantifies how well a task matrix $\\Delta_t$ is represented by the subspace of the task arithmetic matrix $\\Delta_{TA}$. The subspace dimension, $k_M$ as defined in Eq. (6), is determined by the number of singular vectors required to minimize the reconstruction error in terms of Frobenius Norm. Because the singular value spectrum of $\\Delta_{TA}$ is skewed, only a few singular values are large, leading to a low $k_M$ (see Fig. 4(a) - $\\beta=0.0$ (TA), marked by a vertical red dashed line). Relying on these few singular vectors to represent each task matrix produces a highly variable $SAR_{avg}$ across tasks (Figure 4(b) - $\\beta=0.0$), indicating that some tasks are not well captured by this limited subspace.\n\nThe motivation for \"flattening\" the singular values via Iso-C in Sec. 4.1 is to address this issue. By scaling the singular values, the influence of less dominant singular values increases while that of the dominant ones decreases. This adjustment raises the effective subspace dimensionality $k_M$ (as shown by the vertical dashed lines for $\\beta > 0$ in Figure 4a), resulting in a subspace that better represents all task matrices. Consequently, this leads to a higher $SAR_{avg}$ (Fig. 4(b)) and Normalized Accuracy Improvement (Fig. 4(c)). Thus, the skewness in Fig 1(a) explains the variability in $SAR_{avg}$ (and hence merging performance), and the singular value flattening is introduced as a solution to this limitation.\n\n> [Q3]: *Some important baselines are missing in the experiments, such as Fisher merging, RegMean, and a recent SOTA PCB-merging...*\n\nThank you for pointing out PCB method which is a recent and relevant baseline. Originally, we omitted Fisher Merging and RegMean for brevity as they are outperformed by many recent methods. We additionally include the CART baseline requested by Reviewer **BR4L**. The PCB paper reports the average absolute accuracy for merging 8 tasks across 2 model sizes and we compare these results in the Table below: \n\n|Method|ViT-B/32|ViT-L/14|\n|---|---|---|\n|Fisher|68.3|82.2|\n|RegMean|71.8|83.7|\n|PCB|76.3|87.5|\n|CART|83.0|90.8|\n|**Iso-C**|**86.3**|_94.2_|\n|**Iso-CTS**|_86.2_|**94.7**|\n\nIso-C and Iso-CTS outperform all of the added baselines.\n\n> [Q4]: *What are the values of $k$ in Tab.1?*\n\nWe use $\\frac{k}{r} = 0.8$ as a default for all the experiments (see L408-410, right column), where $r$ is a number of singular values for a given layer. Therefore, $k$ can vary across layers according to the $r$ for each particular layer.\n\n> [Q6]: *...I am curious about the performance of Iso-CTS w/o line 12...*\n\nWe present the comparison of the performance of Iso-CTS and Iso-CTS w/o line 12: https://imgur.com/a/39CrGKJ. We observe that isotropic scaling is indeed a crucial component of Iso-CTS. However, the design in Alg. 2 also plays an important role, especially when the number of merged models increases, leading to up to 2.8% improvement on 20 tasks (see Table 1).", "claims_and_evidence": "I am confused about Fig.2, where the authors propose their motivation. The motivation itself makes sense to me, but Fig.2 is confusing. See questions for details.\n\n---\n\n**After rebuttal:**\n\nI think the claims are clear to me right now and are supported by sufficient evidence.", "methods_and_evaluation_criteria": "I think some important baselines are missing. The motivation of the method is unclear to me as well. See questions for details.", "theoretical_claims": "N/A\n\n---\n\n**After rebuttal:**\n\nThey propose some theoretical analysis, which I believe will strengthen the interpretability of this paper and provide insightful views.", "experimental_designs_or_analyses": "There is a dataset in which the performance is inconsistent. Also, the improvement of Iso-CTS over Iso-C is not sufficiently discussed. See questions for some comments.\n\n---\n\n**After rebuttal:**\n\nConcerns are well addressed.", "relation_to_broader_scientific_literature": "Model merging is a popular method to construct a multi-task model without retraining. However, the performance gap between the task-specific and merge models is critical. This paper first analyzes the key reason why the merged model has a worse performance and then proposes a novel method to address it. I believe this paper can help enhance the understanding of model merging and be applied as an effective method to merge models.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "The overall organization and writing of this paper are good. I also like the findings that the subspace alignment is critical to model merging, which I believe is important to the understanding of model merging.  My main concern is two-fold. First, it lacks some important baselines. Second, there are some potential issues regarding the motivations, the analysis of experimental results, and the effectiveness of the extended method. Please see the questions for details.\n\n---\n\n**After rebuttal:**\n\nI think most concerns are well addressed after the rebuttal. I particularly like the interesting findings of the reasons that leads to performance gap in TA. I believe those insights are valuable and critical to the community. Theoretical analysis is provided to enhance the understanding. Experimental design and results are also improved.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. In Fig.2, the author tries to show that the task vector orthogonality is not the reason for TA performance improvement. However, Fig.2(b) is confusing to me. The x-axis is the similarity between $\\Delta_i$ and $\\Delta_{TA}$. Why do we need to compare these similarities? Fig.2(a) shows the task-vector orthogonality, but 2(b) shows some bad NAI, which is enough to conclude the incorrelation.\n2. I may be missing or misunderstanding something, but I have a question regarding the motivation of the proposed method. In Sec.3.3, the author shows that the merging performance is correlated with $SAR_{avg}$. However, in Sec.4.1, the author uses Fig.1(a) as their motivation to “flatten” the singular values. I wonder what the relationship is between these two motivations. The authors claim that the variability of $SAR_{avg}$ is due to the skewness in Fig.1(a), but I didn't see a clear relationship.\n3. Some important baselines are missing in the experiments, such as Fisher merging, RegMean, and a recent SOTA PCB-merging. Recent literature also competes with them. Is there any reason the authors do not compare their method with those baselines?\n4. What are the values of $k$ in Tab.1?\n5. In Fig.4(c), the performance of SUN397 is worst with $\\beta=0.5$. Though this is the only inconsistency, I am curious if there is any explanation.\n6. In Fig.5(a), the performance of Iso-C improves significantly when $\\beta \\to 1$, while in Fig.6, the improvement of Iso-CTS is marginal compared to Iso-C. Does it mean that the most improvement is due to the isotropic singular values but not the design in Alg.2? I am curious about the performance of Iso-CTS w/o line 12 (i.e., when there are no isotropic singular values).\n\n---\n\nI have no questions now. Just suggest that the authors summarize all the reviews and appropriately integrate them in the paper, no matter in the main paper or the appendix. I am very glad to raise my score to 4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ELY1pjIlXy", "reviewer": "Reviewer_BR4L", "summary": "This paper proposes a novel model merging framework that enhances alignment between subspace of task models and merged model. The framwork includes two algorithms, (1) Iso-C that achives isotropic by flattenning the spectrum to the averaged singular values and (2) Iso-CTS in which lowest spectral components are further replaced by task-specific directions. Experiments on merging 8, 14, and 20 CLIP models demonstrated the effectiveness of this framework.\n\n## update after rebuttal\n\nThank you for the very detailed and complete rebuttal. I appreciate authors' newly added theoretical justification and arguments under responses to Reviewer ff9S. Also, thank you for providing the comparisons to those new baselines. It clearly shows the advantages of Iso-C and Iso-CTS. The new results authors provided for LoRA FT models and also T5 models (under responses to Reviewer mnUL) are also convincing. The new results provided during the rebuttal have greatly improved the quality of the paper and I hope the authors can include these results in the final version. I have updated my scoring to reflect this.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are glad that the Reviewer appreciates the soundness of the experimental protocol and analyses, the effectiveness of the proposed approach, and clear writing. We thank the Reviewer for their comments and we respond below.\n\n> [Weakness 1 (W1)]: *Lacks theoretical justification...*\n\nWe provide a detailed explanation why making merged matrix isotropic increases subspace alignment in the response to Reviewer **ff9S** (section Q2).\n\n>[W2]: *SVD baseline is insufficient.*\n\nThank you for pointing out CART which is a recent and very relevant baseline. We added it (alongside baselines suggested by Reviewer **ff9S**, section Q3) to our main Table for 8 tasks (as these are the only results reported in the CART paper):\n\n||ViT-B/32|ViT-L/14|\n|-|-|-|\n|Fisher|68.3|83.7|\n|RegMean|71.8|82.2|\n|PCB|76.3|87.5|\n|CART|83.0|90.8|\n|**Iso-C**|**86.3**|_94.2_|\n|**Iso-CTS**|_86.2_|**94.7**|\n\nIso-C and Iso-CTS outperform all these methods.\n\n>[W3]: *Experiments on a single model family.*\n\nIn this rebuttal we add NLP experiments highlighting the effectiveness of Iso methods on the T5 -- an encoder-decoder language transformer (see the response to Reviewer **mnUL**, section C3).\n\n>[W4]: *...Evaluating the approach on PEFT (e.g. LoRA)...*\n\nThank you for the suggestion - it helps to emphasize the generalizability of our approach. We follow the evaluation protocol of KnOTS[1], a recent SOTA (ICLR 2025) method for merging LoRA fine-tuned models, tested on 8 vision tasks using ViT-B/32 and ViT-L/14. For comparison, we merge the task-specific LoRA weights - provided by the authors - to the pre-trained models, and then we apply Iso-C and Iso-CTS. Below, we present the average normalized accuracy:\n\n||ViT-B/32|ViT-L/14|\n|-|-|-|\n|KnOTS-TIES|68.0|78.2|\n|KnOTS-DARE|63.9|75.6|\n|**Iso-C**|_74.4_|_89.4_|\n|**Iso-CTS**|**75.0**|**89.6**|\n\nIso-CTS achieves SOTA results in LoRA merging setting. Note that our method is a general purpose merging technique while KnOTS is specifically designed for the LoRA merging. This highlights the versatility of Iso methods.\n\n[1] Stoica et al. Model merging with SVD to tie the Knots, ICLR 2025.\n\n>[W5]: *The methods have not been tested in the NLP domain.*\n\nSee our response to Reviewer **mnUL** (section C3) for NLP results.\n\n>[Comment 1 (C1)]: *...can authors provide a complexity analysis?*\n\nLet $\\Delta_t \\in \\mathbb{R}^{m\\times n}$, with $m\\geq n$ and let $T$ and $L$ be the number of tasks and network layers, respectively. For simplicity, assume that each layer has a single matrix, whose dimensions are $m$ and $n$. In the analysis below, the lines refer to Algorithm 2 in the main paper.\n\n- Iso-CTS: One SVD on $\\Delta_{TA}$ (lines 2-3) whose complexity is $O(mn^2)$ and this is applied to each layer, so the complexity is $O(Lmn^2)$; one SVD on each $\\Delta_{t}, t= 1..T$, for each layer (line 5) so the complexity is $O(LTmn^2)$; finally line 11 requires two SVDs on matrices $U \\in \\mathbb{R}^{m \\times r}$ and $V \\in \\mathbb{R}^{n \\times r}$. Since $m \\geq n$, then $r=n$, the complexity is $O(2Lmn^2)$. The total complexity is:\n$$O(IsoCTS)=O(Lmn^2+LTmn^2 + 2Lmn^2)=O(LTmn^2).$$\n- Iso-C: One SVD on $\\Delta_{TA}$ with complexity:\n$$O(IsoC)=O(Lmn^2).$$\n- TSV (our nearest competitor): $T$ SVDs per layer on each task matrix (line 1 Alg. 1, TSV paper), and two additional SVDs per layer(line 10-11 Alg.1 TSV paper) and thus:\n$$O(TSV)=O(LTmn^2+2Lmn^2)=O(LTmn^2).$$\n\nWhile Iso-CTS and TSV share the same asymptotic complexity, Iso-CTS incurs slightly more overhead due to the SVD on $\\Delta_{TA}$ (lines 2-3). Both methods can be further optimized by computing Truncated SVDs for Iso-CTS (line 7) and TSV (line 1 Alg. 1, TSV paper), since only a few components are retained. This reduces the complexity for both approaches. Iso-C is the most computational efficient algorithm.\n\n>[C2]: *Why Table 1 only reports “average absolute accuracy” and “average normalized accuracy” but not NAI?*\n\nWe report these two metrics to stay consistent with previous literature (Consensus TA, TSV-M). In the revised manuscript we will add a Table reporting NAI.\n\n>[C3]: *The authors should provide more motivation...*\n\nThank you for the suggestion. In the revised manuscript we will put more emphasis on \"no task left behind\" achieved by Iso methods highlighting higher performance improvements for tasks underrepresented in TA.\n\n>[Question 1 (Q1)]: *The claim in Section 3.2...*\n\nYes, each pair of task vectors has a near-zero cosine similarity (Fig. 2a). However, our analysis goes a step further by comparing the cosine similarity between individual task vectors and the task arithmetic vector, demonstrating that this measure alone does not correlate with normalized accuracy improvement (Fig. 2b). Since cosine similarity alone does not explain performance gains, we introduce SAR. Unlike cosine similarity, SAR allows for meaningful differentiation among task matrices by highlighting shared subspaces (Fig. 3b). Additionally, SAR positively correlates with NAI (Fig. 3a).", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The multi-task model merging experimental designs and the metrics for evaluating merged model follow the standard procedure of model merging work. The three analyses experiments in Section 5.3 are sound and valid.", "relation_to_broader_scientific_literature": "Yes. This paper provides an analysis of how an isotropic merged matrix can enhance model merging performance, complementing existing spectral-based merging methods by showing solely modifying singular values can be a powerful approach. This work also analyze individual task performance, offering a fresh evaluation angle in areas such as fairness.", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "### Strengths \n\n- Both Iso-C and Iso-CTS exhibit low hyperparameter sensitivity, as demonstrated in the analysis and appendix.\n- The proposed framework enhances model merging performance as well as fairness.\n- The proposed framework uses subspace alignment ratio as the metric for quantifying subspace similarity.\n- The paper is generally clear, and the overall structure is easy to follow.\n\n### Weaknesses\n\n- Lacks theoretical justification or motivation on why enhances the average subspace alignment ratio by making the merged matrix isotropic.\n- The SVD baseline is insufficient. \n\n[1] Choi, J., Kim, D., Lee, C., & Hong, S. (2024). Revisiting weight averaging for model merging. arXiv preprint arXiv:2412.12153.\n\n- The main experiments are conducted on a single model family. \n- Current results are based on merging fully fine-tuned models. Evaluating the approach on PEFT models (e.g. LoRA) would provide a more complete understanding of methods' capabilities.\n- As acknowledged by the authors in the Limitations section, the methods have not been tested in the NLP domain.", "other_comments_or_suggestions": "- Iso-CTS requires multiple SVD operations, can authors provide a complexity analysis of the proposed method?\n- Why Table 1 only reports “average absolute accuracy” and “average normalized accuracy” but not the proposed NAI?\n- The authors should provide more motivation for the performance gains by Iso-C and Iso-CTS and include supporting numerical results to echo \"no task left behind\". e.g., in Figure 3(a), tasks that were less represented in the TA model (such as Cars, DTD, and SUN397) exhibit greater performance improvement after Iso-C, which aligns with expectations.", "questions_for_authors": "- The claim in Section 3.2 (lines 145–147) is somewhat confusing. TA suspected their effectiveness arises from the cosine similarity between the vectorized representations of the task matrices being close to zero. From Fig2(a), they are indeed close to zero?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O6XdEsIEbU", "reviewer": "Reviewer_38Gq", "summary": "This paper demonstrates that alignment between the individual components of task-specific and merged matrices is strongly correlated with performance improvements over a pre-trained model. Building on this finding, the authors propose an isotropic merging framework that flattens the singular value spectrum of task matrices, thereby enhancing alignment and narrowing the performance gap. Furthermore, they incorporate both common and task-specific subspaces to further optimize alignment and boost performance. The proposed approach achieves state-of-the-art results.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are pleased that the Reviewer appreciates the soundness of our introduced metrics, the simplicity and effectiveness of the proposed approaches, and clear writing. We thank the Reviewer for the comments and we respond below to specific points.\n\n>[Reference 1 (R1)]: *Section 4.2, which discusses retaining components from the common subspace and the orthogonal projection in Equation 10, bears resemblance to the shared subspace optimization concept in DOGE [2]. It is recommended to discuss this.*\n\nThank you for pointing us to this recent reference. We were not aware of DOGE paper at the time of preparing a submission as the preprint appeared two weeks before the deadline. Here we compare this approach with ours:\n\n- *Definition of common/shared subspace:* We define the common subspace as top-k components from sum of individual task matrices. DOGE defines shared as concatenation of top-k components from each task matrix followed by SVD, which resembles the TSV method.\n- *Orthogonal projection:* Both Iso-CTS and DOGE use an idea of orthogonal projection. DOGE uses it on the gradient of $\\Delta$ to restrict the optimization process from changing the shared space. Iso-CTS uses the orthogonal projection on the level of weight matrices to determine the task-specific subspace that is orthogonal to the common subspace.\n\nIso-CTS and DOGE both use ideas of common/shared subspace and orthogonal projection in different ways. Moreover, the results of our approaches are better than DOGE. We believe that this discussion is very significant and we will add a detailed version of it to the revised manuscript.\n\n>[Weakness 1 (W1)]: *Lack of further analysis, such as why SVD can be used for model merging due to the redundancy of parameters caused by fine-tuning. It now seems more like an experimental discovery, and these further analyses would elevate the article to a higher level.*\n\nIt is known that fine-tuning of large pre-trained models results in low-rank parameter update. This observation enables efficient fine-tuning of models using inherently low-rank adaptation techniques such as LoRA. Consequently, the recent TSV paper shows how low-rank approximation of the parameter update matrices, obtained using SVD, can be used to facilitate model merging.\n\nIn our paper, we propose to extend the scope of SVD-based analysis for the purpose of model merging. Most importantly, by introducing the SAR metric we show that SVD can help in understanding the overlap between task-specific and merged matrices. Moreover, we show that by modifying the spectrum of singular values of merged matrix we can increase the alignment between task and merged matrix (see the Response to Reviewer **ff9S** (section Q2) for detailed discussion).\n\n>*[Suggestion 1 (S1)]: Supplement experiments on NLP tasks to verify the generalizability of the method and make the article more complete.*\n\nWe present NLP results in response to Reviewer **mnUL** (section C3). Iso-C and Iso-CTS outperform other baselines across two presented settings.\n\n>[Question 1 (Q1)]: *Why can subspace alignment eliminate conflicts and improve performance?*\n\nConsider the Subspace Alignment Ratio between a task matrix and merged task matrix. SAR quantifies an overlap between the subspaces spanned by dominant singular vectors of these matrices. If the SAR is low, the overlap between the subspaces of these matrices is low and the corresponding singular vectors are close to orthogonal. Therefore, the merged matrix cannot reliably represent the dominant components of task matrix if SAR between them is low. Therefore, low SAR leads to low performance on the corresponding task. Conversely, high SAR indicates high subspace overlap, indicating that the merged matrix can reliably represent the important components of the task matrix and results in high performance.\n\n>[Q2]: *How is the ratio of singular values controlled between the common and task-specific subspaces?*\n\nThe ratio of singular values between the common and task-specific subspaces is controlled by the hyperparameter $k$ that is fixed for all the experiments. $k$ is chosen such that $k/r$ for a single layer is equal to 0.8. The final paragraph of Section 5.3 from the paper contains an analysis of impact of $k$ on performance.", "claims_and_evidence": "The proposed Normalized Accuracy Improvement and Subspace Alignment Ratio are supported by evidence. These metrics provide quantitative validation for the proposed isotropic merging framework, showcasing the alignment improvements and their direct impact on model performance.", "methods_and_evaluation_criteria": "It is intuitive to keep task-specific knowledge in the near-zero singular values part of the common subspace and discard the unimportant part.", "theoretical_claims": "This paper does not conduct theoretical analysis.", "experimental_designs_or_analyses": "The experimental design followed Task Singular Vectors, but was limited to vision tasks.", "relation_to_broader_scientific_literature": "The method is a further improvement of Task Singular Vectors, removing noise through SVD decomposition.\n\n[1] Task Singular Vectors: Reducing Task Interference in Model Merging. arXiv.", "essential_references_not_discussed": "Section 4.2, which discusses retaining components from the common subspace and the orthogonal projection in Equation 10, bears resemblance to the shared subspace optimization concept in DOGE [2]. It is recommended to discuss this.\n\n[2] Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent. arXiv.", "other_strengths_and_weaknesses": "**Strengths**: The article is well-written and clear, with simple and effective methods achieving state-of-the-art results in model merging. The proposed methods and metrics are novel and intuitive.\n\n**Weaknesses**: Lack of further analysis, such as why SVD can be used for model merging due to the redundancy of parameters caused by fine-tuning. It now seems more like an experimental discovery, and these further analyses would elevate the article to a higher level.", "other_comments_or_suggestions": "**Suggestions**: Supplement experiments on NLP tasks to verify the generalizability of the method and make the article more complete.", "questions_for_authors": "1. Why can subspace alignment eliminate conflicts and improve performance?\n2. How is the ratio of singular values controlled between the common and task-specific subspaces?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The work proposes to improve model merging by reducing task interference. The method computes the SVD of the summed task vectors (which are actually per-layer weight matrices) and then flattens the singular values to their average. 

Strong improvements on standard computer vision model merging datasets are shown, yielding a similar performance to the recent Task Singular Vectors (TSV) method. 

Reviewers unanimously liked the paper, but several issues were pointed out:
- Reviewer ff9S, Reviewer BR4L, Reviewer 38Gq found the motivation why flattening the singular values works unclear. 
- Reviewer mnUL, Reviewer BR4L pointed out the lack of comparisons on NLP datasets. 
- Reviewer BR4L pointed out a missing complexity analysis and lack of comparison to another SVD-based baseline. 

The authors addressed all these points in their rebuttal, and all reviewers raised their score to 4. I recommend acceptance of the work.
Decision: Accept (poster)