CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Authors: Wenju Sun
Qingyong Li
Yangliao Geng
Boyang Li
Venue: ICML 2025 poster (2025)
Keywords: Multi-task Learning
Model Merging
Knowledge Conflict
Primary area: general_machine_learning->transfer_multitask_and_metalearning

Abstract: Multi-task model merging offers a promising paradigm for integrating multiple expert models into a unified system without additional training. Existing state-of-the-art techniques, such as Task Arithmetic and its variants, merge models by accumulating task vectors—defined as the parameter differences between pre-trained and fine-tuned models. However, task vector accumulation is often hindered by knowledge conflicts, where conflicting components across different task vectors can lead to performance degradation during the merging process. To address this challenge, we propose **Conflict-Aware Task Merging (CAT Merging)**, a novel training-free framework that selectively trims conflict-prone components from the task vectors. CAT Merging introduces several parameter-specific strategies, including projection for linear weights and masking for scaling and shifting parameters in normalization layers. Extensive experiments on vision and vision-language tasks demonstrate that CAT Merging effectively suppresses knowledge conflicts, achieving average accuracy improvements of up to 4.7% (ViT-B/32) and 2.0% (ViT-L/14) over state-of-the-art methods.

CAT Merging:
A Training-Free Approach for Resolving Conflicts in Model Merging
Wenju Sun
Qingyong Li'
Yangli-ao Geng'
Boyang Li
Abstract
Trinmed dan
hith confler
Multi-task model merging offers a promising paradigm for integrating multiple expert models into a unified model without additional training. Existing state-of-the-art techniques, such as Task Arithmetic and its variants, merge models by ac- cumulating task vectors the parameter differ- ences between pretrained and finetuned models. However, task vector accumulation is often hin- dered by knowledge conflicts, leading to perfor- mance degradation. To address this challenge, we propose Conflict-Aware Task Merging (CAT Merging), novel training-free free framework that selectively trims conflict-prone components from the task vectors. CAT Merging introduces sev- eral parameter-specific strategies, including pro- jection for linear weights and masking for scal- ing and shifting parameters in normalization lay- ers. Extensive experiments on vision, language, and vision- language tasks demonstrate that CAT Merging effectively suppresses knowledge con- flicts, achieving average accuracy improvements of up to 2.5% (ViT-B/32) and 2.0% (ViT-L/14) over Sfate-of-thh-aee methods.
task
Pre-tminee model
Prestriined
model
Tack
Mergod
Tink veelor
taak
Trimmed dos Jow mogairuds
Mtrrpd vector based on magainade
Magnitude based Methods
CAT Merging (ours)
Figure I. Illustration of merging two conflicting task vectors. While masking low-magnitude components (the horizontal com- ponent of Task 2) mitigates some interference, high-magnitude dimensions (the vertical component of Task 1) can still induce significant conflicts In contrast, the proposed CAT Merging proac- tively identifies and trims high-conflict components, stahilizing model merging.
expert models into a single unified model without additional training (Matena Raffel, 2022).
A notable advancement in model merging Task Arithmetic (Ilharco et al., 2023b), which introduces the concept of task vectors-deff
the difference vector between the pre- trained and finetuned models in the parameter space. Task Arithmetic demonstrates that task- knowledge can be effectively integrated into the pretrained model through simple arithmetic operations, such as model merging by adding task vectors the pretrained parameters. Despite its effectiveness, the technique is vulnerable performance degradation caused by knowledge conflict (Sun et al., 2025; Ortiz-Jimenez et al., 2023), or the imbalance and contradic- tion among the accumulated task vectors.
1. Introduction
Fine-tuning pretrained foundation models has become standard paradigm for addressing downstream applications (Brown et al., 2020). However, as the number of applica- tions grows, managing and deploying numerous finetuned models introduces significant costs and operational com- plexity. To address this issue, multi-task model merging has emerged promising solution by consolidating multiple
To address knowledge conflict, prior research has explored enhancing Task Arithmetic by trimming unimportant com- ponents within task vectors. One typical importance metric is the parameter magnitude. For instance, Tres-Megging (Yadav et al., 2023) trims small-magnitud elements in task vectors. PCB PCBMerging (Du et al., 2024) refines this impor- tance measure by considering inter-task correlations. Simi- larly, Twin-Merging (Lu et al., 2024) applies singular value decomposition to task vectors; leveraging singular values as the importance metric, it retains components corresponding
Key Laboratory of Big Data & Artificial Intelligence in Transportation (Ministry of Education). School of Computer Sci- ence and Technology, Beijing Jiaotong University, Beijing, China "College of Computing and Data Science, Nanyang Technolog- ical University, Singapore. Correspondence to: Yangli-ao Geng <gengyla ggengyla@bju..cuucn. enn.. Qingyong Li <liqy @
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
to large singular values.
2022), resolving gradient direction conflicts (Chen al.,
2020; Liu et al., 2021; Javaloy & Valera, 2022; Navon et al.,
Despite the significant progress made by magnitude-based
2022; Guo et al., 2022), preventing over-prioritization of certain tasks (Chen et al., 2018; He et al., 2022; Yang et al., 2023).
methods, they often overlook the risk of merging high- magnitude components that may overwrite task-specific knowledge during integration. As illustrated in Figure 1, methods like Ties-Merging prioritize dimensions with larger magnitudes (e.g., the vertical component of the Task vec- tor), which can unintentionally overwrite critical informa- tion from Task 2. This imbalance skews the merged model towards Task 1, degrading performance on Task 2 and un- dermining the overall multi-task capability. In contrast, the me-ginudd horizontal component of Task carries valu- able information while exerting minimal negative impact on Task 1. Trimming these components is therefore counter- productive. This example highlights the dual objectives of effective model merging: (1) suppressing conflicts where dominant knowledge from one task undermines the perfor- mance of others, and (2) preserving the unique and essential knowledge required by each task.
2.2. Multi-Task Learning through Model Merging
Model merging has emerged as a promising approach for integrating knowledge across different models without re- training. Early techniques like Stochastic Weight Averag- ing (SWA) (Izmailov et al., 2018) averages model weights averaging during the final stages of training to enhance generalization. This concept was further advanced by ap- proaches like SWAD (Cha et al., 2021) and Ensemble of Averages (EoA) (Arpit et al., 2022). Additionally, Ilharco et al. (2023a) empirically demonstrate that parameter av- eraging can effectively combine knowledge from models trained on diverse tasks. Based on this, Fisher-Merging (Matena & Raffel, 2022) applies the Fisher information ma- trix (Fisher, 1925) weigh the averaging process. Similarly, RegMean (Jin al., 2023) formulates model merging the minimization of the distances between the merged model and individual finetuned models in the parameter space.
Motivated by the dual objectives outlined above, we propose Conflict- Task Merging (CAT Merging), feature- centric framework that addresses knowledge conflicts by trimming conflict-prone components from task vectors. Specifically, we focus on feature-level conflicts by analyzing task vector components layer by layer. By adhering to the dual objectives above, CAT Merging involves tailored opera- tions for different types of parameters: feature projection for linear weights, and masking for normalization scalers and shifts. These strategies ensure that CAT Merging effectively mitigates knowledge conflicts in training- free manner, rely- ing solely on lightweight forward pass with few unlabeled exemplars. We evaluate CAT Merging on diverse visual, language, and visual-language datasets, demonstrating its superiority over sfte-of-thhearr methods while maintaining robustness with limited exemplars.
The pioneering work of Task Arithmetic (Ilharco et al., 2023b) introduces the concept of "task vectors" parammterr offsets of finetuned models from a pretrained model as the origin. By scaling and combining these vectors, Task Arith- metic effectively balances task-general and task-specific knowledge and achieves notable enhancement in model merging. Building on this idea, Ties-Merging (Yadav et al., 2023) and PCBMerging (Du et al., 2024) enhance the pro- cess by removing components in task vectors with negligible magnitudes. PEFT (Zhang et al., 2023a) and MoLE (Wu et al., 2024) integrate Task Arithmetic with LoRA mod- ules (Hu et al., 2022) to enhance flexibility. Additionally, STA (Zhou et al., 2025) randomly shuffle network layers to eliminate knowledge conflict.
2. Related Work
2.1. Traditional Multi-Task Learning
Some advanced approaches incorporate test-time train- ing techniques to further improve merging performance. AdaMerging (Yang et al., 2024b) trains layer-specific merg- ing coefficients. Additionally, representation surgery (Yang et al., 2024a) and MoE router (Tang et al., 2024) train an extra adapter to align representations. Since test-time adap- tation requires additional training and samples, their appli- cation may be limited. Therefore, this paper focuses on faining-free model merging.
Multi-task learning (MTL) seeks to enhance performance by transferring knowledge from related tasks (Zhang & Yang, 2022). In MTI., the major challenge is referred to as nega- tive transfer (Liu et al., 2017; Zhang et al., 2023b), where the conflict knowledge among tasks results in degraded per- formance compared training them independently. To address this, researchers have explored strategies such sparsification (Ding et al., 2021; Sun et al., 2020; Liu et al., 2019a), modularization (Tang et al., 2020; Ma et al., 2018), and soft parameter sharing (Gao et al., 2020; Hazimeh et al., 2021). Others focus on the optimization process, such as dynamically weighting task- -seeifpei loss (Sener& & Koltun, 2018; Liu et al., 2019;; 2022; Hu al., 2023; Chen al.,
3. Preliminary
Notations. Let Wo = (W)) denote the parameters of pretrained network with L layers, where W repre- sents the parameters of layer 1. The network is finetuned
2

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
independently on K tasks using their respective datasets :..-.; D1...., DK. yielding finetuned parameters wi.---. WK-
where the input features are computed using only the unper- turbed parameters (W1 k: k:.... W This allows the total feature shift at layer I, be controlled under mild Lipschitz assumption:
For given task k, the task vector T-- = {Ti 1 is defined as the difference between the finetuned parameters W and the pretrained parameters Wo:
Akkl f + Ti) (WW), to
Assumption 4.1 (Lipschitz Continuity of Network Layers). (L)
T Wx- Wo = (W1 WilL
On the line segment between Wa and Wa + Tú the function of layer I, denoted as ft is "i-Lipschitz continuous with respect to its input (i.e., the output of layer I - 1).
The dimensions of W and T vary depending on the param- eter type in layer L. Specifically, if layer is a linear layer, then Wh T E Iddixi++,, where di and di+1 are the input and output dimensions of layer I, respectively. For other parameters, such dement-wise scale shift parameters (e.g., in layer normalization), W TV E Rd.
Lemma 4.2. By Assumption 4.1, we have:
f S wul ~ k/R 'II |
(6)
where A8k; fl T.) (WW))
Task Arithmetic. Our objective is to merge K finetuned models {= into a single multi-task model Wid that performs well on all tasks. Task Arithmetic (Ilharco et al., 2023b) achieves this by simply adding task vectors to Wo:
Intuitively, Lemma (4.2) decomposes the feature shift at layer l into (i) the propagated feature shift A from the previous layer and (ii) the layer-wise parameter perturbation A f% caused by T.- We refer the reader to Section A. 1 for the detailed proof.
Wml Wo + TR
(2)
We further consider how perturbations in the features ex- tracted from the network affect the task-specific loss and impose a continuity condition on the loss function:
where a > 0 a manual scaling factor (we let a for notational simplicity; our method allows for different a val- ues). Despite its simplicity, this approach often suffers from knowledge confficts when task vectors interfere, motivating our more refined trimming strategy.
Assumption 4.3 (Lipschitz Continuity of Loss Function). On the line segment between WR and Wx + T, the loss function C is 3-Lipschitz continuous with respect to the final output of the network.
Combining the above, we have
4. Knowledge Conflict
Theorem 4.4 (An Upper Bound on Knowledge Conflict). Suppose that Assumptions 4. and 4.3 hold for any task and task vector To the knowledge conflict follows
Knowledge conflict (Sun et al., 2025) arises in model merg- ing when task vectors are misaligned due to divergent mag- nitudes or directions. Direct aggregation, employed in Task Arithmetic, often leads to performance degradation across tasks. We formally define knowledge conflict in Task Arithmetic as the increase in task-specific loss incurred by incorporating other task vectors. On a given task k, associ- ated with the loss function (-). the conflict introduced by another task vector T; (i + k) quantified as:
L
Σ 1=1 m=|+1
II
(7)
The proof is provided in Section A.2. Theorem 4.4 suggests that to reduce the upper bound on knowledge conflict, we can directly minimize the layer-wise perturbations A| for each layer. In the next section, we develop practical merging algorithm to do so.
ACy:= = Ls (W& + 7i) La (WA).
(3)
As mentioned, a task vector T; can be separated into network layers T, (T: ..... }. where the superscript denotes the layer index. The effect of a perturbation T; propagates through the layers. Without loss of generality, we suppose all layers are sequentially stacked, i.e., the input to layer is the output of layer 1 - 1, where the 1-th layer feature on task k. denoted as (.). can be written as:
5. Methodology
As introduced in Section 4, our approach adopts a layer-wise strategy to analyze and mitigate knowledge conflict locally at each layer. Several existing methods attempt reduce conflict by preserving components deemed important to each task. For example, Fisher Merging (Matena & Raffel, 2022) utilizes the Fisher Information Matrix to retain task- important weights, while Ties Merging (Yadav et al., 2023) and PCB Merging (Du et al., 2024) prune low-magnitude parameters.
(4) (We+ T:) = f' + T:); W T!).
To isolate the local effect of T/, we define:
(W + T:) = 1 (W+); W + T)),
(5)
3

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Although suppressing unimportant parameters can reduce some conflicts, these approaches overlook critical issue: high-magnitude parameters-oficn considered important- can themselves be major sources of knowledge conflict. These components may overpower others during merging, distorting or suppressing the feature representations vital to different tasks. As a result, even when -ow-mannitude parameters are pruned, significant knowledge conflict still exists, degrading multi-task performance.
5.1. Linear Weight Trimming
Linear layers transform features via matrix multiplica- tion. Given X2 E Rxx4 extracted by former layers k..... [ duce the output as follows:
}.
the weight matrix W 12 pro-
f(X:: W) X.W.
(9)
To mitigate knowledge conflicts at this layer, we aim to remove interfering components from other task vectors T. i + k. We introduce a removal basis B& E Rd4+1e, representing c-dimensional core sub-parameter space for task k. For any # k. we define a projection operator & (T;) = T; B3 which discards components T; aligned with Bx: preserving task-specific knowledge while minimizing interference with task
In contrast to existing trimming techniques, we propose method that explicitly balances two objectives: (1) mini- mizing interference between tasks and (2) preserving the knowledge encoded in task vectors. For each task k, we define the task specifii transformation 4% that trims other task vectors in the same layer, i.e., T, (i A k). The objective for (.)(.) is minimize the layer-wish wise knowledge conflict ASk, while maintaining the representational integrity of T- Formally, considering all task vectors yields the following objective for (.):
Using the layer-wise loss from Eq. (8), we formulate the optimization objective for Bg as:
+ W -a()))) 2
min k
- -xx
Inter lask Knowledge Comflict A f
(8)
+ > || + ( (((w))
Σ
X( -..W))))xxx(++-- X,
iyk
Intra Knowledge Deviation
Σ X.. 1
BI X 11:)
where λ controls the trade-off between reducing conflict and preserving knowledge.
(10)
Applying the identity || |I
Based on the computational rules of different parameter types, we design tailored trimming operations. Specifically, for modern transformer-based networks, the parameters can be broadly classified into three categories:
+
| X (see Appendix A.3 for the proof),
recast the objective as maximization problem:
Weight parameters in linear layers, which involve ma- trix multiplication operations;
Σ (IX.BB.B111
max Bh. it
(11)
Scale parameters in normalization layers, which scale normalized features through element-wise Hadamard products;
The optimal removal basis B₁ is constructed from the top-c eigenvectors of the matrix: Σifr T; (X, -XX X,)T, where the full derivation provided in Appendix A.4. Here, is a manually defined dimensionality hyperparameter. higher value of с corresponds to the removal of more in- formation. In low-sample regimes, we recommend setting -= 2 or 3 to avoid introducing noise.
Shift parameters, which modify features through element-wise addition.
Most parameters in modern architectures fall into these categories. Notably, convolutional weights are treated analo- gously to linear weights, since convolution can be expressed as linear operation (Sun et al., 2023). Furthermore, com- plex modules such as attention blocks can be decomposed into multiple linear transfornations e.g., for query, key, and value projections-alloving the trimming strategy to be applied independently to each component. In the following section, we provide detailed wise implementations of (.).) tailored to each parameter type.
5.2. Scaling Parameter Trimming
Normalization layers crucial for stabilizing the training of modern neural networks. Although normalization itself is a parameter-free operation, widely used layers such as batch normalization and layer normalization typically in- clude an affine transformation, which consists of scale and shift parameters. In this section, we focus on merging the

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
scale parameters across tasks. At normalization layer the scaler W E Rd performs element-wise multiplication on the normalized feature Ią. E Rd:
Algorithm 1 The model merging process
Input: Pretrained model Wo; Task vectors (Kk
Unlabeled exemplar-set {MI..... Mx}
Output: Merged model Wml // Collecting the input features
f(zk) W) 24 W,
(12)
for k = 1 to K do Initialize task inputs: X Mr for 1 = to L do
where denotes element-wise (Hadamard) multiplication.
x |+++1 - s(x; W + T
3
To reduce interference at this layer, we introduce a binary removal mask my E (0, 1 applied to task vectors T; (i + k), such that the refined task vector &x = T, - Ti m& has minimal influence on task k. Adapting the layer-wise loss from Eq. (8), we define the objective:
// Compute basis or mask
for k = 1 to K; = 1 to I. do
4 arg ming! Σigs
11 ( Tj + 0 (T)))) fl(Wg + +
+ AWW + +.()))) - [ 2/ Eq. (8)
ΣΣ [
6 // Edit all task vectors
for k to K; = 1 to L; 10 K, do
1 2t
7
T (T!)
+
Merging
Wmn Wo Eb
Σ o
+ >
Tiomml
return Wml
21
(13)
Using
the
identity
I| T. 2g T, o m || o Til 12, we reformulate the
|| o T, o m =
EΣ (Wo+7777--T - -omm))aa--o--
problem as a maximization objective:
-WWo-WW
Σ oomll 24
Tompl1->
Σ om
max mi
Σ i
[T. -T:-T. Ti
2 ΣE
11T mell? l
(14)
21
(16)
This motivates selecting the top-c dimensions in me corresponding to the largest components of the vector: Ligk (Ez, (Ih o 2 E (z, o 7:)2), with remain- ing entries set to 0. The selection of c is match the number of basis B& used in the previous linear weight trimming to reduce noise from limited exemplars. Full mathematical derivations are provided in Appendix A.5.
Similarly, the mask my can be determined by maximiza- tion problem:
Σ IT: ma 1|
Σ T. mm/l² (17) 2L
max mig
Assuming data is approximately balanced across tasks, the optimal mask my is obtained by selecting the top-c compo- nents of Eith (T;)² and setting the corresponding entries of my to 1, with the rest set to 0. Full derivations are provided in Appendix A.6.
5.3. Shifting Parameter Trimming
Shift parameters W e dd adjust features via wise addition, the transformation applied to input da E Rdi is defined
We have now finished discussing the trimming strategy for all major parameter types. During the merging process, we start with layer-by-layer forward pass using only a small number of unlabeled samples-typicalyy two or three per task to collect necessary inputs for parameter trimming. The collected inputs are then processed to compute the corresponding basis or masks for editing task vectors based on the parameter type. The computed basis or masks are
f(z W) = -b W.
(15)
We also adopt a binary removal mask my E {0, 1]d, yield- ing a refined vector (T,) = T₁ - T, o my. The corre-
sponding objective function becomes:
5

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Table Multi-task performance when merging ViT-B/32 models eight vision tasks. The best and second-bews performances are written
in bold and underlined text. The "#best" column represents the number of datasets where the method performs the best.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
#best
Pretrained
62.3
59.7
60.7
45.5
31.4
32.6
48.5
43.8
48.0
Individual
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
Traditional MTI.
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
Fisher Merging
68.6
69.2
70.7
66.4
72.9
51.1
87.9
59.9
68.3
RegMean
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
Task Arithmetic
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
Ties-Merging
59.8
58.6
70.7
79.7
86.2
72.1
98.3
54.2
72.4
TATR
62.7
59.3
72.3
82.3
80.5
72.6
97.0
55.4
72.8
Ties-Merging & TATR
66.3
65.9
75.9
79.4
79.9
68.1
96.2
54.8
73.3
Consensus Merging
65.7
63.6
76.5
81.7
97.0
57.1
PCB Merging
63.8
62.0
77.1
80.6
87.5
78.5
98.7
58.4
75.8
3
CAT Merging (ours)
68.1
65.4
80.5
89,5
85.5
78.5
98.6
60.7
78.3
4
Table 2. Multi-task performance when merging ViT-L/14 models on cight vision tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
#best
Pretrained
66.8
77.7
71.0
59.9
58.4
50.5
76.3
55.3
64.5
Individual
82.3
92.4
97.4
100.0
98.1
99.2
99.7
84.1
94.2
Traditional MTL
80.8
90.6
96.3
96.3
97.6
99.6
84.4
Weight Averaging
72.1
81.6
82.6
91.9
78.2
97.1
62.8
Fisher Merging
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70.0
82.2
RegMean
73.3
81.8
86.1
97.0
88.0
84.2
98.5
83.7
Task Arithmetic
73.9
82.1
86.6
94.1
87.9
86.7
98.9
65.6
84.5
Ties-Merging
76.5
85.0
89.3
95.7
90.3
83.3
99.0
68.8
86.0
TATR
74.6
83.7
87.6
93.7
88.6
88.1
99.0
66.8
85.3
Ties-Merging & TATR
76.3
85.3
88.8
94.4
90.8
88.7
99.2
68.8
86.5
Consensus Merging
75.0
84.3
89.4
95.6
88.3
82.4
98.9
68.0
85.2
PCB Merging
76.2
86.0
89.6
95.9
89.9
92.3
99.2
71.4
87.6
CAT Merging (ours)
78.7
88.5
91.1
96.3
91.3
95.7
99.4
75.7
89.6
applied all task vectors, enabling the final merging in Eq. (2) with minimal knowledge conflict. The detailed implementation of this process is described in Algorithm 1.
6. Experiments
6.1. Settings
Datasets. We select diverse datasets evaluate our work, including eight vision datasets: SUN397 (Xiao et al., 2016). Cars (Krause et al., 2013), RESISC4S (Cheng et al., 2017). EuroSAT (Helber et al., 2019), SVHN (Netzer et al., 2011).
To simplify analysis, we have assumed the scaling factor a (ef. Eq. (2))to be in the above analysis, but setting a to an- other positive value does not affect the resulting conclusion. For example, in Eqs. (10) & (11), by explicitly including a, the optimal vector BR corresponds the eigenvector of the matrix: Eige (aT;T (X,, X₃ AX; (aT)) a Eiyk T. Xx -XX, T.. Since scaling matrix by positive constant a² does not alter its eigenvectors, our theoretical conclusions remain valid. In practice, a is critical hyperparameter that must be tuned carefully.
GTSRB (Stallkamp et al., 201 1), MNIST (LeCun & Cortes, 2010), DTD (Cimpoi et al., 2014), six visual-language datasets: COCO Caption (Chen et al., 2015), Flickr30k Caption (Plummer et al., 2015), Textcaps (Sidorov et al., 2020), OKVQA (Marino al., 2019), TextVQA (Singh et al., 2019), and ScienceQA (Lu al., 2022), and eight NLP tasks in the GLUE benchmark (Wang et al., 2019).
We compare our proposed method with various training-free model merging methods, including the straight- forward Weight Average, Fisher Merging (Matena & Raffel,
Baselines.
6

---

CAT Merging: A Training-Fre Approach for Resolving Conflicts in Model Merging
Table 3. Multi-task performance when merging RoBERTa models eight NL. tasks.
Method
CoLA
MNLI
MRPC
QNLI
QQP
RTE
SST2
STS-B
Average
#best
Task Arithmetic
6.68
66.23
78.46
78.62
72.69
53.43
83.49
27.10
58.34
Ties-Merging
9.46
59.34
74.71
65.93
41.29
47.29
72.13
9.21
47.42
0
TATR
10.20
65.44
72.56
75.73
74.58
55.18
78.87
37.46
58.39
PCB Merging
11.40
50.85
77.63
78.22
55.78
60.29
75.57
67.01
59,59
1
CAT Merging (ours)
33.20
72.33
68.22
82.92
76.05
62.82
89.33
15.57
62.56
6
Table Multi-task performance when merging BLIP models on vision-language tasks.
Method
COCO Caption
Flickr30k Caption
Textcaps
OKVQA
TextVQa
ScienceQA
#best
CIDEr
CIDEr
CIDEr
Accuracy
Accuracy
Accuracy
Pretrained
0.07
0.03
0.05
42,80
21.08
40.50
Task Arithmetic
0.86
0.50
0.39
17.71
0.49
40.10
|
Tics-Merging
0,53
0.27
0.22
27.95
0.57
40.35
0
TATR
0.46
0.31
0.21
28.30
14.74
42.98
0
PCB Merging
0.71
0.52
0.30
36.04
1.88
43.01
CAT Merging (ours)
0.91
0.53
0.36
44.07
19.69
46,36
5
2022), RegMean (Jin et al., 2023), Task Arithmetic (Ilharco et al., 2023b), Ties-Merging (Yadav et al., 2023), Task Arith- metic in Trust Region (TATR) (Sun et al., 2025), Consensus Merging (Wang et al., 2024), and PCB Merging (Du et al., 2024). We also provide the performance some baseline methods, such as the pretrained model, the individual task model, and the traditional multi-Task learning model.
average accuracies of 78.3% ViT-B/32 and 89.6% on ViT-L/14. Notably, substantial improvements are observed on challenging datasets such as EuroSAT (89.5%, ViT-B/32) and GTSRB (95.7%, ViT-I/144. Additional results for ViT- B/16 and comparisons with test-time training methods are provided in the supplementary materials.
Evaluation NLP tasks. We evaluate CAT Merging on eight diverse NLP tasks using RoBERTa, covering both clas- sification and regression (e.g., STS-B), For classification, we report accuracy, while for regression we report the mean of Pearson and Spearman correlations. As summarized in Table 3, CAT Merging consistently outperforms base- line methods, achieving the best results on six of eight tasks, demonstrating strong generalization and robustness in model merging for language tasks.
Implementation details. Our implementation strictly fol- lows Task Arithmetic (Ilharco et al., 2023b). For the vision tasks, we employ the vision encoder from CLIP (Radford et al., 2021) as the pretrained backbone, experimenting with both ViT-B/32 and ViT-L/I4 architectures. Task vectors are directly sourced from the repository provided by Ilharco al. (2023b). After merging the models, we evaluate and report the accuracy of each task, along with the overall aver- age accuracy. For the visual-language tasks, we derive task vectors by finetuning the visual question-answering (VQA) version of BLIP (Li et al., 2022), training each task for 6,000 steps. The architecture of BLIP includes an image encoder, text encoder, and text decoder, with all components finetuned during training. We use CIDEr as the evaluation metric for caption tasks and accuracy for VQA tasks. For the NLP tasks, we use RoBERTa (Liu et al., 2019b) as the backbone. Task vectors are obtained from Lu et al. (2024). Following their setting, we reserve 10% of the training set for validation and employ the original validation data as the test set.
Evaluation on visual-language tasks. Table 4 reports re- sults using the BLIP model on six Ausion-language tasks that potentially conflict with each other (Tiong et al., 2024). CAT Merging achieves top performance on five of six bench- marks, highlighting its effectiveness in multimodal tasks.
6.3. Ablation Study
This section performs an ablation study to evaluate the ef- fectiveness of the parameter-specife trimming techniques in CAT Merging, including the trimming of linear weights, normalization scalers, and shift parameters. Table 5 presents the results with ViT-B/32 and VIT-L/14 architectures.
6.2. Comparison Results
Excluding the trimming of linear weights leads to a signifi- cant performance drop, with average accuracies of 7 71.5% (ViT-B/32) and 85.6% (ViT-I/144. Similarly, removing the trimming of normalization scalers and shifts also causes
Evaluation on vision tasks. Tables 1 and present the merging results on eight image classification benchmarks. CAT Merging achieves state-of-the-art performance, with

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
(a) Task Arithmetic
(b) CAT Merging
Table An ablation study of CAT Merging, where selectively trim three different types of network parameters.
2
10-1
Trimmed parameters
Architecture
Linear weight
Normalization scaler
Shift
ViT-B/32
ViT-L/14
1-
2
71.5
85.6
0
0
V
76.8
89.2
0
1
2
0
10-3
X
77.8
88.7
ac
78.3
89.6
Figure 3. Visualization of knowledge conflict on Cars and RE-
Aog (%, VIT-B85)
Ang Ace (%.VV (4.11111111
SISC45 (i.e., ALCcs,RESSscK LLEESISCAS,C Can.) when merging
78
89.5
ViT-L/14 models with different merging weights and or.
ViT-B/32
77
VIT-L/14
89.0
hyperparameters, including weight À and is provided in Appendix C.4.
8
16
(a) Number of exemplars
6.5. Analysis of Knowledge Conflict
This section investigates the effectiveness of CAT Merg- ing in mitigating knowledge conflict. Specifically, we consider the task vectors corresponding to Cars and RE- SISC45 and merge them using distinct scaling factors, i... W Wo + acTTcas + og RRESISCAS- Figure 3 visualizes the knowledge conflict during merging under different merg- ing coefficients. For Task Arithmetic, regions with minimal knowledge conflict are primarily concentrated where both ac and og are close to In contrast, CAT Merging effec- tively reduces knowledge conflict, enabling the use much larger a values.
75
Avg. Arc
Task Arithmatic
TATR
50
Thes Tiss-Merging
CAT Morging
0.1
0.2
0.3
0.4
0.5
0.6 0.7 0.8 0.9
(b)
Figure 2. (a) Average accuracy (%) of CAT Merging on eight vision tasks with different numbers exemplars per task. (b) Average accuracy (%) on cight vision tasks with ViT/L-14 14 models versus different a (scaling factor in Task Arithmetic, cf. Eq. (2)).
7. Conclusion
performance degradation. With all three types trimmed, we achieve the highest average accuracy. These findings high- light the importance of mitigating knowledge conflict in all types of parameters.
In this work, we propose CAT Merging, a novel training-free technique for multi-task model merging. By introducing parametcr-specific strategies, such as projection for linear weights and masking for normalization and shift parame- ters, CAT Merging effectively resolves inter-task knowl- edge conflicts while preserving task-specific information. Expermmental results across diverse vision, language, and vision languaee tasks demonstrate the effectiveness of CAT Merging. We achieve significant accuracy improvements of up to 2.5% on ViT-B/32 and 2.0% on ViT-L/14 compared to existing methods. CAT Merging also exhibits strong ro- bustness the case with only few exemplars, providing practical solution for real-world multi-task scenarios where retraining is infeasible.
6.4. Sensitivity Analysis
Sensitivity analysis of exemplar number. As shown in Figure 2 (a), CAT Merging is robust to the number of exem- plars. achieves strong results even with just one exemplar per task (76.61% for ViT-B/32 and 89.01% % for ViT-L/14). This is likely due to the patch-based design ViT, which splits one image into several patches, ensuring the diversity of each layer input even with limited samples.
Sensitivity analysis of In Figure 2 (b), with different val- ues of a. CAT Merging achieves more stable performance than Task Arithmetic. The stability should be attributed to suppressing knowledge conflict among task vectors, ana- lyzed further the next section. A detailed analysis of other
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal
8

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
consequences of our work, none which feel must be specifically highlighted here.
Chen, Z., Ngiam, J., Huang, Y., Luong, T., Kretzschmar, H., Chai, Y., and Anguelov, D. Just pick sign: Opti- mizing deep multitask models with gradient sign dropout. In Advances in Neural Information Processing Systems, volume 33, pp. 2039-2050, 2020.
Acknowledgments
This work was supported, in part, by the Fundamental Research Funds for the Central Universities under Grants 2023JBZY037, 2022JBQY007, and 2024YIS047, by the Beijing Natural Science Foundation under Grant L.21001,, by the National Natural Science Foundation of China der Grants 62276019 and 62306028, by the Shenzhen Science and Technology Program Project under Grant DUZD20024900310222225555 and by the National Research Foundation Fellowship (NRF- NRFF13-2021 20000060021-0006, Singa- pore.
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Pro- ceedings of the IEEE, (05(10):1865-188)) 2017. doi: 10.110//PROC..21::267999
Cimpoi, M., Maji. S., Kokkinos, I,, Mohamed, S., and Vedaldi, A. Describing textures in the wild. In IEEE Conference Computer Vision and Pattern Recognition, June 2014.
Ding, K., Dong, X., He, Y., Cheng, L., Fu, C.,, Huan, Z. Li H., Yan, T., Zhang, L., Zhang, X., and Mo, L. Mssm: A multiple-level sparse sharing model for efficient multi- task learning. In International ACM SIGIR Conference on Reseurch and Development in Information Retrieval, pp. 2237-2241, 2021. URL https: //oii org/ 10. 1145/340455 3463022.
References
Arpit, D., Wang. H., Zhou, Y., and Xiong, C. Ensemble of averages: Improving model selection and boosting performance in domain generalization. In Advances in Neural Information Processing Systems, volume 35, pp. 8265 8277,82772222 2022.
Du, G., Lee, J., Li, J., Jiang, R., Guo, Y., Yu, S., Liu, H., Goh, S. K., Tang, H.-K., He, D., and Zhang, M. Parameter competition balancing for model merging. In Advances in Neural Information Processing Systems, 2024.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, D.,
Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G.,
Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., McCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pp. 1877 1901, 2020.
Fisher, R. A. Theory of statistical estimation. In Mathemat- ical proceedings of the Cambridge philosophical society, volume 22, pp. 700-725, 1925.
Gao, Y., Bai, H., Jie, Z., Ma, J., Jia, K., and Liu, W. Mtl- nas: aggosticc neural architecture search towards general- pereral- multi-task learning. In IEEE Conference Computer Vision and Pattern Recognition, June 2020.
Cha, J., Chun, S., Lee, K., Cho, H.-C., Park, S.,, Lee, Y., and Park, S. Swad: Domain generalization by seeking flat minima. In Advances in Neural Information Processing Systems, volume 34, pp. 22405-22418, 2021.
Guo, X., Li, B., and Yu, H. Improving the sample ef- ficiency of prompt tuning with domain adaptation. In Findings of the Conference on Empirical Methods in Nat- ural Language Processing, December 2022.
Chen, S., Crammer, K., He, H., Roth, D., and Su, W. Weighted training for cross-task learning. In International Conference on Learning Representations, 2022. URL https::////........ http///ppeeevveeeeeeeeee n forum? id id-1tM_RM2ntpu.
Hazimeh, H., Zhao, Z., Chowdhery, A., Sathiamoorthy, M., Chen, Y., Mazumder, R., Hong, L.,, and Chi, F. Dselect- k: Differentiable selection in the mixture of experts with applications to multi-task learning. In Advances in Neural Information Processing Systems, volume 34, pp. 29335- 29347, 2021.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 1504.003... 2015.
He, Y., Feng, X., Cheng, C., Ji, G., Guo, Y., and Caver- lee, J. Metabalance: Improving multi-task recommenda- tions via adapting gradient magnitudes of auxiliary tasks, International ACM Web Conference, pp. 2205-2215 2022. doi: 04.1145//485544735522200 URL https:
Chen, 7., Badrinarayanan, V., Lee, C.-Y,, and Rabinovich, A. GradNorm: Gradient normalization for adaptive loss balancing in deep multitask networks. International Conference on Machine Learning, volume 80, pp. 794- 803, 10-15 Jul 2018.
//doi. 1115111..111151
9

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eu- rosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Jour- nal of Selected Topies in Applied Earth Observations and Remote Sensing, 12(7):2117-2226, 2019. doi: S..100/ISTARS.201922999
International Conference on Machine Learning, volume 162 of P Proceedings of Machine Learning Research, pp. 12888-12900, PMLR, 17-23 Jul 2022. URL https: proceedings m
https///vv22122222 html.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. In Ad- vances in Neural Information Processing Systems, vol- ume 34, pp. 1 18878-18890, 2021.
Hu, E. J., yelong shen, Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., and Chen, W. LoRA: Low-rank adap- tation of large language models. In Interaational Confer- ence on Learning Representations, 2022. URL https: / //openreview . net/ forum? id=neeeee.......
Liu, P., Qiu, X., and Huang. X. Adversarial multi-task learn- ing for text classification. In Association for Computa- tional Linguistics, pp. 1-10, 2017. doi: 10.18653/v1/ P17-1001. URL https: acl http//acaaanntoooo org/ P17-1001.
Hu, Y., Xian, R., Wu, Q. Fan, Q. Yin, L.., and Zhao, H. Revisiting scalarization in multi-task learning: A theo- retical perspective. In Advances in Neural Information Processing Systems, volume 36. pp. 48510-48533, 2023.
Liu, S., Johns, E., and Davison, A. End-to-end multi-task learning with attention. In IEEE Conference on Computer Vision and Pattern Recognition, June 2019a.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In International Conference on Learning Representations, 2023a.
Liu, S., James, S., Davison, A., and Johns, E. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Learning Research, 2022. ISSN 2835-
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic. In International Conference on Learning Rep- resentations, 2023b. URL httpps://cpenreview
8856. URL https:/ /twp//pppeeeeeeee//e/ee n forum? i id=KKeCMin5VN.
Liu, Y., Ott, M., Goyal, N., Du, J., Joshi, M., Chen, D., Levy, O., Lewis, M., Zettlemoyer, L., and Stoyanov, V. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1699, 2019b.
Izmailov, P., Podoprikhin, D., Garipov, T., Vetrov, D., and Wilson, A. G. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence, pp. 876-885, 2018.
Lu, P., Mishra, S., Xia, T., Qiu, L.,, Chang, K.W Zhu, S.-C., Tafjord, O., Clark, P., and Kalyan, A. Learn to explain: Multimodal reasoning via thought chains for science question answering. In Advances in Neural Infor- mation Processing Systems, 2022.
Javaloy, A. and Valera, I. Rotograd: Gradient homoge- nization in multitask learning. In International Confer- ence on Learning Representationss 2022. URL https: / www.nneeeeew.eeeeeeeeeeeeeeeeeee forrmm/ -dwTdw 4
Lu, Z., Fan, C., Wei, W., Qu, X., Chen, D., and Cheng, Y. Twin- merging: Dynamic integration of modular expertise in model merging. In Advances in Neural Information Processing Systems, 2024.
Jin, X., Ren, X., Preotiuc- -Peetro, D., and Cheng. P. Dataless knowledge fusion by merging weights of language mod- els. In International Conference on Learning Represen- tations, 2023. URI https:/ / openrevieee wwwreeiiieeee.. net./ forum?ide wwt.iiddFFCnuuuuu.e
Ma, J., Zhao, Z. Yi, X., Chen, J., Hong, L., and Chi, E. H. Modeling task relationships in multi-task leam- ing with multi-gate mixture-of-experts. In Internu- tional ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pp. 1930-1939, 2018. doi: 00011445/221981933220000 org prg/10.11 11 14/1144//222 9819. 3220007.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine-grained categorization. In IEEE International Conference on Computer Vision Workshops, June 2013.
URL https: /oii.
LeCun, Y. and Cortes, C. MNIST handwritten digit database. 2010. URL httt:://tyynnn... com/ exdb/mmn axdb/mnist/..
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok- vqa: A visual question answering benchmark requiring external knowledge. In IEEE Conference on Computer Vision and Pattern Recognition, 2019.
Li, Li, D., Xiong, C., and Hoi, S. BLIP: Bootstrapping language-ine image pre-training for unified lasiunel understanding and generation. In Chaudhuri, K., Jegelka, S., Song, L.., Szepesvari, C., Niu, G., and Sabato, (eds.),
Matena, M. S. and Raffel, C. A. Merging models with fisher- weighted averaging. In Advances in Neural Information Processing Systems, volume 35, pp. 17703-17716, 2022.
10

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Navon, A., Shamsian, A., Achituve, I., Maron, H., Kawaguchi, K., Chechik, G., and Fetaya, E. Multi-task learning as a bargaining game. In International Conference on Machine Learn ing, volume 162, pp. 16428-16446, 17-23 Jul 2022. URL //ppcceec/occccsssss.ooo mwr. mlr.pprss/ v1 #162/navon22a. html.
memory rramework with knowledge projection for task- incremental learning. In IEEE Conference on Computer Vision and Pattern Recognition, June 2023.
Sun, W., Li, Q., Wang, W., ao Geng, Y., and Li, B. Task arithmetic in trust region: training-free model merg- ing approach to navigate knowledge conflicts. arXiv preprint arXiv:2501. 15065, 2025. URL https:// à arxiv.org/aai aat/a/bs//2111 15065.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., al. Reading digits in natural images with unsupervised feature learning. In NIPS workshop on deep learning and unsupervised learning, volume 2011, pp. 4, 2011.
Sun, X., Panda, R., Feris, R., and Saenko, K. Adashare: Learning what share for efficient deep multi-task learn- ing. In Advances in Neural Information Processing Sys- tems, volume 33, pp. 8728-8888 8740, 2020.
Ortiz-iimnnz, G., Favero, A., and Frossard, P. Task arith- metic in the tangent space: Improved editing of pre- trained models. In Advances in Neural Information Pro- cessing Systems, volume 36, pp. 66727-66754, 2023.
Tang, A., Shen, I.., Luo, Y., Yin, N., Zhang, I.., and Tao, Merging multi-task models via weight-ensembling mix- ture of experts. In International Conference Machine Learning, 2024. URL https: openreview www.eeiwwwweee...eee..e.e.e.eeee .net//// forum? wowum?iddnLRKOOTT nLR.no74KK 4RB.
Plummer, B. A., Wang, L., Cervantes, C. M., Caicedo, J. C., Hockemmaier, J., and Lazebnik, Flickr30k entities: Collecting region-to-phrase correspondences for richer mmage-to-sentence models. In IEEE International Con- ference Computer Vision, December 2015.
Tang, H., Liu, J., Zhao, M., and Gong, X. Progres- sive layered extraction (ple): A novel multi-task learn- ing (mtl) model for personalized recommendations. In ACM Conference on Recommender Systems, pp. 269-278, 2020. doi: 10. 00.114453338331333422233 URL https: / //doi.org 0. 145 /111111433333333111111
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., Krueger, G., and Sutskever, I. Learning transferable visual models from natural language su- pervision. In International Conference on Machine Learning, volume 139, pp- 8748 8763, 18-24 Jul 2021. https: httpt::://prrceeeee http///ooceddceeeeeee wogooeedding.......... mlr. press/ v139/radforze htmi.
Tiong, A., Zhao, J., Li, B., Li, J., Hoi. S., and Xiong, C. What are we measuring when we evaluate large vision- language models? an analysis of latent factors and biases. Annual Conference of the North American Chapter of the Association for Computational Linguistics, 2024.
Sener, O. and Koltun, V. Multi-task learning as multi- objective optimization. In Advances in Neural Informa- tion Processing Systems, volume 31. Curran Associates, Inc., 2018
Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., and Bowman, S. R. GLUE: A multi-tast task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations, 2019. URL https: / /ttp///eeeeeeeeeeeeeeeeeeeeeeeee net net/forun? forum? id=rJ4km2R5t7.
Sidorov, O., Hu, R., Rohrbach, M., and Singh, A. Textcaps: A dataset for image captioning with reading comprehen- sion. In European Conference on Computer Vision, pp. 742-758, 2020. ISBN 978-3-030-58536-5.
Wang. K., Dimitriadis, N., Ortiz-Jimenez, G., Fleuret, F., and Frossard, P. Localizing task information for improved model merging and compression. In International Con- ference on Machine Learning, 2024.
Singh, A., Natarajan, V., Shah, M., Jiang, Y., Chen, X., Batra, D., Parikh, D., and Rohrbach, M. Towards vqa models that can read. In IEEE Conference Computer Vision and Pattern Recognition, June 2019.
Wu, X., Huang, S., and Wei, F. Mixture of loRA experts. In International Conference on Learning Representations, 2024. URL httph://////
Stallkamp, J.. Schlipsing, M., Salmen, J., and Igel, C. The german traffic sign recognition benchmark: A multi-class classification competition. In International Joint Confer- ence on Neural Networks, pp. 1453-1460, 2011.
/ http:///pppeeeeeeeeeeeppe net/ net/forum? forum? i dd=uWyKBCY44..
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision, 119:3-22, 2016.
Sun, W. Li, Q., Zhang, J., Wang, W., and Geng, Y.- a. Decoupling learning and remembering: bilevel

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Yadav, P., Tam, D., Choshen, L., Raffel, C., and Bansal, M. merg-ne Resolving interference when merging models. In Advances in Neural Information Processing Systems, 2023.
Yang. E., Pan, J., Wang, X., Yu, H., Shen, L., Chen, X., Xiao, L.,, Jiang, J., and Guo, G. Adatask: A task-aware adaptive learning rate approach to multi-task learning. AAAI Conference on Artificial Intelligence, 3709))00745- 10753, Jun. 2023. URL index. php/ AAA.//xttiieeerri///22271
Yang. E., Shen, L... Wang, Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for multi-task model merging. In International Conference on Ma- chine Learning 2024a. htthtps:: http:///ppppeeeeeeeeeeeeeeee.eet het/forum, ? Sb122k0EM....
Yang, E., Wang, 7.,, Shen, I.., Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merg- ing for multi-task learning. In International Confer- ence on Learning Representations, 2024b. URL https: //openreview net. forum? id=ntP 6NgD3QY.
Zhang. J., chen, s., Liu, J., and He, J. Composing parameter- efficient modules with arithmetic operation. In Advances in Neural Information Processing Systems, volume 36, pp. 12589-12610, 2023a.
Zhang, W. Deng, L., Zhang, L.. and Wu, D. A survey on negative transfer. IEEE/CAA Journal of Automatica Sinica, 10(2):305-322 -29,, 2023b. doi: 10.1102//SS.0022. 106004.
Zhang, Y. and Yang, Q. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineer- ing, 34(12):5586-5609, 2022. doi: 10.110./TKEE.201. 3070203.
Zhou, H., Gokaslan, A., Kuleshov, V., and Hariharan, B. Interfering with interference: Blind shuffling and superposition for better multi-model compression, 2025. URL https: / wwwp.eeee.....ooeooo/eee. net forum? id=4wuvmJRAU4.
12

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
A. Theoretical Proofs
A.1. Proof of Lemma (4.2)
Let WE be the parameters for task k, and let T, be the task vector of task i. The feature shift at layer caused by T2 is defined as:
A = fWW - -T)) - f.
We aim to prove the following bound:
|| | S AillA p111 + A
where ASR fk fl- (W= T3) f/- (WW), and Afk = fl( + Ti) (W
Proof. The feature shift at layer / can be decomposed as:
ASk1 = 1 + T:): W +T,) AOA (WL:
= FIC + T)); W + +T)) fAC (WW): W +T,) A
(WL): +T²)
ACA " (WW):
(18)
Under the norm-induced triangular inequality, we have:
| < 1 (W, T:): W + T,) /W + 7) IVA) T!)
1 (Wa) W (19)
For the first part, we apply the -n-Lisschizz continuity of fl:
(Wk): I - +T): TT): f
+ +IDIl (
For the second part:
- (WW) = ا
Combining these two terms, we conclude:
| S milAA/1'11 1
A.2. Proof of Theorem 4.4
Theorem 4.4 (An Upper Bound on Knowledge Conflict).
Suppose that Assumptions 4.1 and 4.3 hold for any task k and task vector Tú, the knowledge conflict follows
I.
1.
A <
3
II m=l+1
7m
||
(20)
Proof. Recall that in Lemma (4.2), we derive the following recursive Lipschitz bound:
||
S xil
x
f ||
f|
13

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
We recursive substitution starting from layer L down / 1:
| /
7n.1 7
1| (NL-I
I ( f 7L
/ '11 1
(21)
Σ
7m |1l
Using Assumption 4.3, which states that is B-Lipschitz continuous with respect to the network's final output, we conclude:
I.
| S 3
Σ
II 7'm
(22)
1=1 m=1+1
A.3. Frobenius Norm Decomposition of Orthogonal Projection
Assume that has orthonormal columns (i.c., BTB = I), then, for any matrix M, the following equation holds:
MBB - |M - MBB 1 = ||
(23)
Proof.
MMB 2 || MBB 1|2
|MBB F || - )|}
-Tr(MMB")) (TMM M(I - BBJ))
-Tr(BB" M MBB ) ++Tr(I-BBB)JM M(I BBBJ)
=Tr(BB" BB M M)+ ( --+ T M)
=Tr(BB' M MTM) +Tr((1BB')) - (BBB)B)A M)
(24)
=Tr(BB" M MM + (T((I-BB))(I MM M)
-Tr(BBT M () + Tr((I BBT -BT + BB BBTMMTM M)
-Tr(BBT M M) +rr((IBB"T M)
-Tr(M M)
MIIMI
A.4. Solution of (9)
Let Xg E axd be the feature and T, E Rd represent the task vector of the task i. Now our target is learning a group of removal basis By E igxxe for task lo such that:
E| - AII.BT.BBBB 2
(25)
i
Then we have:
14

---

CAT Merging: A Training-Frey Free Approach for Resolving Conflicts in Model Merging
Σ
J.T.T | Fr
*k
TT((..BBBBB -T.T)X..AX..BBB X!)
Σ
Tr X X.) - B x,
Tr (T T. (X X 1X7 x.))
vyt
(B B T. (x) Xx. x,) Ti)
(26)
=Tr
Bg
T
Σ T (Xx Xx 2x,
isk
=Tr
B
Σ I. (X. X2 - AX," X:) Te
Bx
The above equation implies that the largest e cigenvectors of admit an optimal solution.
A.5. Solution of Eq.(
Let 3t E IR be the feature and T; E IRd represents the task vector of the task i for a scaler of a normalization layer. Now our target is learning a group of removal binary mask my E {0, 1] for task such that:
max WER
El T o m//ll -> ///
m
>>En x²
bs o T. mobl?)
(27)
Then we have:
Σ
Σ
1 T om/l'
|| | -
->>
T mill?
( T Ti m, - x
[ I i m
akk 2=1
#i ==1
Σ
Ikk Ti. 2
(oi )i )
(28)
ipfl
(kk, Ti,2)
(gr Di,z) )
mi.9aa
The above equation implies that in my, dimensions corresponding to the largest e values of g, should set to 1, while others should be
15

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
A.6. Solution of Eq.(13)
Let 2 R2 be the feature and T; E Id represents the task vector of the task i for shift parameters. Now our target is learning group of removal binary mask my {0, 13dd for task i such that:
Σ ig
Σ
IT, mxll -
IT 1
(29)
Suppose 0 < À < 1 and all tasks have an equal amount of data n, then we have:
IT, o me!!
- Σ T o m=ll
(IT; mx
- o m=ll²)
ipth
=n(1 A) Σ
[[T o o mxll²)
i
d
(IR Tú mx,)
2-
(oi. mu,
ig
(30)
=n(1 - 1)
Th mk
iy
=n(1
A)
mk,=
==1
n A)
mik02:
2m1
The above equation implies that in my, dimensions corresponding to the largest e values of gz should set to 1, while others should be
B. Experiment Details
This section introduces some additional details of experiments, including the detailes of the experimental environment, datasets, and baselines.
B.1. Environment
All experiments detailed in our manuscript and appendix were conducted on workstation running Ubuntu 16.04, equipped with 2 Intel Xeon 2.60GHz CPUs, 256 GB of memory, and 6 NVIDIA RTX3090 GPUs. We leverage Python 3.8 to implement all the methods.
B.2. Datasets
Our experiments strictly follow Task Arithmetic (Ilharco et al., 2023b) and leverage the following eight widely-used image classification datasets:
SUN397 (Xiao et al., 2016): A large-scale scene classification dataset containing 108,754 images organized into 397 categories. Each category includes least 100 images, making it diverse benchmark for scene recognition tasks.
Stanford Cars (Cars) (Krause et al., 2013): fine-grained car classification dataset featuring 16,185 images 196 distinct car models. The dataset evenly divided into training and test splits, enabling robust model evaluation.
16

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
RESISC45 (Cheng al., 2017): A remote sensing image classification dataset with 31,500 images spanning 45 scene classes. Each class contains approximately 700 images, covering variety of landscapes and man-made structures.
EuroSAT (Helber et al., 2019): A satellite imagery dataset designed for land-use and land-cover classification, consisting of 27,000 labeled and geo-referenced images distributed among 10 categories, such as forests, residential areas, and agricultural fields.
SVHN (Netzer et al., 3011): A digit classification dataset derived from real-world house numbers captured in Google Street View. It includes 10 digit classes, with 73,257 training images, 26,032 test images, and an additional 531,131 samples for extended training.
GTSRB (Stallkamp et al., 2011); The German Traffic Sign Recognition Benchmark, comprising over 50,000 images across 43 traffic sign categories. This dataset widely used for evaluating traffic sign recognition systems.
MNIST (LeCun & Cortes, 2010): A classic handwritten digit classification dataset containing 60,000 training images and 10,000 test images, evenly distributed across 10 digit classes.
DTD (Cimpoi et al., 2014): The Describable Textures Dataset, which includes 5,640 images spanning 47 texture categories, with around 120 images per category. It is designed for texture recognition tasks.
We also leverage the following six vision-language datasets:
COCO Caption (Chen et al., 2015): A large-scale image captioning dataset derived from the MS COCO dataset. It contains over 330,000 images, with each image annotated with five different captions, facilitating training for generating natural language descriptions of images.
Flickr30k Caption (Plummer et al., 2015): A dataset for image captioning and retrieval tasks, consisting of 31,000 images sourced from Flickr. Each image is paired with five descriptive sentences, capturing a variety of objects, scenes, and actions in the images.
TextCaps (Sidorov et al., 2020): A challenging image captioning dataset focusing on reasoning over both visual and textual content in images. It includes 145,000 image-caption pairs, where captions must integrate text from the image to provide meaningful descriptions.
OKVQA (Marino et al., 2019): A smowledge-based visual question dataset designed to evaluate the ability to answer open-ended questions about images using external knowledge. It consists of more than 14,000 questions and corresponding answers requiring reasoning beyond the image content.
TextVQA (Singh al., 2019): dataset for visual question answering where reading and understanding text present in images is crucial. It includes over 45,336 questions across 28,408 images, requiring the integration of textual and visual reasoning to generate accurate answers.
ScienceQA (Lu et al., 2022): A multi-modal dataset designed for science-related question answering. It contains over 21, 21,208 multi-modal multiple-choice questions paired with textual explanations and images across various scientific disciplines, such as biology, physics, and chemistry, supporting reasoning-bes based evaluation.
B.3. Baselines.
Our experiments are associated with several baseline approaches. The details of these baselines are as follows:
Pretrained directly employs pretrained model predict across multiple tasks. Since it does not incorporate any downstream task-specific information during model training, its performance downstream tasks is typically suboptimal.
Individual. In this approach, an independent finetuned model is used for each task. While avoids interference between tasks, it cannot perform multiple tasks simultancously. serves as a reference upper bound for model merging approaches.
17

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Traditional MTL aggregates the original training data from all tasks to train single multi-task model.
Weight Averaging directly averages model parameters from multiple tasks into single model, enabling multi-task learning without additional training.
Fisher Merging (Matena & Raffel, 2022) leverages the Fisher information matrix to assess parameter importance, merging model parameters based on this importance.
RegMean (Jin et al., 2023) refines weight matrices by adjusting and linearly combining rows, utilizing statistical information derived from the training data.
Task Arithmetic (Ilharco al., 2023b) introduces the concept of "task vector," defined the difference between finetuned model parameters and pretrained model parameters. Multiple task vectors are then combined and added to the pretrained model to facilitate multi-tast learning.
Ties-Merging (Yadav et al., 2023) eliminates unimportant parameters from the task vector and resolves sign conflicts among parameters, reducing interference during the final task vector merging process.
TATR (Sun et al., 2025). This method advances task arithmetic by restricting the merging within trust region to mitigate knowledge conflict.
TATR& & Ties-Merging (Sun al., 2025; Yadav et al., 2023). This method combines the trust region restriction in TATR into Ties-Merging to enhance the performance.
Consensus Merging (Wang et al., 2024) computing group of masks for each task vector to minimize the distance in parameter space between the merged model and the finetuned model.
PCB Merging (Du et al., 2024) trims components in the task vector that have small magnitudes and are not significantly related to other tasks.
C. Additional Experiments
C.1. Comparison on ViT-B/16
Table 6 presents the results of various model merging methods using the ViT-B/16 architecture. As we can see, CAT Merging significantly improves the multi-task performance of Task Arithmetic, raising the average performance from 73.8% to 82.1%.
Table Multi-task performance when merging ViT-B/16 models on eight tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Pretrained
63.8
64.6
65.7
54.5
52.0
43.3
51.7
45.1
55.0
Individual
81.8
86.8
96.9
99.7
97.8
99.1
99.7
82.0
92.9
Weight Averaging
67.7
70.0
75.3
79.5
74.9
60.1
94.4
43.8
70.7
Fisher Merging
68.5
69.9
75.2
80.4
73.2
61.2
94.5
50.7
71.7
RegMean
69.1
71.6
77.6
88.8
83.7
70.2
96.9
54.6
76.6
Task Arithmetic
61.1
65.9
74.0
76.2
88.0
73.9
98.4
53.0
73.8
Ties-Merging
69.1
72.5
80.5
84.0
85.0
71.5
98.1
54.9
77.0
TATR
67.4
70.4
77.9
81.7
87.6
77.2
98.3
55.6
77.0
CAT Merging (ours)
72.9
75.9
83.1
92.8
88.2
82.7
98.8
62.7
82.1
C.2. Comparing with Test-time Training based Methods
Table 7 shows that our CAT Merging achieves comparable or superior performance relative to two representative training- based techniques, demonstrating effectiveness without incurring additional computational costs.
18

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Table 7. Multi-task performance when merging on eight vision tasks.
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
ViT-B/32
AdaMerging
58.0
53.2
68.8
85.7
81.1
84.4
92.4
44.8
71.1
TW AdaMerging++
60.8
56.9
73.1
83,4
87.3
82.4
95.7
50.1
73.7
LW AdaMerging
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
LW AdaMerging++
66.6
68.3
82.2
94.2
89.6
89.0
98.3
60.6
81.1
Surgery Merging
63.8
59.9
83.3
97.9
87.0
87.0
98.6
69.4
80.9
CAT Merging (ours)
68.1
65.4
80.5
89.5
85.5
78.5
98.6
60.7
78.3
ViT-1/14
AdaMerging
79,0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
AdaMerging++
79.4
90.3
91.6
97.4
93.4
97.5
99.0
79.2
91.0
Surgery Merging
75.7
84.4
93.1
98.8
91.3
93.4
99.1
76.1
89.0
CAT Merging (ours)
78.7
88.5
91.1
96.3
91.3
95.7
99.4
75.7
89.6
90
Avg Avg. Acc (%)
80
VIT-B/32
ViT-B/32
70
VIT-L/14
ViT-L/14
5
10
50
100
2
4
16
32
64
128
(a)
À
(b)
Figure 4. Average accuracy (%) of CAT Merging eight vision tasks with different values of À (a) and (b).
C.3. Generalization Comparison
This section explores the generalization ability of CAT Merging, Specifically, we merge models using task vectors from six tasks and evaluate their performance on two unseen tasks (MNIST and EuroSAT). The results in Table show that CAT Merging outperforms all baselines on both seen and unseen datasets, with an average performance improvement of 3.4% and 2.6%, respectively. This improvement in generalization is attributed CAT Merging's ability handle knowledge conflicts, ensuring that model updates move toward more globally optimal direction.
Table 8. Generalization results two unseen tasks when merging ViT-B/32 models on six tasks.
Method
SUN397
Cars
RESISC45
DTD
SVHN
GTSRB
Avg Acc
MNIST
EuroSAT
Avg Ace
Task Arithmetic
63.3
62.4
75.1
57.8
84.6
80.4
70.6
77.2
46.2
61.7
Tics-Merging
67.8
66.2
77.2
56.7
77.1
70.9
69.3
75.9
43.3
59.6
TATR
66.0
64.1
77.9
60.1
83.9
81.8
72.3
77.2
47.7
62.5
CAT Merging (Ours)
70.4
68.4
85.3
63.6
82.8
83.8
75.7
77.8
52,3
65.1
C.4. Sensitivity Analysis
This section analyzes the sensitivity of two additional hyper-parameters À and
Sensitivity analysis of weight X. In Figure 4 (a), À significantly impacts performance only when set to 0, where the accuracy drops sharply to 65.33% (ViT-B/32) and 74.84% (ViT-L/14). This indicates that neglecting task-specific knowledge severely degrades the results. For À >> O, the performance remains stable across wide range of values, demonstrating the robustness of CAT Merging.
19

---

CAT Merging: A Training-Free Approach for Resolving Conflicts in Model Merging
Sensitivity analysis of c. c affects the number of task vector components are trimmed. As illustrated in Figure 4 (b), the performance of CAT Merging remains stable for small c, peaking 2 for ViT-B/32 and for ViT-I/14. Larger values lead to partial decline more knowledge is discarded, with ViT-B/32 dropping to 69.89% at 128.
Table Computational complexity comparison (in seconds) for merging VIT-B/32 and ViT-L/14 models across eight vision tasks, measured on single RTX 3090 GPU.
Method
TA w/ Surgery
AdaMerging
TATR
PCB Merging
CAT Merging (ours)
ViT-B/32
12621 36826
8276
176
43
46
ViT-L/14
16299
283
131
150
C.5. Analysis of Computational Complexity
The computational overhead of CAT Merging is reasonable and practically efficient. Specifically, CAT Merging involves two main steps:
Feature Extraction: This step is lightweight and efficient, requiring only a small number (2-3 per task) of unlabeled samples;
Eigendecomposition:
While eigendecomposstion has theoretically higher computational complexity, in practice, we efficiently mitigate this through GPU parallelization. Moreover, CAT Merging only requires the eigenvectors corresponding the top-c (2-4 in our work) eigenvalues, enabling further acceleration through specialized methods (e.g., torch.lobpeg). Empirical results (provided in the table 9, measured on single RTX3090 GPU in seconds) demonstrate that CAT Merging significantly outperforms training-based counterparts (c.g., TA w/ Surgery (Yang et al., 2024a), AdaMerging (Yang et al., 2024b)) in terms of computational efficiency.
Table 10. Balance comparison for merging ViT-B/32 and ViT-L/14 models across eight vision tasks.
Method
Fisher Merging
RegMean
Task Arithmetic
PCB Merging
CAT Merging (ours)
ViT-B/32
13.78 6.79
8.46
8.95
6,85
6.21
ViT-L/14
6.86
5.11
3.49
2.51
C.6. Analysis of the Balance during Merging
As shown in Table and 2, while CAT Merging achieves superior average performance, it does not always yield the highest accuracy on every individual dataset.
Specifically, we observe that Fisher Merging exhibits better performance on certain datasets (e.g.. Cars and SUN397), likely because its weighting mechanism, based on the Fisher information matrix, implicitly prioritizes tasks with weaker performance (larger gradients produce higher Fisher information scores). Conversely, PCB Merging achieves superior performance on datasets like SVHN and MNIST, where masking low- wwgmaanitude vector components implicitly favors tasks with stronger finetuning outcomes (assuming larger vector magnitudes correlate with greater task specialization).
However, both Fisher and PCB merging tend to perform less consistently across other tasks. In contrast, our CAT Merging framework explicitly targets inter-task knowledge conflicts and aims for a balanced integration across tasks. To quantitatively illustrate this balance, we measured the standard deviation of accuracy drops (defined as the accuracy difference between task-specific models and the merged model) across tasks. As shown in the table 10, CAT Merging demonstrates significantly lower variance, reflecting its ability to merge multiple tasks more evenly, without undue preference towards any particular task.
C.7. A Analysis of Knowledge Conflict
This section provides more evidence about the effective mitigation knowledge conflict CAT Merging. Figure visualizes the knowledge conflict during merging under different merging coefficients. As can be seen, CAT Merging consistently has lesser knowledge conflict than Task Arithmetic.
20

---

CAT Merging: A Training-Frey Free Approach for Resolving Conflicts in Model Merging
Task Arithmetic
(b) CAT Merging
(a) Task Arithmetic
(b) CAT Merging
1
2
2
10
CRSSSSSSS
1
10
10
2
0
10-3
2
10-3
asuNN397
dSuU3397
aCars
dCars
(a) SUN397 and Cars
(b) Cars RESISC45
Task Arithmetic
(b) CAT Merging
(a) Task Arithmetic
(b) CAT Merging
2
2
10-1
d'EroSAT
SVHN
10-2
10-2
10 2
10-3
2
10-3
CRRESISC4S
dRESISC4S
ClEroSAT
dguroSAT
(c) RESISC45 and EuroSAT
(d) EuroSAT and SVHN
Task Arithmetic
(b) CAT Merging
(a) Task Arithmetic
(b) CAT Merging
2
10-1
10
OCGTSRB
dANISS
10
10-2
10-2
10
10-3
2
10-3
CSSVHN
Ø'SHNN
ØCGTSRB
aCTSRB
(e) SVHN and GTSRB
GTSRB and MNIST
(a) Task Arithmetic
(b) CAT Merging
2
10-1
aDTo
10-2
10-2
10-3
2
aMNIST
CIMNIST
(g)
MNIST
Figure 5. Visualization of knowledge conflict when merging two VIT-L/14 models.
21


Reviews:
[{"review_id": "GMzMZWOJZQ", "reviewer": "Reviewer_C9mN", "summary": "This paper proposes Conflict-Aware Task Merging (CAT Merging), a training-free method to combine multiple fine-tuned models while alleviating knowledge conflicts that degrade performance when merging. The core idea is to selectively trim conflict-prone components from each task’s weight update (“task vector”) instead of simply adding them. The approach applies parameter-specific strategies – projecting linear layer weight updates and masking normalization scale/shift parameters.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q1.1: Is the Lipschitz continuity assumption becoming less reliable in Transformer architectures?**\n\n**A1.1**: We thank the reviewer for this insightful observation. Indeed, the multiplicative interactions in Transformer architectures complicate the Lipschitz continuity assumption. However, given that both the network parameters and the input data are practically bounded, we can still derive a sufficiently large Lipschitz constant, under which Theorem 5.1 remains valid—albeit with a looser upper bound. Importantly, we would like to emphasize that the primary role of Theorem 5.1 is to provide theoretical insight and motivation for our layer-by-layer trimming strategy, rather than deriving tight practical bounds. Please see response A2.6 for additional details.\n\n**Q1.2: Is the trimming method proposed to solve the magnitude issue?**\n\n**A1.2**: Not exactly. The proposed trimming method specifically addresses knowledge conflicts during model merging rather than magnitude issues alone. Magnitude-based techniques (e.g., Ties-Merging) attempt to resolve conflicts by masking low-magnitude components. However, as illustrated in Fig. 1, simply masking low-magnitude components does not fully eliminate conflicts, since high-magnitude components can also cause significant interference. In contrast, our CAT Merging explicitly identifies and trims components based on their actual contribution to knowledge conflicts, rather than simply their magnitude. This targeted strategy more effectively mitigates conflicts, leading to improved merging performance.\n\n**Q1.3: Does CAT Merging have a high computational complexity?**\n\n**A1.3:** The computational overhead of CAT Merging is reasonable and practically efficient. Specifically, CAT Merging involves two main steps:\n\n1. **Feature Extraction:** This step is lightweight and efficient, requiring only a small number (2–3 per task) of unlabeled samples.\n2. **Eigendecomposition:** While eigendecomposition has theoretically higher computational complexity, in practice, we efficiently mitigate this through GPU parallelization. Moreover, CAT Merging only requires the eigenvectors corresponding to the top-*c* (2-4 in our work) eigenvalues, enabling further acceleration through specialized methods (e.g., `torch.lobpcg`). Empirical results (provided in the table below, measured on a single RTX3090 GPU in seconds) demonstrate that CAT Merging performs much faster than training-based counterparts (e.g., TA w/ Surgery, AdaMerging).\n\n|  | ViT-B/32 | ViT-L/14 |\n| --- | --- | --- |\n| PCB Merging | 43 | 131 |\n| **CAT Merging (ours)** | 46 | 150 |\n| TATR | 176 | 283 |\n| TA w/ Surgery | 12621 | 36826 |\n| AdaMerging | 8276 | 16299 |\n\n**Q1.4: Could activation functions simplify the trimming matrices?** \n\n**A1.4:** We agree that a deeper analysis of activation functions would be beneficial. While CAT Merging does not explicitly model activation functions, it implicitly captures their effects through the layer-wise trimming strategy. For instance, activation functions such as ReLU can cause certain dimensions to consistently remain inactive. Referring to Section 5.2 of our paper, the trimming mask for scaling parameters is computed as:\n$$\\sum_{i\\neq k} \\left( \\sum_{x^l_k} ( x^l_k\\circ T^l_i )^2 - \\lambda \\sum_{x^l_i}  ( x^l_i\\circ T^l_i)^2 \\right)$$ \nIf the $d$-th dimension remains consistently inactive (i.e., $x_k^l[d]  \\equiv x_i^l[d] \\equiv 0$), the corresponding element in the trimming mask naturally becomes zero. Thus, activation functions indirectly simplify the trimming process, even though they are not explicitly modeled in CAT Merging. We will clarify this point explicitly in our revised manuscript.\n\n**Q1.5: Why should only a very low-dimensional subspace (2-4 dimensions) be trimmed as suggested in Figure 4(b)?** \n\n**A1.5:** In Figure 4(b), trimming only a low-dimensional subspace (2–4 dimensions) is sufficient because knowledge conflicts are predominantly concentrated within a few principal dimensions. Specifically, as shown in [eign.png](https://postimg.cc/jCQwQn8Z), the first few eigenvectors represent the most significant directions, accounting for an average of 78.56% (ViT-B/32) and 87.28% (ViT-L/14) of the total eigenvalues. Thus, while conflicts are indeed severe, their severity primarily manifests along these critical dimensions.\n\nMoreover, trimming a higher-dimensional subspace risks unnecessarily degrading the original task performance, as it could remove important task-specific information. Therefore, selecting this low-dimensional subspace effectively balances conflict mitigation and preservation of model performance.\n\n**Q1.6: λ in Figure 4(a) should be tuned in the range of (0, 1).** \n\n**A1.6:** As suggested, we conducted additional experiments with λ values in the (0, 1) range. The results, provided in [sensitive-lambda.png](https://postimg.cc/xcsrvQkS), indicate that model performance consistently improves as λ increases from 0 toward 1, peaking around 5.", "claims_and_evidence": "The claims made in the submission are generally supported.", "methods_and_evaluation_criteria": "The proposed methods and/or evaluation criteria (e.g., benchmark datasets) make sense", "theoretical_claims": "I have carefully reviewed all the theoretical formulations and proofs, and from my perspective, most of them appear to be correct. \n\nHowever, I have concerns regarding Theorem 5.1. The theorem seems to assume that all linear layers are stacked sequentially, which is not always the case in practical architectures. For example, in Transformer attention layers, the mathematical formulation involves a multiplicative interaction:\n\n$$z=σ(XW_Q (XW_K )^T )XW_V=σ(XW ^{KQ}X^T )XW_v,   W ^{KQ}=W_Q W_K^T$$\n\nwhere three linear layers are multiplied together rather than applied in a purely sequential manner. This multiplicative structure makes the Lipschitz continuity assumption less reliable, as the final output’s dependence on input perturbations is quadratic or even cubic (at least $XW^{KQ}X^T$ is quadratic to the input). When the experiment results in this paper mainly focus on Transformer architectures, could the author explain this assumption further?", "experimental_designs_or_analyses": "The experiments look descent.", "relation_to_broader_scientific_literature": "This work situates itself at the intersection of multi-task learning and model merging, building upon and extending prior research in both domains. The authors provide a solid literature review (Section 2) that distinguishes traditional multi-task learning from the newer paradigm of model merging.", "essential_references_not_discussed": "I am not familiar with multi-task literature and therefore unaware of important references.", "other_strengths_and_weaknesses": "Strengths:\n- The paper presents a novel solution to an important problem. This paper takes the concept of knowledge conflicts into consideration in the problem of task merging.\n- The implementation of training-free conflict-aware merging is elegant. This work explicitly formulated and solved the dual-objective conflict minimization per layer.\n- The experiment results are impressive and comprehensive.\nWeakness:\n- Over-prioritization vs magnitude. the paper mentions that differences in task vector magnitudes can lead to over-prioritization of certain task vectors, resulting in problematic merging. However, the proposed trimming method does not seem to fully resolve this issue. For example, for linear layers, the trimmed task vector is $T(I - B^l (B^l)^\\top)$. The multiplication between the matrix $T$ and $B^l (B^l)^\\top$ indicates that a task vector with a large magnitude before trimming will likely still have a large magnitude after trimming. As a result, the same over-prioritization problem mentioned in the Introduction could persist.\n- High computation complexities. Although computing the trim matrix $B_k^l$ and $m_k^l$ does not require explicit training, it relies on eigenvector computation, which has a computational complexity of $O(n^3)$. This can be a significant bottleneck, especially when dealing with large-scale models where the hidden dimension can be extremely high (e.g., 4096). \n- Activation and non-linear functions. All the formulas and implementations in this paper only consider the linear matrices. However, activation functions are an important part of neural networks, and this paper doesn’t discuss it at all. Could these activation functions can further simplify the trim matrices? For example, after ReLU, some parts of the outputs are deactivated, and we don’t need to consider the corresponding trim matrices.", "other_comments_or_suggestions": "More comments.\n- Figure 4(b) shows that the model performs best when c is relatively small (around 2 to 4). However, the trimmed task vector $TB_i^l (B_i^l)^\\top$ lies within the column space of B, meaning that only a very low-dimensional subspace (2–4 dimensions) is being trimmed.  Does this suggest that knowledge conflicts are not as severe as initially claimed?\n- Figure 4(a) shows that the model performs stable when lambda is relatively large. Since a small lambda places greater emphasis on inter-task knowledge conflict, we expect that the model’s performance should also be good if knowledge conflict is important. To fully understand the impact of the trade-off between knowledge preservation and conflict suppression, it would be more informative to include experiments where lambda falls into (0, 1).\n- The value of Eq. (21) is exactly the top c eigenvalues of the matrix $G$. Could authors provide an energy plot that visualizes the eigenvalues of the matrix G? Such a plot would help assess whether the eigenvalues decay sharply or remain relatively large beyond c components.", "questions_for_authors": "Aforementioned in previous sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4pprsWsxfr", "reviewer": "Reviewer_jpFU", "summary": "The paper proposes a novel model training-free model merging algorithm that removes the conflicting components of task vectors. This is done in a round robin fashion; for each task vector, the conflicting components of each other task vector are computed and removed from them. This is done with a projection for linear layers and masking for others. The method requires task-specific data, but authors show that even 1 sample per dataset yields good performance. The paper includes the standard 8-task CLIP benchmark proposed by the original task arithmetic paper as well as a vision-language benchmark.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q2.1: Results on LLM.**\n\n**A2.1:** Thanks for your suggestion. We conducted additional experiments using RoBERTa as the backbone model on the GLUE benchmark. As summarized in A3.3 below, CAT Merging consistently achieves superior average performance compared to existing state-of-the-art merging methods, demonstrating its effective generalization and robustness in language model merging scenarios.\n\n**Q2.2: Should all baselines from Tables 2 and 3 be included in Table 4?**\n\n**A2.2:** Thank you for this suggestion. Table 4 is intended as an ablation study specifically designed to analyze individual components within our proposed method; therefore, including all baselines from Tables 2 and 3 may not align well with its purpose. Could you please clarify whether you instead suggest adding all baselines to Table 3? If so, we will gladly incorporate the additional baselines to ensure a comprehensive and thorough evaluation.\n\n**Q2.3: How do the proofs in Appendices A.3, A.4, A.5 “imply top c eigenvalues or components”, and how is c optimized?** \n\n**A2.3:** Thank you for pointing out this issue. Specifically, the parameter c is a practical, data-dependent hyperparameter chosen to balance performance stability and exemplar efficiency. Given a fixed c, the optimality of selecting the top c eigenvectors can be rigorously justified. For instance, Eq. (21) in A.3 leads to the following optimization form:\n  $Tr(B^\\top GB)=\\sum_d \\lambda_d ||B^\\top v_d||^2$, \nThis is a well-known optimization problem whose solution is directly obtained via the Courant–Fischer theorem, establishing that the optimal choice of $B$ consists precisely of the eigenvectors corresponding to the top c eigenvalues of $G$. We will provide a clearer, step-by-step derivation of this result in the revised appendix.\n\n**Q2.4: Modify the motivation to highlight the important task directions instead of magnitude issues.**\n\n**A2.4:** Thank you for this insightful suggestion. We will clarify our motivation to reflect that CAT Merging explicitly identifies and trims conflict-prone components based on their directional contributions to knowledge conflicts rather than relying solely on vector magnitudes.\n\n**Q2.5: Should not assume α=1.**\n\n**A2.5:** Thank you for raising this point. The assumption of α=1 was introduced solely to simplify the theoretical analysis, as the scaling factor α does not affect the resulting conclusion. For example, in Eqs. (9)&(10), if we explicitly include α, the optimal vector B corresponds to the eigenvector of the matrix:\n$\\sum_{i\\neq k} {(\\alpha T_i)}^\\top( {X_k}^\\top X_k - \\lambda {X_i}^\\top X_i) {(\\alpha T_i)}=\\alpha^2\\sum_{i\\neq k} {T_i}^\\top( {X_k}^\\top X_k - \\lambda {X_i}^\\top X_i) {T_i}$. \nSince scaling a matrix by a nonzero constant α² does not alter its eigenvectors, our conclusions remain valid. Nevertheless, we agree with the reviewer that, in practical model merging, α is a critical hyperparameter that must be tuned carefully. We will provide necessary clarifications in the revised manuscript.\n\n**Q2.6: Isn't the upper bound vacuous in Theorem 5.1?**\n\n**A2.6:** We acknowledge that the bound presented in Theorem 5.1 is relatively loose. However, this bound is primarily intended to provide conceptual insight into the motivation underlying our layer-wise trimming strategy. Specifically, Theorem 5.1 establishes the inequality:\n\n$| L(W) - L( W + \\Delta W) | < \\beta \\sum_{l=1}^L \\Bigl(\\prod_{m=l+1}^L \\gamma_m\\Bigr)\n\\|\\Delta f^l(W^l) - f^l(W^l + \\Delta W^l) \\|$. \n\nThis inequality highlights that reducing layer-specific conflicts (the right-hand side) directly contributes to controlling the difference in model performance (the left-hand side). Thus, even though the bound itself is not tight, it provides theoretical justification for our subsequent conflict-aware trimming strategies at each layer.\n\n**Q2.7: Comparison of runtimes.**\n\n**A2.7:** Please see A 1.3 in our response to C9mN.\n\n\n**Q2.8: Figure 2 needs improvement.**\n\n**A2.8:** Thanks. We update Figure 2 in [sensitivity-experiment.png](https://postimg.cc/HjPTnbtt), which now more clearly demonstrates that our method remains stable in a rational range, with respect to both the number of exemplars and the value of α.\n\n**Q2.9: How are the exemplars selected in Figure 2?**\n\n**A2.9:** The exemplars are selected randomly, and we report the average performance of three random runs.\n\n**Q2.10: Do “shift” parameters mean biases?**\n\n**A2.10:** Yes. They include both the bias parameters of linear layers and the shift parameters in normalization layers. Since they share the same trimming form, we treat both of them as “shift” parameters for brevity. \n\n**Q2.11: Are attention layers treated as linear layers?**\n\n**A2.11:** We decompose it into several linear layers and treat them separately (e.g., three linear layers for Q/K/V calculations).\n\n**Q2.12: Writing issues (verbose formulations and citations).**\n\n**A2.12:** We will revise accordingly in the final version.", "claims_and_evidence": "The claims made in the submission are supported by empirical evidence. They would be stronger if LLM benchmarks would be included like in most model merging works.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria do make sense. The authors should include all baselines from Tables 2 and 3 in their final experiment presented in Table 4 for completeness.", "theoretical_claims": "There are some issues with the theoretical claims in the paper that I encourage the authors to fix and/or clarify. The authors present their method for various types of layers in subsections 5.1, 5.2 and 5.3. A derivation is given and the proofs are provided in the appendix. In all cases, the end results refer to “top $c$ eigenvalues or components”. However, in Appendices A.3, A.4, A.5 the “proof” says that this is **\"implied\"** and does not provide any reasoning about how to actually optimize for $c$. I believe that this makes the derivation too informal and undermines the paper.\n\nFurthermore, the bound provided in Theorem 5.1 is vacuous and does not provide any insight.", "experimental_designs_or_analyses": "The experiments are sound and valid, following the standard practices of the field.", "relation_to_broader_scientific_literature": "The paper reviews adequately the related work and provides multiple recent baselines in the experimental section.", "essential_references_not_discussed": "I think all essential references are discussed.", "other_strengths_and_weaknesses": "## Strengths\n\n* The paper provides an algorithm that can work with as few as 1 sample per task to remove the conflicting components in the task vectors. This is a major strength of the paper and should be highlighted more imo.  \n* The performance of the proposed method is strong across multiple benchmarks and the experimental validation includes multiple baselines.  \n* The analysis of knowledge conflict is illuminating.  \n* Afaik, the idea is original and the result significant. Similar methodologies, such as Ties are referenced in the paper.\n\n## Weaknesses\n\n1. The theoretical claims are not well supported \\- see comment on “Theoretical Claims”.  \n2. The writing of the paper can be improved:  \n   1. Motivation is task vectors with differing magnitudes (see Figure 1). However, the benchmarks are about vision classification with CLIP where the norms are approximately the same for all tasks. Hence, the motivation should be modified to mention the important task directions  \n   2. The mathematical formulations are too verbose and not enough intuition is provided. For instance, in Section 5: no need to have the superscript $l$, it makes the notation cumbersome. Similarly eq9 does not need the sum  \n3. L149-150: assuming alpha=1 is clearly wrong, since the performance of model merging papers highlights the scaling as a very important parameter. It should not be glanced over for brevity  \n4. Theorem 5.1 isnt the upper bound vacuous?  \n5. The paper claims that the procedure is lightweight but does not offer actual runtimes for comparison.", "other_comments_or_suggestions": "* L86 (2nd col) fix Ilharco citation with `\\citet`  \n* L159 fix Guillermo et al.  \n* L159 “perfect weight disentanglement” is not a proper term, since weight disentanglement has not been introduced at all at this stage\n* Include baselines in Figure 2. For figure 2a only present one model because the difference in performance between the two models might make the curves look flatter than they are.", "questions_for_authors": "1. How are the exemplars selected in Figure 2?  \n2. What exactly do you mean by “shift” parameters? The biases?  \n3. Are attention layers treated as linear layers?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nMEHPvSpBa", "reviewer": "Reviewer_jxyf", "summary": "The paper introduces Conflict-Aware Task Merging, a training-free model merging method that addresses knowledge conflicts in multi-task model merging. The meaning of knowledge conflicts is that existing methods, such as Task Arithmetic, suffer from conflicts when integrating multiple fine-tuned task vectors, often resulting in performance degradation. CAT Merging mitigates these conflicts by selectively trimming conflict-prone components from task vectors based on different parameter types: Feature projection for linear weights, Masking for scaling parameters in normalization layers and Masking for shifting parameters. The method is evaluated on vision and vision-language tasks, demonstrating up to 4.7% accuracy improvement on ViT-B/32 and 2.0% on ViT-L/14 compared to state-of-the-art model merging techniques.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q3.1: Writing issues (W1-4).**\n\n**A3.1:**  Thanks for the suggestions. We will revise them and thoroughly double-check the manuscript to avoid similar issues.\n\n**Q3.2: Comparisons on inference speed and computational overhead (W5).**\n\n**A3.2:**  \n\n**The inference speed** remains consistent with that of the individual models, as CAT Merging produces a unified model to conduct inference without adding extra layers.\n\n**The computational overhead** of CAT Merging is reasonable and practically efficient. Specifically, CAT Merging involves two main steps:\n\n1. **Feature Extraction:** This step is lightweight and efficient, requiring only a small number (2–3 per task) of unlabeled samples.\n2. **Eigendecomposition:** While eigendecomposition has theoretically higher computational complexity, in practice, we efficiently mitigate this through GPU parallelization. Moreover, CAT Merging only requires the eigenvectors corresponding to the top-*c* (2-4 in our work) eigenvalues, enabling further acceleration through specialized methods (e.g., `torch.lobpcg`). Empirical results (provided in the table below, measured on a single RTX3090 GPU in seconds) demonstrate that CAT Merging significantly outperforms training-based counterparts (e.g., TA w/ Surgery, AdaMerging) in terms of computational efficiency.\n\n|  | ViT-B/32 | ViT-L/14 |\n| --- | --- | --- |\n| PCB Merging | 43 | 131 |\n| CAT Merging (ours) | 46 | 150 |\n| TATR | 176 | 283 |\n| TA w/ Surgery | 12621 | 36826 |\n| AdaMerging | 8276 | 16299 |\n\n**Q3.3: Comparisons of LLMs (W6 and W8).**\n\n**A3.3:**  Thanks for your suggestion. \n\nTo further validate CAT Merging on language tasks, we conducted additional experiments using RoBERTa as the backbone model on the GLUE benchmark, which comprises eight diverse NLP tasks, including classification and regression (STS-B). We report accuracy for classification tasks and the mean of Pearson and Spearman correlations for the regression task. As summarized in the table below, CAT Merging consistently achieves superior average performance compared to existing state-of-the-art merging methods, demonstrating its effective generalization and robustness in language model merging scenarios.\n\n| **Algorithm** | **cola** | **mnli** | **mrpc** | **qnli** | **qqp** | **rte** | **sst2** | **stsb** | **Average** | #best |\n| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |\n| Task Arithmetic | 6.68 | 66.23 | **78.46** | 78.62 | 72.69 | 53.43 | 83.49 | 27.10 | 58.34 | 1 |\n| Ties-Merging | 9.46 | 59.34 | 74.71 | 65.93 | 41.29 | 47.29 | 72.13 | 9.210 | 47.42 | 0 |\n| PCB Merging | 11.40 | 50.85 | 77.63 | 78.22 | 55.78 | 60.29 | 75.57 | **67.01** | 59.59 | 1 |\n| CAT Merging (Ours) | **33.20** | **72.33** | 68.22 | **82.92** | **76.05** | **62.82** | **89.33** | 15.57 | **62.56** | **6** |\n\n**Q3.4: The theoretical assumptions do not capture the complexities of real-world scenarios (W7).**\n\n**A3.4:** We agree with the reviewers that theoretical assumptions, such as Lipschitz continuity, may not fully capture the complexities encountered in real-world scenarios. Nevertheless, we would like to emphasize that the primary role of Theorem 5.1 is to provide theoretical insight and motivation for our layer-by-layer trimming strategy, rather than deriving tight practical bounds. For further details, please see our response A2.6. In the final version of our manuscript, we will clarify this point further and include an explicit discussion on the potential limitations of our method.\n\n\n**Q3.5: Are the projection and masking the best strategies (W9)?**\n\n**A3.5:**  Good question. Our proposed strategies—projection for linear weights and masking for normalization parameters—are empirically motivated heuristics chosen for their computational efficiency and practical effectiveness in mitigating parameter conflicts. While these methods achieve strong empirical results, we acknowledge they are not theoretically guaranteed to be optimal. In future work, we aim to explore principled approaches, such as weighted averaging or Mixture-of-Experts (MoE), to further refine and theoretically ground our conflict mitigation techniques.\n\n**Q3.6: Comparing with training-based methods (W10).**\n\n**A3.6:**  Thank you for this suggestion. The following table shows that our CAT Merging achieves comparable or superior performance relative to two representative training-based techniques, demonstrating its effectiveness without incurring additional computational costs.\n|  | ViT-B/32 | ViT-L/14 |\n| --- | --- | --- |\n| TA w/ Surgery | 80.9 | 89.0 |\n| AdaMerging | 81.1 | 91.0 |\n| CAT Merging (ours) | 78.3 | 89.6 |", "claims_and_evidence": "S1: The paper is well-written and clearly organized, with detailed algorithmic descriptions (e.g., Algorithm 1) and thorough theoretical derivations (e.g., Theorem 5.1), making it easy for readers to understand the problem, methodology, and experiments. Additionally, the related work section is logically organized and highly readable;", "methods_and_evaluation_criteria": "S3: The method is sound, and the idea of modeling the knowledge conflict through inter-task knowledge conflict and intra-task knowledge deviation is good. Additionally, the proposal and the design of Φ_k are well-designed;", "theoretical_claims": "S2: The mathematical derivations in the paper are well executed, with sufficient details provided throughout. Additionally, the proofs included in the appendix are clear and well-structured, making it easy for readers to follow and understand;\n\nS4: The approach decomposes the global optimization problem into layer-wise sub-problems, which not only simplifies the complex merging process but also provides valuable theoretical insights.\n\nW7: The method’s reliance on theoretical assumptions (e.g., Lipschitz continuity) might not fully capture the complexities encountered in real-world scenarios; a discussion on potential limitations would be valuable.\n\nW9: While projection for linear weights and masking for normalization/shift parameters seem reasonable, the paper does not provide a principled justification for why these are the best strategies.", "experimental_designs_or_analyses": "W1: The paper provides an analysis of previous methods, such as Fisher Merging and Ties-Merging, which are discussed in sections like “Knowledge Conflict” and “Introduction”. However, the sentence in the Introduction --- “As the example shown in the figure 1, magnitude-based methods such as Ties-Merging prioritize task vectors with larger magnitudes (e.g., Task 1) while trimming dimensions with smaller magnitudes, inadvertently discarding critical information from Task 2.” may require further clarification. In contrast, the statement in the Knowledge Conflict section --- “Most existing methods such as Fisher Merging or Ties Merging, implicitly prioritize preserving intra-task knowledge while neglecting the inter-task knowledge conflict” is more direct and could be placed earlier in the paper to better introduce the problem.\n\nW6: The experimental evaluation does not include studies on large language models (LLM), which may limit the understanding of the method’s applicability in broader contexts.\n\nW8: The paper evaluates CAT Merging on ViT and BLIP models, but does not test its applicability to LLMs.\n\nW10: While training-based methods have additional costs, it would be useful to see a comparison against test-time adapted merging techniques to better contextualize the trade-offs.", "relation_to_broader_scientific_literature": "S5：The approach is training-free and only relies on a lightweight forward pass with a small number of exemplars.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "The writing is clear, and the logic is well-structured. The method is novel and sound, and I appreciate the modeling of knowledge conflict, the proposal of the Φ_k operation, and the design of the parameter-specific strategies. Additionally, I think the method is reproducible. Considering the strengths and weaknesses discussed above, I find this paper to be a solid contribution. I also hope that the work presented in this paper will see broader applications in real-world settings.", "other_comments_or_suggestions": "W2: The formatting of Equation (7) appears to have an issue, as there seems to be an extra Δ symbol.\n\nW3: There seems to be an issue with Equation (15), where it likely misses parentheses (( and )). Additionally, in the term ∥T_i^l+T_i^l∘m_k^l ∥^2, there should be a - instead of a +.\n\nW5: Additional comparisons on inference speed and computational overhead during the merging process would offer a more complete evaluation of the method’s efficiency.", "questions_for_authors": "Please refer to the above comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFJt8X0SdO", "reviewer": "Reviewer_oc91", "summary": "## Summary\n\nThe paper introduces CAT Merging, a training-free framework for merging multiple expert models while mitigating knowledge conflicts. Existing methods, such as task vectors, merge models by accumulating task vector weights, but conflicting components across tasks can lead to performance degradation. CAT Merging addresses this by formulating conflict resolution as an optimization problem to trim conflict-prone components. Experimental results show that CAT Merging significantly reduces knowledge conflicts and show (in most cases) accuracy gains of a few percentages over SoTA.\n\n## Update after rebuttal\nI appreciate the response by the authors. I maintain my score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Q4.1: Pay attention to the concurrent work.**\n\n**A4.1:** Thank you for highlighting the concurrent work, \"Interfering with Interference: Blind Shuffling and Superposition for Better Multi-Model Compression,\" which addresses interference during multi-model merging through random layer shuffling and orthogonal transformations.\n\nWe appreciate the relevance, as both this concurrent paper and our CAT Merging framework aim to mitigate interference between task vectors. However, there are fundamental differences between their approach and ours:\n\n**Motivation and Insight**: The concurrent work attributes interference primarily to task vector similarity and proposes randomization-based techniques (layer shuffling and task vector superposition) to increase orthogonality. In contrast, our work explicitly identifies conflict-prone components within task vectors and resolves them through targeted, parameter-specific trimming strategies.\n\n**Methodological Differences**: Their approach depends on randomized transformations, necessitating task-specific decoding (inverse transformations) at inference time. This requirement limits applicability in scenarios involving mixed-task batches. In contrast, CAT Merging enables seamless multi-task inference without task-specific decoding or per-sample routing, making it more suitable for practical deployments involving shared-task batches.\n\nWe will include a discussion of this concurrent study in the related work section of our revised manuscript to position our contributions relative to this relevant work.\n\n**Q4.2: What factors lead to the suboptimal performance of CAT Merging?**\n\n**A4.2:** We thank the reviewer for highlighting this important point. While CAT Merging achieves superior **average** performance—improving accuracy by 2.5% (ViT-B/32) and 2.0% (ViT-L/14) compared to state-of-the-art methods—it is true that it does not always yield the highest accuracy on every individual dataset.\n\nSpecifically, we observe that Fisher Merging exhibits better performance on certain datasets (e.g., Cars and SUN397), likely because its weighting mechanism, based on the Fisher information matrix, implicitly prioritizes tasks with weaker performance (larger gradients produce higher Fisher information scores). Conversely, PCB Merging achieves superior performance on datasets like SVHN and MNIST, where masking low-magnitude vector components implicitly favors tasks with stronger fine-tuning outcomes (assuming larger vector magnitudes correlate with greater task specialization).\n\nHowever, both Fisher and PCB merging tend to perform less consistently across other tasks. In contrast, our CAT Merging framework explicitly targets inter-task knowledge conflicts and aims for a balanced integration across tasks. To illustrate this balance quantitatively, we measured the standard deviation of accuracy drops (defined as the accuracy difference between task-specific models and the merged model) across tasks in the following table. It shows that CAT Merging demonstrates significantly lower variance, reflecting its ability to merge multiple tasks more evenly, without undue preference towards any particular task. We will clarify this analysis and discussion in the revised manuscript.\n\n|  | Fisher Merging | RegMean | Task Arithmetic | PCB Merging | CAT Merging (ours) |\n| --- | --- | --- | --- | --- | --- |\n| ViT-B/32 | 13.78 | 8.46 | 8.95 | 6.85 | **6.21** |\n| ViT-L/14 | 6.79 | 6.86 | 5.11 | 3.49 | **2.51** |\n\n**Q4.3: What is the significance of the removal basis? Is it binary?**\n\n**A4.3:** Thank you for this helpful suggestion. We will clarify this point explicitly in the revised manuscript. Specifically, the removal basis $B$ is a real-valued matrix (not binary). It is optimized to define a task-specific subspace, within which conflict-prone components are identified and suppressed via orthogonal projection. The basis matrix $B$ thus plays a crucial role in effectively mitigating knowledge conflicts during the merging process.", "claims_and_evidence": "Yes. I found the paper very clear in this respect. The presented claims are well supported by theoretical arguments and experiments.", "methods_and_evaluation_criteria": "Yes, the method is thoroughly evaluated across multiple architectures and datasets. Additionally, the paper includes a well-conducted ablation study and sensitivity analysis.", "theoretical_claims": "I checked the correctness of the claims in the main paper. I didn't look into the proofs provided in the appendix.", "experimental_designs_or_analyses": "The experimental results seem sound, but the reviewer is concerned that the performance gains are relatively modest. Given that the proposed method does not achieve the best performance across all tested settings despite its solid theoretical foundation, it would be valuable to analyze and discuss the sources of suboptimality.", "relation_to_broader_scientific_literature": "The authors provide a clear presentation of their method and its contributions to the scientific literature. I would like to bring their attention to the following concurrent work: https://openreview.net/forum?id=4wuvmJRAU4.", "essential_references_not_discussed": "The references are sufficient.", "other_strengths_and_weaknesses": "The paper presents a theoretically grounded approach to conflict resolution in task vectors, making it an interesting read. However, the somewhat low score is primarily due to the method's weak performance compared to the baselines and the lack of clarity on the underlying reasons for this suboptimality.", "other_comments_or_suggestions": "In Sec. 5.1, it would be helpful to clarify the significance of the removal basis B. It is also unclear if B is a binary matrix. Explicitly stating this would improve clarity.", "questions_for_authors": "What factors contribute to the suboptimal performance of CAT Merging?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents Conflict‑Aware Task Merging (CAT Merging), a training‑free method for combining multiple fine‑tuned models by selectively trimming conflict‑prone components of task vectors via layer‑wise projections for linear weights and masking for normalization parameters. Reviewers agree that the problem is important, the approach is novel and well‐motivated, and that the core claims are broadly supported by both theoretical derivations and empirical results on vision and vision‑language benchmarks, with gains over prior model‑merging techniques at effectively zero incremental memory cost. However, several concerns must be addressed: the theoretical analysis (in particular Theorem 5.1) relies on assumptions (e.g., sequential linear layers, Lipschitz continuity) that may not hold for architectures with multiplicative interactions such as Transformers, and some proofs omit constructive arguments for choosing the trimming dimension; the computational cost of eigenvector computations at high hidden dimensions, the omission of activation functions from the analysis, and the lack of runtime comparisons need clarification; the presentation would benefit from clearer notation, inclusion of all baselines (especially in the final table), lambda‑tradeoff experiments, energy plots of eigenvalue decay, and broader evaluation on LLM benchmarks; and minor writing fixes and expanded discussion of suboptimal cases are needed. Given the paper’s solid foundation but these outstanding issues, I recommend a weak accept.
Decision: Accept (poster)