CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Authors: Kunda Yan
Min Zhang
Sen Cui
Qu Zikun
Bo Jiang
Feng Liu
Changshui Zhang
Venue: ICML 2025 poster (2025)
Keywords: model merging; multi-task learning; global task consensus; localized merging
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: A localized model merging method capable of identifying global task consensus.

Abstract: Model merging aims to integrate the strengths of multiple fine-tuned models into a unified model while preserving task-specific capabilities. Existing methods, represented by task arithmetic,  are typically classified into global- and local-aware methods. However, global-aware methods inevitably cause parameter interference, while local-aware methods struggle to maintain the effectiveness of task-specific details in the merged model. To address these limitations, we propose a Consensus Aware Localized Merging (CALM) method which incorporates localized information aligned with global task consensus, ensuring its effectiveness post-merging.  CALM consists of three key components: (1) class-balanced entropy minimization
sampling, providing a more flexible and reliable way to leverage unsupervised data; (2) an efficient-aware framework, selecting a small set of tasks for sequential merging with high scalability; (3) a consensus-aware mask optimization, aligning localized binary masks with global task consensus and merging them conflict-free. Experiments demonstrate the superiority and robustness of our CALM, significantly outperforming existing methods and achieving performance close to traditional MTL.

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Kunda Yan Min Zhang" 2
Sen Cui
Zikun Qu
Bo Jiang" Feng Liu Changshui Zhang'
Abstract
both model efficiency and performance (Liu et al., 2019a;b; Dong et al., 2015; Yang et al., 2024c; Qiu et al., 2024; Xin et al., 2024). This approach has been widely applied in fields such as computer vision (Cai et al., 2024a; Lu et al., 2020; Qiu et al., 2024; Yang et al., 2024c; Cai et al., 2024b), natural language processing (Liu et al., 2019b; Dong et al., 2015; Audibert et al., 2023; Xin et al., 2024), and so on. However, in the context of large foundational models, tradi- tional MTL methods often require the centralized collection and processing of vast amounts of data, leading to high costs in data labeling and computational resources. At the same time, with the widespread use of pre-trained models, down- stream tasks typically fine-tune the same pre-trained model (e.g., ViT (Dosovitskiy, 2020) or BERT (Devlin, 2018)) in- dependently, and the fine-tuned models are usually released without disclosing the specifics of the original training data to protect privacy. In recent years, researchers have changed their focus towards exploring how to effectively integrate multiple independently trained models to achieve MTL with- out re- re-training on the original data.
Model merging aims to integrate the strengths of multiple fine-tuned models into a unified model while preserving task-specific capabilities. Ex- isting methods, represented by task arithmetic, are typically classified into global- and local- aware methods. However, global-aware meth- ods inevitably cause parameter interference, while local-aware methods struggle to maintain the ef- fectiveness of task-specific details in the merged model. To address these limitations, we propose Consensus- A ware Localized Merging (CALM) method which incorporates localized informa- tion aligned with global task consensus, ensur- ing its effectiveness post-merging. CALM con- sists of three key components: (1) elass-balanced entropy minimization sampling, providing a more flexible and reliable way to leverage un- supervised data; (2) an efficient-aware frame- work, selecting small set of tasks for sequential merging with high scalability: (3) consensus- aware mask optimization, aligning localized bi- nary masks with global task consensus and merg- ing them conflict-free. Experiments demonstrate the superiority and robustness of our CALM, sig- nificantly outperforming existing methods and achieving performance close to traditional MTL.
In response, model merging (or fusion) methods have been employed to address challenges in traditional MTL (Yang et al., 2024b; Yadav et al., 2023; Ilharco et al., 2023; Tang et al., 2023; Yang et al., 2024a; Wang et al., 2024; He et al., 2024; Chen & Kwok, 2024). A key approach in this field is task arithmetic (Ilharco et al., 2023), which introduces the task vector to represent task-specific weight adjustments. A task vector created by subtracting task-specific weights from pre-trained weights, producing a unique representation for each task. Research has demonstrated that by strategi- cally merging multiple task vectors and incorporating them into a pre-trained model, a new model can be created to effectively support MTL. As result, most model merging methods rely on task vectors to construct integrated models.
1. Introduction
Multi-task learning (MTL) facilitates knowledge transfer across different tasks through a shared backbone, improving
'Equal contribution Corresponding author Instittte for Artifi- cial Intelligence, Tsinghua University (THUAD) Beijing National Research Center for Information Seience and Technology (BN- Rist), Department of Automation, Tsinghua University, Beijing, P.R. ChinChina "Est China Normal University (ECNU) "The Chinese University of Hong Kong, Shenzhen "School of Computing and In- formation Systems, The University of Melbourne. Correspondence wwwitsssiinuuu..oooouu
Task vector-based model merging methods can currently be categorized into two groups based on the information focus between task vectors: global-aware methods (as illustrated in Figure I(a)) and localized-aware methods (as illustrated in Figure 1(b)). Global-aware methods, such as those uti- lizing arithmetic mean (Ilharco et al., 2023) and learned merging weights (Yang et al., 2024b), process the models in a global manner, performing arithmetic operations on all the parameters of the fine-tuned models. However, parameters from different tuned models inevitably inter-
to: Sen Cui <cuis <zcs@ tsinghusingu....uu... edu.en">.
Changshui Zhang
Proceedings of the 4rnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

CALM: Consensus-Aware Localized Merging for Mlulti-Tak Learning
T1
T2
T1
2
T1
22
Localized-Aware Method CALM
Global Aware
Localized Aware
Localized Aware
CALM
84
Meitod to CALM
Global-4wara
M, = ((rr
M₂ (xx))
M = 2(2))12)
Merging Coefficient 21
Merging Coefficient A2
80
Tmrl 1 + AzTz (a)
Tmm Or + M₂OTz (b)
mmt (1 M) + MOT₂ (c)
0.0
Interpolation Ratio from Inttaa to CALM Mask
0.2
0.4
0.8
0.0 1.0
(d)
Figure 1. (a) Global-aware methods: Process the models holistically by applying arithmetic operations to all parameters of the fine-tuned models. (b) Localized-aware methods: Rely on lask-specific optimized masks extract local information from task vectors. Our CALM method: Utilize masks to extract locally effective information with global task consensus and merge it without information conffict. (d) Experimental validation: Present the accuracy curve of the merged model which gradually interpolates the information extracted by global-aware and localized-aware methods with our approach, validating that CALM integrates more effective localized information.
fere with each other, which leads suboptimal performance in the merged model. Prateek et al (Yadav et al., 2023) vali- dates that redundant parameter updates during fine-tuning often serve as significant source of conflicts.
sus. During merging, the effective information from both task vectors is merged without conflict. Figure (d) presents experimental validation. By interpolating the information extracted by global-aware and localized-aware methods with our approach, we observe that as the merged model's infor- mation approaches ours, its performance improves. This confirms the superiority of the global consensus-basee in- formation extraction in model merging-
In contrast, localized aware methods extract local informa- tion from task vectors, which mitigates conflicts between global parameters. The core of these methods lies in iden- tifying effective local information, with criteria varying across approaches. For instance, Ties-merging (Yadav et al., 2023) emphasizes parameters with larger magnitudes, while Localize-and-stitch (He et al., 2024) prioritizes parameters that preserve local task performance. Although these meth- ods ensure strong local task performance before merging, the overall performance of the merged model tends de- grade significantly, indicating that this local information may not be universally effective for all tasks. This natu- rally raises a pivotal question:
Specifically, CALM consists of three key components: First, we introduce class-balanced entropy minimization sam- pling method. Unlike minimizing the entropy of unsuper- vised data or using supervised training data, this method provides a more flexible and reliable way to leverage un- supervised data. Second, we propose an efficient-aware framework, which only requires selecting a small num- ber of tasks for sequential merging, offering more effi- cient solution with good scalability. Finally, we present a consensus-awarr mask optimization method, which lo- cates masks with global consensus using credible datasets and merges them without conflict. Experimental results demonstrate the superiority and robustness of our approach, significantly outperforming existing baseline methods and approaching the performance of traditional MTL. Code is available at https : / / thub. com hamm//mmkk22//aa
How to identify and extract effective information in model merging enhance performance on all tasks?
Given the above analysis, we propose that effective infor- mation in model merging should possess two key char- acteristics: (I) effective information can be represented by localized parameters, minimizing interference from global merging; and (2) effective information should align with the global task consensus, maintaining its efficacy in the merged model. Based on this criterion, we propose a Consensus-A -eeesussswwaaar ware Localized Merging (CALMM) method. The core idea of CALM is illustrated Figure (c). merging two task vectors, a mask M extracts effective infor- mation from each task vector, which is optimized through the global tasks, ensuring alignment with the global consen-
Our main contributions of this paper are three-fold:
We explore core issue in model merging: how to identify and extract effective information on each task. We argue that effective information should be localized and aligned with global task consensus.
We propose a novel model merging framework, CALM, which leverages reliable unsupervised data extract
2

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
local information aligned with global task consensus, and enables efficient and conflict-free merging.
2.2. Traditional Multi-Task Learning
Traditional multi-task learning (MTL) improves model per- formance and efficiency by transferring knowledge across multiple tasks through shared backbone (Liu et al., 2019b; Dong et al., 2015; Yang et al., 2024c; Zhang et al., 2024; 2023; 2022). However, MTL often faces the problem of negative transfer due to conflicts and interference between tasks (Lin et al., 2019; 2020). Current research address- ing negative transfer primarily focuses on two areas: (I) The classic SharedBottom architecture (Caruana, 1997) per- forms poorly when task correlations are weak, whereas mod- ern architectures alleviate negative transfer through modular design (Ponti et al., 2022; Chen et al., 2023; Swamy et al., 2024), sparsification (Sun et al., 2022; Calandriello et al., 2014; Ma et al., 2022), and soft sharing of the backbone network (Shi et al., 2023). (2) Other research tackles task interference from an optimization perspective, such ad- justing the loss weights for each task (Yu et al., 2020; Liu et al., 2021), resolving conflicts in multi-task gradient di- rections or signs, or suppressing the dominance of learning rates and gradients (Chen et al., 2018). Unlike these tradi- tional methods that focus on loss weights or gradient space, our proposed CALM improves MTI. performance and ad- dresses conflicts through emphasizing local information aligned with global consensus in model merging.
We conduct extensive experiments various datasets and demonstrate that CALM exhibits superior perfor- mance in model merging, surpassing other state-of-the of-the- art global-aware and localized-awan methods.
2. Related Work
2.1. Model Merging for Multi-Task Learning
Model merging represents the process of combining multi- ple independent models into a single model (Ilharco et al., 2023), with two primary application scenarios: (I) merg- ing models trained on the same task to enhance the final model's resistance to forgetting and improve its generaliza- tion (Wortsman et al., 2022; Yao et al., 2023; Zhu et al., 2024; Panigrahi al., 2023). (2) merging models trained on different tasks to achieve MTL (Yang et al., 2024b; He et al., 2024; Matena & Raffel, 2022; Yu et al., 2024; Yang al., 2024a), which is more aligned with real-world applications and is the focus of this paper. These MTL model merging methods can currently be categorized into two groups based on the information focus between task vectors: global-awan methods and localized-aware methods.
Global-aware methods. Global-aware methods, such as those employing arithmetic mean (Ilharco et al., 2023) or learned merging weights (Yang et al., 2024b), process mod- els holistically by applying arithmetic operations across all parameters of fine-tuned models. For instance, AdaMerg- ing (Yang et al., 2024b) leverages unlabeled test data to automatically learn merging coefficients at the task or layer level. Similarly, DARE (Yu al., 2024) reduces redun- dant neuron updates before scaling neurons for merging. However, combining parameters from different fine-tuned models inevitably leads to interference with each other, re- sulting in suboptimal performance in the merged model.
3. Preliminaries
In this section, we first give the notation and definition, then provide a detailed explanation of model merging solutions.
Notation. Let the neural network model be denoted as represented by
f : X x O where the parameters 0 E θ E R", the input by x; E X E Rd, and output by yi y E R.. Here, n denotes the number of parameters, d is the input dimension, and e means the number of output classes. Consider a scenario involving T independent tasks, where each task f has an associated training dataset Di, (X, y). Given a pre-trained model ppre. such as ViT (Dosovitskiy, 2020) BERT (Devlin, 2018), each task's training dataset is used to fine-tune the pre-trained model, resulting in T fine-tuned models, denoted as (0, (T
Localized-aware methods. In contrast, localized-aware methods (Yadav et al., 2023; He et al., 2024; Wang et al., 2024; Chen & Kwok, 2024) emphasize extracting task- specific, localized information from task vectors to reduce conflicts among global parameters. These approaches cen- ter on selecting effective local information, with selection criteria differing between methods. For example, (Yadav et al., 2023) prioritizes parameters with larger magnitudes, whereas (He et al., 2024) focuses on parameters that main- tain local task performance. While these methods help preserve strong local task performance before merging, the overall performance of the merged model often suffers, sug- gesting that this localized information may not generalize effectively across tasks. Therefore, we propose that the ef- fective information extracted in model merging should be localized information with global task consensus.
Traditional multi-task learning. Model merging is to merge the weights (0 into a unified model 0mti with- out retraining, such that the combined model performs ef-
fectively across all tasks. Specifically, given the test datasets Dj, for each task t, the goal is minimize the empirical loss L of the merged model Omn, defined as:
[(0mu)
1 T
Σ (s..)))D
(2)). (1)
D
where 1(.) is the loss function, e.g. cross-entropy.
3

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Task vector-based model merging. Following task arith- metic (Ilharco al., 2023), the task vector 7t is defined as the difference between the fine-tuned model parame- ters 8i, and the pretrained model parameters Opre, i.e., To =0;; Opre. It captures the model parameter updates during fine-tuning, reflecting adjustments for task adapta- tion. Task vector plays crucial role in model merging, providing a compact yet highly informative representation of task-specific modifications that facilitates efficient MTL.
curate entropy estimates can misdirect optimization, leading to suboptimal merged models. To address this, pro- pose class- blass-balanced entropy minimization sampling, which adaptively selects credible samples, using pseudo-labed to ensure reliable information remains intact during merging.
Shannon entropy measure. Given a dataset D = { and a finetuned model with parameters 6, the model's logit output P( zz 9) provides the class probabilities. The Shan- non entropy (Shannon, 1948) for each sample is defined as:
Representative model merging solutions.
C
H 0) - Σ P(ù = c 0) log P(ù = c/22, 8), (2)
Task Arithmetic (Ihharco et al., 2023) is global- aware approach that merges task vectors with the pre- trained model ppre using weighted coefficients A. This allows for a significant improvement over simple aver- aging. The formula is 0mn ppre + À Σ Tr.
where C the total number of classes. High entropy indi- cates greater uncertainty in the model's prediction, while low entropy suggests higher confidence.
Entropy minimization sampling (EMS). EMS aims to se- lect samples with the lowest entropy to construct a credible sample set from the unsupervised dataset. This approach is widely used in fields like active learning (Wu et al., 2022; Xie al., 2022) and anomaly detection (Yoon al., 2023) to reduce uncertainty and enhancing model stability and robustness. Here, we apply EMS to model merging.
Ties-Merging (Yadav et al., 2023) is a localized-aware method that focuses on the parameters with the largest magnitudes in the task vectors. It employs the elect strategy to selectively merge these parameters with minimal conflict, resulting in a merged parameter 8ties- The formula is 0nu pre + Aôties.
Adamerging (Yang et al., 2024b) is global-aware method that optimizes the merging weights of task vectors by minimizing the entropy of unsupervised samples. This approach refines the merging process across tasks and allows for layer-level adjustments. The formula is Onti= Opre + Σt" deTi.
Given multiple task-specific unsupervised datasets (D) our goal is to select low-entropy samples for each task t. which can be formulated as:
arg min D
H( Σ 2.ED!
(3)
D
4. Methodology
This selection improves model merging robustness at the data level. Additionally, to prevent interference with entropy information during parameter merging, we assign pseudo- labels to unsupervised data based on logits:
This section details the implementation of CALM. Sec- tion 4.1 explains the method for obtaining a reliable unsu- pervised dataset, while Section 4.2 introduces the efficient- aware framework, which extracts only a small number of task execution sequences for merging. Building on this, Section 4.3 presents a method for identifying localized pa- rameters with global task consensus using binary masks. The overall process is summarized in Algorithm 1.
arg max P(j c)z,,
(4)
Thus, obtain a credible dataset with pseudo-labels that re- main unaffected when merging, denoted as (2'
Class-balanced entropy minimization sampling (CB- EMS). To address class imbalance that may occur from selecting only low-entropy samples, we propose class- balanced entropy minimization sampling (CB-EMS). The core idea is to select an equal number of low-entropy sam- ples from each class to ensure balanced representation. For the dataset of class c, D we define:
4.1. Hlass-Balanced Entropy Minimization Sampling
Utilizing additional sample information to aid model merg- ing often significantly improves performance. For example, Adamerging (Yang et al., 2024b) learns from the entropy of unsupervised samples, allowing for balanced task perfor- mance, while Localize-and-Sticee (He et al., 2024) relies on training set data to identify effective parameter points. In contrast, acquiring a subset of unsupervised samples is more practical in real-world scenarios.
Dra U E D Ho,, S (DN 0z)), (5) c=1
However, we argue that the entropy values used in Adamerg- ing (Yang et al., 2024b) are not entirely reliable, as shown in Figure& 8, and may be affected by the merged model. Inac-
where H(R) (Di (/,)) is the entropy of the k-th lowest en- tropy sample in D'. Finally, we assign pseudo-labels to create the class-balance credible dataset (D D)
4

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Sequential Task Vector
Efficientt Mergod Task Vector
Masked Model Merging
Task Vectors (re3i-1
M
M
T
(b) Efficent-Aware Framework
Tany
M Seo
M MOT,
Masked Model Optimization
Finetuned Models
Logits
Credible Samples
Hy < Rivn!
Eu
R, Ross
Ppre 8
Unsupervised Datasets (D,)
(od)
Ry Howay
(2
DD)
Opre + Teeq
(a) Class-Blanced Entropy Minimization Sampling
(c) Consensus-Aware Mask Optimization
Figure 2. Illustration of Our CALM Approach. CALM involves three stages: (a) Efficient Aware: Task vectors split, with most merged efficiently into T. via Task Arithmetic, and a smaller part used for sequential merging; (b) Class-Balanced Entropy Minimization Sampling: Finetuned models extract credible samples from the unlaheled dataset while maintaining class halance; (c) -onsensus-Aware Mask Optimization:
Sequential tasks are merged into the model using a mask, which is then refined using the credible samples.
4.2. Efficieet-Aware Framework
ciently integrates new tasks without needing recombine all previous tasks, thereby showing greater practicality.
We propose an efficient- aware model merging framework. Unlike traditional methods, for given set of tasks, repre- sented by task vectors (Te ) we divide these vectors into two parts: S and S, with the majority of tasks belonging to S. For the tasks in S, we perform efficient merging utilizing task arithmetic (Ilharco et al., 2023), resulting in an initial merged model defined as Te = A EiEs Tt.
4.3. Consensus-Aware Mask Optimization
Based on the CB-EMS dataset and the efficient-aware frame- work, this part aims to address how to identify and extract effective information during the model merging process. As demonstrated by the validation experiments in Figure 1, effective information is not uniformly present across all parameter points. Instead, extracting appropriate localized parameters can significantly improve the performance of model merging. Furthermore, the extraction of effective information should not be limited individual tasks; due to the interference among tasks in model merging, the effec- tiveness should be the consensus of all tasks globally.
The remaining tasks in S are merged sequentially. We use (0) as the base model, denoted Taeq and then incrementally merge each task from S. For each task j E S, the current merged model Tooq (j-1) updated by merging with the task vector Ty. resulting in maqq
(i) [ ( T,), where f() is applicable model merging method. This process is repeated until all sequential tasks have been merged, yielding the final merged task vector Tmul- Figure 2(b) illustrates the merging process when 5 contains only one model.
Building on this understanding, we propose two guiding principles for our design approach: First, model merging should focus on localized parameters. Second, the selection of these localized parameters should take into account global task information to achieve consensus.
The efficient-aware framework demonstrates substantial ad- vantages, leading to a significant enhancement in merging efficiency, while simultaneously mitigating the rapid in- crease in computational complexity as the number of tasks expands. Therefore, it shows impressive scalability, particu- larly when dealing with an increasing volume of tasks.
Mask is effective tool for focusing on local parame- ters. Several existing studies have explored the idea of using masks. For instance, Ties-Merging (Yadav et al., 2023) ap- proximates the selection of the largest parameter points through a mask, while Localize-and- aims to find a mask that better represents single task. In this study, propose novel masked model merging strat- egy. Given a current merged task vector Tseq (0-1) sequential task vector T, awaiting merging, and binary mask M, we define the masked model merging as follows:
Furthermore, our framework shows high flexibility in han- dling new tasks. In practical scenarios, task asynchrony is very common, with different tasks often completing at dif- ferent times. Traditional methods typically require all tasks to be completed before merging, which inevitably leads to delays. Additionally, in resource-constrained environments, it is challenging to merge all tasks simultaneously using conventional approaches. In contrast, our framework effi-
(He et al., 2024)
(f) Teeq
= (1 M)) C ( + MOT
(6)
5

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Our masked model merging strategy has three main advan- tages: (1) effectively extracts information from the new task Tys (2) removes interference present in Tseg - from the efficient merging process; and (3) ensures conflict-free parameter merging. Compared existing methods, our approach is better suited for complex tasks and improves the robustness of model merging. Since the mask is useful, the next challenge is how to obtain an effective mask.
Algorithm 1 CALM
Input: Number of tasks T. pretrained model Opre. fine- tuned models [ task-specific unsupervised datasets
( efficient merging coefficient X, regularization
parameter a.
1: Step 1: Compute Task Vectors
2: for each task from 1 to T do
3: Compute task vector 7 = 0ij Ppee
Mask optimization should consider global tasks consen- sus. Current localized-awand methods often focus solely on individual tasks, but effectiveness in a single task does not guarantee effectiveness after merging. Therefore, it is erucial to evaluate effectiveness from global task perspec-
4: end for
5: Step 2: Class-Balanced EMS
6: for each task from T do
7: Obtain class-balanced credible samples as Eq. 5;
8: Assign pseudo-labels to create Di (X,)));
tive. In the context of serializing the merging task j. let S. denote the current set of visible tasks, which includes task j and all preceding visible tasks. Leveraging the credible dataset obtained through the CB-EMS method, for each visi- ble task t, E Sy, we can compute the empirical loss Lr, Therefore, the optimization objective for the mask is:
9: end for
10: Step 3: Efficient-Aware
11: Randomly select few tasks as sequential tasks S;
12: The rest form efficient merged tasks S:
(0) 13: Efficiently merge S to obtain Tseg = X > TV
14: Step 4: Consensus-Aware Mask Optimization
15: visible tasks S, = S, real-valued mask RC
min Σ C., (0l2)) + alIMM|,, E.
16: for each task j in do
17: Add task j to S.:
18: Masked model merging:
+()) where 060 Ppre + Teeg. As shown, our mask optimization
19: 012-1) = 6pe + (R)) O (1--)) + oR) @ Ty:
takes into account all current visible tasks, allowing effective consensus to be extracted a broader scale. In addition,
20: Masked model optimization:
R arg min Σ ES. Lr, (8(6 + (R]oo(R)]l
21:
we introduce the L1 norm ||M || to promote sparsity in the mask, with as the hyperparameter of the L1 term.
22: Binarization: M = Round(o (R*) ));
(1) 23: Update: Tang = (1 - M*) O Tseg
(9-1)
+ M Tji
Through the masked model merging and optimization pro- cesses, we successfully obtained the binary mask M re- quired for merging the task j. and completed the merging of task j. resulting in (i) However, effectively optimizing a binary mask is challenging. In practice, following (Pan- igrahi et al., 2023; He et al., 2024), we re-parametrize the binary mask as the sigmoid of real-valued vector R, i.e., M = (R) to facilitate the optimization process. Combin- ing the above mask optimization method with the efficient- aware framework, we implement serialized merging ap- proach, as detailed in Algorithm 1. The final binary mask is obtained by rounding the sigmoid output, represented as M = Round(o (R)), resulting in the next-step task vector:
24: end for
25: Output: Merged model @(final) = Opre + Tsey
(final)
5.1. Experimental Setup
Baselines. We compare CALM with eight representative model merging approaches. These baselines fall into three categories. The first includes traditional model merging methods that do not rely on task vectors, such as Simple Averaging, Fisher Merging (Matena & Raffel, 2022), and RegMean (Jin et al., 2023). The second category comprises alobal-aware methods, including Task Arithmetic (Ilharco et al., 2023), AdaMerging (Yang et al., 2024b) and Pareto Merging (Chen & Kwok, 2024). We evaluate two strategies for AdaMerging: task-wise (7W AdaMerging) and layer- wise (LW AdaMerging). The third category encompasses localized-awarr methods, such as VIES-Mggging (Yadav et al., 2023), Consensus TA (Wang et al., 2024), Consen- sus TIES (Wang et al., 2024) and Localize-and-Stee et al., 2024). Additionally, we present three comparison results: the performance of pretrained models without merg- ing (the lower bound), individual models and traditional MTL, which serve as the upper bound.
(1)) = (1 - M²) @ T Sang
()-1) + M @ Tj.
8
seq
Finally, we obtain the merged model as follow:
Olfinal) Opre + seq
(ffinal).
(He
5. Experiments
This section outlines our experimental setup and presents performance comparisons. Additionally, we provide com- prehensive analysis of the robustness and convergence of our CALM, as well as the effectiveness of CB-EMS.
Datasets. Following previous work on generic datasets (II-
6

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Pretrained
62.3
59.7
60.7
45.5
31.4
32.6
48.5
43.8
48.0
Individual
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
Traditional MTI.
73.9
74.4
93.9
98.2
95.8
98.9
99.5
77.9
88.9
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
50.1
65.8
RegMean (Jin et ul., 2023)
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
Fisher Merging (Matena & Raffel, 2022)
68.6
69.2
70.7
66.4
72.9
51.1
87.9
59.9
68.3
Task Arithmetic (Ilharco et al., 2023)
55.2
54.9
66.7
78.9
80.2
69.7
97.3
50.4
69.1
TW AdaMerging (Yang et al., 2024b)
58.0
53,2
68.8
85.7
81.1
84.4
92.4
44,8
71.1
LW AdaMerging (Yang et al., 2024b)
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
Pareto Merging (Chen & Kwok, 2024)
71.4
74.9
87.0
97.1
92.0
96.8
98.2
61.1
84.8
Ties-Merging (Yadav al., 2023)
59.8
58.6
70.7
79.7
86.2
72.1
98.3
54.2
72.4
Consensus TA (Wang et al., 2024)
63.9
64.1
75.5
79.4
81.6
69.9
98.0
55.1
73.7
Consensus TIES (Wang et al., 2024)
62.3
62,2
74.5
80.0
87.7
77.5
98.6
55.3
74.8
Lacalize-and-Stich (He et al., 2024)
67.2
68.3
81.8
89.4
87.9
86.6
94.8
62.9
79.9
CALM (ours)
72.6
74.8
91.9
98.6
95.2
96.4
99.1
72.8
87.7
Table 1. Multi-task performance with CLIP VTT- architecture on eight vision classification tasks.
harco et al., 2023; Yang et al., 2024b; He et al., 2024), we evaluate the performance of CALM on visual clas- sification tasks and 12 natural language tasks. The eight visual classification datasets include SUN397 (Xiao al., 2016), Cars (Krause et al., 2013), RESISC45 (Cheng et al., 2017), EuroSAT (Helber et al., 2019), SVHN (Netzer et al.,
MFGA
Ind
TREC
Time Merg lnn
lack ithhetic
Li Admmrgging
LIC
Consentux
Loos and
SUBL
OR
CALI
201 GTSRB (Stallkamp et al., 201 1), MNIST (LeCun,
1998), and DTD (Cimpoi et al., 2014). The natural language tasks consist of 12 GLUE tasks (Wang, 2018), including six single-sentence tasks: SST-2 (Socher et al., 2013), CR (Hu & Liu, 2004), MR (PaNgB, 2005), MPQA (Wiebe al., 2005), TREC (Voorhees et al., 1999) and SUBJ (Lee & Pang, 2004), and six pairwise-sentene tasks: QNLI (Wang, 2018), SNLI (Bowman et al., 2015), MNLI (Williams et al., 2017), RTE (Wang. 2018), MRPC (Dagan et al., 2005) and QQP (Iyer et al., 2017). To avoid information leakage from the test set, 8 visual tasks use unsupervised training sam- ples for optimization. For 12 NI.P tasks, the validation set is used, while some datasets without validation set use training samples. All data are unsupervised.
UNLI
051-2
SML
MALI
MAYC
RTE
Figure 3. Multi-task performance with RoBERTa-bree architecture on twelve NLP tasks, with results presented as absolute accuracy (%) compared to individual tasks.
superior performance of our CALM. Our analysis of the results presented in Table leads to two primary insights:
Models and details. For the visual experiments, we use the ViT-B/32 architecture from CLIP (Radford et al., 2021) as the pre-trained model, while for the NLP experiments, we use the RoBERTa-bass (Liu, 2019) model, with pre- trained and fine-tuned models consistent with previous periments (Yang et al., 2024b; He et al., 2024). The merging process randomly selects two tasks for sequential merging, while others apply task arithmetic (Ilbarco et al., 2023) with a coefficient of 0.3. For the visual tasks, 90% of the credible samples are used, and for the NLP tasks, 80% are used. Each task optimized for 100 iterations, with the regularization parameter x= 1. For details, please refer to Appendix A.
CALM significantly outperforms other baselines and ap- proaches traditional MTL performance. On most tasks, CALM demonstrates 1%-3% performance improvement over existing global-aware and localized-aware model merg- ing methods. Furthermore, when compared to traditional MTL, the performance of CALM is nearly par with the theoretical upper bound achievable through retraining with data, a result attained with minimal computational cost.
CALM ensures more balanced performance across tasks. While baseline methods excel on tasks like MNIST and SVHN, they suffer substantial performance declines (up to almost 20%) on tasks like RESISC45 and DTD when compared to the individual models. In comparison, CALM demonstrates minimal accuracy degradation on any task, achieving a more balanced performance. This highlights the reliability of CALM in multi-task model merging.
5.2. Visual Experimental Results
Our experimental evaluations, conducted across various visual classification datasets, conclusively demonstrate the
7

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Individual
Average Aocuran any Standard Deviation
Aco on 8 vicual tasks nip
Average Accuracy
96
54
Individuel
80
Sampling Proportion
CD EMS
Sequential
Merging of Task Number
Figure 4. Variation of average accuracy with sampling proportion for CB-EME across visual and NLP tasks.
Figure 6. Average accuracy and standard deviation on visual tasks with different numbers of sequential merging tasks.
OB-ENS
Unsucervieed
Figure depicts the effect of sampling proportion average accuracy across visual and NLP tasks. The accuracy initially increases and then decreases as the sampling proportion for CB-EMS rises, indicating that while more samples can improve performance, too many unreliable samples harm aggregation. The results demonstrate that 30% sampling rate is sufficient for state-of-the-an cof-thhear((SOAA performance in visual tasks, while 50% sampling rate achieves individual task performance, also at SOTA, for NLP tasks.
Supervi Sampl
Dav
way
60-
50-
Figure 5 compares CB-EMS with its ablation methods: the unsupervised EM method used by Adamerging and the su- pervised method with ground- truth labels. CB-EMS per- forms similarly to the supervised method and significantly outperforms the unsupervised EM method, suggesting that it retains majority of the correct information. Furthermore, the standard deviation reveals that CB-EMS exhibits greater robustness, with a more stable merging process.
SUN297
Cara
RES 9045
EuroSAT
SVHN
BT5RB
WNIST
DTD
Figure 5. Average accuracy and standard deviation for entropy minimization sampling and its ablation methods.
5.3. NL.P Experimental Results
the NLP experiments, we compared several representative baselines with our method, as shown in Figure 3. The results in the figure are presented absolute accuracy (%) compared individual tasks, where accuracy of 100% indicates that the performance of the merged model closely matches that of the individual fine-tuned models.
5.5. Analysis of Efficient-Aware Framework
In our sequential framework, two factors may introduce sen- sitivity into the experimental results: the order of sequential task merging and the number of tasks merged sequentially, In this section, we present experiments demonstrating that CALM exhibits substantial robustness in both aspects.
The experimental results demonstrate that our CALM ap- proach achieves nearly the same performance as the individ- ual models on most tasks, and even outperforms individual models some tasks, such as RTE and MRPC, leading to overall improvement in performance after merging. Additionally, CALM consistently performs well across all datasets, exhibiting superior consistency compared to the baselines, which show weaker performance on some tasks.
Figure 6 presents the average accuracy and variance of CALM on visual tasks with different numbers of sequen- tial merging tasks [S]. As the number of sequential tasks increases, computational complexity rises, and the aver- age accuracy shows slight increase, stabilizing around 87.7%. This indicates CALM's robustness the number of sequential tasks, achieving effective model merging even with fewer tasks. The standard deviation also increases with more sequential tasks, introducing some randomness. How- ever, the maximum standard deviation is about 0.4%, which does not significantly impact performance. Thus, CALM remains robust to the order sequential tasks, with more stable performance when fewer tasks are used.
5.4. Analysis of CB-EMS
The experimental results have demonstrated that consensus- aware mask process successfully extracts effective informa- tion for model merging. Next, we analyze the influence of the sampling information on the results.
8

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Model Parameters
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Opre
62.3
59.6
60.3
45.7
31.6
32.6
48.3
44.4
48.1
prr + T1
48.8
54.9
46.3
35.4
46.7
32.9
74.8
35.0
46.9
Opre + (1-M') 11
60.4
64.8
69.8
85.6
76.5
68.6
89.5
54.0
71.2
Pprc. + T22
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
pree + T2
69.9
69.5
80.0
88.1
72.7
71.7
90.4
59.7
75.3
Op pre + (1-M))TT) M T
71.9
73.9
88.3
96.9
93.4
93.9
98.0
69.0
85.7
Table Multi-task performance under different model parameters. contrast Table 1. this experiment involves only sequential merging task, where denotes the task vector for this task, M represents its optimized binary mask, and T1 corresponds to the task vector from the remaining tasks via task arithmetic.
SUN397
0.7
Cars
0.6
RESISC45
0.5
Tasks
EuroSAT
0.4
SVHN
0.3
GTSRB
0.2
0.1
DTD
12
15
18
21
27
30
33
30
30
42
35
A8
57
80
88
75
78
84
87
400
93
99
201
108
111
114
117
120
123
126
129
132
135
138
141
144
147
150
Effective Weight Proportions per layer
Figure 7. Effective weight proportions per layer tasks visualized via
mask.
5.6. Analysis of Consensus-Aware Mask
5.7. Visualization of Effective Weight Proportions
As shown in Table 2, we validate the effectiveness of the extracted consensus-aware mask. To intuitively demonstrate the mask's value for each task vector, we employ only one sequential merging task, while merging the remaining tasks efficiently via task arithmetic. Here Ppre denotes the pre- trained model, is the task vector from the remaining seven tasks via task arithmetic, T2 is the target task vector, and M the optimized binary mask. We evaluate the target task performance under five configurations.
As illustrated Figure 7, the effective weight proportions of each layer are depicted when each task serves as a serialized task, providing an intuitive representation of the importance of each layer. From the figure, it can be observed that there strong consistency actoss tasks: if a layer is important for one task, it is likely important for others as well. Addition- ally, some layers exhibit significant differences in weight proportions, highlighting the specificity between tasks.
As illustrated in the table, CALM effectively accomplishes model merging by precisely localizing 5% of critical local parameters with global consensus. Removing 5% of inter- ference parameters from improves the average accuracy by nearly 25% and enhances target task adaptability, demon- strating its capability to accurately identify interference re- gions; restricting 72 to 5% of masked parameters maintains strong performance, indicating minimal loss of task-relevant information and successful retention of critical transfer sig- nals; simultancously applying Ppre + (1 -M*) ? тi and M* ④ 72 achieves conflict-free model merging and outper- forms independent adjustment strategies, verifying effective multi-task merging capability of CALM.
6. Conclusion
In this study, explore how to identify and extract effective information in model merging. By analyzing global-aware and local-aware methods, propose that effective infor- mation found in localized parameters aligned with global task consensus. Based on this insight, we introduce CALM, new approach that uses reliable unsupervised data and an efficient-aware framework to identify and integrate param- eters with global consensus. Extensive experiments show CALM's superiority existing methods. This study pro- vides new perspective model merging, focusing on aligning localized knowledge with global consensus, and of- fers a potential solution for multi-task and transfer learning.

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Impact Statement
Chen, Z., Shen, Y., Ding, M., Chen, Z., Zhao, H., Learned- Miller, E. G., and Gan, C. Mod-squad: Designing mix- tures of experts modular multi-task learners. In Pro- ceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition, CVPR, pp- 11828-11837, 2023.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Acknowledgements
Cheng, G., Han, J., and Lu, X. Remote sensing image scene classification: Benchmark and state of the art. Proc. IEEE, 105(10):1865-155)
This work is funded by the National Science and Technology Major Project (No. 2022/D01 14903) and the Natural Sci- ence Fundation of China (NSFC. No. 62476149). Sen Cui would like acknowledge the financial support received from Shuimu Tsinghua scholar program. FL is supported by the Australian Research Council (ARC) with grant num- ber DE240101089, LP240100101, DP230101540 and the NSF & CSTRO Responsible Al program with grant number 2303037.
2017.
Cimpoi, M., Maji, S., Kokkinos, I., Mohamed, S., and Vedaldi, A. Describing textures in the wild. In Computer Vision and Pattern Recognition, CVPR, pp. 3606-3613, 2014.
Dagan, I., Glickman, O., and Magnini, B. The pascal recog- nising textual entailment challenge. In Machine learning challenges workshop, pp. 177-190. Springer, 2005.
References
Devlin, J. Bert:
Pre-training of deep bidirectional trans- formers for language understanding. arXiv preprint 2018.
Audibert, A., Amini, M. R., Usevich, K., and Clausel, M. Low-rank updates of pre-trained weights for multi-task learning. In Findings of the Association for Computa- fional Linguistics, ACL. pp. 7544-7554, 2023.
Dong, D., Wu, H.,, He, W., Yu, D., and Wang. H. Multi-task learning for multiple language translation. In Associa- tion for Computational Linguistics, ACL, pp. 1723-1732, 2015.
Bowman, S. R., Angeli, G., Potts, C., and Manning, C. D. A large annotated corpus for learning natural language inference. arXiv preprint arXiv: urXiv:1508.053: 508.05326, 2015.
Dosovitskiy, A. An image is worth 16x16 words: Trans- formers for image recognition at scale. arXiv preprint arXiv:2010. /9999, 2020.
Cai, C.,, Ye, Z., Feng, L., Qi. J., and Liu, F. Bayesian Bayesian-guided label mapping for visual reprogramming. In Advances in Neural Information Processing Systems, 2024a.
He, Y., Hu, Y., Lin, Y., Zhang, T., and Zhao, H. Localize- and-stitch: Efficient model merging via sparse task arith- metic. arXiv preprint arXiv::4088 13656, 2024.
Cai, C.,, Ye, 7.., Feng, L.., Qi, J., and Liu, Sample-specific masks for visual reprogramming- based prompting. In International Conference on Machine Learning (ICML), 2024b.
Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. IEEE J. Sel. Top. Appl. Earth Obs Remote. Sens., 12(7):221-2220 2019.
Calandriello, D., Lazaric, A., and Restelli, M. Sparse multi- task reinforcement learning. Advances in neural informa- tion processing systems, NeurlPS, 27, 2014.
Hu, M. and Liu, B. Mining and summarizing customer reviews. In Proceedings of the tenth ACM SIGKDD in- ternational conference on Knowledge discovery and data mining, pp. 168-177, 2004.
Caruana, R. Multitask learning. Machine learning, 28: 41-75, 1997.
Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, Editing models with task arithmetic. In International Conference on Learning Representations, ICLR, 2023.
Chen, W. and Kwok, J. You only merge once: Learning the pareto set of preference-aware model merging. arXiv preprint arXiv:2408.122 12105, 2024.
Chen, 7., Badrinarayanan, V., Lee, C.-Y,, and Rabinovich, A. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. International conference on machine learning. ICML, pp. 793-803. PMLR, 2018.
Iyer, S., Dandekar, N., and Csernai, K. First
quora dataset release: Question pairs, 2017. URL
qwwraaaaaa.uuuo...ooo.... com/
Accessed: 2025-01-29.
10

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Jin, X., Ren, X., Preotiuc-Pietro, D., and Cheng, P. Data- less knowledge fusion by merging weights of language models. In International Conference on Learning Repre- sentations, ICLR, 2023.
Netzer, Y., Wang, T., Coates, A., Bissacco, A., Wu, B., Ng, A. Y., et al. Reading digits in natural images with unsupervised feature learning, In NIPS workshop deep learning and unsupervised feature learning, volume 2011, pp. 4. Granada, 2011.
Krause, J., Stark, M., Deng, J., and Fei-Fei, L. 3d object representations for fine- geeee-rrandd categorization. In 2013 IEEE International Conference on Computer Vision Work- shops, ICCV Workshops, pp. 554-561. IEEE Computer Society, 2013.
PaNgB, L. Exploiting class relationships for sentiment categorization with respect rating sales. IN: Proceeding- sofACI, r05, 2005.
Panigrahi. A., Saunshi, N., Zhao, H., and Arora, S. Task- specific skill localization in fine-tuned language models. In International Conference Machine Learning, ICML., pp. 2701 17011-270333 2023.
LeCun, Y. The mnist database of handwritten digits, 1998. URL http:// /aann.yan.... lecun. wwan//xxb//aeee /. Accessed: 2024-11-05.
Lee, I.. and Pang. B. Sentiment analysis using subjectivity summarization based on minimum cuts. In Proceedings of ACL-04, 42nd meeting of the association for computa- lional linguistics, 2004.
Ponti, E. M., Sordoni, A., Bengio, Y., and Reddy, S. Com- bining modular skills in multitask learning. arXiv preprint arXiv:2202.1391 2022.
Qiu, H., Huang. J., Gao, P., Lu, I.., Zhang, X., and Lu, S. Masked autodecoder is effective multi-task vision generalist. In Computer Vision and Pattern Recognition, CVPR, pp. 14152-14161, 2024.
Lin, X., Zhen, H.-I.., Li, 7 Zhang, Q-Fi, and Kwong, S. Pareto multi-task learning. Advances in neural informa- tion processing systems, 32, 2019.
Lin, X., Yang, Z., Zhang, Q. and Kwong, S. Controllable pareto multi-task learning. 2020.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference on machine learning, ICML, pp. 8748-8763, 2021.
Liu, B., Liu, X., Jin, X., Stone, P., and Liu, Q. Conflict- averse gradient descent for multi-task learning. Advances in Neural Information Processing Systems, NeurlPS, 34: 18878-18890 2021.
Shannon, C. E. A mathematical theory of communication.
Liu, S., Johns, E., and Davison, J. End-to-end multi-task learning with attention. In Computer Vision and Pattern Recognition, CVPR, pp. 1881--880, 2019a.
The Bell system technical journal, 27(3):379-423, 1948.
Shi, H., Ren, S., Zhang, T., and Pan, J. Deep multitask learning with progressive parameter sharing. In Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, CVPR, pp. 19924-19935, 2023.
Liu, X., He, P., Chen, W., and Gao, Multi-task deep neu- ral networks for natural language understanding. arXiv preprint arXiv:1901. 15544, 2019b.
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts, C. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631-1642, 2013.
Liu, Y. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv: 1692, 364, 2019.
Lu, J., Goswami, V., Rohrbach, M., Parikh, D., and Lee, S. 12-in-1: Multi-task vision and language representation learning. In Computer Vision and Pattern Recognition, CVPR, pp. 10437-10446, 2020.
Stallkamp. J., Schlipsing, M.. Salmen, J., and Igel, C. The german traffic sign recognition benchmark: a multi-class classification competition. In The 201 international joint conference on neural networks, pp. 1453-1460. IEEE, 2011.
Ma, H.. Yuan, X., Ding. Z.. Fan, D., and Fang. J. Over- the-air federated multi-task learning via model sparsifica- tion, random compression, and turbo compressed sensing. IEEE Transactions on Wireless Communications, 22(7): 4974-4988, 2022.
Sun, X., Hassani, A., Wang, 7.,, Huang, G., and Shi, H. Dis- parse: Disentangled sparsification for multitask model compression. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR, pp. 12382- 12392, 2022.
Matena, M. and Raffel, C. Merging models with fisher- weighted averaging. In Neural Information Processing Systems, NewrlPS, 2022.

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Swamy, V., Satayeva, M., Frej, J., Bossy, T., Vogels, T., Jaggi, M., Käser, T., and Hartley, M.-A. Multi- modn multimauti,, multi-task, interpretable modular networks. Advances in Neural Information Processing Systems, NeurlPS, 36, 2024.
Yadav, P., Tam, D., Choshen, L.,, Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merg- ing models. In Neural Information Processing Systems, NeurlPS, 2023.
Yang, E., Shen, L.,, Wang, Z., Guo, G., Chen, X., Wang, X., and Tao, D. Representation surgery for muuli--tskk model merging. International Conference on Machine Learning, ICML, 2024a.
Tang, A., Shen, L.,, Luo, Y., Ding, I.., Hu, H., Du, B., and Tao, D. Concrete subspace learning based interference elimination for multi-task model fusion. arXiv preprint arXiv:23/2.06//3, 2023.
Yang, E.,, Wang, 7., Shen, L.,, Liu, S., Guo, G., Wang, X., and Tao, D. Adamerging: Adaptive model merging for multi-task learning. International Conference on Learn- ing Representations, ICLR, 2024b.
Voorhees, E. M. al. The trec-8 question answering track report. In Trec, volume 99, pp. 77-82, 1999.
Wang, A. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. arXiv preprint arXiv: 07461, 2018.
Yang, Y., Jiang, P.-T., Hou, Q., Zhang, H., Chen, J., and Li, B. Multi-task dense prediction via mixture of low-rank experts. In Computer Vision and Pattern Recognition, CVPR, pp. 27927-27933 2024c.
Wang, K., Dimitriadis, N., Ortiz-Jimenzz, G., Fleuret, F., and Frossard, P. Localizing task information for improved model merging and compression. International Confer- ence on Machine Learning, ICML, 2024.
Yao, H., Zhang, R., and Xu, C. Visual-language prompt tuning with wewwledgeggiider context optimization. In Computer Vision and Patlern Recognition, CVPR, pp. 6757-6767, 2023.
Wiebe, J., Wilson, T., and Cardie, C. Annotating expres- sions of opinions and emotions in language. Language resources and evaluation, 39:165-210: 2005.
Yoon, S., Jin, Y-U.,, Noh, Y.-K., and Park, F. Energy-based models for anomaly detection: A manifold diffusion re- covery approach. Advances in Neural Information Pro-
Williams, A., Nangia, N., and Bowman, S. R. A broad- coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv: 1704.05220 2017.
cessing Systems, 36:49445-49466, 2023.
Yu, L.., Yu, B., Yu, H., Huang, F., and Li, Y. Language mod- els are super mario: Absorbing abilities from homologous models as free lunch. In International Conference on Machine Learning, ICML., 2024.
Wortsman, M., Ilharco, G., Li, M., Kim, J. W., Hajishirzi, H., Farhadi, A., Namkoong, H., and Schmidt, L. Robust fine-tuning of zero-shot models. In Computer Vision and Pattern Recognition, CVPR. pp. 7949-7961, 2022.
Yu, T., Kumar, S., Gupta, A., Levine, S., Hausman, K., and Finn, C. Gradient surgery for multi-task learn- ing, Advances in Neural Information Processing Systems, NeurIPS, 33:5824-5836 2020.
Wu, J., Chen, J., and Huang, D. Entropy-based active learning for object detection with progressive diversity constraint. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9397- 9406, 2022.
Zhang, M., Huang, S., Li, W., and Wang, D. Tree structure- aware few-shot image classification via hierarchical ag- gregation. In European conference on computer vision, pp. 453-470, Springer, 2022.
Xiao, J., Ehinger, K. A., Hays, J., Torralba, A., and Oliva, A. SUN database: Exploring a large collection of scene categories. Int. J. Comput. Vis., 119(1):3-22, 2016. doi: 10. 1 001--633---78-----
Zhang, M., Yuan, J., He, Y., Li, W., Chen, Z.,, and Kuang, K. Map: Towards balanced generalization of iid and ood through aodel-agnostic adapters. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 11921-1193,, 2023.
Xie, B., Yuan, I.,, Li, Liu, C. H., Cheng, X., and Wang, G. Active learning for domain adaptation: An energy-based approach. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp. 8708-8716, 2022.
Zhang, M., Li, H., Wu, F., and Kuang, K. Metacoco: A new few-shot classification benchmark with spurious correla- tion. arXiv preprint arXiv:2404. 19644, 2024.
Xin, Y., Du, J., Wang, Q., Lin, Z.,, and Yan, K. Vnt-adapter: Parameter-efficieet transfer learning for multi-task dense scene understanding. In Proceedings of the AAAI Con- ference on Artificial Intelligence, AAAI, volume 38, pp. 16085-16093, 2024.
Zhu, D.,, Sun, Z., Li, Z., Shen, T., Yan, K., Ding, S., Kuang, K., and Wu, C. Model tailor: Mitigating catastrophic forgetting in multi-modal large language models. Inter- national Conference on Machine Learning, ICML., 2024.
12

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
A. Experimental Details
A.1. Datasets
Following the standard practices in the multi-task model merging field (Ilharco et al., 2023; Yadav et al., 2023; Yang et al., 2024b; He al., 2024), we employe eight image classification datasets and twelve NLP datasets evaluate the effectiveness of our proposed method. A detailed introduction to these eight datasets is provided below.
Visual datasets:
SXiao7 et al., 2016) is a large-scale scene recognition dataset with 397 categories and over 108,000 images, covering diverse range indoor and outdoor scenes support scene-level attribute analysis.
Stanford Cars(Krause et al., 2013) consists of 16,185 images from 196 car models, labeled by make, model, and year, aimed at fine-grained vehicle classification tasks.
RESISC45(Cheng et al., 2017) is a remote sensing dataset with 31,500 images spanning 45 scene categories, represent- ing different land-use types such as residential, industrial, and agricultural areas.
(HrlSAT et al., 2019) contains 27,000 satellite images from Sentinel-2 data, classified into 10 land-use categories, useful for applications in earth observation and environmental monitoring.
SVHN(Netzer et al., 2011) includes over 600,000 images of house numbers from Google Street View, categorized into classes (digits 0-9), with complex natural scene backgrounds, posing challenges for digit recognition.
GTSRB(Stallkamp al., 2011) contains more than 50,000 images of 43 types of traffic signs, used for evaluating traffic sign detection and classification, especially in autonomous driving systems.
(LeST(LeeCun, 1998) is a well-known dataset with 70,000 grayscale images of handwritten digits (0-9), serving as a benchmark for evaluating models in digit classification tasks.
DTD(Cimpoi al., 2014) consists of 5,640 images of 47 texture classes, each representing different texture types like "striped" or "dotted," commonly used for texture recognition and visual attribute understanding.
NLP datasets:
SST-2(Socher et al., 2013) is sentiment analysis dataset consisting of 67,349 sentences from movie reviews, labeled as either positive or negative, commonly used for binary sentiment classification tasks.
CR(Hu Liu, 2004) (Customer Review) is sentiment classification dataset containing 1,000 product reviews, labeled as positive negative, used to analyze customer opinions.
MR(PaNgB, 2005) (Movie Review) contains 10,662 movie reviews, labeled as positive negative, widely used for sentiment analysis in text classification tasks.
MPQA (Wiebe al., 2005) is a sentiment analysis dataset with 10,000 sentences from news articles, annotated for subjective and objective sentences, and used for polarity and subjectivity classification.
TREC(Voorhees, et al., 1999) is question classification dataset with 6,000 labeled questions categorized into six broad topic categories, used for task-fpecific question classification.
SUBJ(Lee & Pang, 2004) contains 10,000 subjective and objective sentences, labeled as subjective or objective, used for subjectivity classification tasks.
QNLI(Wang, 2018) (Question Natural Language Inference) is dataset with 104,000 pairs of questions and sentences, used to test if a sentence contains the answer to the given question (semantic entailment task).
SNLI(Bowman et al., 2015) (Stanford Natural Language Inference) consists of 570,000 sentence pairs, labeled with entailment, contradiction, or neutral, used for natural language inference tasks.
13

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
MNLI(Williams et al., 2017) (Multi-Genre Natural Language Inference) is dataset containing 433,000 sentence pairs from various genres, used to classify if two sentences in entailment, contradiction, neutral relationship.
RTngWang, 2018) (Recognizing Textual Entailment) consists of 2,500 sentence pairs, focused on determining if one sentence logically follows from another, and is used for natural language inference tasks.
MRPC(Dagaa et al., 2005) (Microsoft Research Paraphrase Corpus) contains 3,600 sentence pairs, labeled as either paraphrases or non-paraphrases, used for paraphrase detection tasks.
QQP(Iyer et al., 2017) (Quora Question Pairs) consists of 400,000 pairs of questions from Quora, labeled duplicate non-duplicate, commonly used for question pair similarity and paraphrase detection.
A.2. Baselines
CALM is evaluated with respect to the following baseline approaches, which are introduced in further detail below.
Pretrained.
The model is pretrained on general-purpost datasets without any task-specific information. It serves the lower bound for evaluating model merging methods, validating their effectiveness by demonstrating moderate task performance.
Individual. These models fine-tuned each specific task based the pretrained model. They serve both as foundational modules for model merging and upper bound to evaluate the performance loss during the merging process.
Traditional MTL. This is multi-task learning model trained directly on the combined training dataset. serves an upper bound for model merging methods, as these approaches aim achieve similar performance without reusing training data.
Weight Averaging averages the parameters of all individual models without considering parameter conflicts. It shows moderate improvement, slightly outperforming the pretrained model.
RegMean (Jin et al., 2023) aims to ensure that the merged model remains close to individual fine-tuned models in L2 distance by using the covariance matrix of the training dataset to guide the merging process.
Fisher Merging (Matena & Raffel, 2022) uses the Fisher information matrix, derived from training data, guide the merging process. It treats the Fisher matrix measure of each parameter's importance in the model.
Task Arithmetic (Ilharco et al., 2023) introduces the concept of task vectors, which are calculated as the difference between fine-tuned and pretrained models. It preserves task-specific information and merges task vectors using weighted aggregation achieve significant performance gains.
TW AdaMerging (Yang et al., 2024b) applies task-wise AdaMerging, where each task is assigned merging weight. The merging weights are optimized using an unsupervised test dataset, enhancing the merging process over the Task Arithmetic method.
LW AdaMerging (Yang et al., 2024b) utilizes layer-wise AdaMerging, providing finer granularity by assigning merging weights to individual layers. This approach effectively resolves parameter conflicts and achieves state-of-the- art performance on benchmark datasets.
Ties-Merging (Yadav et al., 2023) focuses merging parameters from a localized perspective. It uses strategies like Trim, Elect Sign, and Disjoint Merge to identify the most crucial parameters from each model while minimizing conflicts during the merging process.
Pareto Merging (Chen & Kwok, 2024) formulates model merging as multi-objectivv optimization problem, generating diverse set of Pareto-optimal models in single process. This approach outperforms existing methods by tailoring models to user-specific preferences.
Consensus TA (Wang et al., 2024) proposes identifies the task supports given a collection task vectors. This method, combined with Task Arithmetic, aggregates the selected task supports.
14

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Consensus TIES (Wang et al., 2024) proposes replacing the original Task Arithmetic module with Ties-Merging. achieving strategy for selecting effective parameter points.
Localize-and-Stitch (He et al., 2024) aims to identify important localized parameters by using partial training data for optimization. strives represent task information effectively with fewer parameters, compared the TIES- TIES-Merging method.
A.3. Implementation Details.
CALM employs an efficient-aware sequential merging approach, and we describe our experimental setup using the main experimental configuration as an example. First, we randomly select two tasks as the sequential tasks, while the remaining task vectors are merged using task arithmetic, formulated as тo A % where X = 0.3, following Task Arithmetic.
Next, we conduct sequential merging for the two randomly selected tasks. Using the glass-balances entropy minimization sampling method, We select 90% of the samples from the unlabeled samples of the visual task as the credible sample set, and 80% of the samples from the text task as the credible sample set. We initialize real-valued mask R of the same size as the model and set le-5 of the parameter points to be active. The mask R is then iteratively trained with the credible sample set, which contains pseudo-labels, using batch size of 128 and a learning rate of 1e7-a large learning rate to ensure effective information feedback to the mask. For each iteration, only two batch of the reliable sample set per task is used, with a total of 100 iterations.
The trained real-valuee mask converted binary mask using the sigmoid function, and a new task vector merged accordingly. Once all sequential tasks are merged, the final merged model is obtained. The same process followed for other numbers of sequential tasks.
A.4. Computing Resources
Part of the experiments is conducted local server with Ubuntu 16.0 system. has two physical CPU chips which are Intel(R) Xeon(R) Gold 6248 CPU @ 2.50GHz with 20 cpu cores. The other experiments are conducted on remote server. It has 8 GPUs which are GeForce RTX 3090.
B. -lasneal Entropy Minimization Sampling Analysis
EMS vs. CB-EMS: validation comparison. Figure 8 shows the accuracy of pseudo-labels versus ground truth across eight tasks at different EMS sampling rates. As the entropy increases, the accuracy of the pseudo-labels decreases significantly, indicating a strong correlation between entropy and sample confidence. Therefore, compared to Adamerging, our selected samples exhibit greater robustness. Additionally, comparing (a) and (b) reveals that the accuracy of CB-EMS decreases more rapidly, This suggests that there are significant differences in entropy across classes, making EMS prone to class imbalance.
156
SUS3P?
Oars
Garo
RESI SC45
RES) 5045
25
SWAN
SWNN
GTSRB
WIST
ST
DID
20
30
100
62
100
Sangl ing
Ratio
(a) CB-EMS
(b) EMS
Figure& Accuracy of EMS and CB-EMS pseudo-labels labels compared ground truth across different sampling rates
15

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
B.1. Entropy Imbalance Analysis
Figure 9 illustrates the boxplots of entropy for each class in the DTD and RESISC45 datasets. The boxplots visually represent the median entropy for each class, with the bottom and top of the box indicating the first quartile (Q1) and third quartile (Q3), respectively. The whiskers extend to the minimum and maximum values within the dataset. From these boxplots, it is evident that there are significant differences in entropy across classes- -omee classes have consistently low entropy, while others show predominantly higher values. Thus, employing the class-balanced entropy minimization sampling method is justified, as it helps prevent sample selection from being dominated by only a few classes.
Enerpyy Disribution
Entrogy
12
Figure 9, Binned boxplot of entropy distribution for DTD and RESISC45 datasets
B.2, Entropy Minimization Reliability Analysis
This subsection aims demonstrate that directly using the entropy minimization method is insufficient due the unreliability of the resulting entropy. As illustrated in Figure 10, both the adamerging and CALM methods start with initial model equivalent to task arithmetic. Therefore, we visualize the confusion matrices of this initial model for the DTD and RESISC45 tasks. From the figure, it is evident that the initial model yields suboptimal predictions for these tasks, with many classes being almost entirely misclassified. Consequently, the entropy estimated from these samples points in an incorrect direction.
Using an entropy minimization approach this context could further enhance the credibility of these incorrect entropy values, causing these erroneous signals to negatively impact model merging. In contrast, the CALM method alleviates the impact of these misleading entropies by first sampling reliable dataset, thus mitigating the harm. Moreover, CALM sets pseudo-labels to ensure that entropy information remains unaffected during model merging, ultimately leading more reliable outcomes.
Confusion
Figure 10. Confusion matrix of initial model predictions DTD and RESTSC45 tasks
16

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
C. Supplementary Experiment
C.1. Merging Strategy Analysis
CALM introduces a consensus-awarr conflict-free merging strategy, and we explore the effectiveness this merging approach below. Given binary mask M, CALM's merging method defined as Tmil = (1 M) Tsey M @ Ty. The advantage of this approach is that all tasks' task vectors participate in mask training, while ensuring that the parameters of the new task vector do not conflict with those of the original task vectors. In ablation study, also investigate the performance of merging strategies with only (1 11-MM -M) and only M.
The experimental results are shown in Table 3. It can be observed that the merging strategies using only M or only (1 - M) exhibit distinct characteristics. The strategy with only M maintains the state of the original tasks better, as it does not affect the information the existing tasks, while the new task is impacted due to parameter interference. In contrast, the strategy using only (1 .M) retains the new task information well but results in decrease in the original tasks' performance due information loss and parameter conflicts. By comparison, CALM achieves better balance by combining both approaches, leading to superior performance overall.
Method
Merging Strategy
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
LW AdaMerging
global
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
Localize-and-Sticl
local
67.2
68.3
81.8
89.4
87.9
86.6
94.8
62.9
79.9
CALM
only only - M
72.9
89.6
98.1
95.1
98.8
69.9
86.6
CALM
69.0
70.6
87.2
96.6
93.0
93.3
98.0
77.5
85.7
CALM
hoth (ours)
72.6
74.8
91.9
98.6
95.2
96.4
99.1
72.8
87.7
Table 3. Multi-task performance cight vision classification tasks under different merging strategies.
C.2. Learning Rate Analysis
Table 4 investigates the influence of learning rate on our proposed method. In the main experiment, the learning rate is set to 1c7. We further evaluate the effects of learning rates 5e6 and 2c7, and compare the results with those obtained by the localize-and-stice method under identical learning rates. The results indicate that the impact of different learning rates on CALM negligible, with each dataset showing consistent performance fluctuations. This demonstrates the stability of our method across varying learning rates, underscoring the robustness of CALM.
Method
Learning Rate
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Localize-and-Stice
le7
67.2
68.3
81.8
89.4
87.9
86.6
94.8
62.9
79.9
CALM
5e6
72.4
74.6
91.8
98.7
95.2
96.4
87.5
CALM
2c7
72.2
74.3
92.1
98.7
95.3
96.2
99.1
73.2
87.6
CALM
1e7
72.6
74.8
91.9
98.6
95.2
96.4
99.1
72.8
87.7
Table 4. Multi-task performance eight vision classification tasks under different learning rates.
C.3. Regularization Coefficient Analysis
Table 5 investigates the impact of the L.1 norm regularization coefficient the performance of CALM. This coefficient controls the extent to which regularization influences the final result, with larger coefficients encouraging a sparser mask. The results indicate that the regularization coefficient has minimal effect on overall performance. While larger coefficients may lead to slight decline CALM's performance, CALM generally maintains stability across different values of the regularization coefficient, demonstrating its robustness.
17

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
Method
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg
Pretrained
66.8
77.7
71.0
59.9
58.4
50.5
76.3
55.3
64.5
Individual
82.3
92.4
97.4
100
98.1
84.1
94.2
Traditional MTL
80.8
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Weight Averaging
72.1
81.6
82.6
91.9
78.2
97.1
62.8
79.6
RegMean (Jin et al., 2023)
73.3
81.8
86.1
97.0
88.0
84.2
98.5
60.8
83.7
Fisher Merging (Matena & Raffel, 2022)
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70.0
82.2
Task Arithmetic (Ilharco et al., 2023)
73.9
82.1
86.6
94.1
87.9
86.7
98.9
65.6
84.5
Ties- fies-Merging (Yadav et al., 2023)
76.5
85.0
89.3
95.7
90.3
83.3
99.0
68.8
86,0
AdaMerging (Yang al., 2024b)
79.0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
Pareto Merging (Chen Kwok, 2024)
82.5
91.4
92.0
98.5
95.4
99.0
78.3
92.0
CALM (ours)
80.2
90.8
94.6
98.8
96.2
97.5
99.3
79.3
92.1
Table Multi-task performance with CLIP ViT-I/14 architecture on eight vision classification tasks.
Method
Coefficient
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Localize-and and-Stitchhh
67.2
68.3
81.8
89.4
87.9
86.6
94.8
62.9
79.9
CALM
0.5
72.9
75.2
92.3
99.0
95.3
96.3
99.1
72.9
87.8
CALM
2
72.5
74.8
91.8
98.7
95.3
96.3
99.1
72.1
87.6
CALM
lc7
72.6
74.8
91.9
98.6
95.2
96.4
99.1
72.8
87.7
Table Multi-task performance on eight vision classification tasks under different regularization coefficients.
C.4. ViT-L/14 Experimental Results
Following the experimental settings used in previous work on model merging (Ilharco et al., 2023; Yadav al., 2023; Yang al., 2024b), we extend our results on the same datasets and methods using the CLIP ViT-L/14 architecture, as shown in Table 6. Given the larger scale of this model, the fine-tuned task information is well preserved, allowing even simpler approaches like weight averaging to achieve relatively strong model merging performance. Methods like adaMerging are already upproaching the upper limit of model merging for this architecture, as demonstrated by individual and traditional MTL. baselines, leaving limited room for further improvement.
Nonetheless, our approach still achieves state-of-the-art performance, improving the average accuracy by 0.1% over Pareto Merging, with only a 1.5% gap from the theoretical upper bound. This consistent, strong performance highlights both the effectiveness and stability our method. Upon further analysis of individual datasets, it is evident that some datasets, such Cars and MNIST, have nearly reached their theoretical upper bound, where CALM shows similar results to baselines. However, for datasets with greater room for improvement, such as RESISC45 and SVHN, CALM demonstrates significant performance gains. It is noteworthy that our method shows a slight decrease in performance on the DTD dataset, likely due to the fact that extracting reliable information for DTD as serialized task requires more iterations. Increasing the iteration count presents potential opportunity for further improvement in our approach.
C.5. Convergence Analysis
We analyze the convergence of the CALM iterative process. Figure 11 exhibits the task sequence for the CALM(33 = setup, where MNIST and DTD datasets are sequentially merged into the model. Each task undergoes 100 iterations, with performance evaluated across tasks every iterations. The red curve represents average accuracy at each evaluation point, Initially, the model is initialized using task arithmetic (Ilharco al., 2023) over the first six tasks, resulting in the initial merged model. After 100 iterations, DTD is merged as the sequential task.
From Figure 11, we observe that CALM achieves rapid convergence, with performance improving significantly after just 10 iterations, almost reaching a stable state. The model exhibits high stability, with consistent and smooth increase in accuracy. When DTD is added, the model experiences a slight perturbation, but previous performance is largely maintained. As training continues, the model's predictive ability improves further, indicating that CALM converges quickly and remains resilient to the integration of new tasks.
18

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
130
00
SUMIF
EurodAl
SViN
DTD
Average Accuracy
Iterati
Figure 11. Accuracy and average accuracy curves over iterations for each dataset.
D. Binary Mask Analysis
In the preceding sections, we have analyzed the effectiveness and robustness of the CALM method in model merging from macro perspective. Below, we shift our focus to the binary mask representation to examine the specific characteristics of the parameter points identified by CALM.
D.1. Binary Mask Activation Trend
We selected the CALM( = S| = 1) configuration to explore the characteristics of the binary mask, which is generated for the final task to extract effective parameters while removing the influence the previous seven merged tasks. Figure 12 illustrates the variation in the proportion of parameters set to in the binary mask throughout the iterations. As shown in the figure, CALM identifies approximately 3.5%-4% of parameters as effective, indicating high degree of redundancy in and this ratio increases
the model parameters. Initially, the mask randomly selects only le-5 of parameters be gradually during iterations, demonstrating consistency across different datasets.
Additionally, an interesting phenomenon is observed: for more challenging tasks, such SUN397, Cars, and DTD, the proportion of selected effective parameters is relatively higher, whereas for simpler tasks, like MNIST and SVHN, the ratio lower. This aligns with our expectations, as more difficult tasks require more parameters to adequately represent their complexity.
0.42
3.12
2342
1./0
0.98
3.35
2.50
1.56
0.03
2.00
2.30
1.20
0023
2.03
2.55
1.55
0.02
1.00
2315
GTSRB
MIIIST
DTD
3.02
2350
0.00
3.005
2.000
2.000
0.11
1315
0.11
2011
2010
1310
0.06
3005
2.00
130
22
20
Excatios
Iovation
1x07500
Figure 12. Effective weight proportions across iterations.
19

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
D.2. Magnitude and Importance Correlation Analysis
For long time, many studies have intuitively assumed that parameters with larger magnitudes in the task vector are more important, providing theoretical guidance for some data-free approaches, such ties- ties-merging (Yadav et al., 2023). In CALM, we identified better local parameter points with the assistance of unsupervised samples, which allows to further analyze the relationship between parameter magnitude and its importance within the task vector. As shown in Table 7, the proportion of effective parameters with mask value that rank within the top k% in terms of magnitude is presented for each task. there were no correlation, the proportion in the table should approximately equal k. However, the results indicate that the magnitude of parameters in the task vector is correlated with their more likely to be important. However, this does not imply that parameters with the highest magnitudes are always critical; rather, magnitude is merely one of several factors influencing importance. Therefore, methods like ties-merging do not necessarily identify the most significant parameters, but require sample information to more accurately determine these key parameters.
with larger magnitudes are
Magnitude Selection
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Top 1%
4.9
4.6
3.5
2.3
3.6
4.9
3.9
3.9
Top 5%
17.5
16.0
14.1
10.7
14.1
16.4
14.6
14.9
Top 10%
29.2
26.8
24.6
19.9
24.6
27.0
24.8
25.5
Top 20%
46.6
43.3
41.0
35.0
40.9
42.8
40.4
41.6
Top 50%
76.8
73.9
72.0
66.3
71.4
71.9
70.1
71.9
Table Proportion of effective parameters within the mask ranked the top k% of magnitude across eight vision classification tasks.
E. Time Cost Analysis
Benefiting from the efficient-awar framework, despite employing sequential merging approach, we still achieve remarkable model merging performance with relatively low time cost. Analyzing from a convergence perspective, demonstrated in Section 5.5, CALM achieves nearly 10% improvement within just 10 iterations, almost reaching convergence. This process takes only about 4 minutes on single GeForce RTX 3090, which is significantly faster compared to other model merging methods that rely on sample-based optimizations. After 10 iterations, our method converges gradually to the optimal solution with high stability. This feature allows for flexible adjustment of the number of iterations depending on computational resources and time constraints in practical scenarios.
Below, we compare the overall runtime of model merging optimization methods that require data. As shown in Table 8, we report the runtime costs for different numbers of serialized tasks. The grcalize-and-stin method optimizes for single task with 80 iterations, resulting in relatively low time consumption. By randomly selecting one serialized task for merging, achieve comparable runtime, while our performance far surpasses that of localize-and-sticch. As the number of serialized tasks increases, the runtime for CALM exhibits sub-linear growth trend. This is due to the fact that each scrialization step only considers previously seen tasks. Even in the most complex scenario, our runtime remains lower than that of adamerging, highlighting the efficiency of our approach.
Method
Run-time Consumption
Iterations
Adamerging
210.23min
500
CALM( S
7)
163.47min
700
CALM( S 2)
7 71.24min
200
CALM( S 1)
36.42min
100
Localize-and-Stitch
31.76min
80
Table 8, Run-time consumption of across different methods.
F. Privacy Discussion
Model merging offers superior privacy protection compared to traditional multi-tas taati-task learning, as it does not require retraining with the original training datasets. The information we use is identical to adamerging, relying solely on unsupervised test data, ensuring there is risk of data leakage or privacy breaches. Moreover, CALM provides greater security and flexibility
20

---

CALM: Consensus-Aware Localized Merging for Multi-Task Learning
compared methods that either extract information from training data or directly the training dataset. Specifically, CALM does not require direct interaction with data providers, which reduces the complexity of data access and minimizes privacy risks. Furthermore, unsupervised test samples may originate from diverse environments or distributed sources, making the approach adaptable for decentralized applications. Importantly, there is no need for additional manual labeling, enhancing both scalability and ease of implementation.
21


Reviews:
[{"review_id": "gJHqRp7Nzk", "reviewer": "Reviewer_y6gA", "summary": "Localized Information with Global Consensus: CALM proposes a method to extract localized parameters that align with global task consensus, ensuring that the merged model maintains effectiveness across all tasks.\n\n* A new sampling technique that leverages unsupervised data more effectively by balancing class representation and minimizing entropy.\n\n* A scalable and efficient framework for merging models sequentially, reducing computational complexity while maintaining performance.\n\n* A method to optimize binary masks that align with global task consensus, enabling conflict-free merging of task-specific parameters.\n\n* The authors demonstrate the superiority of CALM through extensive experiments on both vision and NLP tasks, showing that it outperforms existing global-aware and localized-aware methods, and approaches the performance of traditional MTL without the need for retraining.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question 1: A more formal theoretical analysis of the effectiveness of localized information with global consensus***\n\n***Answer 1***: Thanks for the inspiring question.  We perform a theoretical analysis of the following three aspects based on error.\n\n- **Localized information reduces interference in model merging. Such interference arises from intra-task noise and conflicts across task vectors (e.g., cancellation due to opposing parameter values).**  \nDenote the interference by $\\epsilon$, so that task vector$$\\tau_j = \\Delta\\tau_j+\\epsilon,$$ where $\\Delta\\tau_j$ is the effective information for the global optimal update (nonzero in dimensions $I$) and $\\epsilon$ is concentrated in the complementary set $J$. Define the binary mask $M_L$ such that $(M_L)_i=1$ if $i \\in I$ and \n$(M_L)_i=0$ if $i \\in J$. Thus, the merged task vector becomes $$\\Delta \\tau_L = M_L\\odot\\tau_j = \\Delta\\tau_j+M_L\\odot\\epsilon,$$\nreducing the interference energy from $||\\epsilon||^2$ to $E_L=||M_L\\odot\\epsilon||^2.$\nAlthough methods like Ties-Merging and Consensus TA aim to lower interference, their effectiveness is limited by the absence of global task information.\n\n- **Global consensus enables the update to closely align with the optimal direction common to all tasks, reducing the overall loss.**  \nFor all visible tasks $S_v$, we utilize global consensus to optimize a mask $M_G$ by minimizing\n$$\n\\min_{M_G} \\sum_{t_v \\in S_v} L_{t_v}(\\theta_{pre} + \\tau_{seq}^{(j-1)} + M_G \\odot \\tau_j).\n$$\nAssuming a quadratic approximation near $\\theta_{pre} + \\tau_{seq}^{(j-1)}$, we have\n$$\nL_{t_v}(\\theta_{pre} + \\tau_{seq}^{(j-1)} + M_G \\odot \\tau_j) \\approx L_{t_v}(\\theta_{pre} + \\tau_{seq}^{(j-1)}) + \\langle g_{t_v}, M_G \\odot \\tau_j \\rangle + \\frac{1}{2}(M_G \\odot \\tau_j)^T H_{t_v} (M_G \\odot \\tau_j),\n$$\nwhere $H_{t_v}$ is the Hessian. Assume $H_{t_v}$ is the identity matrix and $\\Delta\\tau_j$ denote the global optimal update. The update error is\n$$\nE_{G} = ||M_G \\odot \\tau_j - \\Delta\\tau_j||^2.\n$$\nThe gain from global consensus is defined as\n$$\n\\Delta_{G} = ||\\tau_j - \\Delta\\tau_j||^2 - ||M_G \\odot \\tau_j - \\Delta\\tau_j||^2.\n$$\nThe optimization process shows that global consensus effectively reduces the overall loss.\n\n- **Our CALM method integrates localized information and global consensus to effectively mitigate local parameter interference while capturing information beneficial for global tasks.**  \nLet the obtained binary mask be $M^*$; then the update error is\n$$\nE_{CALM} = ||M^* \\odot \\tau_j - \\Delta \\tau_j||^2.\n$$\nBased on the optimization objective, the error can also be expressed as\n$$\nE_{CALM} = \\min(E_{loc}, E_{glob}) - \\Delta_{syn},\n$$\nwhere $\\Delta_{syn} > 0$ represents the additional error reduction from joint optimization.\n\n**Thus, CALM achieves both local and global gains, effectively reducing the overall error.**\n\n---\n\n***Question 2: Additional ablation studies on the individual components of CALM.***\n\n***Answer:*** Thank you for your valuable question. \n \nDue to length limitations, we will present additional ablation study results in **Reviewer oEok Question 2**.\n\n---\n\n***Question 3: The sensitivity of CALM to hyperparameters (the regularization parameter $\\lambda$ and the sampling rate for CB-EMS).***\n\n***Anwser:*** Many thanks for the insightful suggestions.\n\nBelow are the model merging results on eight vision datasets under different regularization parameters $\\lambda$ and sampling rates.\n\n|$\\lambda$|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|Avg Acc|\n|-|-|-|-|-|-|-|-|-|-|\n|0.5|72.9|75.2|92.3|99.0|95.3|96.3|99.1|72.9|**87.8**|\n|2|72.5|74.8|91.8|98.7|95.3|96.3|99.1|72.1|**87.6**|\n|1|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|**87.7**|\n\nCALM is insensitive to $\\lambda$, with merging performance remaining stable. \n\n**Thus, $\\lambda$ between 0.5 and 2 is acceptable**.\n\n|Sampling Rate|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|Avg Acc|\n|-|-|-|-|-|-|-|-|-|-|\n|0.1|69.7|69.0|88.6|97.1|93.9|94.1|96.8|61.4|**83.8**|\n|0.2|70.8|70.3|89.2|97.4|94.0|94.7|97.2|65.4|**84.9**|\n|0.3|71.5|71.8|89.9|97.9|94.3|95.0|97.5|67.5|**85.7**|\n|0.4|71.9|72.4|90.2|98.0|94.3|95.1|97.8|69.1|**86.1**|\n|0.5|72.2|73.3|90.8|98.2|94.6|95.6|98.0|68.8|**86.4**|\n|0.6|72.6|73.5|91.0|98.2|94.5|95.6|98.1|70.9|**86.8**|\n|0.7|72.4|74.5|91.2|98.4|94.8|95.8|98.2|71.7|**87.1**|\n|0.8|72.7|74.9|91.7|98.6|95.0|96.0|98.5|71.8|**87.4**|\n|0.9|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|**87.7**|\n|1.0|72.9|75.3|91.8|98.6|95.2|95.9|98.8|71.8|**87.5**|\n\nPerformance first increases then decreases with the sampling rate, peaking at 0.9. This aligns with our theory. \n\n**In practice, the optimal sampling rate depends on data quantity and quality; for high-quality data, a higher rate (e.g., 0.8–0.9) is preferred, otherwise a lower rate is recommended**.\n\n---\n\n***Question 4: efficient-aware-> efficiency-aware.***\n\n***Answer:*** Thank you for your constructive advice.\n \n**We will seriously consider making changes in the final version.**", "claims_and_evidence": "Yes, most claims are support by empirical evidence.", "methods_and_evaluation_criteria": "The evaluation criteria follows existing works and is reasonable in practice.", "theoretical_claims": "No theoretical claims.", "experimental_designs_or_analyses": "Yes, the experimental designs are sound.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "**Strengths**:\n\n* The authors provide extensive experimental results across multiple datasets, demonstrating that CALM consistently outperforms existing baselines and achieves performance close to traditional MTL.\n\n* The efficient-aware framework is a practical contribution, especially in scenarios where tasks are completed asynchronously or computational resources are limited. The sequential merging approach reduces the complexity of merging multiple tasks.\n\n\n\n\n**Weaknesses**:\n\n* While the empirical results are strong, the paper lacks a deeper theoretical analysis of why localized information with global consensus leads to better performance. A more formal theoretical framework or proof could strengthen the paper.\n\n* Although the paper includes some ablation studies (e.g., comparing CB-EMS with other sampling methods), it would benefit from a more detailed analysis of the individual components of CALM (e.g., the impact of the efficient-aware framework vs. the consensus-aware mask optimization).\n\n\n* The paper does not discuss the sensitivity of CALM to hyperparameters (e.g., the regularization parameter \n$\\lambda$ or the sampling rate for CB-EMS). A discussion on how sensitive the method is to these choices would be useful for practitioners.\n\n\n* efficient-aware->   efficiency-aware?", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CeFbf52xnv", "reviewer": "Reviewer_Rsjn", "summary": "This paper focuses on model merging in multi-task learning, aiming to identify locally shared information with global task consensus while addressing existing limitations of parameter conflict in global information and diluted local features during merging. \nThe method comprises three components: 1) Class-Balanced Entropy Minimization Sampling constructs a reliable unsupervised dataset; 2) An Efficiency-Aware Framework enables resource-effective model merging; 3) Consensus-Aware Mask Optimization designs mask refinement with global consensus. \nExperiments on both vision and language datasets demonstrate the method's effectiveness, outperforming state-of-the-art approaches.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question 1: The authors need to provide a clearer definition and explanation of global task consensus to establish the core idea of this work.***\n\n***Answer:***  Thanks for your valuable question. We clarify the concept of global task consensus as follows:\n\n- **Definition:** Global task consensus refers to shared and generalizable knowledge patterns or parameter adaptation directions among multiple independently fine-tuned models during model merging.\n- **Key Properties:**\n  - **Cross-Task Validity:** Consistent parameter adaptation directions across tasks during fine-tuning.\n  - **Task-Specific Feature Preservation:** Avoidance of significant performance degradation on any task after model merging.\n- **Validation:** Due to length limitations, we provide theoretical error analysis to validate its effectiveness. For details, please refer to **Reviewer y6gA Question 1**.\n\n---\n\n***Question 2: Can CALM adapt to existing frameworks without an efficiency-aware framework?***\n\n***Answer:***  Thank you for your valuable question.\n\nThe efficiency-aware framework enables CALM to achieve efficient model merging with strong scalability. Our method can also adapt to scenarios without this framework, maintaining compatibility with existing approaches. Below we present two feasible solutions:\n\n- **Single-Task Merging: Randomly select one task for model merging.** As shown in Figure 6, the results demonstrate that applying CALM to a single task achieves comparable or even better performance than sequential multi-task merging. Thus, CALM can be applied to a single task without requiring sequential merging.\n- **Multi-Task Simultaneous Merging: Utilize multiple binary masks to merge all tasks simultaneously.**  We can slightly modify the CALM method to align with existing model merging frameworks, with the following optimization objective. Within a single optimization equation, we optimize a binary mask for each task and enforce that the sum of binary masks equals an all-ones matrix. This allows simultaneous extraction of information from all tasks while maintaining global consensus. During optimization, we still employ real-valued masks $R$ and classify each parameter to tasks via softmax.\n$$\n\\min_{\\{M\\}} \\ \\sum_{t=1}^T\\mathcal{L}(\\theta_{mtl})+\\alpha\\sum_{t=1}^T||M_t||_1\n$$ \n\n$$\ns.t. \\ \\theta_{mtl}=\\theta_{pre}+\\sum_{t=1}^TM_t\\odot\\tau_t,\\  \\sum_{t=1}^TM_t=\\textbf{1}_{n \\times n}.\n$$\n\n---\n\n***Question 3: Does the binary mask capture global consensus?***\n\n***Answer:***  Thanks for the practical comment.\n\nIn Appendix D, we provide visualizations and analysis of the binary mask properties. Here, we present a concise experiment to demonstrate the effectiveness of binary mask.\n\nFor each task, we apply CALM with the following definitions: $\\theta_{pre}$ denotes the pre-trained model, $\\tau_{1}$ is the task vector from the remaining seven tasks via task arithmetic, $\\tau_{2}$ is the target task vector, and $M$ is the optimized binary mask. We evaluate the target task performance under five configurations:\n\n|Model Parameters|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|\n|-|-|-|-|-|-|-|-|-|\n|$\\theta_{pre}$|62.3|59.6|60.3|45.7|31.6|32.6|48.3|44.4|\n|$\\theta_{pre}+\\tau_1$|48.8|54.9|46.3|35.4|46.7|32.9|74.8|35.0|\n|$\\theta_{pre}+(1-M)\\odot\\tau_1$|60.4|64.8|69.8|85.6|76.5|68.6|89.5|54.0|\n|$\\theta_{pre}+\\tau_2$|79.2|77.7|96.1|99.8|97.5|98.7|99.7|79.4|\n|$\\theta_{pre}+M\\odot\\tau_1$|69.9|69.5|80.0|88.1|72.7|71.7|90.4|59.7|\n|$\\theta_{pre}+(1-M)\\odot\\tau_1+M\\odot\\tau_2$|71.9|73.9|88.3|96.9|93.4|93.9|98.0|69.0|\n\nCALM extracts task vectors with approximately 5% parameters. We have the following conclusions:\n- **Interference suppression** (Rows 2-3): Removing 5% parameters from $\\tau_{1}$ via $(1-M)\\odot\\tau_{1}$ significantly improves target task adaptation, demonstrating effective identification of interference parameters.\n- **Task-specific preservation** (Rows 4-5): Confining $\\tau_{2}$ to 5% masked parameters $M\\odot\\tau_{2}$ maintains strong performance, indicating precise capture of essential transfer signals.\n- **Conflict-free merging** (Rows 3,5,6): Simultaneous application of $(1-M)\\odot\\tau_{1}$ and $M\\odot\\tau_{2}$ achieves performance gains without parameter conflicts, verifying CALM's effective multi-task merging capability.\n\n---\n\n***Question 4: Additional experiments of language multi-task benchmarks.***\n\n***Answer:*** Thanks for the very constructive comments.\n\n**CALM demonstrates consistent behavior across language and visual benchmarks.** We give an example, shown in the table below. Both domains exhibit nearly identical trends in average accuracy versus CB-EMS sampling rates - initially increasing then decreasing with higher sampling. Additional experiments confirm this cross-domain consistency.\n\n|Sampling Rates|0.1|0.2|0.3|0.4|0.5|0.6|0.7|0.8|0.9|1.0|\n|-|-|-|-|-|-|-|-|-|-|-|\n|language benchmarks|80.1|80.8|81.0|81.2|81.2|81.3|81.3|**81.7**|81.4|81.5|\n|visual benchmarks|83.8|84.9|85.7|86.1|86.4|86.8|87.1|87.4|**87.7**|87.5|", "claims_and_evidence": "Most claims made in the submission are clear. However, the authors should further clarify the exact definition of \"global consensus\" and explain how CALM specifically achieves it.", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "No errors in theoretical claims. However, whether character subscripts should be italicized needs consistency.", "experimental_designs_or_analyses": "The experiments include 8 visual classification tasks and 12 NLP tasks, which aligns with most model merging methods.  However, the lack of analysis on the binary mask—such as whether it captures global consensus or the properties of this consensus—weakens the support for the authors’ key insight.", "relation_to_broader_scientific_literature": "The main contribution of this paper lies in applying the global optimization approach from Adamerging to localized model merging methods, such as Ties-Merging, and Localize-and-Stitch, thereby avoiding the issue where these data-free methods struggle to precisely capture task-specific characteristics.", "essential_references_not_discussed": "The author should add a comparison between CALM and some of the latest model merging-related papers, like [1] and [2].\n\n[1] Mitigating Parameter Interference in Model Merging via Sharpness-Aware Fine-Tuning\n[2] Model merging with SVD to tie the Knots", "other_strengths_and_weaknesses": "Other Strengths:\n1. The paper is well-structured and well written, and the figures and tables are detailed, clearly conveying the authors' insights and methodology.\n2. I agree that the issue the authors focus on is significant in model merging.\n3. The proposed method achieved state-of-the-art experimental results, and the experiments are thorough.\n\nOther Weaknesses:\n1. The experiments based on language multi-task benchmarks could be more comprehensive to check if the findings align with those in visual benchmarks.", "other_comments_or_suggestions": "Please refer to Other Strengths And Weaknesses and Questions For Authors", "questions_for_authors": "The following are the questions and concerns I hope the authors can address further:\n1. The authors need to provide a clearer definition and explanation of global task consensus to establish the core idea of this work.\n2. Can CALM adapt to existing frameworks without an efficiency-aware framework?\n3. Does the binary mask capture global consensus? The authors need to explore this further.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zIyqxIRVXt", "reviewer": "Reviewer_oEok", "summary": "The paper introduces a test-time adaption method named CALM, which optimizes an equally-sized mask on pre-fine-tuned task-specific models through a reliable unsupervised dataset. The mask aims to extract locally shared parameters with global task consensus, offering a new perspective for model merging by exploring how to identify more effective local parameters. Furthermore, CALM implements a serialized merging framework that achieves remarkable efficiency in scenarios requiring individual merging of new tasks.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question 1: The sampling rates of CB-EMS in experiments appear inconsistent; how would this be resolved in practical applications?***\n\n***Answer:***  Thanks for the practical comments. We would like to explain it as follows:\n\n- **CB-EMS Sampling Rates Show Cross-Task Generalizability.**  \nExperimental results demonstrate that CALM exhibits consistent performance trends across language and visual benchmarks: accuracy first increases then decreases with higher sampling rates, peaking at 0.8 (language) and 0.9 (visual). This validates CALM's cross-domain adaptability and task-agnostic performance patterns.\n\n- **Optimal Sampling Rates Depend on Unsupervised Data Properties.**  \nThe choice of optimal sampling rates is dynamically determined by the scale and quality of unsupervised data. For small datasets, higher rates (0.8-0.9) help mitigate data scarcity through increased sample diversity. When individual task models show high prediction confidence on the unsupervised data, high sampling rates remain viable despite potential noise. Conversely, lower rates (0.4-0.5) are recommended for unreliable predictions to prioritize data credibility.\n\n- **Supervised Data Guides Sampling Rate Optimization.**  \nSupervised data enables adaptive sampling: set sampling rate equal to task models' accuracy on the supervised set. This balances data trustworthiness and utilization by filtering noise while keeping useful data. **Below we demonstrate the performance improvements enabled by this method. Setting the sampling rate to match the accuracy of the supervised dataset significantly enhances data credibility.**\n||SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|\n|-|-|-|-|-|-|-|-|-|\n|Individual(Full Data)|75.3|77.7|96.1|99.7|97.5|98.7|99.7|79.4|\n|Sampling Rate|75.3|77.7|96.1|99.7|97.5|98.7|99.7|79.4|\n|Individual(Sampled Data)|88.3|87.3|97.4|99.9|98.5|99.2|99.8|87.5|\n\n---\n\n***Question 2:  A detailed ablation study on the individual components of CALM.***\n\n***Answer:*** Thanks for the very constructive comments.\n\n- **Ablation Study of CB-EMS.**  \n**As shown in Figure 5, we present ablation study results of CB-EMS.** The experimental results demonstrate that adding the CB-EMS method achieves performance comparable to supervised learning results and is significantly better than unsupervised EM methods using entropy.  \n**Below, we provide the \"Class-balanced\" ablation results.** Incorporating the Class-balanced component effectively avoids class imbalance and shows significant performance improvement.\n|Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|**Avg Acc**|\n|-|-|-|-|-|-|-|-|-|-|\n|CB-EMS|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|**87.7**|\n|EMS|70.6|73.0|88.2|93.3|88.1|75.9|93.7|70.0|**81.5**|\n\n- **Ablation Study of Efficient-Aware Framework.**  \n**We conduct additional ablation studies by adding the Efficient-Aware Framework (EAF) to existing model merging methods and removing it from CALM.** EAF has no effect on order-insensitive methods (e.g., Task Arithmetic) but effects order-sensitive methods (AdaMerging). Without EAF, CALM reduces to randomly selecting a task for mask learning, with results as follows.  \n**Existing methods exhibit significant performance degradation under the Efficient-Aware Framework** since they require simultaneous consideration of all task vectors.   \n**Without this framework, CALM still maintains robust performance.** Notably, CALM still requires EAF for large-scale task merging.\n|Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|**Avg Acc**|\n|-|-|-|-|-|-|-|-|-|-|\n|AdaMerging (+EAF)|70.7|55.3|61.3|66.5|74.0|58.1|99.1|36.2|**65.2**|\n|AdaMerging (-EAF)|64.5|68.1|79.2|93.8|87.0|91.9|97.5|59.1|**80.1**|\n|CALM (+EAF)|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|**87.7**|\n|CALM (-EAF)|71.9|74.1|91.6|98.7|95.2|96.2|98.9|71.2|**87.3**|    \n\n- **Ablation Study of Consensus-Aware Mask Optimization.**  \nConsensus-Aware Mask Optimization contributes most performance gains as our core component and cannot be fully ablated. We instead conduct ablation studies on our core insights and key components.\n    - **Ablation Study of Localized Information and Global Consenus**  \nAs core insights, CALM degenerates into Localize-and-Stitch (extracting localized task-specific information) when focusing solely on localized patterns, and degenerates into TW AdaMerging (applying identical masks to all parameters) when only global consensus is considered.\n    - **Ablation Study of Regularization**  \nEvaluating the impact of regularization in the optimization equation.\n    - **Ablation Study of Binary Mask**  \nPlease refer to **Reviewer Rsjn Question 2**.  \n|Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|**Avg Acc**|\n|-|-|-|-|-|-|-|-|-|-|\n|Local-only|67.2|68.3|81.8|89.4|87.9|86.6|94.8|62.9|**79.9**|\n|Global-only|58.0|53.2|68.8|85.7|81.1|84.4|92.4|44.8|**71.1**|\n|W/o Regularization|70.1|72.4|90.8|97.6|94.2|95.3|98.2|70.5|**86.3**|\n|CALM|72.6|74.8|91.9|98.6|95.2|96.4|99.1|72.8|**87.7**|", "claims_and_evidence": "The paper focuses on addressing the question: \"How to identify and extract effective information in model merging to enhance performance on all tasks?\" The authors propose that effective information can be represented through localized parameters and aligned with a global task consensus.At first glance, this claim appears reasonable. The authors support their arguments through two primary approaches: first, by analyzing the limitations of existing global and localized model merging methods; second, by experimentally demonstrating the feasibility of extracting global consensus. Although the paper lacks further theoretical validation, the authors emphasize that due to the inherent difficulty in interpretability within model merging, their claims can still be supported experimentally.", "methods_and_evaluation_criteria": "The three contributions of this method hold good novelty and practical value for the model merging community. CB-EMS can serve as a general and feasible approach for unsupervised data in model merging; the Efficient-Aware Framework introduces a new serialized merging approach; while binary mask training follows Localize-and-Stitch, employing global task optimization is indeed more reasonable. Therefore, this method demonstrates clear advantages over previous approaches. If the authors further develop interpretable theories, it would be more beneficial to the community. For experiments, the paper provides sufficiently comprehensive tests, and its evaluation criteria align with classical model merging works.", "theoretical_claims": "Eq.2-Eq.5 in the paper provide the derivation of CB-EMS, and Eq.6-Eq.9 outline the optimization process of Consensus-Aware Mask Optimization. There is no proof process, so no proof errors exist.", "experimental_designs_or_analyses": "The main experiments consist of four parts: Visual multi-task, NLP multi-task, CB-EMS analysis, and Efficient-Aware Framework analysis. The core experiments largely validate CALM’s feasibility, showing strong performance on both visual and NLP tasks. The analysis sufficiently examines the robustness and effectiveness of each proposed module. One question: since the CB-EMS method does not exhibit stable sampling rates, how should sampling be handled in practice?", "relation_to_broader_scientific_literature": "As for research on model merging, there are mainly two approaches. One is data-free methods, which do not require additional task data and rely on experience and strategies to merge models. The other is data-based methods, which utilize extra task information to achieve better merging results. Obviously, the CALM method falls into the latter category, but it fully learns from the former approaches by using task information to develop merging strategies. This combination offers a useful perspective for model merging research.", "essential_references_not_discussed": "I think the references in this paper are comprehensive.", "other_strengths_and_weaknesses": "Strengths:\n(1) The authors present a novel perspective by addressing the fundamental challenges of model merging, proposing a method that balances global task consensus with local parameter alignment.\n(2) The framework of the proposed method is exceptionally clear, with each component being structurally complete and innovating upon existing model merging approaches in all three modules.\n(3) The experimental results demonstrate substantial performance improvements over baselines, even compared to data-based methods.\n\n\nWeaknesses:\n(1) The sampling rates of CB-EMS in experiments appear inconsistent; how would this be resolved in practical applications?\n(2) A detailed ablation study on global consensus is limited. The authors should explore the concept of global task consensus more deeply.", "other_comments_or_suggestions": "Ensure minimal overlap among elements in figures, as exemplified in Figure 1.", "questions_for_authors": "See weakness", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0frQ427Ao8", "reviewer": "Reviewer_hiCE", "summary": "The authors introduce a novel model merging approach called CALM to address multi-task learning integration. The core idea involves identifying localized parameters aligned with global task consensus through three key components:\n1. Class-Balanced Entropy Minimization Sampling (CB-EMS): A method to extract reliable unsupervised datasets while preserving class balance.\n2. Efficient-Aware Framework: A sequential merging strategy that reduces computational complexity.\n3. Consensus-Aware Mask Optimization: A binary mask mechanism to extract effective localized parameters, where global consensus is optimized using CB-EMS datasets.\nExperiments across diverse vision and language multi-task scenarios validate the method’s robustness and effectiveness.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "***Question 1: Does the Efficient-Aware Framework apply to existing model merging methods?***\n\n***Answer:*** Thanks for the inspiring question.\n\nThe Efficient-Aware Framework (EAF) introduces a novel serialized merging approach for model merging, which impacts existing methods as follows:\n\n- **For task quantity/order-insensitive methods**: EAF does not directly affect merging results and can be naturally applied to methods like Task Arithmetic and Localize-and-Stitch. These approaches typically disregard global task information during merging.\n\n- **For task quantity/order-sensitive methods**: Existing techniques (e.g., Ties-Merging, Adamerging) can adopt EAF through pairwise merging substitution. However, their performance will significantly degrade due to EAF's inability to comprehensively integrate multi-task information. Experimental validation appears in **Reviewer oEok Question 2**. This limitation further constrains their scalability for new tasks.\n\n---\n\n***Question 2: How is global task consensus captured through Mask Optimization in practice?***\n\n***Answer:*** Thank you for the valuable question. We would like to explain it as follows:\n\n- **Theoretical perspective**:  The Mask Optimization process extracts gradient directions beneficial for global tasks, aligning with the global task consensus principle. Detailed theoretical analysis is provided in **Reviewer y6gA Question 1**.\n\n- **Empirical perspective**: Mask Optimization effectively compresses task-relevant information while eliminating inter-task interference. Experimental results are available in **Reviewer Rsjn Question 3**.\n\n---\n\n***Question 3: Additional experiments on extensive visual tasks.***\n\n***Answer:*** Thanks for the constructive comments.\n\nWe supplement six new datasets (**CIFAR100, Flowers102, OxfordIIITPet, STL10, KMNIST, FashionMNIST**), expanding the vision tasks to 14 in total, to validate CALM's stability under increased task diversity. Results are as follows.\n\n|Method|SUN397|Cars|RESISC45|EuroSAT|SVHN|GTSRB|MNIST|DTD|\n|-|-|-|-|-|-|-|-|-|\n|CALM ($\\|S\\|=13, \\|\\overline{S}\\|=1$)|69.6|68.6|89.2|97.7|93.6|93.2|98.2|67.5|\n|CALM ($\\|S\\|=12, \\|\\overline{S}\\|=2$)|71.4|73.4|91.0|98.3|94.6|95.2|98.4|70.4|\n|CALM ($\\|S\\|=11, \\|\\overline{S}\\|=3$)|72.0|74.1|91.4|98.1|94.7|95.4|98.5|71.0|\n|CALM ($\\|S\\|=10, \\|\\overline{S}\\|=4$)|72.3|74.0|91.5|98.1|94.9|95.5|98.6|71.7|\n|**Method**|**CIFAR100**|**Flowers102**|**OxfordIIITPet**|**STL10**|**KMNIST**|**FashionMNIST**|**Avg Acc**||\n|CALM ($\\|S\\|=13, \\|\\overline{S}\\|=1$)|79.4|83.4|91.5|97.6|84.4|90.7|**86.0**||\n|CALM ($\\|S\\|=12, \\|\\overline{S}\\|=2$)|81.9|85.0|91.4|97.9|87.1|90.8|**87.6**||\n|CALM ($\\|S\\|=11, \\|\\overline{S}\\|=3$)|82.6|84.8|91.6|97.9|85.6|90.6|**87.7**||\n|CALM ($\\|S\\|=10, \\|\\overline{S}\\|=4$)|83.2|84.7|91.8|97.9|86.2|90.7|**87.9**||\n\n**Key Findings from Experiments on 14 Visual Tasks:**\n\n- **Scalability:** CALM achieves robust performance (average accuracy ≈87% for the original 8 tasks) through sequential merging of a minimal task subset, demonstrating sustained effectiveness under task scaling.\n- **Flexibility:** Superiority over existing methods is attained even with limited merging steps, highlighting its efficiency in resource-constrained scenarios.\n- **Robustness:** The ordering of sequential merging tasks shows negligible impact on final performance, confirming task-agnostic stability.\n\n---\n\n***Question 4: Additional analysis of the Consensus-Aware Mask Optimization module.***\n\n***Answer:*** Thanks for the practical comments.\n\nIn the review, we provided a more detailed analysis of the Consensus-Aware Mask Optimization module, including:\n\n- **The definition of global consensus**: refer to **Reviewer Rsjn Question 1**;\n- **Experimental analysis of the binary mask**: refer to **Reviewer Rsjn Question 3**;\n- **Ablation Study of Consensus-Aware Mask Optimization**: refer to **Reviewer oEok Question 3**.\n\n---\n\n***Question 5: Essential References Discussion.***\n\n***Answer:*** Thank you for your constructive advice.\n\n**Comparison between CALM and ***Model Breadcrumbs: Scaling Multi-Task Model Merging with Sparse Masks:*****\n\n- **Commonality:**\nBoth methods leverage task vectors and acknowledge the significance of local parameters for multi-task model merging.\n- **Divergence:**\nModel Breadcrumbs filters anomalous parameter values based on local prior characteristics (e.g., magnitude distribution thresholds), whereas CALM dynamically extracts task-agnostic knowledge to identify optimal local parameters that align with global task consensus.\n- **Advantage of CALM:**\nCALM more effectively resolves conflicts among local parameters by integrating cross-task knowledge. The consensus-aware optimization ensures that selected parameters exhibit enhanced reliability and generalizability across diverse tasks, thereby offering a more principled and robust solution.", "claims_and_evidence": "1. From the insight, the author proposes that local parameters with global task consensus are the effective information in model merging. The author provides explanations and experiments, and I hope the author will further explore this global task consensus, which will be elaborated later.\n2. The author proposes using Class-Balanced Entropy Minimization Sampling to build reliable unsupervised datasets. This part has sufficient proof.\n3. The author proposes an Efficient-Aware Framework to achieve more efficient model merging. The explanation here is adequate but requires more experimental support.\n4. The author introduces the Consensus-Aware Mask Optimization method to extract local parameters with global consensus. I agree with the value of masks in model merging, and the experimental results prove its effectiveness, since the improvement mainly comes from this method.", "methods_and_evaluation_criteria": "The authors offer a detailed explanation in the method section, including effective figures and algorithm workflows. CB-EMS originates from the classic Entropy Minimization Sampling approach and can be seen as an improvement on the Adamerging entropy.\nThe Efficient-Aware Framework is a serialized framework, with the final Mask Optimization stage aimed at refining task-specific binary masks applicable to all tasks. The method itself has no critical flaws, but the written of Mask Optimization and its corresponding algorithm do not precisely align. I believe the evaluation criteria are appropriately set.", "theoretical_claims": "I have reviewed the formulas and equations in the method section. The paper's theoretical claims focus on explaining the method workflow, with no clear issues.", "experimental_designs_or_analyses": "The authors followed existing experimental setups, testing on standard visual and NLP benchmark datasets, and achieved significant results. However, some existing model merging methods have expanded to more tasks—further exploration on these benchmarks is warranted. The analysis of CB-EMS and Efficient-Aware is complete, but the authors should add more analysis for the Consensus-Aware Mask Optimization module.", "relation_to_broader_scientific_literature": "From a broader scientific perspective, model merging is similar to fields like federated learning and model fusion, but differs in that its multi-task training needs to be based on the same pretrained model. Existing methods fully leverage this characteristic by proposing task vectors [1], and how to optimally utilize these task vectors has become a key focus across approaches. The optimization strategy proposed by the authors – using an unsupervised test set to identify more effective parameter points (working points) for task vectors – builds upon existing research [2,3,4] and applies the optimization strategy to local parameters.\n\n[1]Ilharco, G., Ribeiro, M. T., Wortsman, M., Schmidt, L., Hajishirzi, H., and Farhadi, A. Editing models with task arithmetic.\n\n[2]Yadav, P., Tam, D., Choshen, L., Raffel, C. A., and Bansal, M. Ties-merging: Resolving interference when merging models.\n\n[3]Yang, E., Wang, Z., Shen, L., Liu, S., Guo, G., Wang, X.,and Tao, D. Adamerging: Adaptive model merging for multi-task learning.\n\n[4]He, Y., Hu, Y., Lin, Y., Zhang, T., and Zhao, H. Localize-and-stitch: Efficient model merging via sparse task arithmetic.", "essential_references_not_discussed": "Maybe [1] can be added into discussion.\n[1] Model breadcrumbs: Scaling multi-task model merging with sparse masks.", "other_strengths_and_weaknesses": "## Strengths\n1. This paper presents clear and impactful insight, with strong introductory explanations that offer valuable insights for model merging.\n2. The introduction of EMS and sequentialized merging is innovative. While not entirely novel, to my knowledge, these ideas are the first proposed in the context of model merging, marking an exploratory contribution.\n3. The proposed global task consensus is conceptually related to \"task-shared information\" but distinct: global consensus focuses on non-conflicting parameters, whereas task-shared information emphasizes parameter fusion. This distinction is thought-provoking.\n## Weaknesses\nHowever, I still have some questions that, if addressed, would significantly strengthen the work:\n1. The Efficient-Aware Framework is innovative and proven effective. However, whether this framework applies to existing model merging methods requires further investigation.\n2. The authors should provide a clearer explanation and brief exploration of how global task consensus is captured through Mask Optimization in practice.\n3. Some existing model merging methods have expanded to more tasks， such as 14- 20 visual tasks—further testing on these benchmarks would solidify the claims.\n4. Additional analysis of the Consensus-Aware Mask Optimization module is necessary such as the other two modules.", "other_comments_or_suggestions": "Ensure alignment between the main text and algorithm diagrams, or provide more prominent explanations.", "questions_for_authors": "My questions have been comprehensively outlined in the Weaknesses.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces CALM, a novel model merging approach for multi-task learning that identifies localized parameters aligned with global task consensus. Extensive experiments on vision and NLP benchmarks demonstrate CALM’s superiority over existing methods, approaching the performance of traditional multi-task learning without retraining.

CALM makes significant empirical and practical contributions to model merging, with novel components (CB-EMS, consensus masking) advancing the field, strong scalability demonstrated across 26+ tasks, and thorough rebuttals addressing key concerns.

While deeper theoretical analysis would strengthen the work, all reviewers think that the consistent experimental gains and methodological innovations justify acceptance.
Decision: Accept (poster)