EMR-Merging: Tuning-Free High-Performance Model Merging
Authors: Chenyu Huang
Peng Ye
Tao Chen
Tong He
Xiangyu Yue
Wanli Ouyang
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Model Merging
Model Compression
Multi-task Learning
Supervised Finetuning
Primary area: optimization_for_deep_networks
TL;DR: Existing model merging methods usually suffer from significant performance degradation or requring additional tuning. We realize tuning-free model merging, which shows impressive performance under various experimental settings.

Abstract: The success of pretrain-finetune paradigm brings about the release of numerous model weights. In this case, merging models finetuned on different tasks to enable a single model with multi-task capabilities is gaining increasing attention for its practicability. Existing model merging methods usually suffer from (1) significant performance degradation or (2) requiring tuning by additional data or training. In this paper, we rethink and analyze the existing model merging paradigm. We discover that using a single model's weights can hardly simulate all the models' performance. To tackle this issue, we propose Elect, Mask & Rescale-Merging (EMR-Merging). We first (a) elect a unified model from all the model weights and then (b) generate extremely lightweight task-specific modulators, including masks and rescalers, to align the direction and magnitude between the unified model and each specific model, respectively. EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance. We find that EMR-Merging shows outstanding performance compared to existing merging methods under different classical and newly-established settings, including merging different numbers of vision models (up to 30), NLP models, PEFT models, and multi-modal models.

EMR-MERGING: Tuning-Free High-Performance
Model Merging
Chenyu Huang Peng Yel.3t, Tao Chen't Tong He?, Xiangyu Yue Wanli Ouyang
1 Fudan University Shanghai AI Laboratory
3 The Chinese University of Hong Kong
cyhuang240m.
Abstract
The success of pretrain-finetuli paradigm brings about the release of numerous model weights. In this case, merging models finetuned on different tasks to enable single model with multi-task capabilities is gaining increasing attention for its practicability. Existing model merging methods usually suffer from (1) significant performance degradation or (2) requiring tuning by additional data or training. In this paper, we rethink and analyze the existing model merging paradigm. We discover that using a single model's weights can hardly simulate all the models' performance. To tackle this issue, propose ELECT, MASK & RESCALE- MERGING (AMR-MERGING) We first (a) elect a unified model from all the model weights and then (b) generate extremely light-weight task-specific modulators, including masks and rescalers, to align the direction and magnitude between the unified model and each specific model, respectively. tuning-free, thus requiring no data availability or any additional training while showing impressive performance. We find that EMR-MERGING shows outstanding performance compared to existing merging methods under different classical and newly-establishedd settings, including merging different numbers of vision models (up to 30), NLP models, PEFT models, and multi-modal models.
MERGGEGGING is
Introduction
With the rapid development of deep learning, different model architectures 36 22 proposed, along with multiple training strategies [89
88 are
Pre-trained models' capabilities are Finetuning models on downstream tasks from pre-trained model has become standard paradigm in both NL.P and vision fields 20
86
enhanced, thus showing increasing significance [54
19
51
22 5 87], which usually leads to improved performance with less labeled data. With the development of open-source repositories such as Huggingface (79), timm (77) and torchvision 44
the number of pre-trained and finetuned checkpoints exponentially rise. However, applying individual models to different tasks results in high storage and deployment costs. Multi-task learning (MTL) partially solves this problem by jointly training a model using multiple datasets [70] 93 95], but
suffers from (i) high computational costs and (ii) data unavailability due to privacy 33 Recently,
model merging attempts to solve these drawbacks by combining weights instead of additional training, thus showing vital significance and broad application prospects.
simple strategy of model merging is averaging the model weights [0), but it usually causes obvious performance degradation, as shown in Fig. [ To this end, there multiple model merging methods proposed to improve the performance of the merged model, which can be roughly divided into three
Correspnn Authortestchen@ffuaan.ddd edu.cc. 'quual Contribution. PProecct Lead.
OOu code is available at httpp::/gihbbb..o/////eee/eeeeee 1//M W/MR Merging
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

categories: (i) Weighted averaging of model weights include Fisher-Merging [46 and RegMean B3
They use pre-computed Fisher information matrices [23] and inner-product matrices 33 to tune the
coefficients for weighted averaging. (ii) Task vector-based methods that add task vectors together instead of model weights, include Task Arithmetic [30], Ties-Merging [84), and AdaMerging [85]. Mres-nggging handles the interference issue and AdaMerging adaptively tunes the merging coef- ficients. (ii) Pre-processing techniques include DARE [90]. It reduces interference by dropping most elements and rescaling the others in task vectors. Despite the promising results, there are two unresolved problems with the existing model merging methods: (1) The performance gap between the merged model and individual models or MTL is still obvious, shown in Fig. I (2) The performance improvement of existing methods depends on tuning by data or training, as shown in Tab. T
To boost the performance of model merging, we rethink and analyze the existing model merging paradigm. We discover that the goal of all the existing methods to obtain a sin- gle model applicable to all the N tasks, as follows:
90-
Insividual MTL
Average Accurncy (%)
80
WM = M ([WI..WN)))
(I)
73
where [W] are the model weights to be merged, M denotes the merging func- tion, and War is the merged model weight. This paradigm may inevitably lead to a non- negligible gap between the merged model and each individual model, especially when there numerous models models challeng- ing tasks. We argue that using a single model weight to simulate all the model weights is sub-optimal. To tackle this issue, we propose a brand new merging paradigm: We first ex- tract unified model weight from all the mod- els' weights, and then we calculate and store sigmificant but lightweight task mask-specific parts of each model weight. This process can be written as:
(4)
Simple Averiing Merging
Fisher
RegMeon
Tisk
Ties
Ada
Ada
(MR(OUr))
Arilhmetic Merging
MergingMergingt+
The average accuracy of the multi-task performance of different model merging methods on eight vision tasks. Among all the merging methods, our EMR-MERGING is the only one comparable to the performance of MTL and even individual models.
Figure 1:
Table I: Prerequisites for each method's working.
Training- Data Tuning
Valid Data Tuning inputs
Tuning by Training
Methods
labels
(2) where Wani represents the common and shared part of all model weights and Ex] denote the task-specific parts of each model
W unnis [E,..EN] = M (w))
Weight Averaging
x
x
Traditional MTL
4
x
Merber-Merring
x
Reg Mcan S
x
x
x
Task Arithmetic
30
Ties- Mergingg
8
weight. M' is the revised merging function following our paradigm.
AdaMerging
81
7
8
EMR-Merging
Based on the above paradigm, we propose MERRIIGGING (ELECT, MASK & RESCALE- -ERRINNG...
We first elect a unified model from all the model weights The election strategy is choosing the maximum absolute value of each parameter on the specified sign direction to minimize interference and avoid additional tuning. Then we generate additional lightweight task-specific modulators, including masks and rescalers. Their functions are respectively to align the direction and magnitude of the unified model with the original task-specific model. We find that applying the task- fask-specific modulators to the unified model can better approximate the task-specific model, thus improving performance. The detailed process, theoretical and empirical analysis of the proposed method are illustrated in Section 3 By applying our method, the performance of model merging is significantly enhanced and is comparable to MTL or individual models, as shown in Fig. I Mcanwhile, EMR- MERGING requires no data, tuning, or any additional training, shown in Tab. II
We first demonstrate the effectiveness of the proposed E.R- MERGIIG under the existing setting of (1) merging Vision Transformer (ViT) [22] models of different sizes on 8 vision tasks, (2) merging parameter-efficient finetuning (PEFT) models on language tasks, and (3) merging GPT-2 (55] models on 7 language tasks. Our method shows significant performance improvement under these settings, even when compared to the strongest baseline. We further validate the method's effectiveness under wwlylesttalished and more challenging settings including: (4) merging ViTs on 30 vision
2

---

Merging Procedure
(b) Inference Procedure
Task
Tank3
TaK TeCTTAee
Mart
Tisk- Task-ppol's Dreccion and Ang Rudo Mod//0t99
Modden
Mabi
(i) Task Tpecfep Direction Amplitude Modubtioo
Rrvvle 123
WTN
Figure 2: Framework overview. In the (a) Merging Procedure, we merge task-specific vectors into unified task vector and lightweight task-specific modulators modulate direction and amplitude. During the (b) Inference Procedure, we apply the corresponding mask and rescaler to the unified task vector obtain a specific task vector. The process of (c) Task--ppecfic Direction and Amplitude Modulation includes obtaining task-specifie masks and scalers.
lasks, (5) merging RoBERTa 43 models on 8 NLP tasks, and (6) merging BEiT3 75 models on multi-modal tasks.
Our contributions can be summarized as: (1) We propose a novel merging method called EMR- MERGING, which merges task-specific models into unified model and lightweight task-specific modulators (i.c., masks and rescalers), requiring no data, tuning, or additional training. (2) The proposed MERGING is cumple-but-effective and its effectiveness is validated on various classical benchmarks and newly-established benchmarks under various vision, NLP, PEFT, and multi-modal settings. (3) We show that the masks and rescalers of EMR-MERGING for aligning task-specific direction and amplitude task vectors are applicable to most kinds of merging methods.
2 Related Work
Model Merging obtains model using the existing task-specific model weights instead of training
3
30 84 85]
66) 90 (6). Simply averaging [80] usually causes severe performance degradation. Various
methods are proposed to handle this problem. Fisher-Merging [46] and RegMean (33 use fisher information matrices (3 and inner-product matrices 333 to calculate the merging coefficients
for weighted merging. However, they require additional matrices released by model owners or manually computed. Task Arithmetic [30] merges models by adding together task vectors. which is the difference between the finetuned and pre-trained models. Ties-Merging (84) and AdaMerging (85) based on task vectors. Ties-ne resolves interference and AdaMerging adaptively learns the merging coefficients. However, the performance of Task Arithmetic and Tees-nggging highly depends on manually tuning the merging coefficients and AdaMerging needs additional training to obtain them. DARE (90] reduces interference by randomly dropping most elements and rescaling remaining ones in each task vector before merging. However, DARE's performance only validated under the setting of merging a limited number of tasks and the performance gain is also limited, addition, all existing methods merge models into single one, and have not been verified under experimental settings of more models to merge, models more difficult tasks, and

---

multi-modal models. In this paper, we propose -MMMGEGGI which requires no tuning while showing impressive performance under various settings.
Multi-Task Learning trains a single model using training data from multiple tasks together 70 93
[93] MTL typically necessitates access the labeled data of multiple tasks for training the model from scratch. Though enabling the model multi-task capabilities, MTL suffers from not only (i) the expensive computational cost for training, especially for large models, but also (ii) the limited data availability due to data privacy [85] In comparison, model merging solves the mentioned problems by combining the model weights without using training data or additional training, thus obtaining a multi-task model while sharply reducing the costs.
down-stream tasks is becoming standard 22 5. Depending on whether all the parameters of models are adjusted, SFT can be divided into conventional full finetuning (FFT) and parameter- efficient finetuning (PEFT), which is proposed to reduce the number of trainable parameters for downstream tasks by adjusting the inserted small modules called adapters while keeping the whole model frozen [28 12 42). PEFT is becoming the prevailing method to adapt pre-trained large models because of its efficiency 94 There are a large number of pre-trained, full finetuned model weights, and PEFT module weights available on public repositories [79] 77 44]. In this paper, the proposed EMR- MERRGING based on the common gretrain-fineture paradigm and we show the applicability of our method to both full finetuned models and PEFT modules.
Supervised Finetuning from pre-trained models paradigm in both NI.P and vision fields 20
51
19
3 Method
3.1 Motivation
Given N tasks [Ti..Ty]. the goal of model merging is to obtain a model applicable to all the tasks using finetuned models [W]..WN] from the same pre-trained model W pre each task. Existing methods focus on merging the models into a single model M. Please check Appendix [oor detailed information on the existing merging methods. However, single model can hardly represent all the model weights, thus causing severe performance drops. We discover that the combination of a unified task vector and lightweight task-specific modulators can settle this issue approximating the task-specific vectors better without any additional tuning. The size of proposed task-specific modulators is discussed in Section 4.4 which is much smaller than that of a model.
significant extent by
3.2 ELECT, MASK & RESCALE-MERGING
The overall framework of EMR- is shown in Fig. 2 We follow the setting of task vector- based methods 30 84 85) and we merge models using task vectors. For task T,, i [1..N]. the corresponding task vector is defined as т, W W pre where T, Rdd
Electing a unified task vector We first create aggregate elected sign vector Thni sgn E2 TR) by choosing the sign with the higher total magnitude of each parameter across all relevant task vectors. Then we choose the maximum absolute value of each parameter with the sign consistent with Yuns from all the task vectors and obtain absolute value vector Eurd E Rd. By combining Tuni and cani. the unified task vector can be obtained by Tani = Tuni @ Guni- The electing procedure can reserve the maximum amplitude and sign information shared by the task vectors, thereby maximally reducing interference. The unified task vector Tuni corresponds the W Wuns in Eq Before being applied task T;, the Tuni needs to be modulated in advance by task-specific modulators, which are corresponding to E, in Eq. 2 The generation of task-specific modulators is described below:
Task-specific masks. Next, we compare the unified task vector Tani with each task vector 7.. The task- specific mask M, = (Tx @ Tuns > 0) for task sets the elements whose signs are not correspondent with Tuni to zero and the rest to one. The function of the masks is to align the direction of the unified model with the task-specific model. The masks share the same structure with the task-specific models but due to their I-bit nature, the size of a mask is much smaller than that of task vector.
Task-specific Rescalers Then, for each task, compute a rescaler parameter keep the average absolute value of the elements in Ti and M, O Tani equal. The function of the rescalers X; = sum(abs((T)) is to align the parameter magnitude of the unified model with the task-specific Mam(abe(MMorr)

---

Elect
Dlect Mask
EMR Merging
Individual
Elect
Elect Mask
MMR-MMiging
Individual
EunSAT
RESINCAS
RESISCAS
EumSAT
SHVV
4
Figure 3: Partial (a) t-SNE and (b) Grad-CAM visualization results of EMR-MERGING'S procedures.
model. The significance of rescaling is also reported by DARE 900, which claims that after dropping most elements in a task vector, rescaling the rest leads to better results compared to not.
Before being applied to task, task-specific modulation is required to be conducted to the unified task vector. After that, we add it to the pre-trained parameter values W pre. The inference steps of
applying the merged model to task are as follows: W, = Wpre + é. where f = A Me @ Tuni- should be noted that during the whole process, no additional tuning is needed, thus requiring no data or additional training. We summarize the algorithm flow in Appendix A
3.3 Theoretical analysis
Our goal is to merge model weights by minimizing the distance between the merged model Wuni and each individual model W;, where the distance can be calculated by:
Dis 1
W Wunil' N
Σ
||
Tuni
12
N
where т, refers to the task vector for task T; and Tuni is the unified task vector.
Analysis 1: Effectiveness of Masks. After applying the masks M; (Ta @ Tuni 0) to the unified model Tunis the distance Dis can be formulated as:
ΣN
||,
Me O Tunill
Dis
< Dis
(4)
N
where Dis refers to the distance before applying the masks. Eq. 4 demonssratess that the distance between the merged model and each individual model can reduced after applying the masks.
Analysis 2: Effectiveness of Rescalers. After applying the rescalers X
sum sam (ammaaahrrrrrrr M.OrF)) the
sum (abs(r.))
masked task vectors M, Tani, the distance Dis M,A is formulated as:
Di IT
A M, O Tuni 2
D.
Dis
Eq. demonstrates that the distance between the merged model and each individual model can be minimized after applying the rescalers. Please check Appendix [ for detailed proof.
3.4 Empirical analysis
In Fig. visualize partial results of merging eight ViT-B/32 models different tasks using I-SNE 699 and Grad-CAM [61]. It can be seen that each procedure of EMR-MERGING can help improve the performance of the merged model and perform closer individual models. Specifically, a more obvious distinction is shown in t-SNE and a more precise target is focused by Grad-CAM.
Please check Section 1.1.I for experimental details and Appendix for forr more visualization results.

---

Table 2: Multi-task performance when merging ViT-B/32 models on eight tasks.
Methods
SUN397
Cars
RESISC4S
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Acc
Individual
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
Traditional MTL
73.9
74.4
91.9
98.2
95.8
98.9
99.5
77.9
88.9
Weight Averaging
65.3
63.4
71.4
71.7
64.2
52.8
87.5
50.1
65.8
Fisher Merging
10
69.2
70.7
66.4
72.9
51.1
87.9
59.9
68.3
RegMean
65.3
63.5
75.6
78.6
78.1
67.4
93.7
52.0
71.8
Task Arithmetic
62.1
72.0
65.1
52.2
70.1
Ties-Merping 84
64.8
62.9
74.3
78.9
83.1
71.4
97.6
56.2
73.6
AdaMerging
64.5
68.1
79.2
93.8
87.0
91.9
97.5
59.1
80.1
AdaMerging++
68.3
82.2
94.2
89.6
89.0
60.6
81.1
EMR-MERGING (Ours)
75.2
72.8
93.5
99.5
96.9
98.1
99.6
74.4
88.7
Table 3: Multi-task performance when merging ViT-L/14 models on eight tasks.
Methods
SUN397
Cars
RESISCAS
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Ace
Individual
82.3
92.4
97.4
100
96.1
99.2
99.7
84.1
94.2
Traditional MTL
80.8
90.6
96.3
96.3
97.6
99.1
99.6
84.4
93.5
Weight. Averaging
72.1
81.6
82.6
91.9
78.2
70.7
97.1
62.8
79.6
Fisher Meggin
4G
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70.0
82.2
RegM
3+
73.3
81.8
86.1
97.0
88.0
84.2
98.5
60.8
83.7
Task Arithmetic
30
74.1
82.1
86.7
93.8
87.9
86.8
98.9
65.6
84.5
Ties Merging
84
76.5
85.0
89.3
95.7
90.3
83.3
99.0
68.8
86.0
AdaMerging
79.0
90.3
90.8
96.2
93.4
98.0
99.0
79.9
90.8
AdaMerging++
83
79.4
90.3
91.6
97.4
93.4
97.5
99.0
79.2
91.0
EMR-MERGING (Ours)
83.2
90.7
99.7
97.9
99.1
99.7
82.7
93.7
In Fig. we compare sign con- flicts, L2 distance, and cosine sim- ilarity between the merged model weights obtained by different merg- ing methods and the task- specific model weights. It can be seen that EMR-MERGING significantly re- duces sign conflicts and 1.2 dis- tance and improves the cosine similarity, indicating that EMR- MERGING approximates each task- specific model weight effectively. The configuration of Fig. 4 can be found in Appendix F
Task Arthhmeic
Ties
AdaMerging*+
BMR Merging
0.15
0.4
0.2
0.0
(n) Sign Conftiet
(b) L2 Distance
(6) Cosine Similarity
Figure Comparison of (a) sign conflicts, (b) L2 distance, and (c) cosine similarity of model weights obtained by different methods and task-specific model weights.
4 Experiment Validation
Baseline methods. We compare the proposed EMR-MERGING with: (1) Individual Models, (2) Tra- ditional MTL, (3) Weight Averaging, (4) Fisher Merging [46), (5) RegMean 333, (6) Task Arith-
metic [30] (7) Ties-Merging 884 (8) AdaMerging [5] For more details about baseline methods,
please check Appendix
4.1 Merging vision models
4.1.1 Merging 8 ViTs.
Settings. We follow the setting from Task Arithmetic (30], Ties-Merging (84, and AdaMerging (85). We employ ViT-B/32 and ViT-L/14, two variants of CLIP (54) models' visual encoders, as the pre-trained models. The performance of each method is evaluated by eight image classification
tasks, including SUN397 [83], Cars [35]. RESISC45 [10], EuroSAT [27]. SVHN 9I], GTSRB [65]
MNIST 38), and DTD [11. All the datasets are evaluated by accuracy.
Results. The experimental results of merging ViT-B/32 and ViT-L/14 on eight tasks are shown in Tab. and Tab. 3 We observe that MER-MEGGGNG shows significant performance improvement

---

Tack
The
ADA
Indvsbul Ul
The
ADA
Kapbleaa
Figure 5: Partial visualization results of different merging methods, (a) t-SNE and (b) Grad-CAM
Table 4: Task-specific and average performance when merging ViT-B/16 models on 30 tasks.
MNIST
Citur-o
Vegesables
Food:
Kvasios2
lntel lmnges
EueoSAT
Indirittol
09.71
97:38
100.00
$7.93
04.51
94.63
$5.96
59.04
06.27
59:05
Weight Avecoping
27.63
42.91
K3.20
06:13
25.27
K2.00
7.54
24.37
61.00
RopMoon
95.10
93.00
Aithmeet
91.77
73.06
31.05
19.03
9:34
14.56
93.61
Ties Merging (4)
23.21
42.82
02.31
13.23
31.09
19.39
5.30
10.96
72.86
91.38
Menine
61.22
87.54
07.97
75.23
22.76
91.02
0.42
44.00
69.13
96.91
EMR MERGING (Ours)
96,69
99.97
95.27
72,48
96.24
97.76
99.27
Degs
Facbion
Pleaers
ST1-10
C1B--00-3001
FMNIST
DTD
RESISI145
Weight Auazuring
42.80
20446
51.25
73.14
68.91
52.24
37.86
3.35
14.63
13.56
RegMein F3
42.89
83.42
34.62
63.64
95.26
78.94
49.78
48.07
30.53
34.65
Taok Aritbraic
47.65
14.23
15.90
MaagMnnggng
24.33
KMR MERGING (Ours)
81.89
92.41
87.15
16.17
97.08
98.41
76.1
92313
61115
90.11
MangoLea/BD
Beam
Cios-100
OT5RB
SVHBN
SUN397
KenyarFoed13
Animal- 10N
Garlug:
Fruis-360
Indirihtd
100.00
01:35
89.85
95.34
06.22
78.94
$5.53
92.52
03.36
99.63
Weush Avensoing
73.98
57.42
RegMonn
Tink Arthmetis 500
62.02
84662
90.20
57.01
17.41
55.88
36.32
51.14
25.23
6.15
Thes Meu-Merging.. [0
76.56
67.32
7561
00.34
18.54
52.09
19.90
19.1
3.91
1.50
MaMMrging
95.15
91.38
64.19
59.50
25.20
64.09
48.06
06:35
18.54
7.56
EMR
(Ours)
76.19
Averoge
Individul
Neeraing
RegMous 13
Arthunesi [0]
Merping [4] /MMMerring [5 EMR MMRRMEBGIING (Ours)
42.52
08.14
60.25
$9.54
compared existing merging methods, respectively 7.6% and 2.7%. Notably, MARMMEEGGNG requires no additional training, tuning, or any dataset accessibility while outperforming AdaMerging and Ties-Merging. which require additional training or careful hyper-parametts tuning using datasets. Under this setting, EMR- MERGING performs very close to or even better than traditional MTL, which is normally considered as a reference upper bound for model merging work [85]. For visualized comparison, we provide some visualization results of different merging methods using t-SNE and Grad-CAM in Fig. 5 can be seen that among all the merging methods, the visualization results of EEMR-MERGITT are the closest to individual models, which corresponds to quantitative results. Please check Appendix D for more visualization results.
4.1.2 Merging 30 VITs.
Settings. To further explore the performance of EMR- MERGGNG, we establish a new benchmark on merging vision models, expanding the number of task-specific models from eight to 30. We
employ ViT-B/16 (2 pre-trained on ImageNet-21k [8] as the pre-trained model. The performance
is evaluated by image classification datasets including MNIST (38), CIFAR-10 [36], Vegetables M.
Food-101 (6), Kvasir-v2 [53]. Cars [35, Intel Images 4. EuroSAT 22 Weather [82]. Cats and
dogs [15). ManDoLcafBD 22. Beans 37 CIFAR- 100 36), GTSRB (65] SVHN 911. Dogs 34
Fashion MNIST [8). Oxford Exford-IIIT-Per (50), Landscape Recognition 177, Flowers Recognition 45
STL-10 12 CUB-200-201 (73), EMNIST [13], DTD M. RESISC45 (1O), SUN397 [83, Kenyan-
Food13 32 Animal- 644, Garbage Classification R and Fruits-360 [47), covering tasks from
common food classification to disease detection. All of them evaluated by accuracy.
Results. The experimental results are shown in Tab. challenging setting of merging 30 models, all the existing methods show significant performance drops compared to individual models. Even RegMean, which performs best among existing methods, still exhibits performance decay of nearly 25%. However, EMR-MERGING reduce this value
can be clearly seen that under this
7

---

Table 5: Results of merging RoBERTa models eight datasets from GLUE benchmark.
Methods
Sentene Tasks
Similarity MRPC
and Paraphrase Tasks
Inference Tasks
CoLA
SST2
STSB
QQP
MNLI
QNLI
RTE
Individual
0.6018
0.9404
0.8922
0.9063
0.9141
0.8720
0,9271
0.7906
Weight Averaging
0.1396
0.6411
0.6936
0.3184
0,7536
0,4219
0.587
0.5523
RegMcan [3
0.3667
0.906
0.7574
0.6268
0.8355
0,7002
0.8235
0.5848
Task Arithmetic 30
0.1878
0.8589
0.7990
0.7403
0,8378
0.5908
0.6967
0.6209
Ties Tees-Merging
0.2048
0.8440
0.8113
0.5819
0,8570
0.6465
0.7481
0.4296
EMR-MERGING (Ours)
0.3996
0.9335
0.8627
0.8277
0.8972
0.8545
0.8957
0.7437
Table 6: Multi-task performance when merging GPT-2 models on seven text classification tasks.
Method
CoI.A
MNLI
MRPC
QNLI
QQP
RTE
SST-2
Avg.
Indivudual
76.8
82.1
80.4
88.3
89.6
65.3
91.2
82.0
Weight Averaging
55.0
55.1
51.0
57.6
76.7
44.8
52.5
56.1
Fisher Merging 46
54.8
58.0
39.5
63.3
81.5
49.1
64.7
58.7
RegMean [33
61.7
70.4
65.4
78.8
56.0
79.7
68.8
Task Arithmetic
10
68.7
68.6
69.6
70,5
81.8
47.3
83.6
70.0
Tics Merging
84
68.4
71.4
68.4
69.6
82.4
47.7
81.8
70.0
EMR-MER MERGINGGG (Ours)
72.8
81.1
79.2
84.8
88.1
66.5
90.3
80.4
to 3.48%. This shows that the proposed method maintains the performance comparable individual models when merging vision models even if the number of tasks increases.
4.2 Merging language models
4.2.1 Merging fully finetuned RoBERTa models
Settings. We partially follow the setting from DARE [90]. However, instead of merging two or three models at a time, we merge all eight models finetuned on each task. RoBERTa-base [43] model is selected as the pre- trained model. The performance of each method is evaluated by eight tasks from
GLUE 74] benchmark, respectively CoLA [76]. SST-2 663, MRPC [2I], STS-B (9) QQP [II,
MNLI 78 QNLI (56), and RTE (24] Among them, CoLAA is evaluated by the Matthews correlation coefficient, STS-B is evaluated by the average value of Pearson and Spearman correlation coefficients, and the rest tasks are evaluated by accuracy.
The experimental results are shown in Tab. 5 It can be seen that EMR-NNGGING outperforms all the existing methods every task, verifying the applicability of the proposed method to language models. Note that the reported results of Ties-Merging, Task Arithmetic, and RegMean are the best among multiple hyper- syper-paramettr settings. Please check Appendix D.4 for more detailed information. should also be noted that we find that under our setting of merging multiple models, DARE may not help improve the performance. Similar results were also reported by 25 This may be due to DARE's random dropping strategy can no longer resolve conflicts among task vectors under the setting of merging multiple models. Please check Appendix (D3 for DARE's experimental results.
Results.
4.2.2 Merging fully finetuned GPT-2 models
Settings. We follow the setting from FusionBench 688, a benchmark for model merging. We merge GPT-2 (55) models seven tasks from GLUE 774, each with different head for classification. Under this setting, each task is evaluated by accuracy.
Results. The experimental results are shown in Tab. 6 EMR-MERGING outperforms all the merging methods by over 10% and decreases the performance degradation caused by model merging from 12% to 1.6%. This validates the applicability of EMR- MERGINGGING to fully finetuned GPT2-scale language models.
4.2.3 Merging PEFT models
Settings. We follow the setting from Ties-Merging 84
(IA)3 42 is PEFT method that uses
learned vectors to scale the base model activations. We use TO-3B 60) as the base model and merge (IA)3 modules. The performance is evaluated using eleven datasets, including RTE [24] CB [16),
8

---

Table 7:
Results of merging (IA)* models on eleven NLP tasks.
Methoah
Valiariion
RTB
CB
Winngrande
WSC
WSC
COPA
H-SWAG
Stury Cloze
ANLI-RI
ANL1-22
ANLI-R3
Arg. Aee
Individual
K2.7
95.5
75.1
71.7
05.3
85.3
45.3
94.9
70.2
46.5
53
71,4
Traditional MTL
85.6
95.8
75.5
61.1
30.6
98.1
42.3
97.6
70.5
49.5
47.7
73.1
Fisher Menrie
66
83.5
53.3
56.7
54.2
58.3
83.1
42.2
94.1
45.9
41.0
42.2
62.2
RagMore 23
81.2
58.3
53.8
55.2
53.5
80.9
40.1
92.5
43.3
39.2
40.2
5X
Tiok Anllmnck
74.1
83.3
62.8
4911
49.3
87.5
41.5
95.3
00.5
49.4
5000
63.9
Ties Merging
76.0
83.3
67.9
57.6
59.7
81.7
42.8
90.3
66.9
51.3
51.1
66.4
Weight Avcraging
x
81.2
58.3
53.8
55.2
53.5
80.9
40.1
92.5
43.3
39.2
40.2
58
Tisk Aitllnekk
X
76.5
79.2
57.7
51.6
51.4
66.2
31.4
81.5
59.8
47.5
48.2
99.2
Ties Mersing [4]
81.2
87.5
60.8
59.9
58.3
80.1
42.6
91.1
39.1
46.5
47.4
64.9
EMR MMIR-MERGING (Ours)
X
81.8
$7.5
66.6
36.1
65.3
82.4
44.7
93.6
65.7
43.8
50.8
67.1
Table 8: Results of merging multi-modal BEiT3 models on five vision-language tasks.
M.thock
COCO-Retiieal Accumcy(
CIDP(())
COCO-Caplioing
METFOR())
ROUGF-111)
ImageNet-1lk Classification
Accoracy(1)
NLVR2 Accuracy(")
Accurscy(1))
0,8537
Wiighe Averaging
0.011
0,159
0.6771
0.6285
Arilhmetic HD)
0000
0.118
0.176
0.7081
0,6033
Tias-Maggnng [84)
0.01
0,167
)
0.534
0,7742
0,7211
Winogrande [59], WiC [52,, WSC [39, COPA [58), H-SWAG 92 Story Cloze from RI to R3. All the datasets are evaluated by accuracy.
62 and ANLI 48
Results. The experimental results shown in Tab. EMR-MERGING outperforms all the merging methods. Compared to methods without validation, EMR-MERGING improves the average accuracy each task by 2.2%. When compared to methods that require validation data tune hyper-parameters or compute matrices for weighted merging, EMR-MERG MER-MEIGING still improves the average performance by 0.7%, validating the applicability of method PEFT models.
4.3 Merging multi-modal models
Settings. We merge BEiT3- base (75) models finetuned on five datasets from different kinds of tasks, respectively ImageNet-Ik 1k [18] (Image Classification). VQAv2 [26] (Visual Question Answer- ing), NLVR2 [67] (Visual Reasoning), COCO Captioning [11 (Image Captioning). and COCO
Retrieval 41) (Image-Text Retrieval). Among them, COCO Captioning evaluated BLEU4 44)
CIDEr 722. METHOR 3, and ROUGE-L [40]. The other tasks are evaluated by accuracy.
Results. The experimental results are shown in Tab. S It can be seen that EMR-MERGING performs best on all the vision-language tasks regardless which evaluation metric applied among all the in merging multi-modal models.
merging methods, validating the effectiveness of
4.4 Merging different number of models
In Fig. we compare the number of parameters and performance using indi-
Avg. Normalized Pert. 8
vidual models, Ties-Merging, and EMR- MERGING when merging different num- bers of ViT-B/32 models under the setting of no-validation.
Num. of Pammm (M)
Indivilull Miskis TES Marging
0.9
Mirging
150-
Number of parameters. Compared other merging methods, EMR-MERGING requires little additional storage for task- specific modulators. However, compared single 32-bit model, the additional stor- age caused by task-specific 1-bit mask equals a binarized network, whose size is 32 times smaller than single 32-bit model [14. Additionally, the storage required by a task-specific rescaler, which is a single pa- rameter, is negligible. In Fig. (]a). we compare the number of parameters when merging different numbers of models, and we observe that EMR-MERGING'S parameter number slightly more than Ties fres-Merging but significantly fewer than individual models.
Numher of Tasks
Number of Tasks
Figure Comparison of the (a) number of parameters and (b) average normalized performance when using in- dividual models, Ties-Merging, and EMR-MERGING.

---

Table 9: Ablation the Electing procedure of EMR-MERGING
Methods
SUN397
Cars
RESISC4S
EuroSAT
SVHN
GISRB
MNIST
DID
Avg Acc
Task Arithmetic
63.8
62.1
72.0
77.6
74.4
65.1
94.0
52.2
70.1
Task Aritlunetic M&R
67.6
80.2
91.3
79.3
75.7
96.0
57.9
76.9 [1 6.8)
Ties-Merging
78.9
71.4
56.2
Ties-Merging w/ M&R
68.8
68.9
82.2
91.6
81.4
80.0
96.6
59.3
78.6 [1 5.0]
AdaMerging++
66.6
68.3
82.2
94.2
89.6
89.0
98.3
60.6
81.1
AdaMerging++ w/ M&R
74.0
76.2
93.1
98.2
93.3
96.3
99.4
71.2
87.7(1 6.61
EMR MERGING (Ours)
75.2
72.8
93.5
99.5
96.9
98.1
99.6
74.4
88.7
Table 10: Ablation on the Masking and Rescaling procedures of EMR-MERGING
Methods
SUN399
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Ace
(Elect)
31.7
34.7
65.9
85.7
98.2
42.2
Ours (Elect & Mask)
70.7
65.9
92.2
98.7
96.9
97.6
99.6
72.3
86.8 [2 27.5]
Ours (Ellect & Rescale)
58.2
57.2
69.1
81.6
85.2
73.0
98.4
52.2
71.9 [1 12.6]
Ours (Elect, Mask & Rescule)
73.2
72.8
93.5
99.5
96.9
98.1
99.6
74.4
88.7
17 29.41
Performance. The performance comparison when merging different numbers of models is shown in Fig. (b). Compared to Ties-Merging, the performance of EMR-MERGING is higher and decreases more slowly the task increases. Note that EMR-MERGING outperforms individual models under the 2-task setting. Similar findings are reported by DARE [9]] More details are shown in Appendix D.5
4.5 Ablation Study
We perform ablations all the components of EMR-MERGING as follows.
Ablation on Electing procedure. Tab, IJsows the results of merging eight ViT-B/32 models when the Electing procedure is replaced by other task vector-based merging methods. The effectiveness of our Electing strategy verified by outperforming the combination of other merging methods with masking and rescaling. Another interesting finding is that post-processing procedure, masking and rescaling can help improve the performance of task vector-based merging methods, respectively 6.8%. 5.0%, and 6.6% for Task Arithmetic, Ties-Merging, and AdaMerging++.
Allaiion on Masking and Rescaling procedures. Then, we further validate the importance of Masking and Rescaling procedures by disabling either or both of them. The results are shown Tab. TO It can be seen that simply electing results in a severe performance drop while adding improve the performance by 27.5% and 12.6%, respectively. Furthermore,
Masking and Rescaling compared to separately applying either these two procedures, jointly applying Masking and Rescaling leads to greater improvement, up 29.4%.
5 Conclusion
In this paper, we study on tuning-free and nigh-perffrmance model merging. We first attribute the severe performance degradation of existing merging methods to that single model can hardly simulate all the models' performance. Then we propose ELECT, MASK RESCALE-MERGING (EMR-MERGING which does not require any data access or additional training for tuning. The effectiveness EMR-MERGIN MERGING validated by comprehensive experiments on various classical benchmarks and netly-established benchmarks under vision. NLP, PEFT, and multi-modal settings.
6
Acknowledgement
This work is supported by Shanghai Natural Science Foundation (No. 23ZR 1402900), National Natural Science Foundation of China (No. 62071122 and No. 62306261), National Key Research and Development Program of China (No. 2022/D0160101). The computations this research were performed using the CFFF platform of Fudan University.
10

---

References
[I] M. I. Ahmed, S. M. Mamun, and A. U. Z. Asif. Denn-based vegetable image classification using transfer learning: A comparative study. In 202/ 5th International Conference on Computer, Communication and Signal Processing (ICCCSP), pages 235-243. IEEE, 2021.
pages
[2] S. Ahmed, M. Ibrahim, M. Nadim, M. M. Rahman, M. M. Shejunti, T. Jabid, and M. S. Ali. Mangoleafbd: comprehensive image dataset classify diseased and healthy mango leaves. Data Brief, 47:108941, 2023.
[3] S. Banerjee and A. Lavie. Meteor: automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or sunmarization, pages 65-72, 2005.
[4] P. Bansal. Intel image classification. Available on http::/www koggle. com/puncerr006600nee----icm classification, Online, 2019.
[5] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. Bernstein. Bohg. A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.0722 2021.
[6] L. Bossard, M. Guillaumin, and L. Van Gool. Food- 101- - mining discriminative components with random forests. In Eumpean Conference on Computer Vision, 2014.
[7] T. Brown, B. Mann, Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 1371 2020.
[8] CCHANG. Garbage classification.
2018.
[9] D. Cer, M. Diab, Agirre, I. Lopez-Gazpio L. Specia. Semeval-2017 task 1: Semantic textual similarity- muilllgt and cross-lingual focused evaluation. arXiv preprint arXiv: 1708000055 2017.
[10] G. Cheng, Han, and Lu. Remote sensing image scene classification; Benehmark and state of the art. Proceedings of the IEEE, (05(110:1865-188 2017.
[11] M. Cimpoi. S. Maji. L. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Proceedings of IEEE conference on computer vision and pattern recognition, pages 3606-3613, 2014.
[12] A. Coates, A. Ng. and H. Lee. An analysis of single-laye layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMIR Workshop and Conference Proceedings, 2011.
[13] G. Cohen, S. Afshar, J. Tapson, and A. Van Schaik. Emnist: Extending mnist to handwritten letters. In 2017 international joint conference on neural networks (IJCNN), pages 2921-2926. IEEE, 2017.
[14] M. Courbariaux. I. Hubara, D. Soudry, R. El- Yaniv, and Y. Bengio. Binarized neural networks: Training deep neural networks with weights and activations constrained tot 1 or-1, arXiv preprint arXiv: 0202.02830. 2016.
[15] W. Cukierski. Dogs vs. cats, 2013. URL htts::/kaggle. comkconpettioosddoss--vvoa..
[16] M.-C. De Marneffe, M. Simons, and J. Tonhauser. The commitmentbank: Investigating projection in naturally occurring discourse. In proceedings Sinn und Bedeutung, volume 23, pages 107-124, 2019.
[17] DeepNets.
Landscape recognition. https: http://wwwwwlw.l.ee/e/ http://wwww.aaagge...o//aaaaaaae http:////oaaaassssss///aaaaaaaae
mondssape-recogmiiioo--ii
dataset- 12k-imagess
[18] J. Deng, W. Dong, R. Socher, L.JJ. K. Li, and L. Fei Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages 248-255. leee, 2009.
[19] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 1810:08800004805 2018.
[20] J. Dodge, G. Ilharco, R. Schwartz, A. Farhadi, H. Hajishirzi, and N. Smith. Fine-tuning pretrained language models: Weight initializations, data orders, and early stopping, arXiv preprint arXiv:2002.06305 2020.
[21] B. Dolan and C. Brockett. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005). 2005.

---

[22] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale, arXiv preprint arXiv:2010.1
2020.
[23] R. A. Fisher. On the mathematical foundations of theoretical statisties. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character. -688. 1922.
[24] D. Giampiccolo, B. Magnini. I. Dagan, and W. B. Dolan. The third pascal recognizing textual entailment workshop on textual entailment and paraphrasing, pages
challenge. In Proceedings of the 1-9, 2007.
[25] C. Goddard, S. Siriwardhana, M. Ehghaghi, L. Meyers, V. Karpukhin, B. Benedict, M. McQuade, and J. So- lawetz. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint 2024.
13257,
[26] Y. Goyal, T. Khot, D. Summers- Staye D. Batra, and D. Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6904-6913, 2017.
[27] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217-222 2019.
[28] N. Houlsby, A. Giurgiu, Jastrzebski, B. Morrone, Q De Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pages 7790-2799, PMLR, 2019.
[29] E. J. Hu, P. Wallis, Z. Allen-Zhu Y. Li, S. Wang. L. Wang. W. Chen, et al. Lora: Low-rank adaptation of large language models. In International Conference on Learning Representations, 2021.
[30] G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations, 2022.
[31] S. Iyer, N. Dandekar, K. Csernai, et al. First quora dataset release: Question pairs. data. quora. com. 2017.
[32] Jalal, K. Wang, S. Jefferson, Y. Zheng, E. Nsoesie, and M. Betke. Scraping social media photos posted in kenya and elsewhere to detect and analyze food types. In Proceedings of the 5th International Workshop on Multimedia Assisted Dietary Management, pages 50-59, 2019.
[33] X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representutions, 2022.
[34] A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine fine-grained image categorization. In First Workshop on Fine-Graiiee Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011.
[35] J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554 561, 2013.
[36] A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[37] M. A. Lab. Bean disease dataset, Jamuary 2020.
[38] Y. LcCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mnist/, 1998.
[39] H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. Thirteenth international conference on the principles of knowledge representation and reasoning, 2012.
[40] C.-Y. Lin. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pages 74-81, 81, 2004.
[41] T.-Y. Lin, M. Maire, S. Belongie, Hays, P. Perona, Ramanan. P. Dollár, and C. Zitmick. Microsoft coco: Common objects in context. In Computer Vision 2014: 13th European Conference, Zurich, Switzerland, September 6- 12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[42] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Fhw-shot efficete fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950- 196000999..61611 2022.
12

---

[43] Y. Liu, M. Ou, N. Goyal. J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis. L. Zettlemoyer, and V. Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint urXiv:1977.11992 2019.
[44] T. maintainers and contributors. Torchvision: Pytorch's computer vision library. httpp://gittuttt..iia
pytorch/vision 2016.
[45] A. Mamaev. Flowers recognition.
Flovers- rrccrniieooontiioon
[46] M. S. Matena and C. A. Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703- 17716, 2022.
[47] H. Muresan and M. Oltean. Fruit recognition from images using deep learning. Acta Universitatis Sapientiae, Informatica, 10(1))26-42, 2018.
[48] Y. Nic, A. Williams, E. Dinan, M. Bansal, Weston, and D. Kicla. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910. 14500, 2019.
[49] K. Papineni, S. Roukos, T. Ward, and W.J. Zhu. Bleu: method for automatic evaluation of machine translation. In Proceedings the 40th annual meeting the Association for Computational Linguistics. pages 311-318, 2002.
[50] O. M. Parkhi, A. Vedaldi, A. Zisserman, and C. Jawahar. Cats and dogs. In 2012 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEEE, 2012.
[51] S. Paul and P.-Y. Chen. Vision transformers are robust learners. In Proceedings of the AAAI conference on Artificial Intelligence, volume 36, pages 2071-2081, 2022.
[52] M. T. Pilehvar and J. Camacho-Colladee Wic: the word-in-contex dataset for evaluating context-sensitive mcaning representations. arXiv preprint arXiv: 09121, 2018.
[53] K. Pogorelov, K. R. Randel, C. Griwodz, S. L. Eskeland, de Lange, D. Johansen, Spampinato, D.-T. Dang-Nguyen, M. Lux, P. T. Schmidt, et al. Kvasir: multi-class image dataset for computer aided gastrointestinal disease detection. In Proceedings of the 8th ACM on Multimedia Systems Conference, pages 164 169,2017.
2017.
[54] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin. Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8888888 9748.8763. PMLR, 2021.
[55] A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever. Language models are unsupervised multitask learners. 2019.
[56] P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. Squad: 100,000+ questions for machine comprchension of text. arXiv preprint arXiv: 1606.05250 2016.
[57] A. Ramé, K. Ahuja, J. Zhang, M. Cord, Bottou, and D. Lopez-Paz. Model ratatouille: Recyeling diverse models for out-of -istriibbittinii generalization. In International Conference on Machine Learning, pages 18656-28679 PMLR, 2023.
[58] M. Roemmele, C. A. Bejan, and A. S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI Spring Symposium Series, 2011.
[59] K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Communications of the ACM, 6499)999-006, 2021.
[60] V. Sanh, A. Webson, C. Raffel, H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint 2021.
[61] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra, Grad-cam: Visual explanations from deep networks gradient- based localization. In Proceedings of the IEEE international conference on computer vision, pages 8-686,662220 2017.
[62] R. Sharma, J. Allen, O. Bakhshandeh, and N. Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2; Short Papers), pages 752-757 2018
13

---

[63] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, and C. Polls. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642, 2013.
[64] H. Song, M. Kim, and J.-G. Lee. Selfie: Refurbishing unclean samples for robust deep learning. In International conference on machine learning, pages 5907-5915. PMLR, 2019.
[65] J. Stallkamp, M. Schlipsing, J. Salmen, and C. lgel. The german traffic sign recognition benchmark: a multi-class classification competition. In The 201 international joint conference neural networks, pages 1453-1460. IEEE, 2011.
[66] G. Stoica, D. Bolya, Bjomer, T. Hearn, and J. Hoffmann Zipit! merging models from different tasks without training. arXiv preprint arXiv:2305.0300 2023.
[67] A. Suhr, S. Zhou, A. Zhang. 1. Zhang. H. Bai, and Y. Artzi. A corpus for reasoning about natural language grounded in photographs. arXiv preprint 1811 11881111111,11111111111111111111 2018.
[68] A. Tang, L. Shen, Y. Luo, H. Hu, B. Du, and D. Tao. FusionBench: A Comprehensive Benchmark of Deep Model Fusion, June 2024.
[69] L. van der Maaten and Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, 9(86)25579-265 2008.
[70] S. Vandenhende, S. Georgoulis, W. Van Gansbeke, M. Proesmans, D. Dai, and L. Van Gool. Multi- task learning for dense prediction tasks: A survey, IEEE transactions on pattern analysis and machine intelligence, 44(7):3614-363 2021.
[71] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, I.. Jones, A. N. Gomez, E.. Kaiser, and I. Polosukhin. Attention all you need, Advances in neural information processing systems, 30, 2017.
[72] R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: cansensus-basee image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566-4575, 2015.
[73] C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The caltech-uesd birds-200-22 200-2011 dataset. 2011.
[74] A. Wang, A. Singh, J. Michael, F. Hill, Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint / 1804.07464 2018.
[75] W. Wang, H. Bao, L. Dong, J. Bjorck, Z. Peng, Q. Liu, K. Aggarwal, O. K. Mohammed, S. Singhal, S. Som, and F. Wei. Image as foreign language: BEiT pretraining for vision and vision-language tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2023.
[76] A. Warstadt, A. Singh, and S. R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 7,625-641,200 2019.
[77] R. Wightman. Pytorch image models. 2019.
hitptoorrrc--mmammmmmmddeeee models
[78] A. Williams. N. Nangia, and S. R. Bowman. A broad challenge corpus for sentence understanding through inference. arXiv preprint arXiv: w7Xiv:104.00426. 2017.
[79] T. Wolf, L. Debut, V. Sanh, Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Fun- towicz, et al. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint 2019.
[80] M. Wortsman, G. IIharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Loppes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornhlith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference Machine Learning. pages 23965-23998. PMLR, 2022.
[81] H. Xiao, K. Rasul, and R. Vollgraf. mashion-mnist: a novel image dataset for benchmarking machinc learning algorithms. arXiv preprint arXiv: 2017.
[82] H. Xiao, F. Zhanng Z. Shen, K. Wu, and J. Zhang. Classification of weather phenomenon from images by using deep convolutional neural network. Earth and Space Science, $(5):22000EA001164, 2021. 2021.
14

---

[83] J. Xiao, J. Hays. K. A. Ehinger, A. Oliva, and A. Torralba. Sun database: Large-scale scene recognition from abbey 200. In 2010 IELE computer suciety conference on computer vision and pattern recognition. pages 3485-3492. IEEE, 2010.
[84] P. Yadav. D. Tam, L. Choshen, C. Raffel, and M. Bansal. Ties-merging: Resolving interference when merging models. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[85] E. Yang, Z. Wang, L. Shen, S. Liu, G. Guo, X. Wang, and D. Tao. Adamerging: Adaptive model merging multi-task learning. In The Twelfth International Conference on Learning Representations, 2023.
[86] P. Ye. T. He, S. Tang, B. Li, T. Chen, L. Bai, and W. Ouyang. Stimulative training++: Go beyond the performance limits of residual networks. arXiv preprint arXiv:2305.02507 2023.
[87] P. Ye, C. Huang, M. Shen, T. Chen, Y. Huang, Y. Zhang, and W. Ouyang. Merging vision transformers from different tasks domains. arXiv preprint arXiv:2312.16240. 2023.
[88] P. Ye, Li, Y. Li. T. Chen, Fan, and W. Ouyang- b-darts: Beta-decay regularization for differentiable architecture search. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10874-10883, June 2022.
[89] P. Ye. S. Tang. B. Li, T. Chen, W. Ouyang. Stimulative training of residual networks: social psychology perspective of loafing. Advances in Neural Information Processing Systems, 35:3596-3608, 2022.
[90] L. Yu, B. Yu, H. Yu, F. Huang, and Y. Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. arXiv preprint arXiv:2311.0883 2023.
[91] N. Yuval. Reading digits in natural images with unsupervised feature learning. In Proceedings of the NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011.
[92] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint arXiv: 1905.07830, 2019.
[93] B. Zhang, Yuan, B. Shi, T. Chen, Y. Li, and Y. Qiao. Uni3d: unified baseline for multi-dataset 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9253 9262, 2023.
[94] J. Zhang, J. Liu, J. He. al. Composing parameter-efficient modules with arithmetic operation. Advances in Neural Information Processing Systems, 36: 36:12589-12610, 2023.
[95] Y. Zhang and Q. Yang. A survey on multi-task learning. IEEF Transactions Knowledge and Data Engineering, 34(12):5586-566 5609, 2021.
15

---

Appendix for EMR-MERGING
A Algorithm flow of EMR-MERGING
We summarize the procedure of EMR- in Algorithm
Algorithm 1 MERGING Procedure
Input: Finetuned models WI.N,, pretrained model W pre
Output: Unified task vector Tunis task-specific masks MI..,, task-specific rescalers A for ( ii.,..... N do
D Create taak vectora.
= W W pre
end
Step
vector.
Juni sgn -=1 Tx)
Cuni = zeros d)
for L in ,... N do
for p in d do
if
p of then
P = max uni (Cantyabs abs (Tems)
end
end
end
Tuni
Yani ④ Cuni-
for t in ...g N do
b Stop 2: Generate task task-apecific masks.
for p inl, ..., do
M (T! O T > 0)
end
> Step 3: Generate tack taek-specific rescalers.
A
sum(abx(7.))
(M
end
B Theoretical analyses
In Section 3 we claimed that the task-specific modulators can lower the distance between the merged model and task-specific models. Here provide detailed theoretical analyses.
Our goal is to merge model weights W, Wi.. by minimizing the distance between the merged model Wani and each individual models W., i E [1..N] without using any dataset (X;, Yi]. where the distance can be calculated by:
Dis E/ I|4
W Wonkl 1
(6)
N
The premise merging is that all the models are fine-tuned from the same pre-trained model. Thus, Eq. 66ccan can be re- re-written:
Dis En || Tumill
N
where refers the task vector for Task i. Tuni is the merged task vector. We demonstrate the effectiveness of the task-specific modulators by step.
Analysis 1: Effectiveness of Masks. Suppose we apply a mask M, to the unified model Tuni to disable elements Tuni that have the opposite sign the corresponding elements in Tanis which can be written as:
16

---

0.3
AdMlrgingg*
025
0.6
Blex Mask
EMRR-Merggig
0.15
0.1
0.8
Siga Couilici
12 Dibane
Cusin: Simileity
Figure 7: Comparison of (a) sign conflicts, (b) L2 distance, and (c) cosine similarity of model weights obtained by different methods (including AdaMerging++ and each procedure of EMR (AM--MEGGING) and task-specific model weights. The detailed configuration is shown in Appendix F
Table 11: Multi-task performance when merging ViT-B/16 models on eight tasks.
Methods
SUN397
Cars
RESISC45
FurSAT
SVHN
GTSRB
MNIST
DTTD
Avg Acc
Task Arithmetic
61.1
65.9
74.0
76.2
88.0
73.9
98.4
53.0
73.8
Ties-Merging
84
69.1
72.5
80.5
84.0
85.0
71.5
98.1
54.9
77.0
AdaMerging
8S
70.2
80.7
81.6
94.8
91.6
95.8
98.5
66.2
84.9
AdaMerging
85
71.8
94.3
91.9
94.5
98.7
85.7
EMR-MERGING (Ours)
78.6
82.6
95.5
99.2
97.6
98.8
99.6
78.3
91.3
M, (Ts ① Tani > 0)
(8)
By applying the masks M, i [1..N], the distance becomes:
Dis Σ |
Mj Tumill
(9)
Furthermore, it can written as:
Dis EN | OT,
M, O Tumell?
Σ
ll (1 M,) O TT12
N
N
(10)
ΣN
M| @ (abs (Tx) abs (Tuni)) 12 ΣN
(1 - M.) o abs (TT) 12
N
N
where abs(-) returns the absolute value of each element in the input. For case of comparison, the distance without applying M, can be formulated as:
Dis EN M O (abs (T.) abs (Tunc)) I|? EN (1 (M) @ (abs (T.) + abs (Tunc)) I|a
N
N
(11)
Dis EN [1 M.) @ abs (
N
Thus, we demonstrate that D < Dis, indicating applying task-specific masks can reduce the distance between the merged model and individual models, thus showing effectiveness.
Analysis 2: Effectiveness of Rescalers. Suppose we apply a rescaler X; > 0 to the masked unified task vector M2 Tunis the distance becomes:
M. EN ||Te d Mi Tamill2
N
(12)
N |labs (7s) d abs (M, Tani) 1
17

---

Task
Ties Merging
ADA
Avenging
RegMaan
Arithmetic
Merging
Flect
Elect Mask EMR-MMrggigg
Individual
Cars
MNIST
EaroSAY
RESISCAS
SUN397
NHHS
CTSRB
DID
Figure 8: t-SNE visualization results of different merging methods.
Table 12: Multi-tase performance when merging ViT-B/32 models on 9 vision tasks (ImageNet- IK added).
Methods
SUN397
Cars
RESISC45
EumSAT
SVHN
GTSRB
MNIST
DTD
ImageNet- IK
Avg Ace
Individual
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
82.0
89.6
Weight Avenging
61.8
56.4
65.9
66.2
62.7
44.5
81.8
49.0
61.5
61.1
Task Arithmetic 30]
51.8
30.9
55.8
64.3
69.0
42.2
92.7
46.8
66.6
57.8
Ties-Merging [S4]
53.3
34.1
57.0
55.8
72.3
43.2
90.5
46.5
68.9
58.0
EMR-MERGING (Ours)
77.0
75,2
92.9
92.7
79.7
90.2
97.6
76.2
79,8
84.6
To minimize the distance in Eq. be calculated by:
12
we set the first derivative of Dis with respect to A; to 0, thus A;
sum((bs(T..))
A sum (amn(ab(MM))) M, O Tani))
(13)
which exactly matches our setting of A;. This indicates that our setting of rescalers A can minimize the distance between the merged model and individual models, which is: D.A Dis thus showing effectiveness.
It is also reflected in Fig. 7 that after Masking and Rescaling, the sign conflicts and L.2 distance between the merged model and task-specific models are reduced and the cosine similarity can improved.
18

---

Tisk
Ties Merging
ADA
Averaging
RegMican
Marging++
Blect Mask EMR-MMrrging
Curs
MINIST
EuroSAT
RESISCAS
SUNSA
GISRD
DTD
Figure 9; Grad-CAM visualization results of different merging methods.
C Baseline Methods
Individual Models refer to task-specific models before merging.
Traditional MTL uses datasets from all the tasks to train single model jointly.
Weight Averaging element-wisely averages all the model weights. Its effectiveness when applied fine-tuned model weights from the same pre-training has been verified 80 57 33
Fisher Merging 46 uses Fisher information matrices 23 calculate the importance of each parameter and weighted merges them based on their importance.
RegMean 33 weighted merges models based on closed-form solution to the merging problem. When merging K linear model weights W,, where (wr) W 1..K, the merging problem can be formulated as: min W
ΣK || X, - W v, where the merged model weights, and X; denotes the input of ith model. The closed-form solution to the problem is: W <E x?',) (Z X,X, W.). Inner-produc matrices need to be computed before merging.
Task Arithmetic 30 defines task vectors as the difference between finetuned model weights and the pre-trained model weights. Suppose model 0; is finetuned from pre-trained model 6pre. the task vector is 8, 6pre- When merging f1..k, k, the merged model is + Opre. where is the merging coefficient.
Ties- Mies-Merging 844 (Trim, Elect Sign & Merge) believes that the conflicts among the task vectors severely effect the merged model's performance. Ties-Merging solves this problem by climinating redundant parameters and resolving symbol conflicts.
19

---

Table 13: Performance of RegMean and Task Arithmetic when pre-processeed using DARE 90
Single-Sentence Tasks CoL.A
Similarity and Puraphrase Tusks
Inference Tasks
Methods
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTE
Individual
0.6018
0.9404
0.8922
0.9063
0.9141
0.8720
0,9271
0.7906
EMR-MERGING (Ours)
0.3996
0.9335
0.8627
0.8277
0.8972
0.8545
0.8957
0.7437
RegMcan 33
0.3667
0.906
0.7574
0.6268
0.8355
0.7002
0.8235
0.5848
w/ DARE (drop 10%%
0,5046
0,5298
0,3603
0.1533
0,4955
0.3245
0.4924
0.4477
w/ DARE (drop 30%)
0,4535
0,6135
0.3186
0.0471
0,4219
0.3325
0.505
0,5126
w/ DARE (drop 50%)
0.2758
0.5138
0.3211
-0.0965
0.3685
0.3338
0.508
0,5235
w/ DARE (drop 70%)
0
0,4908
0.3162
0.0021
0.3682
0.3184
0,5056
0,4838
w/ DARE (drop 90%)
0.4908
0.3162
-0.0776
0.3682
0.3187
0.5158
0.4910
Task Arithmetic (30
0. 0.1878
0.8589
0.7990
0.7403
0.8378
0.5908
0.6967
0.6209
w/ DARE (drop 10%)
0.2424
0.8509
0,7966
0.7234
0.8382
0.5869
0.7368
0.6101
w/ DARE (drop 30%)
0.3040
0.8452
0.7941
0.6311
0.8333
0,5515
0.786
0.6137
w/ DARE (drop 50%)
0.2451
0.8188
0.7990
0.4262
0.8099
0.4591
0.7269
0.6029
w/ DARE (drop 70%)
0
0.7225
0.6373
0.1353
0.7321
0.3453
0.6495
0,5162
w/ DARE (drop 90%%
0
0.4908
0,3162
0.1422
0,3682
0.3185
0,5114
0.4729
Ties- Mies-Mggging [44
0.2048
0.8440
0.8113
0.5819
0.8570
0.6465
0.7481
0.4296
DARE (drop 30%)
0.5103
0,3382
-0.0024
0.3961
0.3238
0.5277
0.4838
DARE (drop 50%)
0.0464
0.6021
0.5343
0.0192
0.6846
0,3410
0.5841
0,4982
DARE (drop 70%)
0.1342
0.7833
0.7672
0.1667
0,8180
0,4172
0.691
0,5271
w/ DARE (drop 90%)
0.2618
0.8383
0.8039
0.6082
0,8336
0,5551
0.7692
0.5235
AdaMerging [85 uses unsupervised method to learn the merging coefficients for each task vector (Task-wise AdaMerging) or each layer (Layer-wise AdaMerging). AdaMerg- ing++ is realized by adopting Ties-Merging [84] before learning the merging coefficients.
DARE [90] (Drop and Rescale) validates the extremely redundant properties of language models. As pre-processing technique, DARE randomly drops most (90% or even 99%) delta parameters (task vectors) before merging potentially mitigate the interference of parameters among models.
D More experimental results
D.1 Merging ViT-B/16 models on tasks
We follow the settings in Section 4.1.J and merge ViT-B/16 models. Tab. II shows the accuracy of merging ViT-B/16 models eight vision tasks. The proposed EMR-MERGING brings about 5.6% performance improvement compared to Adamerging++ 85 further demonstrating the effectiveness of EERR-NG
D.2 Merging ViT-B/32 models on tasks (ImageNet-1K added)
To further explore the performance of MER-MEEGGNG, we follow the settings in Section 4.1.1 and add one more task, ImageNet-IK [18]. We merge models on these nine tasks using different merging methods. The results are shown in Tab. 12 and EMR-Merging shows a much more significant improvement compared to existing merging methods (up to 20%).
D.3 DARE's experimental results and causes
DARE's experimental results when combined with RegMean and Task Arithmetic are shown in Tab. 13 It can be seen that when applied to merge eight models, DARE works on a few tasks under low dropping rate settings but generally fails. We attribute failure to random dropping strategy's unapplicability to merging multiple models. Under the setting of merging two or three models, randomly dropping most parameters task vectors can significantly reduce interference but conflicts are lot more difficult to avoid when merging multiple models.
20

---

Table 14: Performance of Task Arithmetic (3), Ties-Merging 844 Ties- Ties-Merging 84 and RegMean [33 under different merging coefficient. P is the drop rate for DARE. is the non-diagonal multiplier for RegMean.
w/ DARE 90
settings. À for task vector- vector-based methods the
Single-Sentence Tasks
Similarity and Parsphrase Tasks
Inference Tasks
Methods
CoLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTE
Individual
0.6018
0.9404
0.8922
0.9063
0.9141
0.872
0.9271
0.7906
EMR-MERGING (Ours)
0.3996
0.9335
0.8627
0.8277
0.8972
0.8545
0.8957
0.7437
Task Arithmetic
À
0.0464
0.742
0.6691
0.2344
0.771
0.3567
0.6919
0.556
À = 0.3
0.1878
0.8589
0.799
0.7403
0.8378
0.5908
0.6967
0.6209
À = 0.5
-0.0089
0.7913
0.7794
0.5686
0.8271
0,4631
0.5387
0.4693
À 0.7
-0.0079
0.6525
0.7819
0.1292
0.8146
0.3949
0.5279
0.5054
0.9
-0.0207
0.7202
0.4167
-0.1283
0.8012
0.2913
0.5294
0.5162
X 1.0
0,5619
0.3554
-0.2496
0.7939
0.259
0.5338
0.5162
Ties-Merging
0,4908
0.3162
0.0214
0.3682
0.3186
0.5105
0.4729
À = 0.3
0.5631
0.5049
-0.0074
0.4696
0.35
0.5649
0.4621
À = 0.5
0.2232
0.7592
0.7696
0.1149
0.827
0.4486
0.6939
0.4368
À 0.7
0.2507
0.8291
0.7917
0.3774
0.8488
0.5858
0.7507
0.4188
À 0.9
0.2048
0.844
0.8113
0.5819
0.857
0. 0.6465
0.7481
0.4296
X = 1.0
0.1712
0.8406
0.799
0.6444
0.859
0.6409
0.7069
0.426
Ties-Merging w/ DARE
À 0.2, P= 0.3
0.4920
0.3162
0.0053
0.3682
0.3186
0.5131
0.4477
À = 0.2, == 0.5
0.0043
0.3162
0.0036
0.3690
0.3202
0.5226
0.4946
À = 0.2, p= 0.7
0.0464
0.6388
0.5735
0.0301
0.0047
0.3383
0.5984
0.5090
À 0.2, P- 0.9
0.2402
0.8165
0.7843
0.2696
0.8112
0.4384
0.7223
0.5415
= 0.3, = 0.3
0.5103
0.3382
-0.0024
0.3961
0.3238
0.5277
0.4838
À = 0.3. P= 0.5
0.0464
0.6021
0,5343
0.0192
0.6846
0.3410
0.5841
0.4982
À 0.3, P 0.7
0.1342
0.7833
0.7672
0.1667
0.8180
0.4172
0.691
0.5271
À 0.3,
0,2618
0.8383
0.8039
0.6082
0.8336
0.5551
0.7692
0.5235
À = 0.4, ===0=0 0.3
0.0656
0.6216
0,5588
0.0192
0.7301
0,3461
0,5891
0.5162
À = 0.4, P = 0.5
0.1172
0.7374
0.7451
0.1045
0.8157
0.3913
0.6667
0.5126
À 0.4, P 0.7
0.2440
0,8234
0.7843
0.3955
0.8371
0.5496
0.7216
0.4838
0.4, 0.9
0.1380
0.8440
0.8064
0.7044
0.8365
0.5835
0.6529
0.5054
Reg Mean
0.7
0.3005
0,9037
0.7525
0.6349
0.8322
0.6794
0.8157
0.5632
a
0.3346
0.9014
0.7549
0.6375
0.8339
0.6841
0.8173
0.5704
a 0.9
0.3445
0.9048
0.7525
0.6362
0.8361
0.6918
0.821
0.5632
1.0
0.3667
0.906
0.7574
0.6268
0.8355
0.7002
0.8235
0.5848
D.4 Results under different hyper-paramerter settings
presented the best performance of Ties- Task Arithmetic, and RegMean among multiple hyper-yppaae perepparaeeeer settings. Here we present more experimental results of Ties-Merging. Task Arithmetic, and RegMean under different hyper- parameter settings in Tab. 14
In Section 4.2.1
D.5 Detailed information for merging different number of models
we showed partial results merging different number of VIT-B/32 models by Fig. Here provide quantified and task-specific performance results in Tab. 15
In Section 4.4
21

---

Table 15: Merging different number of VIT-B/32 models.
Methods
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Avg Ace
Individual
Tasks
75.3
77.7
76,5
Tasks
75.3
77.7
96.1
83.0
4 Tasks
75.3
77.7
96.1
99.7
87.2
Tasks
75.3
77.7
96.1
99.7
97.5
89.3
Tasks
75.3
77.7
96.1
99.7
97.5
98.7
90.8
Tasks
75.3
77.7
96.1
99.7
97.5
98.7
99.7
92.1
Tasks
75.3
77.7
96.1
99.7
97.5
98.7
99.7
79.4
90.5
Ties-Merging
Tasks
69.2
68.2
68.7
Tasks
69.2
68.0
78.9
72.0
Tasks
68.9
67.9
86.0
75.5
Tasks
68.6
67.1
83.5
73.0
Tasks
68.0
77.9
80.1
74,4
69.9
72.8
Tasks
66.6
65.7
75.7
76.7
81.0
69.2
96.4
75.9
Tasks
64.8
62.9
74.3
78.9
83.1
71.4
97.6
56.2
72.4
(Ours)
Tasks
78.9
76.1
77.5
Tasks
77.9
75.2
95.3
82.8
Tasks
77.4
74.9
94.8
99.7
86.7
Tasks
77.2
74.2
94.7
99.7
97.1
88.6
Tasks
76.4
73.4
94.2
99.7
97.0
98.5
89.9
Tasks
75.8
73.3
93.6
99.6
96.9
98.2
99.6
91.0
Tasks
75.2
72.8
93.5
99.5
96.9
98.1
99.6
74.4
88.7
Table 16: Sparsity (ratio of non-zero items) of the masks and the values of the rescalers when merging ViTs on 8 vision tasks and RoBERTa models on 8 language tasks.
Sparsity
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
VIT-B/32
0.7194 0.6832
0.7121
0.7106 0.6734
0.6994 0.6579
0.7195 0.6748
0.7062 0.6444
0.7132
0.7058
VIT-L/14
0.6699
0.6614
0.6620
Rescalers
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
VIT-B/32
0.7489
0.7635
0.7489 0.7537
0.7476
0.7962
0.7652 0.7313
0.7981 0.7763
0.7624
VIT-I/14
0.7656
0.7652
0.7384
0.7874
0.7638
Sparsity
CoLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTE
RoBERTa
0.6264
0.6547
0.6498
0.6150
0.7620
0.7739
0.6243
0.5979
Rescalers
CoLA
SST2
MRPC
STSB
QQP
MNLI
QNLI
RTE
RoBERTa
0.2458
0.4698
0.5033
0.2078
0.8891
0.8987
0.4683
0.1466
D.6 Sparsity of masks and values of rescalers.
We show the sparsity of the masks and the values of the rescalers when merging eight ViTs and eight RoBERTa models Tab. [6
E More visualization results
In Section 3 showed some visualization results using t-SNE 69 and Grad-CAM 61 Here we provide more visualization results of both existing merging methods and EMMR-ERGING. 1-SNE and Grad-CAM visualization results are shown in Fig and Fig. respectively.
22

---

Configuration of Fig. and Fig. 7
hope to compare the sign conflicts, L2 distance, and cosine similarity of the merged model weights and individual model weights. To calculate the sign conflicts, we element-wised compare the merged model weights each individual model weights and record the ratio of the elements whose signs conflict. We report the average value of the sign conflicts between merged model and each individual model. calculate L2 distance or cosine similarity, we first flatten the merged model weights and each individual model weights as 1-dimension vectors. Then we calculate the 1.2 distance or cosine similarity between the merged model and each individual model and report the average value.
In Fig. and Fig.
G Limitations and future works
Despite the convincing results, the proposed method suffers from several limitations. On the one hand, compared existing methods, EMR-MERGING requires little additional memory store the light-weight task-specific modulators. On the other hand, as a common limitation task vector-basee methods, EMR-MERGING cannot be generalized models trained from-scratch because the task vector is based on the pretrain-finetune paradigm.
Further improving the performance of the merged model and generalizing model merging to models trained from-scratch or even models with different structures are significant directions for future work. Additionally, combining model merging with low bit-width quantization has broad application prospects and is also potential future work.
23

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
The main claims made in both abstract and Section accurately reflect the paper's contributions and scope.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please check Appendix G
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]

---

The full set of assumptions and a complete (and correct) proof are detailed in Appendix [
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We disclose the information in Section and we provide the code and data for the convenience of reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
25

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes)
Justification:
We provide the code for reproduction. Please check Abstract.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipps.c/ppulli//guu /ttoe//uuudeeooeoouuummiiiiiii ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We specify them in Section 4 and in our released code.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Error bars are not reported.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
26

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [No]
Justification: The required computer resources are decided by the structure of the models to merged.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification:
The research is conducted with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification:
Not applicable to societal impacts.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
27

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We cite the original papers or websites that produced the code package or dataset.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
28

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
29


Reviews:
[{"review_id": "UUy5CK3xI2", "reviewer": "Reviewer_Lqs2", "summary": "This paper aims to improve the performance of model merging in the multi-task learning domain. By electing a unified model, identifying a mask and a rescaling factor for each task, the proposed method EMR-Merging is able to significantly improve the merging performance over the previous state-of-the-art, such as Adamerging, Ties-Merging, Task Arithmetic, etc. Besides the CV model merging, EMR-Merging also effectively merges the large language models, approaching the individual fine-tuned performance.", "questions": "Please see the above weaknesses.", "limitations": "Provided.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. I like the idea that we need to elect a unified model first, and then identify different masks for different tasks. Directly merging models can lead to function conflicts. \n\n2. The experiments are extensive, including VIT model, language models, and multi-modal models, with considerable good performance.\n\n3. The ablation study is also provided.", "weaknesses": "1. I am curious why the proposed way works for the unified model. One specific case is that, if the the average of the task vector is close to zero, but not exactly equal to zero, e.g., 0.0001, which will give us a positive sign. If the magnitude of this weight is large, a positive sign multiplied by this large magnitude weight can lead to a large error. Can the authors elaborate more the intuition behind the unified model, mask, and rescaler? \n\n2. Can you explain the process of obtaining the mask? I could not find the definition of $\\tau_F$ after reading the paper in detail.\n\n3. What is the configuration of Figure 4? More details are needed here. \n\n4. How to interpret the t-SNE visualization?\n\n5. The distance between figures is over-reduced. It can be confusing and misleading. For instance, figure 5 and figure 4.\n\n6. How expensive to extend EMR-Merging to LLMs, such as LLaMa3?\n\n7. How does the proposed method perform compared to SOTA https://arxiv.org/pdf/2402.00433?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work and admitting the value of our contribution.\n\n---\n## Weakness 1: why the proposed way works, what if the average of the task vector is close to zero, and the intuition of the method.\n\n**Ans**: When we elect the sign vector, we first element-wisely add the task vectors and obtain the elected sign, i.e., $\\gamma_{uni} = sgn(\\sum_{t=1}^{N} \\tau_t)$ when merging $N$ task vectors. Then we element-wisely choose the maximum absolute value of each parameter with the sign consistent with $\\gamma_{uni}$ and obtain the elected unified task vector $\\tau_{uni}$ . It contains significant elements from all the task vectors.\n\nNext, we calculate the masks by: $M_i= \\left( \\tau_i \\odot \\tau_{uni} > 0 \\right)$ for task $i$.\n\nFinally, we calculate rescalers by $\\lambda_i = \\frac{sum(abs(\\tau_{i}))}{sum(abs(M_i \\odot \\tau_{uni}))}$. We apply absolute values in both the numerator and denominator, avoiding the situation that the average of a task vector is close to zero. Additionally, the $\\tau_{uni}$ contains the maximum values of the elected direction. This will also prevent the denominator from approaching zero. We observe that the rescalers are normally not larger than 1. We provide the values of the rescalers under various settings in Table R8 of the Rebuttal PDF. \n\nWhen designing EMR-Merging, we hope to reserve the shared and significant features from all the task vectors in the unified task vector $\\tau_{uni}$. We reserve the shared features among task vectors by electing the sign vector and reserve the significant features by electing the elements of the largest absolute values with the sign consistent with the sign vector. Ties-Merging shows that sign conflicts cause significant performance degradation. DropOut [R3] and DARE [R1] shows that simply dropping without magnitude alignment can also effect the performance. Inspired by this, we use masks to avoid sign conflicts between the unified task vector and task-specific ones and use rescalers to align the magnitude. \n\n---\n## Weakness 2: the process of obtaining the masks; the definition of $\\tau_{F}$.\n\n**Ans**:  The mask for task $i$ is calculated by: $M_i= \\left( \\tau_i \\odot \\tau_{uni} > 0 \\right)$. In the method section, the $\\tau_{uni}$ was mistakenly written as $\\tau_{F}$ when introducing the process of obtaining the masks. Sorry for the mistake.  However, in the theoretical analysis, we use $\\tau_{F}$ to refer to collection of the merged task vector by existing methods (e.g., Ties-Merging) or the elected task vector $\\tau_{uni}$. The definition of $\\tau_{F}$ is to demonstrate the effectiveness of the masks and rescalers regardless of the electing process. This can explain why other methods can be combined with Mask & Rescale and obtain performance improvement in Table 8.\n\nWe find that this may lead to misunderstanding and we will uniformly replace $\\tau_{F}$ by $\\tau_{uni}$ in the revision.\n\n---\n## Weakness 3: the config of Fig. 4.\n\n**Ans**: In Fig. 4, we hope to compare the sign conflicts, L2 distance, and cosine similarity of the merged model weights and individual model weights. \nTo calculate the sign conflicts, we element-wisely compare the merged model weights to each individual model weights and record the ratio of the elements whose signs conflit. We report the average value of the sign conflits between the merged model and each individual model.\nTo calculate the L2 distance (cosine similarity), we first flatten the merged model weights and each individual model weights as 1-dimension vectors. Then we calculate the L2 distance (cosine similarity) between the merged model and each individual model and report the average value.\n\nWe will add the configuration in the revision.\n\n---\n## Weakness 4: interpret the t-SNE.\n\n**Ans**: t-SNE is introduced to visualize high-dimensional data. In our paper, we visualize the outputs of the last transformer block (before the classification head). The results can represent the feature extracting capabilities by the separation among the classes. In Fig. 5, EMR-Merging shows the clearest separation among all the merging methods and is close to individual models. This is correspondent with the experimental results.\n\n---\n## Weakness 5: the small distance between figures.\n\n**Ans**: Thanks for your advice and we will increase the distance between figures in the revision.\n\n---\n## Weakness 6: how expensive to extend to LLMs?\n\n**Ans**: The proposed EMR-Merging can easily be adapted to various settings including merging LLMs.\n\n+ In our paper, we show the experimental results when applying EMR-Merging to T0-3B models finetuned by $(IA)^3$, demonstrating the applicability of EMR-Merging to both LLMs and PEFT models. \n\n+ To further demonstrate the high performance and broad applicability of EMR-Merging, we apply EMR-Merging to a new benchmark, which merges GPT-2 models (fully-finetuned) on 7 NLP tasks and it achieves the SOTA performance. The performance improvement is up to more than 10%. The results are in General Response.\n\n---\n## Weakness 7: comparison to WEMoE [R2]\n\n**Ans**: We did not compare EMR-Merging to WEMoE [R2] in our paper because WEMoE had not been officially published by ICML when we submitted the paper.\n\nWe compare the parameter numbers and prerequisites of WEMoE and EMR-Merging in Table R1 of the Rebuttal PDF. The proposed EMR-Merging needs no data, training, or tuning while WEMoE requires unlabeled test data to train partial parameters from the framework. Meanwhile, EMR-Merging achieves promising performance. The results of merging ViTs on the eight vision tasks are shown in Table R2 of the Rebuttal PDF. EMR-Merging achieves the SOTA performance on the benchmark of merging ViT-L/14 models. Moreover, EMR-Merging can be easily extended to other settings.\n\n---\n## References\n\n[R1] Language models are super mario\n\n[R2] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts\n\n[R3] Dropout: a simple way to prevent neural networks from overfitting", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iye7cGpFT0", "reviewer": "Reviewer_S3RQ", "summary": "Model merging directly fuses multiple independently trained models at the weight level to obtain a single comprehensive model, which is a current research hotspot. This paper proposes a new model merging method EMR-MERGING to reduce the performance gap between the merged model and the independent models.", "questions": "- Is it possible to select only the maximum value in the Electing operation? Is it also possible to select the average value?\n- In Figure 4(c), regarding Cos similarity, TV > Ties-Merging > AdaMerging, but in terms of accuracy, TV < Ties-Merging < AdaMerging. Figure 4(b) is similar. These phenomena seem to be contradictory. How does the author explain this?", "limitations": "The authors discuss some limitations of the method: for example, it consumes additional memory compared to other model merging methods, and all rely on the same pre-trained model. However, as in Weaknesses, this method may lose the ability to parallelize inference on multiple tasks, and it is unclear how much benefit it has compared to sparse independent models.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed EMR-MERGING method does not require additional training, data, and tuning, which enhances the applicable scenarios of the method.\n- This paper conducts extensive experiments, including visual models, NLP models, and multi-modal models. In particular, the visual model is extended to 30 tasks.\n- This paper has a clear structure and the proposed methods are easy to implement.", "weaknesses": "- **Loss of parallel ability**: Existing model merging methods obtain a unified model for all tasks, so samples from all tasks can be included in one inference (i.e., a batch of data). However, EMR-MERGING needs to configure an independent model for each task during inference, thus losing the ability of parallel inference. In real scenarios, a batch of data may come from different tasks rather than a single task.\n- **Important baseline missing**: It is unclear what the benefits and advantages of EMR-MERGING are compared to \"sparse independent models\". Ties-Merging shows that removing 90% of the neurons in each independent model has almost no change in performance. So why not just use the sparse version of the task vector instead of using EMR-MERGING?\n- **Symbol or spelling error**: \n  - The symbols are not uniform, and the uniform vector is sometimes $T_F$ and sometimes $T_{uni}$.\n  - Table 11: “huper-parameter settings” -> “hyper-parameter settings”\n  - Sec 4.2.2: “using 1eleven datasets” -> “using eleven datasets”\n\n- Finally, there is a lack of datasets and checkpoints for CV (30 tasks), NLP, and multimodal in the anonymous link. If the authors can release them, it will help the community research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work and kind comments.\n\n---\n## Weakness 1: parallel ability.\n**Ans**: 1) Most multi-task model merging methods cannot handle the situation where multi-task samples are included in one inference because only one classification head can be applied during one inference. All the existing merging methods do not merge classification heads [R1-R3] because the classification heads may not share the same architecture. The usual practice of merging methods is applying the proper classification head manually to the merged model before testing on a specific task.  2) We follow this setting and additionally apply a mask and a rescaler to the unified model before testing on a specific task. Once the problem of manual classification head selection is handled, the modulators can also be selected automatically, thus realizing parallel inference. 3) A potential solution is using a gating network to sample-wisely determine which task an input comes from before each inference [R4].\n\n---\n## Weakness 2: comparison to multiple Sparse Models\n\n**Ans**: Here we show a detailed comparison.\n\n+ The performance of EMR-Merging is more promising.\n\nWe compare the performance of Sparse Models and EMR-Merging on merging (2-8) ViT-B/32 models and 30 ViT-B/16 models in Table R3 of the Rebuttal PDF. The $K$ for the Sparse Models refers to keeping the top $K\\%$ elements following TIES-Merging [R2]. It can be seen that EMR-Merging performs better than Sparse Models when merging vision models. Additionally, we compare the performance of Sparse Models and EMR-Merging on merging BEiT3 models in Table R4 of the Rebuttal PDF. The performance of Sparse Models severely decreases on some tasks while EMR-Merging shows promising performance on all tasks.\n\n+ EMR-Merging leads to smaller storage costs when merging multiple models.\n\nThe comparison of parameter numbers is shown in Figure R1 of the Rebuttal PDF.\nEMR-Merging shows both fewer parameter numbers and better performance when compared to Sparse Models (K=10). \nThough Sparse Models (K=5) can realize fewer parameters, their performance is severely reduced.\nIt should be noted that the number of parameters in EMR-Merging increases more slowly with the number of tasks due to the lightweight task-specific modules.\nTherefore, when the number of tasks further increases, the parameter numbers of EMR-Merging tend to be fewer than Sparse Models (K=5).\n\n+ The elected unified task vector of EMR-Merging can also be sparse.\n\nDuring EMR-Merging, after we elect $\\tau_{uni}$ and before calculating the rescalers, we can make $\\tau_{uni}$ sparse by keeping the top $10\\%$ ones. This can further reduce the storage cost of EMR-Merging. We identify this method by EMR-Merging (Sparse).\nThe performance and parameter numbers of EMR-Merging (Sparse) can be found in Table R2 and Figure R1 of the Rebuttal PDF.\nIt can be seen that EMR-Merging (Sparse) sharply reduces the parameter numbers while maintaining high performance, which is still better than Sparse Models while showing the fewest parameter numbers.\n\n---\n## Weakness 3: typos.\n\n**Ans**: In the method section, the $\\tau_{uni}$ was mistakenly written as $\\tau_{F}$. Sorry for that. However, in Theoretical Analysis, we use $\\tau_{F}$ to refer to the collection of the merged task vector by existing merging methods and the elected task vector $\\tau_{uni}$. This is to demonstrate that regardless of which method is applied for merging (or electing), masking and rescaling can lower the distance between the merged model and individual models, thus explaining why other methods can be combined with Mask & Rescale and get improved in Tab. 8. \n\nTo avoid misunderstanding, we will uniformly use $\\tau_{uni}$ in the revision. We will also fix the other typos and thanks for pointing them out.\n\n---\n## Weakness 4: datasets and checkpoints.\n\n**Ans**: We have released the checkpoints and datasets in the anonymous link of the manuscript. Hopefully, the released checkpoints and newly established benchmarks can help community research.\n\n---\n## Question 1:  Is the maximum or the average strategy possible?\n\n**Ans**: Additional ablation studies on the electing process are shown in Table R9 and R10 of the Rebuttal PDF. Applying the maximum strategy may result in better performance on partial tasks including vision tasks and partial multi-modal tasks and worse performance on the rest tasks. In contrast, the average strategy results in worse performance on most tasks. \n\nNote that the main contribution of EMR-Merging is the idea of decoupling model merging into unified parts and task-specific parts. The unified vector can be obtained by not only electing but also existing merging methods, which is shown in Tab. 8.\nEMR-Merging shows tuning-free, data-free, and plug-and-play properties. The effectiveness of EMR-Merging is verified through theoretical and empirical analysis.\n\n---\n## Question 2:  the contradictory phenomena.\n\n**Ans**: 1) The proposed EMR-Merging first considered using masks and rescalers to align the signs and magnitudes. Therefore, the sign conflicts can be avoided and the L2 distances can be maximally reduced, as we illustrated in Theoretical Analysis. This explains why EMR-Merging shows the lowest sign conflicts, L2 distance, and the highest cos-sim in Fig. 4. 2) In Fig. 7, we show the sign conflicts, L2 distance, and cos-sim of each step of EMR-Merging. Combined with Tab. 9, it can be seen that during the 3 steps of EMR-Merging, we reduce sign conflicts and L2 distance and increase cos-sim, and the performance is correspondingly increased. 3) Existing methods may not focus on reducing sign conflicts and L2 distances, thus their performance may not correspond with sign conflicts and L2 distances.\n\n---\n## References\n[R1] Editing models with task arithmetic\n\n[R2] Ties-merging: Resolving interference when merging models\n\n[R3] Adamerging: Adaptive model merging for multi-task learning\n\n[R4] Merging Vision Transformers from Different Tasks and Domains", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QYK37RdCBB", "reviewer": "Reviewer_Qs8J", "summary": "The authors identify two issues in current model merging methods: significant performance degradation from the multi-task counterpart and requirement of additional data and training. To tackle those issues, they propose EMR merging, which first creates a unified task vector, then selects masks based on each task-specific model’s correspondence with the unified task vector, finally rescales it. EMR merging shows strong empirical performance among several benchmarks.", "questions": "1. What is the sparsity of the masks? I know that it is kind of task/model dependent, but it would be nice to show those results.\n2. Do EMR still need the hyperparameter tuning on the scaling factors? Or the one from rescalder is final? Because in DARE, they tune the scaling factors even after they compute the rescaling factors.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well written and the algorithm is coupled with adequate amount of analysis. \n2. The proposed algorithm is computationally lightweight, yet obtains significant performance improvement.", "weaknesses": "1. The novelty is limited. The three steps are essentially a combination of TIES and DARE, where only the masking step is slightly different from TIES. Due to this high similarity, it is unclear why the proposed method performs so much better than TIES and DARE. It would be better to provide a more disentangled analysis for each of the three steps through ablation studies, so that the readers can have a better understanding where exactly the performance improvement comes from.\n2. Several related works are missing [1,2] (both on ArXiv in Feb 2024). It is important to do a thorough comparison, especially when the paper basically claims SoTA results.\n3. Not all methods are compared on all benchmarks. For instance, AdaMerging is not compared with on NLP benchmarks. Although the original paper of AdaMerging does not provide results in NLP, I do not see any algorithmic obstacle that prevents it from being implemented in the NLP setting.\n\n[1] Representation Surgery for Multi-Task Model Merging. Yang et al. ICML 2024.\n\n[2] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. Tang et al. ICML 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work and helpful comments.\n\n---\n## Weakness 1: limited novelty. The three steps are a combination of TIES and DARE.\n\n**Ans**: The proposed method is totally different from DARE+TIES-Merging.\n\n+ The motivation for EMR-Merging is different from existing methods.\n  \nWe first decouple model merging into unified parts and task-specific parts. Existing model merging methods can be formulated as: $W_M = \\mathcal{M}\\left(\\left[ W_1..W_N \\right]\\right)$, while our EMR-Merging can be formulated as: $W_{uni}, \\left[ E_1..E_N \\right] = \\mathcal{M'}\\left(\\left[ W_1..W_N \\right]\\right)$.\nWe elect the unified task vector to reserve the most significant elements and use the masks and rescalers to align the signs and the magnitudes of the merged model, thus bringing about significant improvements. As Reviewer Lqs2 noted, the idea of electing a unified model and identifying different modulators for different tasks is interesting because directly merging models can lead to function conflicts. Differently, TIES and DARE are both proposed to reduce interference but a single model may not optimally simulate the task-specific model weights on all the tasks.\n\n+ The process of EMR-Merging is different from TIES and DARE.\n\nTIES and DARE drop partial elements based on magnitude or randomly. The goal of dropping is to reduce interference while our EMR-Merging can avoid sign conflicts by pre-computed masks. DARE's rescaling is to compensate for the dropping process while ours is to align each task-specific model. Additionally, the resclaers for DARE are normally larger than 1 while the rescalers for EMR-Merging are normally smaller than 1 as shown in Table R8 of the Rebuttal PDF. This reflects the difference in the functions of the rescalers of EMR-Merging and DARE.\n\n+ EMR-Merging shows tuning-free and plug-and-play properties.\n\nTIES+DARE introduces multiple hyper-parameters while EMR-Merging requires no hyper-parameter tuning, making it highly practical for real-world applications (Reviewer b5vt). \nEMR-Merging also shows plug-and-play capabilities and the performance of EMR-Merging has been verified under vision, NLP (including PEFT and FFT), and multi-modal settings.\n\n+ Experimental comparisons.\n  \nWe additionally compare the performance of EMR-Merging and TIES+DARE in Table R11 of the Rebuttal PDF. It can be seen that the performance TIES+DARE is sensitive to hyper-parameters and our tuning-free EMR-Merging achieves better performance.\n\n+ The reason for the improvement.\n  \nAs to the reason for the performance improvement, we believe that it is because EMR-Merging uses masks to avoid sign conflicts and rescalers to lower the L2 distance between the merged model and task-specific ones. Please check Appendix B for more information. Additionally, we have conducted ablation studies on all three steps of EMR-Merging in Tab. 8 and 9 of our paper. The results show that: 1) exiting merging methods can obtain performance improvement through Mask and Rescale. 2) Both masking and rescaling can boost the performance of the elected vector.\n\n---\n## Weakness 2: comparison to the WEMoE [R3] and Representation Surgery [R4].\n\n**Ans**: It should be noted that we did not compare EMR-Merging to WEMoE [R3] and Representation Surgery [R4] in our paper because they had not been officially published by ICML when we submitted the paper.\n\nWe compare the parameter numbers and prerequisites of WEMoE, Representation Surgery, and EMR-Merging in Table R1 of the Rebuttal PDF. The proposed EMR-Merging needs no data, training, or tuning while WEMoE and Surgery require unlabeled test data to train partial parameters in their frameworks. Meanwhile, EMR-Merging achieves promising performance. The results of merging ViTs on the eight vision tasks are shown in Table R2 of the Rebuttal PDF. EMR-Merging achieves the SOTA performance on the benchmark of merging ViT-L/14 models. Moreover, EMR-Merging can be easily extended to other settings including NLP and multi-modal.\n\n---\n## Weakness 3: not all methods are compared on all benchmarks.\n\n**Ans**: AdaMerging needs to train the merging coefficients. Shifting it to NLP settings requires re-building the framework and re-training. Since AdaMerging is only implemented in vision settings officially and its implementation in other settings is not yet realized by some newly-released toolkits and benchmarks including MergeKit [R1] and FusionBench [R2], we only compare the performance of AdaMerging under vision settings. We argue that the rebuttal time is very limited and re-building and re-training all the methods including AdaMerging under all the benchmarks is beyond the scope of this paper.\n\nHowever, we are glad to release the benchmarks, checkpoints, and the code of EMR-Merging under all the experimental settings. We have released all the checkpoints and datasets used in our experiments in the anonymous github repository of the manuscript. We have released the code under vision settings and we will release the code for all the other experimental settings soon.\n\n---\n## Question 1: the sparsity of the masks.\n\n**Ans**: We show some results of the sparsity of the masks under different settings in Table R5, R6, and R7 of the Rebuttal PDF.\n\n---\n## Question 2: does EMR-Merging still need the hyperparameter tuning on the scaling factors?\n\n**Ans**: We promise that EMR-Merging requires **no tuning on any hyper-parameter under any experimental setting**. Actually, EMR-Merging has no hyper-parameters. The unified task vector and the task-specific modulators are all calculated using model weights. We have released the anonymous code under vision settings and we welcome you to reproduce our experiments.\n\n---\n## References\n\n[R1] Arcee's MergeKit: A Toolkit for Merging Large Language Models.\n\n[R2] FusionBench: A Comprehensive Benchmark of Deep Model Fusion.\n\n[R3] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts.\n\n[R4] Representation Surgery for Multi-Task Model Merging.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aVHAsaHYDg", "reviewer": "Reviewer_b5vt", "summary": "The paper presents a method called EMR-MERGING (ELECT, MASK & RESCALE-MERGING) for merging models finetuned on different tasks into a single model with multi-task capabilities without the need for additional tuning or training. This method addresses the limitations of existing model merging techniques, which often suffer from performance degradation or require additional data and tuning.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- **Theoretical and Empirical Analysis:** The paper provides a solid theoretical foundation for the effectiveness of the proposed method, including detailed proofs and empirical analyses. This adds robustness to the claims made by the authors.\n- **Tuning-Free:** One of the major advantages of this method is that it does not require any additional data, tuning, or training, making it highly practical for real-world applications where data availability is limited or privacy concerns restrict access.", "weaknesses": "- **Additional Computational Overhead:** The proposed method introduces extra computational overhead during inference due to the use of masks and rescalers. This aspect should be discussed in the paper, particularly regarding its impact on the overall efficiency and performance in real-world applications.\n- **Storage Considerations for PEFT Models:** Pre-trained models finetuned on downstream tasks using Parameter-Efficient Finetuning (PEFT) methods typically require only the storage of the pre-trained model weights and additional adapter weights. These adapter weights consume significantly less storage space compared to the masks utilized in EMR-MERGING. The authors should address this comparison and discuss the storage implications in detail.\n- **Impact of Model Quantization:** With the trend towards quantization of models, reducing their precision from 32-bit to 8-bit, 2-bit, or even 1-bit, the storage overhead of masks becomes increasingly significant. Quantization is an essential step in the development of large models, and as such, the relative storage cost of the masks in EMR-MERGING will be more pronounced in the future. The authors should consider discussing the impact of model quantization on their method and the potential challenges it poses.\n- **Limited Dataset Size in Experiments:** The datasets used in the experiments are relatively small, which undermines the persuasive power of the results. For example, in the case of visual tasks, large models like CLIP's ViT-L exhibit strong generalization capabilities and can achieve high zero-shot classification performance on these small datasets. As a result, the performance differences among various model merging methods might not be significant. The authors should consider using larger, more diverse datasets to better demonstrate the effectiveness and robustness of their approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your hard work and constructive comments.\n\n---\n## Weakness 1: Additional Computational Overhead.\n\n**Ans**: The unified task vector, masks and rescalers are computed during the merging process, which is before the inference process. During the inference process, before we evaluate the performance on a single task, we simply apply the specific mask and rescaler to the unified task vector. This only causes little additional computational costs.\n\n---\n## Weakness 2: Storage Considerations for PEFT Models\n\n**Ans**: When merging PEFT models, the task vectors are only generated by the adapters. EMR-Merging only needs to store the masks applied to these adapters because the other weights in the model are fixed. Therefore, compared to storing $N$ adapters, using EMR-Merging to store a unified one and $N$ masks for the adapters can be a strong alternative.\n\n---\n## Weakness 3: Impact of Model Quantization\n\n**Ans**:\n\n+ The focuses of model merging and Quantization overlap but are not the same.\n\nModel merging focuses on reducing the deployment and storage costs by giving a model multi-task capabilities without training while quantization focuses on reducing computational and storage costs of a single model on a specific task by reducing parameter precision.\n\n+ Barriers to combining the model merging and low-bitwidth quantization.\n\nThough combining model merging and low-bitwidth quantization can further reduce the storage costs and computational costs, all the existing model merging methods cannot be directly applied to low-bitwidth quantized models including INT8, INT4, etc. This is due to that model merging needs to subtract or average the weights from different models but after quantization, the low-bitwidth weights cannot be directly operated because these weights may have gone through different quantization functions due to different models' specific weight distribution. Combining model merging with low-bitwidth quantization has broad application prospects and is a potential future work.\n\n+ Merging FP-16 models.\n\nIn order to explore the applicability of EMR-Merging to FP16 models, we conduct experiments on RoBERTa models loaded in FP16 and the results are shown in Table R12 of the Rebuttal PDF. EMR-Merging still shows promising performance, demonstrating its potential to be applied to quantized models.\n\n+ Quantizing the unified task vector.\n\nAlthough the quantized model cannot be directly merged, we find that the unified task vector can be simply quantized. We layer-wisely convert the elements in the unified task vector into low-bitwidth ones linearly. We use this method to merge RoBERTa models and the results are shown in Table R13. We find that EMR-Merging 's performance is not affected when the unified task vector is quantized to 4-bit and is still promising when the unified task vector is quantized to 2-bit. This can further reduce the storage cost of EMR-Merging.\n\n+ The increasingly significant storage overhead of masks.\n\nA potential limitation of EMR-Merging is that, with the trend towards quantization of models, the storage of the masks can be increasingly significant. We will add this to the Limiataion section. \nHowever, compared to storing multiple individual non-1-bit quantized models, the storage costs by the masks (1-bit) and rescalers (a single value) are still fewer.\nWhen applied to merge multiple models, EMR-Merging may still tend to be a competitive alternative.\nWe will add this discussion to the Future Work section. \n\n---\n## Weakness 4: Limited Dataset Size in Experiments\n\n**Ans**:\n+ The good performance of EMR-Merging is not merely due to the strong zero-shot capabilities of CLIP models because other merging methods normally suffer from significant performance degradation compared to individual models while EMR-Merging can achieve the performance close to individual models.\n+ We have conducted some experiments on large-scale datasets including COCO, ImageNet, and VQA v2 when merging multiple BEiT3 models and EMR-Merging demonstrates its good performance on all these large-scale datasets.\n+ Your opinion that using a larger and more diverse dataset may better demonstrate the effectiveness of EMR-Merging is reasonable. Based on merging eight vision tasks using CLIP ViT-B/32, we add a task, i.e., ImageNet-1K. We downloaded a CLIP ViT-B/32 model finetuned on the task in timm and merged the nine models. The results are in General Response, Additional Result #2. EMR-Merging shows a much more significant improvement compared to existing merging methods (up to 20%). We will release this benchmark as a new model merging benchmark.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General Response:

We thank all the reviwers for their time and constructive comments. We appreciate the reviewers' praise of the strengths of our paper including:

- the idea that decoupling model merging into unified parts and task-specific parts to avoid conflicts (reviewer Lqs2).
- extensive expreiments and considerable good performance (reviewer Lqs2, S3RQ).
- adequate amount of theoretical and empirical analysis (reviewer b5vt, Qs8J).
- tuning-free and data-free properties (reviewer b5vt, S3RQ)

-----

In the responses, we show additional experimental results including:

1. Merging GPT-2 models on seven NLP tasks.

   We merge GPT-2 models finetuned on seven NLP tasks from GLUE benchmark following the settings from FusionBenchmark. EMR-Merging improves the best performance of compared model merging methods by up to 10%. The results are in General Response, Additional Results #1.

2. Merging ViT-B/32 models on nine vision tasks (ImageNet-1K added).

   We additionally add ImageNet-1K to the original eight tasks. We merge nine CLIP ViT-B/32 models and EMR-Merging shows a much more significant improvement compared to existing merging methods (up to 20%). The results are in General Response, Additional Results #2.

3. Detailed comparison with Sparse Models.

   We compare the parameter number and performance of EMR-Merging and Sparse Models. In addition, we find that the elected unified task vector of our method can be sparsed to further reduce parameter numbers without affecting performance much. The results are in Table R3, R4, and Figure R1 in the Rebuttal PDF.

4. More ablation studies on the Electing process.

   We conduct additional ablation studies on the electing process of EMR-Merging. In addition, we find that the elected unified task vector can be quantilized to 4-bit or even 2-bit to further reduce parameter numbers without affecting performance much. The results are in Table R13 in the Rebuttal PDF.

5. Detailed comparison with WEMoE and Representation Surgery.

   We show detailed comparison of EMR-Merging with WEMoE [R1] and Representation Surgery [R2] in parameter numbers, prerequisites, and performance. EMR-Merging shows the SOTA performance on merging ViT-L/14 models and requires no data or training. The results are in Table R1 and R2 of the Rebuttal PDF.

   

----

# Some Additional Results

## 1. Merging GPT-2 Models on seven NLP tasks

Here we provide the results of EMR-Merging on a new benchmark [R3], merging GPT-2 models on seven NLP tasks from GLUE. All the tasks are evaluate by accuracy. Our EMR-Merging improves the performance of model merging by up to 10%. 

| Merging GPT2 Models    | CoLA     | MNLI     | MRPC     | QNLI     | QQP      | RTE      | SST-2    | Avg      |
| :--------------------- | :------- | :------- | :------- | :------- | :------- | :------- | :------- | -------- |
| Individual             | 76.8     | 82.1     | 80.4     | 88.3     | 89.6     | 65.3     | 91.2     | 82.0     |
| Simple Average         | 55.0     | 55.1     | 51.0     | 57.6     | 76.7     | 44.8     | 52.5     | 56.1     |
| Fisher Merging         | 54.8     | 58.0     | 39.5     | 63.3     | 81.5     | 49.1     | 64.7     | 58.7     |
| RegMean                | 61.7     | 70.4     | 65.4     | 69.7     | 78.8     | 56.0     | 79.7     | 68.8     |
| Task Arithmetic        | 68.7     | 68.6     | 69.6     | 70.5     | 81.8     | 47.3     | 83.6     | 70.0     |
| Ties-Merging           | 68.4     | 71.4     | 68.4     | 69.6     | 82.4     | 47.7     | 81.8     | 70.0     |
| **EMR-Merging (Ours)** | **72.8** | **81.1** | **79.2** | **84.8** | **88.1** | **66.5** | **90.3** | **80.4** |

## 2. Merging ViTs on nine vision tasks

On the basis of merging ViTs on eight vision tasks, we additionally add ImageNet-1K to the these tasks. The finetuned model on ImageNet-1k is released by timm. We merge nine CLIP ViT-B/32 models and EMR-Merging shows a much more significant improvement compared to existing merging methods (up to 20%).

| ViT-B/32 on nine vision tasks | SUN397   | Cars     | RESISC45 | EuroSAT  | SVHN     | GTSRB    | MNIST    | DTD      | IN-1k    | Avg      |
| ----------------------------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- | -------- |
| Individual                    | 75.3     | 77.7     | 96.1     | 99.7     | 97.5     | 98.7     | 99.7     | 79.4     | 82.0     | 89.6     |
| Averaging                     | 61.8     | 56.4     | 65.9     | 66.2     | 62.7     | 44.5     | 81.8     | 49.0     | 61.5     | 61.1     |
| Task Arithmetic               | 51.8     | 30.9     | 55.8     | 64.3     | 69.0     | 42.2     | 92.7     | 46.8     | 66.6     | 57.8     |
| Ties-Merging                  | 53.3     | 34.1     | 57.0     | 55.8     | 72.3     | 43.2     | 90.5     | 46.5     | 68.9     | 58.0     |
| **EMR-Merging (Ours)**        | **77.0** | **75.2** | **92.9** | **92.7** | **79.7** | **90.2** | **97.6** | **76.2** | **79.8** | **84.6** |

The impressive performance of our EMR-Merging on various benchmarks demonstrates its good performance and applicability.

----

# References

[R1] Merging Multi-Task Models via Weight-Ensembling Mixture of Experts. ICML 2024.

[R2] Representation Surgery for Multi-Task Model Merging. ICML 2024.

[R3] FusionBench: A Comprehensive Benchmark of Deep Model Fusion. arXiv:2406.03280.

Meta-review: The paper introduces a new method for model merging, a currently hot research topic, which considers how to directly fuse multiple independently trained models at the weight level to obtain a single comprehensive model. The proposed approach addresses the significant performance degradation from the multi-task counterpart and requirement of additional data and training in existing methods.

Strengths of the paper and the proposed method include: 
- the paper is well-written.
- method does not require additional training, data, and tuning, which enhances the applicable scenarios of the method. This makes it practical for real-world applications where data availability is limited or privacy concerns restrict access.
- experiments are extensive, including VIT model, language models, and multi-modal models. In particular, the visual model is extended to 30 tasks.
- informative ablation studies are provided.
- the methods are computationally lightweight and straightforward to implement, yet they perform well
- The paper provides a solid theoretical foundation for the effectiveness of the proposed method, including detailed proofs and empirical analyses. This adds robustness to the claims made by the authors.

Regarding the weaknesses identified by the reviewers, the rebuttal and subsequent discussion effectively answered most of these, including the additional computational overhead during inference and comparison to parameter-efficient fine tuning methods.

The discussion about the applicability of the method to quantized networks was quite fruitful, and the proposed changes to the document based on this will add to the depth of the presentation of the method.

The rebuttal effectively addresses the novelty of the proposed method, showing that it is quite different than the methods indicated by the reviewer.

The issue of not being able to do multiple tasks in parallel is moot, since competing methods cannot do this either.
Decision: Accept (spotlight)