TIES-Merging: Resolving Interference When Merging Models
Authors: Prateek Yadav
Derek Tam
Leshem Choshen
Colin Raffel
Mohit Bansal
Venue: NeurIPS 2023 poster (2023)
Keywords: Model Merging
Fusing
Collaborative Training
Robust Fine-tuning
Federated Learning
TL;DR: Different tasks might prefer different directions in the weight space and this can lead to conflicts when merging models. We identify this problem and propose an extremely lightweight dataless merging method.

Abstract: Transfer learning – i.e., further fine-tuning a pre-trained model on a downstream task – can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter’s values across models. To address this, we propose our method, TrIm, Elect Sign & Merge (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, highlight the importance of signs, and show that estimating the signs using the validation data could further improve performance.

TIES-MERGING: Resolving Interference When
Merging Models
Prateek Yadav!
Derek Tam
Leshem Choshen Colin Raffel Mohit Bansal
University of North Carolina at Chapel Hill IBM Research
MIT
leshem, choshen bmm......o.. com
(praty, atzedsox, craffel, mbansal (css. unc.
Abstract
Transfer learning i.e.. further fine-tuning a pre-trained model downstream task - can confer sigmificant advantages, including improved downstream perfor- mance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform single task and do not benefit from another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper. we demonstrate that prior merging techniques inadvertently lose valuable information due two major sources of interference: (a) interfer- ence due to redundant parameter values and (b) disagreement on the sign of a given parameter's 's values across models. To address this, we propose our method, TRIM, ELECT SIGN & MERGE (Tres-MERGINO) which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-MERGING outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference.
Introduction
Pre-trained models (PTMs) have become widespread in many real-world applications [91, 6]. Using PTMs typically involves fine-tuning them to specialize on a specific task [69, 12]. which can lead to improved performance with less task-specific labeled data. These benefits have resulted in the release of thousands of finetuned checkpoints [81] derived from popular PT'Ms such as ViT [14] for vision and T5 [58] for language. However, having separate fine-tuned model for each task has various drawbard bwwbacks: (1) for each new application, a separate model has to be stored and deployed [17, 89], and (2) models trained in isolation cannot leverage information from related tasks improve in-domain performance or out-of-domain generalization [66, 58, 75]. Multitask learning [66, 57] could address these concerns but requires costly training and simultaneous access all tasks [17]. Moreover, can be complex and resource-intensie to determine how best to mix datasets to ensure that multitask training is beneficial for all tasks [55, 54, 80, 52, 2, 17].
Our code is available at
mergn
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Aligned Values
Parameter
Influential
Model 2
Merged Model
Redundant values
(3) Disjoint Merge
Task Vectors
Trimmed Task Vectors
(1) Trim
Sign Vector 7m
(2) Elect Sign
Merged Task Vector Tau
A depiction of the steps involved in MEESMEEGGNG. We visualize each parameter in model as a square, The arrows depict the update (task vector, 7) to a parameter produced by fine-tuning on different tasks (coded by colors), with direction denoting sign and length denoting magnitude. We first trim the task vector values based on their magnitude, then we elect the sign for each parameter (Ym. green vector containing +1 or - 1) by resolving sign conflicts. Finally, we pick only the values that align with the elected sign and take their mean as the final parameter value.
Figure 1:
Recently, a growing body of research has focused on model merging [40]. One application of merging involves combining multiple task-specific models into single multitask model without performing additional training. Previous works merge models by summing the individual model weights with different weighting schemes, either via simple average [9, 28, 83], via more sophisticated means that incorporate parameter importance [45] or account for permutation invariances [1, 31, 70, 74, 42]. Combining fine-tuned models in this way can be seen adding together task vectors [29] that computed by subtracting the pre-trained model's parameter values from those of the fine-tuned model.
While weighted averaging of model parameters has proven effective for merging, all of these methods ignore the possibility that values may interfere across models, thereby harming the performance of the merged model. In this paper, first demonstrate that interference can stem from two major causes (see Fig. 2), both of which can reduce parameter magnitudes in the merged model and eliminate sub- tle distinctions among values: (1) INTERFERENCE FROM REDUNDANT PARAMETERS: Previous studies on model pruning [25, 76] have shown that during fine-tuning, many model parameters can change over the course of fine-tuning [63] but only have a small impact on performance. However, when merging parameter that is influential for one model but re- dundant (i.e. not influential) for other models, the influential value may obscured by the redundant values, lowering the overall model performance (O in
Original Values
Merged Values
Model
Model
TIES
No Interference
Redundant
Sign Conflict
Figure 2: Different types of conflict and
merged outputs produced by either averaging or MERR-NG.GI The parameters causing interference are denoted by dotted arrows.
Fig. 2). (2) INTERFERENCE FROM SIGN DIS DIAAGREMENT. A given parameter might have a positive value for some models and negative value for others. Consequently, employing simple averaging might compromise the performance on both tasks (O in Fig. 2). In both of these situations, simply aggregating the values lead to interference that shrinks the parameter's 's value in the merged model. This interference between influential parameters might explain why the performance gap between the merged model and teltitask--rainned model increases the number of models increases [31].
To address these sources of interference, we propose TIES- MERGING (TRIM, ELECT SIGN MERGE) method, a method for merging models by combining task vectors that has three steps (visualized in Fig. 1): First, we trim each task vector to retain only the influential parameter values by setting the redundant values in each task vector to zero (or, equivalently, resetting the fine-tuned

---

parameter value back to the value from the pre-trained model), After this step, sign conflicts may still persist among influential parameter values, as visualized in Fig. Our second step therefore resolves the sign confficts between different values and our last step only averages parameters whose sign agrees with the direction of the largest total movement across models.
We demonstrate the effectivvnesss of our proposed TIES TIES-MERGING method in various setups with: (1) different modalities, including language and vision benchmarks, (2) distinct model sizes and families, such as T5-base and T5-large [58] as well as ViT-B/32 and ViT-L/I4 [14], (3) in-domain and out-of- domain tasks, (4) full finetuning or parameter-efficien finetuning, and (5) in the presence or absence of a validation set for setting merging hyperparameters. We show that TES-MERGING outperforms other merging methods, such as Task Arithmetic [29], RegMean [31], Fisher Merging [45], and weight averaging [9, 82] across all these experimental settings. Notably, for in-domain evaluation, TTES-MERGING outperforms the strongest baseline by an average of 2.3% and 1.7% absolute in NLP and vision settings (Table 1), respectively. For -ft-of-domain generalization (Table 2), TIES- MERGING outperforms the strongest baseline by 1.0% and 4. 4.4% absolute for T5-base and T5-large models respectively. In Section 7, we perform ablations our method components and demonstrate the impact of interference on parameter values. Additionally, we showcase the increased advantage of TIES-MERGING over task arithmetic [29] as the number of tasks increases. Finally, we examine the importance of obtaining the correct sign vector. Our results and analysis establish MEE-MMGGIING as powerful and effective method for combining fine-tuned models into single multi-task model.
2 Related Work
Loss Landscape and Weight Interpolation. While the loss function of a neural network is generally non-convex, recent work has demonstrated that the parameter values from different training runs can sometimes be interpolated without increasing the loss (i.e. they are mode-connected) [15, 20, 21, 32, 22]. For example, Frankle et al. [19] showed that if part of the optimization trajectory is shared between two neural networks then they can be interpolated without lowering accuracy. On the other hand, Neyshabur et al. [48] showed that naively interpolating two neural networks with completely disjoint optimization trajectories can result in a catastrophic drop in their accuracies. Entezari et al. [16] hypothesized that if account for the permutation symmetry of neural networks, then all neural networks of given architecture trained on the same dataset are linear mode connected. Ainsworth al. [1]. Singh and Jaggi [70]. Wang et al. [79] therefore used techniques based on finding permutations [79, 1] and optimal transport [70] to better align neural networks trained from scratch so that they can be interpolated without increasing the loss.
Model Merging and Different Use Cases. Different fine-tuned models initialized from the same pre-trained model effectively share part of the optimization trajectory, and can therefore often be merged without accounting for permutation symmetry [82, 83, 29, 31]. Therefore, merging fine-tuned models can improve performance on single target task [30, 23, 82, 9], improving out-of-domain generalization [31, 29, 7, 4, 60, 59], creating a multitask models from different tasks [31, 29, 38), for federated learning [46, 41]. compression [39], multimodal merging models [72], continual learning [86, 85], and other settings [38, 13]. The range of applications has led to a proliferation of methods to improve beyond simple parameter averaging. RegMean [31] proposed a closed-form solution for the merged model's parameters by solving a local linear regression problem for each individual linear layer in the model. However, this requires transmitting additional data statistics that are the same size as the model and requires additional inference steps to calculate them. Fisher Merging [45] goes beyond simple averaging to identify the importance of individual parameters using Fisher Information Matrix [18, 3, 34] and uses it to weigh the parameters in each model when merging, However, this shows little gains when merging multiple checkpoints and also requires computing gradients which has a high memory cost. Task Arithmetic [29] presented a method for merging models by generating task vectors and performing arithmetic operations, such as addition, to obtain a multitask checkpoint. A concurrent work by Ortiz-liménez et al. [51] provided theoretical insights on model merging based on the weight disentanglement property that arises during pretraining. They showed that finetuning models in their tangent space enhance this property, leading to better- models. Our method follows these past works on model merging but additionally takes into account the interference between different parameters during merging.
3

---

0.3
Awerade Ped
Foc. of Params with Conflict
0.2
60
55
20
40
60
80
100
2
3
6
8
9
11
Keeping Top-K96 Parameters
Number of Models
Figure Performance depends on small
Figure 4: Sign conflicts occur even after trim-
fraction of high-magnitude parameters. For
ming and increase with the number of models.
each task vector, we keep only the largest - top-
We plot the fraction parameters that have a sign conflict after trimming versus the number of mod-
4% parameters and plot the average performance across eleven tasks. Keeping only the top- top-20% of the parameter does not degrade the performance.
els being merged.
Background and Motivation
Problem Setting Given set of tasks (41.--.; (n) and pre-trained model such T5 [58] ViT [14], we either finetune the entire model or employ a parameter-efficient finetuning (PEFT) method [43, 26]. In both cases, we denote the trainable parameters the finetuned parameters as Og. In this paper, we assume access to finetuned model parameters On for multiple tasks and devise a method merge the weights of these models into single multitask model proficient on both in-domain and out-of- of-domain datasets. We follow Ilharco et al. [29] and perform merging with task vectors. Specifically, for a task t, the task-vector тN Rd is defined as = ff - fiii-- This operation allows us to focus on the changes that happen during the fine-tuning phase of each task-specific model and is equivalent to computing weighted average of the models' weights with appropriate scaling.
the initialization Oinit and
Redundancies in Model Parameters. First, we demonstrate that in a given task vector, many values are redundant (denoted by O in Fig. 2), and removing them does not affect the performance of the task. Specifically, Fig. shows the average performance across eleven task-specific models when "trimming" each task vector to retain only the top-k% values and resetting the rest to their initial value (i.e. setting the corresponding value in the task vector 0). Fig. 3 shows the average performance across varying values of k. demonstrating that keeping only the top-20% of values delivers comparable results to retaining all parameters. For additional details and the results on the T5 model, please refer to Appendix C.3. This shows that many parameter changes introduced during fine-tuning are redundant. Hence, disregarding those values during merging could prevent interference with the influential parameters without compromising the task's performance.
Algorithm 1 TIES-MERGING Procedure.
Input: Fine-tuned models {0.)=1 Initialization Ounir. k. and X. Output: Merged Model 6n forall in!,... do
Create task vectors.
= 0g Oui
step 1: Trim redundant parameters.
à + keep topk reset_ rest to (T,
ft + sgn(ft)
je + lr
end
Step
Final Signs.
7'm sgn ( it)
Step 3: Dis jisjoint Merge.
forall p iin,, iinl.... ..., d do
AA (1 E [n] of m
TP TA AL
end
Ubtain nerged checkpoint
0m <- Onni + X Tm return 0..
Disagreement between Parameter Signs: Different fine-tuned models might introduce opposing changes to a parameter their task vectors, causing interference due to conflicting signs (denoted by O in Fig. 2). Fig. 4 presents analysis of the frequency of sign conflicts when merging varying numbers of models. We first trim the task vectors for eleven tasks by keeping only the top 20% of influential parameters. Then, plot the percentage of parameters that have sign conflict

---

increase the number of models to be merged from 2 to 11. Notably, sign conflicts occur even when merging only models from different tasks or when merging multiple models from the same task (see Appendix Figure 10), and the likelihood of a sign conflict increases with the number of models being merged. For additional details and the results on the T3 model, please refer to Appendix C.3.
TIES-MERGING: TRIM, ELECT SIGN & MERGE
To address the aforementioned issues, we present TIES- MEE-INGGG (TRIM, ELECT SIGN MERGE), which aims to address the kinds of interference mentioned above before performing merging.
4.1 Preliminaries
task vector Tt Rdd represents direction and the amount of movement required in the d- dimensional parameter space relative to the initialization that leads to a low loss region for the task Each entry in Tr (corresponding to a particular parameter) can be thought of as an axis in the d-dimensional space. The sign of parameter denotes the direction along this axis (positive or negative) that decreases the loss on task t. Hence, a given task-vector Tę can be decomposed into a sign vector "/e R and magnitude vector p4 R Ti Y O j4t, where @ is the elementwise product. Formally, f = sT,( where sgn(æ) |x|| = x and returns a value of +1, O, or -1. The magnitude vector le is defined as μ4 [Tel and the value μ tells the movement required in the i-th dimension from the initialization.
4.2 Steps in TIES-MERGING
To merge multiple task-specific models Jn Given these task vectors, -EER-MGGGGNG method follows three steps in order to perform a merge (see Fig. 1 for a diagram and Algorithm 1):
first create corresponding task vectors In
1. Trim:
For each task we trim the redundant parameters from the task vector Ti create ft by keeping the top-k% values according to their magnitude and trimming the bottom (100 - % of the redundant parameters by resetting them to O. This can be decomposed further as = 9/ Pr.
2. Elect:
Next, we create an aggregate elected sign vector Ym for the merged model that resolves the disagreements in the sign for each parameter across different models. To create the elected sign vector, we choose the sign with the highest total magnitude across all relevant models. For each parameter
{1,2... .... d}. we separate the values {f?" based on their sign (+1 or -1) and take their sum calculate the total mass (i.e., total magnitude) in the positive and the negative direction. We then assign rh as the sign with greater total movement. This can be efficiently computed using 7h = sgn( sgn( Er f!").
3. Disjoint Merge:
Then, for each parameter p, we compute a disjoint mean by only keeping the parameter values from the models whose signs are the same as the aggregated elected sign and calculate their mean. Formally, let A²" = (1 E [n]] f% 2 }. then T |AP ΣLEAP
AP
Note that the disjoint mean always ignores the zero values.
Given the final merged task vector Tya we scale it and add it to the initial parameter values to obtain the merged model parameters O,, Om Ouit Ti, where À is scaling hyperparameter (as used in past work [29]).
5 Experimental Setup
Baseline Methods. We compare TIES-MERGING with four baseline merging methods: (1) Sim- ple Averaging [9, 82] calculates the element-wise mean of all the individual models and can be expressed as 8,. = Ei= 0e/n. (2) Fisher Merging [45] uses diagonal approximation of the Fisher Information Matrix F. [34, 3, 18] to measure the importance of each parameter for task I, where F. = BeeeD,,Errpoo (1)=),,,(11)) (u|=) Ve, (log Pe, (z) The final merged model is obtained by reweighting each parameter in each fine-tuned model by the corresponding value in the model's approximate Fisher matrix 6m Ei= F,0// / Fr. (3) RegMean [31] computes closed-fom solution to
5

---

Method (L)
Validation
PEFT
Full Finetuning
Model (-+)
(AA)
T5-Base
T5-Large
ViT-B/32
ViT-L/14
FINE-TUNED
71.4
82.8
88.8
90.5
94.2
MULTITASK
73.1
83.6
88.1
88.9
93.5
AVERAGING 182, 91
65.9
59.6
65.8
79.6
TASK ARITIMETIC [29]
X
73.2
73.5
60.4
83.3
TIES-MERGING
X
69.7 [-3.2]
74.4 [+0.9]
72.4 [+6.6]
86.0 [+2.7]
FISHER MERGING [45]
62.2
68.9
64.6
68.3
82.2
REGMEAN [31]
58.0
71.2
73.2
71.8
83.7
TASK ARITHMETIC [29]
63.9
73.2
73.3
70.1
84.5
TIES-EERGING
66.4 [+2.5]
73.9
[+0.7]
76.9 [+3.6]
73.6 [+1.8]
86.0 [+1.5]
Table 1: Comparing model merging methods across multiple fine-tuning settings and modalities (NLP and Vision) with and without the availability of a validation set.
a least-squares regression problem that aims to minimize the distance between the merged model's ac- tivations and the individual models' activations as 6m. (Ei=1 X; X,) Σ (X² X,0t). where X₂ is the input activation of a given layer. (4) Task Artithmetic [29] scales and then adds the task vectors to the initial model to produce the merged model as 9. O + x > -t= Tr. In addition to these baselines, we present the performance of the individual fine-tuned models involved in the merging process well as the performance of multi-task model trained the concatenation of all tasks' datasets. For more details on compute resources, dataset licenses, and the finetuning procedures, refer to Appendix C.1, C.2, and C.6.
Merging in Absence of the Validation Set. Prior works [29, 45, 82) model merging assume access to a validation set, which is utilized to compute the Fisher matrix or tune ayper-parameters.. To avoid the need for validation set, RegMean [31] proposed storing and transmitting inner product matrices of the training data for each task that are the same size as the original model. This can quickly become expensive for large models the storage and transmission scale linearly with model size and the number tasks.
To consider the setting where no validation set is available, we developed a generic recipe of TIES- MERGING with fixed hyperparameters that could applied any setting without hyperparameter tuning on validation set. The recipe keeps the top-20% parameters in the task vector resetting the rest to O and sets 1. We chose this recipe based on results in the parameter-efficient fine-tuning only apply it to the unseen settings of full model fine-tuning on ViT (vision) (PEFT) setting, models. We also compare WHES-MERGING with the Task Arithmetic method and T5 (language) without validation set by utilizing the recommended value of À 0.4 [29]. For further details on how this recipe was created please refer to Appendix C.4.
6
Main Results
Our main goal is to merge multiple task-specific models into single multitask model that can perform well on both in-domain and out-of-domain scenarios. In this section, we evaluate the performance of TiES-MERGING with other methods across multiple different experimental settings.
Merging PEFT Models. Consider the setting where task vectors are computed based on parameters introduced during parameter-efffcient fine-tuning. Specifically, we focus on (IA)² [43]. a PEFT method that scales the base model activations with learned vectors. We follow Liu et al. [43] and use
TO-3B [66] as the base model and finetune (IA)3 models on the train split of eleven datasets including sentence completion (COPA [61], H-SWAG [88], and Story Cloze [68] datasets), natural language inference (ANLI [49], CB [44], and RTE [11]), coreference resolution (WSC [37] and Winogrande
[64]), and word sense disambiguation (WiC [53]). When fine-tuning (IA)² parameters added to the
TO-3B model, we use prompt templates from the Public Pool of Prompts (P3 [5]) to convert each example in each dataset to prompted text-to-text format where each label corresponds to a different string, For experiments with (IA)², for each dataset, report the median score across all templates.

---

1.1
Avg. Nornaalzzed Perf.
Simple
Task Artihmeli
0.9
Model
TS-Bas
T5-Large
Zeroshot
31.1
27.6
0.8
Simple Averaging [9, 82]
31.7
30.4
0.7
Fisher [45]
33.8
32.0
RegM [31]
34.3
36.0
2
3
5
7
Task Arithmetic [29]
31.9
32.3
Number of Tasks
TIES-MERGING
35.3
[+1.0]
40.4 [+4.4]]
Table 2: TEES-MERGING generalizes better.
Figure 5: TIES-MERGING scales better. Aver- age performance when merging different num- ber of tasks.
Out of Distribution Generalization for T5-Base and Large on six held-out tasks.
Table I using TIES-MERGING merge models trained with (IA)² exceeds the performance of all other merging methods with a validation set, TIES-MERGING shows an average enhancement of 2.5% across 11 tasks compared to the top baseline. For detailed results, refer to Appendix Table 8.
Merging Fully Finetuned Vision Models. For image classification, we adhere to the experimental setting from Ilharco et al. [29, 28]. We employ two variants of the CLIP model [56] with ViT-B/32 and ViT-L/14 models [14] as visual encoders. Subsequently, we finetune the visual encoder on the eight tasks derived from Ilharco et al. [28, 29], Radford al. [56] while keeping the text encoder fixed, This setting considers variety of classification domains such as remote sensing, traffic classification, and satellite imagery recognition. Specifically, we work with the following datasets: Cars [35], DTD [10], EuroSAT [24], GTSRB [71]. MNIST [36]. RESISC45 [8], SUN397 [84], and SVHN [47].
Table 1 shows that using TIES- TIES-MERGING MERGING menge fully fine-tuned ViT-B/32 and ViT-L/14 models leads to an average improvement of 1.8% and 1.5% over tasks, given the availability
validation set. In the absence of validation set, MERGING improves by 6.6% and 2.7% over other methods for ViT-B/32 and ViT-L/14, respectively. Notably, STES-MERGIN without validation outperforms Task Arithmetic [29] with validation by 2.3% and 1.5% for VIT-B/32 and ViT-L/14. For more detailed results, refer to Appendix Table 11 and 12.
Merging Fully Finetumed NLP Models. For the NLP domain, we the T5-base and T5- [57] models, which are encoder-decoder transformers [77] pretrained via masked language modeling on large text corpus. We finetune both T5-base and T5-large on seven tasks: question answering (QASC [33], WikiQA [87], and QuaRTz [73]), Paraphrase Identification (PAWS [90]), Sentence Completion (Story Cloze [68]), and Coreference Resolution (Winogrande [64] and WSC (37]).
Table shows that using TTES-MERGING on T5-base and T5-large models with a validation set produces an improvement of 0.7% and 3.6% respectively over 7 tasks compared to the state-of of- of--th--rt. Moreover, for TS-large MANS-MERGING without validation outperforms all baselines (even with a validation set) by 1.1% For more detailed results, refer Appendix Table 9 and 10.
Out-of-Domall Generalization. In many use-cases, multitask models are used for their ability to generalize better domain shift. Hence, we use the T5-base and T5-large models merged on the seven in-domain datasets from the previous experiments and evaluate them on six held-out datasets from TO mixture [65] measure out-of-domain generalization. Specifically,
report the average
performance over the following tasks and datasets: Cosmos QA [27], Social IQA [67]. and QuAIL [62] for question answering: WiC [53] for word sense disambiguation; and COPA [61]. and H-SWAG [88] for sentence completion. Table 2 shows that TIES-MERGING MERGING outperforms the strongest bascline for both T5-base and T5-Large by 1.0% and 4.4% respectively, demonstrating better out-of-domain generalization. For more elaborate results please refer to Appendix B.6 and Table 13 and 14.
Merging Different Number of Tasks. We evaluate the performance of the merged model on in-domain tasks as we vary the number of tasks being merged. In Fig. 5, we normalize the accuracy of each task by its fine-tuned model performance and report the average normalized accuracy on the in-domain tasks. We compare with the strongest baseline Task Arithmetic [29] as well as
7

---

RTE
MRPC
WNLI
Init Method
RTE
MRPC
WNLI
Averaging 59.9
78.2
56.3
PTM Init
66.4
81.8
56.3
Fisher 65.7
81.4
52.1
Average
75.8
86.5
56.3
Ensembling
70.8
86.0
45.1
Task Arithmetic
78.3
86.2
50.7
Task Arithmetic
71.8
86.0
59.2
TIES-MERGING
80.1
88.0
54.9
TIES-MERGING
72.2
86.8
58.8
Table 4: A TIES-merged model is a better ini- tialization for finetuning. For each task, merge the checkpoints from the 7 other GLUE tasks and then finetune and evaluate on the se- lected task.
Table 3: Model soups experimental setup. TIES improves performance when merging check- points on the same tasks. For each task, we merge 10 checkpoints from Huggingface hub and evaluate on the one task they were trained on.
Mean magnitude
Mean
Trim + Disjoint Mean
Mean
Elect
Disjoint Mean
Mean magnitude
0.08
0.15
0.04
0.02
0.05
0
>1
0.5.0.6
0.6
0.7-0.8
0.8
0.9-1.0
Parameters With Nonzero Values
Bins of Sign Agreement
(a) Redundant Parameter Interference.
(b) Sign Interference.
Figure 6: Trimming Parameters and Electing Signs prevents interference. Demonstration of parameter interference between different models and its impact on parameter values. The Standard Mean (red) shrinks magnitudes and does more when there is less agreement on the sign (right) or the parameter is influential for multiple tasks (left).
simple averaging [82]. Each data point signifies merging a subset of the tasks, with the solid line representing the mean performance across multiple subsets. For similar results with the TS-base model, please refer to Appendix C.5 and Figure 13.
From Fig. 5, we observe the following: (1) As the number merged tasks increases, the performance of all methods decreases. (2) When merging two tasks, both TIES-MERGING and Task Arithmetic achieve an average normalized accuracy close to one, indicating negligible performance loss. In contrast, Simple Averaging suffers from 10% performance drop. (3) As the number of tasks increases, the merging performance of Task Arithmetic declines more rapidly than TIEs-MERGING. This suggests that task interference is present when merging multiple tasks and that TIES-MERGING is more effective at mitigating this issue.
Merging Checkpoints of the Same Task For Better Robustness We perform additional experi- ments to merge multiple checkpoints trained on the same task (as done in ModelSoups [82]) to see it can improve robustness. Typically, ensembling is used to combine different models on the same task for better generalization. We use the experimental setting and the code from Fisher Merging [45] to merge top- 10 fine-tuned base sized BERT models from huggingface for RTE, MRPC, and WNLI datasets from GLUE, From the results presented Table 3, we observe that SEEEMMEGGING works the best in all cases except WNLI, where it only slightly underperforms Task Vectors. Notably, TIES-MERGING provides a dramatic boost over both Fisher Merging, averaging. and outperforms ensembling in all cases. Moreover, in Appendix B.4, show that interference exists even between differently finetuned checkpoints of the same tasks.
Merging Models for Better Initialization. Next, we perform experiments following the setting [9], where we merge checkpoints from different tasks for better initialization when fine-tuning on a downstream task. We take the finetuned bert perrt-base-uncan tasks (wnli, sst2, rte, gnli, mrpc, cola, mnli, qqp) from Huggingface [81]. We consider three of these GLUE tasks (RTE, MRPC, WNLD as our downstream tasks. When fine-tuming particular
checkpoints for 8 GLUE [78]

---

Top-20
Top-30
Top-50
Bestom-80
Bettom-70
Bottom- 50
Average Performance
Method
Average
70
Fine-Tuned
71.4
Multitask
73.1
60
Averaging 19, 82]
58.0
Task Vectors [29]
63.9
50
TIES-MERGING
66.4
TIES-MERGING (Oracle Sign) 72.0 [+5.6]
0.0
0.2
0.4
0.6
0.8
1.0
Probability of flipping the Sign
Table 5: TIES-MERGING
perform close to multitask models if the signs can be estimated correctly. We use the signs from the multitask vector as the elected sign and perform merging and report the perfor- mance.
Figure 7: Flipping the signs of high magnitude pa- rameters leads to catastrophic performance drops. Average Performance when flipping the directions of Top-k% and Bottom-k% parameters for each task. We report the results averaged over eleven (IA)² tasks.
downstream task (say RTE), we merge all the checkpoints from the other seven tasks together (apart from the chosen task). From Table 4, we find that TIES-MERGING works well in this setting and outperforms all other merging methods by significant margin (apart from Averaging for WNLI).
7 Additional Results and Analysis
7.1 Types of Interference and Their Effect on Merging
(a) Importance of Removing Redundant Parameters. To better disentangle the effect of re- dundant parameters on the resulting magnitude of merged parameters, we separate the parameters into three groups: redundant parameters (using trimming threshold of 20%), parameters that are influential to exactly one model, and parameters that are influential to more than one model. We then compare the parameter values when they are directly merged versus when they are first trimmed and then (disjointly) merged without electing signs. Specifically, we only take the mean non-trimmed values. The results are shown for the PEFT setting in Fig. 6a, which demonstrates that redundant parameters cause interference. Specifically, we find that when a parameter is not an influential parameter for any of the task- models, the mean value is low, and therefore may be considered noise. However, when only one model sees the parameter as influential, the merged value can still be low since other models assign small value to this parameter. The merged value is larger when more models see the parameter as influential. When trimming, we see this interference mostly avoided, and the average size is mostly the same whether This is because we remove the effect of noisy parameters that unnecessarily decrease the magnitude (see in Fig. 2). In the Appendix B.5, bring a more detailed view, including comparison to applying TIES-MERGING and show reducing interference encourages diversity in specific parameter values (std) together with the similarity of their influence (mean).
more models consider parameter influential.
(b) Importance of Resolving Sign Interference. To quantify the impact of sign interference, we group the parameters by their sign agreement. A value of 0.5 indicates an equal number of positive and negative signs for given parameter across different models, whereas implies all the parameters have the same sign. We compare the parameter values when those are merged, or when sign disagreement is first resolved by election and then they are (disjointly) merged. The results in the PEFT setting are shown in Fig. 6b, where we demonstrate that the ELECT step preserves the relative parameter magnitudes to avoid sign interference. Specifically, we find that resolving signs increases the overall parameter magnitudes across different ranges of sign agreements. Parameters with low agreement tend to smaller on average regardless of the interference. One potential cause could be that the sign from noisy parameters pulls the average down, as seen in Fig. 6a. We show in Appendix B.5 that combining both methods indeed reduces some the difference, but not all, suggesting that high agreement is correlated with overall influential parameters.

---

7.2 Relevance of Signs of the Top-k % Parameters
In this experiment, we work with the (IA)² models and aim to quantify the importance of the top-k% parameters and their directions on a task's performance. For each task vector тr, we flip the direction of each of the top-k% parameters (by magnitude) with a probability p to obtain řt. Flipping the direction is done by multiplying the parameters with -1. Then we add back this direction flipped it to the initialization to get 0,- mn + 9.. Finally, we evaluate 9r and compute the average performance over all tasks for each value of k and p. As a baseline, we also flip the directions of the bottom (100 k)% of the parameters. We report the results averaged over three independent runs.
In Fig. 7, we plot the average performance as function of p. the probability of flipping the direction. A probability of 0 means that the direction of none of the top-k% parameters is flipped and a value of 1 means that the direction of all the top-&% parameters is flipped. For the top-20/300 of the parameters (solid lines), we observe that the performance monotonically decreases as we increase the probability of flipping the direction. parameters (dashed lines) has little impact on the performance. These results establish the importance of having the right directions for the parameters with high magnitude and show the catastrophic performance drop that happens with incorrect directions.
contrast, flipping the directions of the bottom-80/700 of the
7.3 Ablation of TIES-MERGING Components
We perform ablations on the individual components of to assess their importance. In Table 6,
Method
T5-base
(IA)²
we start with TIES-MERGING and remove one component at time and report the performance on the validation set for full model merging (T5-base) and merging PEFT models ((I)² on TO3B). Removing elect while keeping the disjoint mean refers to taking the mean of values with signs +1 and - but not including the O values of the trimmed task vectors in the mean. Removing disjoint mean but trimming and electing refers taking the mean of the values with the elected signs and the O for the trimmed values. Removing scaling means using À = 1. Table shows that all components of the method are crucial for optimal performance. Specifically, scaling and the disjoint mean emerge as the most crucial, causing performance declines of 2.5% and 1.9% in TS-base, and 5.2% and 3. 3.2% in (IA)², respectively.
TIES-MERGING
74.5
70.7
TRIM
73.0
70,6
ELECT
73.1
69.6
DISJOINT MEAN
72.6
67.5
SCALE
72.0
65.5
Table 6: Ablation on all the steps of MERGGNG
7.4 Importance of Estimating Correct Signs When Merging Models
Given the importance of sign vectors, we now aim to understand the performance that can be obtained by TIES-MERGING if we can use the oracle sign vector from the multitask model. To test this, we train a multitask (IA)² model, mmtli. on the eleven tasks under consideration (as in $ 6). We then create the multitask vector Tmult and the multitask sign vector Ymult- Next, while merging models using MER-MMGGGN we assume access to the oracle
Ymult. Hence, we skip the conflict resolution step and directly set 7m = Tmaltt Surprisingly, from Table 5, we observe that when merging tasks by using the oracle sign vector, we get performance of 72% compared to 73. 1 1% for the multitask trained model. Moreover, on average the merged model performs better task-specific models. This implies that if we can obtain the correction directions for the merged model, then can closely bridge the gap to multitask models. In Appendix B. and Table 7, we attempt to estimate the multitask-sign-veete by using limited data.
8 Conclusion
We introduced MEESMEGGINGG to address interference when merging models. -TES-NGGGING trims mwggittte changes fine-tuned model's values and then resolves sign disagreements across the models being merged. We found experimentally that TIES- MERRING enhances the performance of the merged multitask model across various settings and domains, despite being simple with fixed hyperparameters. Our study also sheds light on the impact of different types of interference on model parameters and emphasizes the importance of signs the merging process. For some discussion on limitations and future directions please refer to Appendix A.
10

---

Acknowledgements
We thank Yi-Lin Sung, Shiyue Zhang, Archiki Prasad, and the reviewers for their valuable feedback on this paper. This work is supported by NSF-AI Engage Institute DRL21 1263, NSF-CAREER Award 1846185, DARPA MCS Grant N66001-19-2-4031, and NSF Grant 2145822. The views, opinions, and/or findings contained in this article are those of the authors and not of the funding agency.
References
[1] K. Ainsworth, J. Hayase, and S. Srinivasa. Git re- basin: Merging models modulo permutation symmetries, 2022. https:/ arxiv. org abs 2209 04836.
[2] A. Albalak, C. Raffel, and W. Y. Wang. Improving few-shot generalization by exploring and exploiting auxiliary data. arXiv preprint arXiv:2302.0667.. 2023.
[3] S. Amari. Neural learning in structured parameter spaces natural riemannian gradient. In NIPS, 1996.
[4] D. Arpit, H. Wang, Y. Zhou, and C. Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. arXiv preprint arXiv:21 10832, 2021.
[5] S. H. Bach, V. Sanh, Z.-X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, Févry, et al. PromptSource: An integrated development environment and repository for natural language prompts. arXiv preprint 2rXiv:220..0/279, 2022.
[6] R. Bommasani,
Hudson, Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bossclut, E. Brunskill, et al. On the opportunities and risks of foundation models, 2021. https: aariiv.iivv/////ai////aaaa/aaatv www/arrgeaa..ooooooeeeee 2100 0 07258.
[7] J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems, 34:22405-22418, 2021.
[8] G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engineers (IEEE), 2017. / /ttsss//eeeexxlrreee ieee. org/ abstracte http////tmuuennn/cceeeeeee
[9] L. Choshen, E. Venezian, N. Slonim, and Y. Katz. Fusing finetuned models for better pretraining, 2022. https::////o
222/////b22224444 2204 03044.
[10] M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recogni- tion (CVPR), 2014. https:/ httt:::://peeaccessss thecvf. html htt.pppp_.scccbbbo...rrrrreeeeea htt.. html.
[11] I. Dagan, O. Glickman, and Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop, 2005. pttcs::/linng wprisiiiieee winnenng.... com/ chapter 10.
[12] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint 17X0v.1810.04805, 2018.
[13] S. Don- Yehiya, E. Venezian, C. Raffel, N. Slonim, Y. Katz, and L. Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, 2022. attps://arxivw abs 22ts.221...0 01378.
[14] A. Dosovitskiy, L. Beyer, Kolesnikov, D. Weissenborn, Zhai, Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. https: /ttp:s/////peeeieeeeeeeeee http::/.eeetttoeeeeeeeeeeee wttt://ormmiidd-i....mo. d-niniidddiiiii aNTTy.

---

[15] F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning (ICML), 2018.
https: / arxiv orgg abs////////// 1803. 808.00885. 00885.
[16] R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity neural networks. arXiv preprint urXiv:2110.062990 2021.
[17] C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems, 34:27503-27516 2021.
[18] R. A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical transac- rions of the Royal Society of London. Series A, containing papers of mathematical physical character, 222(594-004)33399333 368, 1922.
[19] J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), 2020.
ht https::/proceee http://poceeeieigeeee/e mlr... press/ srress/ville /ttpv/vl//////////// wttp//ranneaeaaaaeeeeeeeaa htmll.
[20] C. D. Freeman and J. Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint 16Xiv:6611.00 1..50100,0 2016.
[21] T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS), 2018. https:/
/10/1100.111 11.80210022 10026.
[22] A. Gueta, E. Venezian, C. Raffel, N. Slonim, Y. Katz, and L. Choshen, Knowledge is a region in weight space for fine-tuned language models. arXiv preprint 2rXIv:2332.00863,, 2023.
[23] V. Gupta, S. A. Serrano, and D. DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. International Conference on Learning Representations, 2020.
[24] P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. https: aaaaaaiivvv.... org/ abs / 1109/ 00029.
[25] T. Hoefler, D. Alistarh, T. Bun-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruming and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research, 22(1):00822-1105 1005, 2021.
[26] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. ArXiv, abs/2100.096883 2021.
[27] L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi. Cosmos qa: Machine reading compre- hension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP,, pages 2391-2401, 2019.
[28] G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems (NeurIPS), 2022. https: /ttts:///rxiv.... aarii..iiivv///ii//.aaaaaaaaa org/ abs/
2208. 05592.
[29] G. Ilbarco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing mod- els with task arithmetic. In The Eleventh International Conference on Learning Representations, 2023. URL https: /ttp:::////peeeiiewe/e wtt.eiewwwweeeeeeeeeeeeee.eeeeee hwtp:/eee/tttoooeeeeeeeee www.moormmi/o 6t 0 jrj.
[30] P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty Artificial Intelligence (UAI), 2018. https: http//axivvv/. wttp:/iww.rrg/b/////.33 05407.
[31] x. Jin, X. Ren, D. Preotiuc-Pietm and P. Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL https: httLtttt//peeeeeeeeeee http///ppeeeei...e/ wwtpeeewwwwwwweeeeeeeeeeeeeeeeee wttpeeet///eteeeeeeeeeeeeeeeeeee
12

---

[32] K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur. REPAIR: REnormalizing per- muted activations for interpolation repair. The Eleventh International Conference on Learning Representations, 2023. URL https::////... http///pppeeeiieeeeeeeeeeeee wwt.we/ww...mmmem////// wtt..ett//orrmmmaaggu0sssssee d 62ggexX.
[33] T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8082-8090, 2020.
[34] Kirkpatrick, R. Pascanu, Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinske Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 2017. https: 00796.
//arxiv. org/obgg///..o
[35] Krause, M. Stark, Deng. and Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision Workshops (ICML), 2013. https:/ /www. cv www-foonaatiooorrgg..ooo www.or//rr.ccee////oooo/
workshops_aie Bacppps_2133//// 2013/W1 /ttp//tt//aaueeeeeeee /tt////aaue___eeeee httpaaae____oocccceeeereeeeeette IccV_paper html.
[36] Y. LeCun, The mnist database of handwritten digits, 1998. exdb/mnist/.
com/
[37] H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. Thirteenth International Conference the Principles of Knowledge Representation and Reasoning, 2012.
[38] M. Li. S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. Smith, and L. Zettlemoyer. Branch train-merge Embarrassingl parallel training of expert language models, 2022. https: //arxiv. org/ abs / 22082202222 03306.
[39] P. Li, Z. Zhang, P. Yadav, Y.-L. Sung. Y. Cheng, M. Bansal, and T. Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy, 2023.
[40] W. Li, Y. Peng, M. Zhang, L. Ding, H. Hu, and Shen. Deep model fusion: A survey, arXiv preprint arXiv:2309. 15698, 2023.
[41] X. Li, K. Huang, W. Yang. S. Wang, and Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations, 2019.
[42] Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.0754. 2015.
[43] H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems, 35:1950- 1965, 2022.
[44] M.-C. d. Marneffe, M. Simons, and J. Tonhauser. The CommitmentBank: Investigating projection naturally occurring discourse. Proceedings of Sinn und Bedeutung 23, 2019.
[45] M. Matena and C. Raffel. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems (NeurIPS), 2021. attps://arxits org/ / abs/ 2111.098 09832.
[46] B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication--fficiinn learning of deep networks from decentralized data. In Artificial intelligence and statistics, pages 1273-1282. PMLR, 2017.
[47] Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in nat- ural images with unsupervised feature learning. In Advances in Neural Information Pro- cessing Systems (NeurIPS) Workshops, 2011. https: /ttpss://stoarae /ttttttgeee.ee. googl eapi s. com/ pub-tocls-pube cutps-ppuuuuuuullii.eeeepe-eeeee inw-puulicaiionddaae
37648//3377.... pdf.
[48] B. Neyshabur, H. Sedghi, and C. Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
13

---

[49] Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint vrXiv:1910.1459, 2019.
[50] H. Orgad, B. Kawar, and Y. Belinkov. Editing implicit assumptions in text-to-image diffusion models, arXiv preprint arXiv:2303.0800 2023.
[51] G. Ortiz-Jiménez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. NewrIPS, 2023. https: http://cc.s///aaa...rr////////// /0tp../0//////35:122277777
[52] J. Phang. T. Févry, and S. R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088. 2018.
[53] M. T. Pilehvar and J. Camacho-Colladoss WiC: The
dataset for evaluating sensext-eensitive meaning representations. In Proceedings of NAACL-HLT, 2019.
[54] C. Poth, J. Pfeiffer, A. Rücklé, and I. Gurevych. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-1000 10605, Online and Punta Cana, Dominican Republic, Nov. 2021.
[55] Y. Pruksachatkun, Phang, H. Liu, P. M. Htut, X. Zhang, R. Y. Pang, C. Vania, K. Kann, and S. R. Bowman, Intermediate-tase task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online, July 2020.
[56] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021. /ttps://arkee org/ abs/ 2 2000.0002...00....
[57] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Zhou, W. Li, and J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020. http: / www.r.orr..aa..ae..ee..ee.eeeeee httpp//rrr/pppers////20-00000000 html.
[58] C. Raffel, N. M. Shazeer, Roberts, K. Lee, Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXiv, abs/1910.10683, 2020.
[59] A. Ramé, K. Ahuja, J. Zhang, M. Cord, L. Bottou, and D. Lopez-Paz. Model ratatouille: Recy- cling diverse models for -adt---diitiibttion generalization. arXiv preprint arXiv:2212.10445, 2022.
[60] A. Ramé, M. Kirchmeyer, Rahier, Rakotomamonjy. Gallinari, and M. Cord. Diverse weight averaging for out- dut-t--distributtin generalization. ICML, 2023.
[61] M. Roemmele, C. Bejan, and A. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series, 2011.
[62] A. Rogers, O. Kovaleva, M. Downey, and A. Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial
Intelligence, EAA/ 2020, New York, NY, USA, February 7-12, 2020, pages 8722-8731. AAAI Press, 2020. URL. htttps://aat Bactaai.org/ojaa.. wwwp/o/////////o//ooo//oo/sostot http///sss/dexxx/////////o///xxs inw.. php/ http//////aaaaaaa///a///a 6398.
[63] S. Ruder. An overview of gradient descent optimization algorithms. arXiv: Nal:::60900444 2016.
arXiv preprint
[64] K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. Proceedings of the AAAI Conference on Artificial Intelligence, 2020.
14

---

[65] V. Sanh, A. Webson, C. Raffel. S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja. al. Multitask prompted training enables zero-shot task generalization. arXiv preprint ar 10.08207, 2021.
[66] V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations (ICLR), 2021, attts:://axxt org/ / abs/ /212 O. 08207.
[67] M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social iqa: Commonsense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
pages 4463-4473, 2019.
[68] R. Sharma, J. Allen, O. Bakhshandeh, and N. Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 752-757, 2018.
[69] E. Shnarch, A. Halfon, A. Gera, M. Danilevsky, Y. Katsis, L. Choshen, M. S. Cooper, D. Epel- boim, Z. Zhang, D. Wang, et al. Label sleuth: From unlabeled text to a classifier in a few hours. In Conference on Empirical Methods in Natural Language Processing, 2022.
[70] S. P.. Singh and M. Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems, 33:22045-22055 2020.
[71] J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In International Joint Conference on Neural Networks (IJCNN), 2011. https: wwweepeeeeepeeeeeeeeeeeeeeeeeeee feee. org/ comment 6033395.
[72] Y.-L. Sung. L. Li, K. Lin, Z. Gan, M. Bansal, and L. Wang. An empirical study of multimodal model merging. Empirical Methods in Natural Language Processing (Findings), 2023.
[73] O. Tafjord, M. Gardner, K. Lin, and P. Clark. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing SEMNLP-ICCNLA pages 5941-5946, 2019.
[74] N. Tatro, P.-Y. Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems, 13:15300-1531,, 2020.
[75] Y. Tay, M. Dehghani, Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al. U12: Unifying language learning paradigms. In The Eleventh International Conference Learning Representations, 2022.
[76] G. Thimm and E. Fiesler. Evaluating pruning methods. In International Symposium on Artificial Neural Networks, 1995.
[77] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, F.. Kaiser, and I. Polosukhin. Attention is all you need, Advances in Neural Information Processing Systems (NeurlPS), 2017. https: / aawvvv viv. org/ abs/ 1 1706..06000 03762.
[78] A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: multi-task bench- mark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR), 2018. https: : / www.ivivivvvv//ieo/.aaaaaaaaavva .org/ wwwp///./....... 07461.
[79] H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
[80] O. Weller, K. Seppi, and M. Gardner. When to use multi-task learning vs intermediate fine- tuning for pre-trained encoder transfer learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pupers), pages 272-282, Dublin, Ireland, May 2022.
15

---

[81] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et Huggingface's 's transformers: State-of-the -frt---rtt natural language processing. 2019. https: / att////rxxii www.v.vvvrreo.. abs 011///110.00771177 03771.
[82] M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves aceuracy without increasing inference time. In Inter national Conference on Machine Learning (ICML). 2022. https:/ /ttpe///axxvvvvie org/ abs/ 2203. 054
[83] M. Wortsman, G. Ilharco, M. Li, W. Kim, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022. https: / arxiv. org/ abs aww//////.000 01 903.
[84] Xiao, K.
Ehinger, Hays, A. Torralba, and Oliva. Sun database: Exploring large collection of scene categories. International Journal of Computer Vision (IJCV), 2016. https: / http//iiin..iiiinnn..e..
springer.
[85] Yadav and M. Bansal. Exclusive supermask subnetwork training for continual learning.
In Findings of the Association for Computational Linguistics: ACL 2023, pages 569-587, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-aclee ac1.36. URL httph::///// http://aaaaanntoooooooooooa org /2tgg/222..222 wwn.ings-nnnenee acl. 36.
[86] P. Yadav, Q. Sun, H. Ding, X. Li, D. Zhang, M. Tan, P. Bhatia, X. Ma, R. Nallapati, M. K. Ramanathan, M. Bansal, and B. Xiang. Exploring continual learning for code gen- eration models. In Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 2: Short Pupers), pages 782-792, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: https: / htt///aaaaaatttoooaaaeaeaa org/ /ttg//23......oo acl- 68.
URL
[87] Y. Yang. W.-t. Yih, and C. Meek. WikiQA: A challenge dataset for open-domain question answering. Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 201 2018-2018, Lisbon, Portugal, Sept. 2015. Association for Computational Lin- guistics. doi: 10. N0.18653/v1/D15. 1237. URL ht : / ac anthology wrw/D1 5
[88] R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv: /90.000000000 2019.
[89] Y. Zhang and Q. Yang. A survey multi-task learning. JEEE Transactions Knowledge and Data Engineering, 34(12):5586-560, 2021.
[90] Y. Zhang, J. Baldridge, and L. He. PAWS: Paraphrase Adversaries from Word Scrambling, In Proc. of NAACL, 2019.
[91] Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, Zhu, H. Xiong, and Q. He. comprehensive survey on transfer learning. Proceedings of the IEEE, 2020. https: http///avvvv////////////////aavt http://wwwww..o//b//////o/ org/abs/ 1911. 02.00266
16

---

Appendix for TIES-MERGING
A Limitations and Future Works
Our works share the same general limitations of existing merging methods, like (1) a limited theoretical understanding of why and when weight interpolation works, what the important underlying factors, and its proper connections with mode connectivity. Recent works like [50] have demonstrated interesting relationships between weight disentanglement and mergingability of models; (2) that merging relies on common initialization and model architecture; and (3) merging individual-task models to create multitask still lags behind the simultaneous multitask training. Moreover, it is not clear how to select the checkpoints for merging in order to create multitask models useful for specific domains. In addition, while our method provides a way to choose signs when merging task vectors, we still find that using the signs from a multitask model performs better. Some potential future works include figuring out good way to estimate multitask signs without having access to the multitask model as this has the potential to bridge the gap between multitask merging and multitask training (as demonstrated Section 7.4).
B Additional Results
Method
Estimating Sign
Average
Multitask
Samples
Init.
Fine-Tuned
71.4
Multitask
73.1
Averaging [9, 82]
58.0
Task Vectors [29]
63.9
TIES-MERGING
66.4
32
scratch
66.5 [+0.1]
TIES-MERGING
32
mean
67.7 [+1.2]
All
scratch
72.0 [+5.6]
Table 7: Merging Performance can be improved by estimating the Sign Vector by performing
few-shot multitask training. We use the estimated sign as the elected sign and perform merging.
B.1 Enhancing Performance by Estimating the Multitask Sign Vector.
Considering the findings, inquire whether it is possible efficiently acquire multitask sign vectors without extensive multitask training. Our proposal involves utilizing a limited number validation samples from each task to cheaply train multitask model and subsequently derive the relevant sign vector. We create two multitask (IA)² models: one developed from scratch and another initialized using the average of task-specific (IA)² models intended for merging. We use 32 validation examples from each task to train this model. In Table 5, we notice using the sign vector from the fewshot multitask model initialized with mean yielded a performance increase of 3.8% and 1.3% compared to Task Arithmetic and TIES-MERGING, Interestingly, training fewshot multitask training from scratch did not yield significant improvements over TIES-MERGING without sign estimation. We believe that exploring this area further may result in improved merging techniques.
B.2 Effect of Slyper-Parameter À and K on the Performance.
In Figure (left and middle), we plot the effect of on the performance when merging T5-base and T5-large models trained on GLUE (Similar to Table- I). For TIES-MERGING, we vary around the value 1 because TIES takes the mean of task vectors, whereas task arithmetic adds up the task vectors. Hence, a value of for TIES is similar to using #tasks 1 for Task Arithmetic [29]. The range
of 0.8-1.8 for TIES was selected based on preliminary experiments on the PEFT setting (as mentioned in Section 5). We find that WERS-EGGGNG much less sensitive changes in (with an accuracy range of 68-75% across the considered values of λ) compared to Task Arithmetic (with an accuracy
17

---

Merging
TIES Merging
TS-Base T54arge
T5-Large
Ang. Purformaanc va. K(TS-Boao)
70
78
70
60
74
55
AWV
86
1.2
10
Lambda
Lambda
Figure 8: Performance as function of hyperparameters. For more details please refer to the response to our general response.
RIE
WNLI
0.05
20
30 [Fract
40
50
60
70
80
10
30
40
50
60
50
50
30
40
50
60
70
parsretee Serpe.)
(Frec of parimetem Sappe))
I (Frrc of parurmmer kegs.
Figure 9: Sign conflict increases as we trim less parameters. For each task, we merge 10 different checkpoints from hunggingface hub and plot the sign conflict as function of keeping only the top-k% parameters.
Figure 10: Sign Conflict exists even when merging multiple checkpoints for the same task. The first three plots are for RTE, MRPC, WNLI datasets when merging 10 Huggingface checkpoints, while the last one is when merging different tasks (Figure 4 from the main paper).
range of 55-75). Figure 8 (right) demonstrates the effect of & as we increment the value of is in steps of 10 and skip k 0 as that would select parameters. We observe that as & increases the performance drops and then saturates. However, we would like to note that this curve might change based on the distribution of the parameter values in the task vector.
B.3 Sign Conffict Increases as We Trim Less Parameters
checkpoints from huggingface finetuned on for three different glue tasks (RTE, MRPC, and WNLI) and plot the sign conflict as function of k. As we keep more and more parameters, the sign conflict increases and reaches almost 80%. This is also expected as there are many more nonzero parameters that can create conflict even if their magnitude is small.
In Figure 9, we merge 10 beer--base-uncall
B.4 Sign Conflicts Exists Between Different Checkpoints for the Same Task
In Figure 10, we show that sign conflicts exist even within models trained on the same task. We plotted the sign conflict (similar to Figure 4) between the 10 checkpoints of RTE, MRPC, and WNLI from Huggingface. As the number of checkpoints increases, sign conflict increases. We also compare this with the sign interference when merging different task checkpoints and find a similar degree of interference in all of these cases. Hence, sign conflicts exist even within models trained on the same dataset. We suspect that this is because models are highly overparameterized and hence there are multiple subnetworks (subsets of parameters) that could lead to the same performance where different finetuning runs update the same parameters in different directions.
18

---

Mean
Elect + Dis. Mean
TIES
0.25
Mean
Trim Dis. Mean
TIES
Mean magnitude
Mean magnitude
0.2-
0.15
0.15
0.1
0.05
0.05
>1
Parameters with nonzero values
0.5-0.6 0.6-0.7 0.7-0.8 0.8-0.9 0.9-1.0 Bins of Sign Agreement
(a) Redundant Parameter Interference for (IA)* with
STD.
(b) Sign Interference for (IA)² with STD.
Figure 11: Effect of different types of Merging on the Magnitudes of the Parameters. Here we additionally compare with TIES-MERGING and also provide the standard deviation of parameter values. A high std implies that there is some diversity in magnitude values across different parameters.
Mean
Plect D.s. Mean
Moan magnitude
Thac Parama with Corfict
Mean magritude
Mean Tri Dis.
TIES
0.006
0.2 3..15
0.08
0.004
0.1 0.05
0.002
0.02
Modes
0.5-0.5 0.6-0.7 0.7-0.8 0.8-0.9 0.9-1.0
(a) Fraction of Parameters with Sign conflicts for T5-Base model
Parameters
Sign Agreement
(b) Redundant Parameter Interfer-
(c) Sign Interference for T5-Base model with STD.
versus number of models.
ence for T5-Base with STD.
Figure 12: Plots for TS-Base model.
Method
Validation
Average
winogrande
cloze
anli-rl
anli-r2
anli-r3
Zeroshot
40.5
Fine-Tuned
Multitask scrateh)
73.1
94.]
70.5
49.8
47.7
Multitask (32, scratch)
60.9
74.
792
59.3
892
63.9
80.9
39.5
91.6
49.8
41.9
40.1
Multitusk (32, mean)
65.2
79.8
83.3
61.6
$1.2
66.7
85.3
41.1
94.4
$8.1
16,0
16,5
Averaging
58
$1.2
58.3
$3.8
$5.2
53.5
80.9
40.1
92.5
43.3
39.2
40.2
Tisk Arilhmetic
59.2
76.5
79.2
S7.7
$1.6
$1.4
66.2
31.4
81.5
$9.8
47.5
18.2
TIES-MERGING
Merging
RegMean
43.3
39.2
40.2
Task Arithmetic
63.9
74.
83.3
62.8
49.1
49.3
87.5
41.5
95.3
60.8
49.4
50.0
TIES-MERGING
66.4
78.0
83.3
67.9
57.6
59.7
81.7
42.8
90.3
66.9
51.3
51.1
Table Test set performance when merging IA3 models on eleven tasks. Please refer to Section for experimental details.
B.5 Detailed Results for Types of Interference and Their Effect on Merging
In Section 7.1 and Figure 6, we showed the effect of redundant parameters and sign conflicts on parameter magnitudes when comparing simple averaging vs disjoint mean after either trimming or electing and showed that performing these operations helps with the parameter magnitudes. In Figure 11, we additionally compare with MERGINGGG and show that performing both trimming and electing usually results in higher magnitude and also higher standard deviation in parameter magnitudes. Higher std denotes that all parameter values the merged model are the same and that there is significant variation in the magnitude which is in contrast to simple averaging as it decreases the magnitude of not redundant parameters and reduces the magnitude of the influential parameters in the merged model. Similar plots for the T5-base model are provided in Figure 12.
19

---

B.6 Comprehensive Task-Level Results
We provide the task level for all the in-domain evaluation experiments in the main Table 1. Ta- ble 8, 9, 10, 1, and 12 provide the task level results IA3 [43], T5-Base, T5-Large [58]. ViT-B/32, and ViT-L/14 [14] respectively. The task level results of the out-of-domain experiments for T5-Base and LarLarge can be found in Table 13, and 14, Lastly, Figure 13, shows the scaling of the T5-Base model we merge different numbers of tasks.
Method
Vailidation
Average
paws
qasc
quartz
story_che wiki qa
winogrande
wsc
Zeroshot
53.5
49.9
35.8
53.3
48.1
76.2
S0
61.1
Fine-tuned
94.3
98.3
95.5
64.1
62.5
Multitask
83.6
94
97.9
82.5
86.7
95
64.1
65.3
Averaging
65.9
66.4
82.6
49.5
94.1
50.4
58.3
Task Arithmetic
X
73.9
73.3
93.5
68.2
76.5
93.7
55.5
56,9
TIESMERGING
X
69.7
74
83.3
70.3
64,2
84.7
55.9
55.6
Fisher Merging
V
68.9
69.3
85.7
63.6
56.4
93.8
50.9
62.5
RegMean
V
71.2
76.8
96.2
62.5
55
94.8
51.9
61.1
Task Arithmetic
73.2
73.4
93.3
67.1
71.7
94.1
52.9
59.7
TIES-MERGING
73.9
79.3
88.6
71.8
72.9
82.5
61.3
61.1
Table 9: Test set performance when merging T5-base models on seven tasks. Please refer to Section for experimental details.
Method
Validation
Average
paws
qase
quartz
story_clozz
wiki_qa
winogrande
wse
Zeroshot
51.7
55.4
14.3
54.1
54.1
71
49.3
63.9
Fine-tumed
88.8
94.4
98.9
87.8
90.8
96
74.7
79.2
Multitask
88.1
94.2
98.5
89.3
92
95.4
73.5
73.6
Averaging
X
59.6
61.3
82.6
70.5
53.7
63.2
49.7
36.1
Task Arithmetic
X
73.5
79.2
96.8
80.2
83.6
58.6
60.2
55.6
MEE-MMEGGING
X
74.4
80.5
96.2
81.8
78.6
62
61.9
59.7
Fisher Merging
64.6
60.4
81.7
75
60.1
88.6
50
36.1
RegMean
73.2
96.9
80.7
78.6
82.6
51.8
36.1
Task Arithmetic TIES-MERGING
73.3
77.8
96
78.6
86.4
59.1
62.3
52.8
76.9
81.5
96.2
80.1
83.6
64.9
66.5
65.3
Table 10: Test set performance when merging Ta-large models on seven tasks. Please refer to Section 6 for experimental details.
Method
Validation
Average
SUN397
Cars
RESISC45
EuroSAT
SVHN
GTSRB
MNIST
DTD
Individual
90.5
75.3
73.7
96.1
99.7
97.5
98.7
99.7
79.4
Multitask
88.9
74.4
77.9
98.2
98.9
99.5
93.9
72.9
95.8
Averaging
65.3
71.4
71.7
64.2
52.8
50.1
Task Arithmetic
X
60.4
36.7
41
53.8
64.4
80.6
66
98.1
42.5
TTRS-MERGING
X
72.4
59.8
58.6
70.7
79.7
86.2
72.1
98.3
54.2
Merging
72.9
51.1
59.9
RegMean
71.8
65.3
63,5
75.6
78.6
78.1
67.4
93.7
52
Task Arithmetic
70.1
63.8
62.1
72
77.6
74.4
65.1
94
$2.2
TIES-MERGING
73.6
64.8
62.9
74.3
78.9
83.1
71.4
97.6
56.2
Table 11:
Test set performance when merging ViT-B/32 models eight tasks. Section for experimental details.
Please refer to
C Implementation Details
C.1 Compute Resources Used and Runtimes
executed our experiments on Nvidia A6000 GPUs equipped with 48GB RAM. Single-task (IA)3 models for eleven tasks required 1-2 hours per model, while the multitask vector took around 24 hours on four GPUs. The T5-Base and TS-Large models, based on dataset size, needed between 15 minutes and 2 hours per task, and approximately cight hours for the multitask checkpoint. Vision
20

---

Method
Validation
Average
SUN399
Cars
RESISC45
EuroSAT
SWHN
GTSRB
MNIST
DID
Fine-tuned
82.3
92.4
100
98.1
99.2
99.7
84.1
Multitask
93.5
90.6
84.4
99.2
99.1
99.6
96.3
80.8
97.6
Averaging
X
79.6
72.1
81.6
82.6
91.9
78.2
70.7
97.1
62.8
Task Arithmetic
83.3
72.5
79.2
84,5
90.6
89.2
86.5
64.3
X
86
76.5
85
89.3
95.7
90.3
83,3
99
68.8
Fisher Merging
82.2
69.2
88.6
87.5
93.5
80.6
74.8
93.3
70
RegMean
83.7
73.3
86.1
97
88
84.2
98.5
60.8
Task Arithmetic
84.5
74.1
82.1
86.7
93.8
87.9
86.8
98.9
65.6
GIES-MERGING
76.5
89.4
95.9
90.3
83.3
68.8
Table 12: Test performance when merging ViT-L/14 models on eight tasks. 6 for experimental details.
Please refer to Section
models ViT-B/32 and ViT-L/14 were utilized, as supplied by Ilharco al. [29]. Merge experiments were efficient, with evaluations consuming less than 2 minutes for the T5-Base, T5-Large, ViT- B/32, and ViT-L/14 experiments. The assessment of (IA)² models, due the necessity of using multiple templates from prompt sources and median result calculations across all templates, required approximately one hour per dataset evaluation.
Model
Average
cosmos qa
social
iqa
quail
wic
copa
h-swag
PAWS
35.9
18.8
25
24.8
68.8
56.2
21.9
QASC
34.9
15.6
21.9
25.1
75
53.1
18.8
QUARTZ
37.4
31.2
18.8
24.3
71.9
59.4
18.8
Story Cloze
35
6.2
25
25.6
75
65.6
12.5
Wiki QA
24.5
18.8
21.9
24.9
28.1
43.8
9.4
Winogrande
28.3
18.8
25
25.7
34.4
43.8
21.9
WSC
31.7
21.9
21.9
24.6
62.5
46.9
12.5
Pretrained
31.1
21.9
18.8
24.1
65.6
43.8
12.5
Averaging
31.7
21.9
21.9
24.6
68.8
37.5
15.6
Fisher Merging
33.8
15.6
21.9
24.9
65.6
53.1
21.9
Task Arithmetic
31.9
15.6
31.2
25.7
28.1
68.8
21.9
RegMean
34.3
23.1
28.1
24.9
48.4
62.5
18.8
TIES-MERGING
35.3
21.9
25
25.7
50
65.6
23.8
Table 13: Out-of-Distributto performance of T5-Base model checkpoints on six tasks. Please refer Section 6 for experimental details.
C.2 Employed Datasets and Associated Licences
We use the following datasets in the paper with the following licenses. ANLI [49], WiC [53]. WSC [37], and Story Cloze [68], QuaRTz [73], Cars [35], GTSRB [71] are under Creative Commons License. Winogrande [64]. QASC [33] are under Apache license. COPA [61] is under BSD-2 Clause license. H-SWAG [88], EuroSAT [24]. is under MIT Licence. MNIST [36] is under Gnu General Public License. We could not find the licences of DTD [10], RESISC45 [8], SUN397 [84],
SVHN [47], CB [44], RTE [1]]), and PAWS [90] but they are publically for research use.
C.3 Details of the Motivation Experiments
For both Figure 3, and 4 in Section 3, we perform experiment on the eleven (IA)3 models used in our PEFT merging experiments ($ 6). For Figure similar Fig. demonstrating the fraction of parameters with sign conflict for T5- base model, please refer to Fig. 12a.
21

---

Model
Average
cosmos qa
social
iqu
quail
wic
copa
h-swag
PAWS
32.3
25
28.1
25.6
56.2
46.9
12.5
QASC
33.4
21.9
28.1
25.5
43.8
62.5
18.8
QUARTZ
28.7
25
25
25.1
25
53.1
18.8
Story Cloze
32.1
21.9
34.4
26.8
46.9
53.1
9,4
Wiki QA
27.1
25
28.1
25.2
28.1
46.9
9,4
Winogrande
32.4
31.2
18.8
25.6
43.8
62.5
12.5
WSC
29.7
25
25
25.1
37.5
56.2
9.4
Pretrained
27.6
21.9
21.9
24.9
28.1
56.2
12.5
Averaging
30.4
31.2
25
26.3
31.2
59.4
9.4
Fisher Merging
32
34.4
25
26.1
40.6
56.2
9.4
Task Arithmetic
33.3
21.9
34.4
24.6
40.6
59.4
18.8
RegMean
36
34.4
28.1
25.3
62.5
50
15.6
TIES-MERGING
40.4
31.2
43.8
26.6
59.4
59.4
21.9
Table 14: Out-of-Distributon performance of T5-Large model checkpoints on six tasks. Please refer to Section 6 for experimental details.
Average Normalized Perf.
Simple
Task Arithmetic
0.9
0.8
0.7
Number of Tasks
Figure 13:
T5-Base with increasing number of task being merged. Average performance when merging a different number of tasks.
C.4 Merging in the absence the Validation Set
In our investigation into scenarios where a validation set is not available, we have devised a recipe and identified the optimal hyperparameters, employing the PEFT experimental procedure detailed in Section 6. This approach was applied to the eleven task-specific models presented in the same section, utilizing the TIES- MERGING method for tuning the hyperparameters. Our preliminary estimates for the hyperparameters were & = 20 and À close to 1. The hyperparameter search was conducted using the eleven task sseciccc (IA)3 models, with & {10, 20, 30), and spanning from 0.8 to 3.0. in increments of 0.1. The results of this comprehensive search indicated an optimal value of 20, with values of À 0.9, 1.0, and À 1.1 demonstrating equivalent performance. To maintain simplicity in our model, we chose a À value of 1. Thus, the final selection of parameters for TIES-MERGING
20, signs based on mass, the disjoint mean, and a value of
C.5 Merging Different Number of Tasks
Here provide some additional details the experiments when merging different numbers of tasks. In Fig. 5, we perform the experiment with T5-Large when merging the seven tasks considered in Tab. and described in Sec. 6. The x-axis shows the different number of tasks being merged. Note that when merging 7 tasks, we have total of (7) combinations. However, in our experiment, sample at most 10 distinet combinations for each value of T. similar plot for the T5- -5-Base model shown in Fig. 13.
22

---

In Figure 5, for each number of tasks we take at most 10 random subsets of the 8 tasks we were considering. The solid line the average the merged model's performance from these different runs. Below we provide the optimal X values for the different subsets of tasks we merged for both TIES-MERGING and Task Arithmetic, note that for averaging the #tasks always. Fach entry in the list is the optimal for a particular subset of tasks selected on the validation set.
(2 tasks) TIES + [1.7, 1.9, 2, 2, 1.1, 1,5, 1,6, 1.8, 1.9, 11
(2 tasks) Task Arithmetic [1, 0.9, 1, 1, 0.9, 1, 0.9, 0.9, 0.9, 1]
(3 tasks) TIES + [1.2, 2, 1.5, 1.9, 1.8, 1.7, 1,4, 2, 1.9]
(3 tasks) Task Arithmetic [1, 0.7, 0.,,,0.,,,1,1, 0.7, 0.7, 0.9, 1]
(4 tasks) TIES +
2.3, 18,
(4 tasks) Task Arithmetic + (0,8 0.7, 0.7, 0.8, 0.6, 0.7]
(5 tasks) TIES- [2, 1.7, 2.6,
(5 tasks)Task Arithmetic - [0.7, 0.8, 0.6, 0.8, 0.7, 0.6, 0.6, 0.6, 0.6, 0.7]
(6 tasks) TIES [1.6, 1.7, 17,
(6 tasks) Task Arithmetic - [0.6, 0.5, 0.5, 0.5,0.5,00.. 0.6]
(7 tasks) TIES [1.7]
(7 tasks) Task Arithmetic + [0.5]
C.6 Training Details
In our research, we utilized two variants of the T5 model, specifically the T5-base and T5-large models, which were trained to a maximum of 75,000 steps. An effective training batch size of 1024 was implemented, alongside learning rate (Ir) of 0.0001. We instituted an early stopping mechanism with patience threshold of to prevent overfitting. During the training process, bfloat16 was adopted to curtail GPU memory expenditure, and the maximum sequence length was set at 128. In contrast, for the PEFT configuration of the (IA)² approach on the TO-3B model, we modified our parameters. An effective training batch size of 16 was deployed along with an evaluation batch size of 32, while maintaining the learning rate at 0.0001. To accommodate the model's complexity, the early stopping patience was augmented to 10. We do not use any Ir scheduler and weight decay for any of our model training.
For the purpose of evaluation, we perform rank classification. In this method, the model's log probabilities for all potential label strings are ranked. The model's prediction is deemed accurate the choice ranked highest aligns with the correct answer. It should be noted that rank classification evaluation can accommodate both classification tasks and multiple-choice tasks.
23


Reviews:
[{"review_id": "b6tDdTMNex", "reviewer": "Reviewer_rBCc", "summary": "This paper proposes to resolve the interference of model merging, a solution to combine multiple task-specific models into a single multitask model. It demonstrates two major sources of interference, including redundant parameter values and sign conflict and proposes solutions to resolve the interference. ", "questions": "Refer to questions.", "limitations": "The authors do NOT adequately address the limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper demonstrates some interesting research insights and the solutions are simple and clear.", "weaknesses": "1: I question the necessity of Section 7.3.  According to the network pruning literature, a high magnitude is always an implication of importance/sensitivity, and a widely used pruning metric. Flipping the sign of or pruning away the Top-k% parameters can cause significant performance drop is well-known to the literature. It seems to have weak connection with this paper’s main argument. \n\n2: I did not find the discussions regarding limitations and future work.\n\n3: More analysis on the experimental results are required. For example, what are the effects of the proposed on each task respectively? Why do some tasks perform better than others?  \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Weakness 1:** Necessity of Section 7.3 and pruning away the Top-k% parameters can cause a significant performance drop is well-known in the literature.\n\n**Answer:** As mentioned in L166-168, we would like to clarify that we are pruning the task vectors (i.e. the difference between the fine-tuned model and the pre-trained model), not the model's parameters. Prior work in the pruning literature mostly focused on pruning model parameters. Furthermore, we provide experiments and analysis to provide more intuition and motivation behind pruning task vectors. For example, Figure 3 shows that deleting the bottom 80% of the values from the task vector doesn’t affect performance while Figure 7 shows that the signs of the remaining top 20% of the parameters in the task vector are very crucial.\n___\n**Weakness 2:** Limitations and future works\n\n**Answer:** Please refer to our general response Question 3.\n___\n**Weakness 3:** What are the effects of the proposed on each task respectively? Why do some tasks perform better than others?\n\n**Answer:** We provide per-task results in Appendix Tables 6-12. For the effect of merging on individual tasks, we hypothesize that the improvements over other methods are in cases where there is significant interference between tasks. However, when merging multiple tasks it is harder to predict the effect on individual tasks and would be a good research question for future study.\n"}, {"review_id": "n57idIEkAo", "reviewer": "Reviewer_Rfna", "summary": "Merge mutiple **fine-tuned** neural networks, each from an unique task (dataset), into one neural network by weight averaging. Denate the weights of pretrained network as $\\theta_0$, fine-tuned network as $\\theta_{\\tau}$, the weight updating direction $\\tau = \\theta_{\\tau} - \\theta_0$ \n\nThe paper introduces two tricks 1) for each $\\tau$ only keep the top $k$% elements in $\\tau$ according to their magnitude. (keep the others as zero), 2) during the weight averaging, only retain the dominate direction (+1 or -1) for each elements. \n\nThe experiments shows the improvements of the proposed Ties-Merging method over other weight averaging methods, such as naive weight averaging, task arithmetic, and regmean. But Ties-Merging is still significantly lagging behind MultiTask training.\n\n", "questions": "- Simple averaging is $1/n \\sum_{\\tau} \\theta_{\\tau}  = \\theta_0 + 1/n \\sum_{\\tau}\\tau$\n- Task Arithmetic  is $ \\theta_0 + \\lambda \\sum_{\\tau}\\tau$\n\nWhat is the parameter $\\lambda$ of task arithmetic in Figure 5? If it is $0.4$ as line 210, why there is a big gap between simple averaging and task arithmetic in figure 5, ( given that $1/2=0.5, 1/3=0.33$)", "limitations": "no negative societal impact. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The writing is clear and easy to read. \n- The introduced two tricks are resonable and easy to understand. \n- Comparing with naive weight averaging, task arithmetic, Fisher averaging, and regmean, the proposed ties-merging performs very well. \n\n", "weaknesses": "- Even though the proposed Weight Averaging method out-perform other weight averaging methods, it still legs behind MultiTask training. \nFurthermore, considering both the training cost and inference time, it is hard to see the benefit of the proposed method over mutitask training. As to other baseline methods, such as naive weight averaging, Fisher averaging and regmean, they are designed to solve IID (one fine-tuning task instead of mutiple different tasks) not OOD. These two facts reduce the strongness of the experiments.  (But it is fine. Because the two introduced tricks are good enough). \n\n\n\n\n\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Weakness 1 (Part 1):** Method still legs behind MultiTask training.\n\n**Answer:** While we agree that the gap between TIES merging and multitask training is an important limitation, we note that multitask learning requires simultaneous training on all the training data at once. In contrast, when merging models, we only require access to individual-task models and do not require additional training. These individual task models can be trained asynchronously and recycled to create many different merged models, and are frequently shared e.g. on the Hugging Face Model Hub (where hundreds of thousands of fine-tuned models are available). Our methodological contribution identifies an important aspect of interference when merging task-specific models and shows notable improvements over existing methods.\n___\n**Weakness 1 (Part 2):** The training cost and inference time make it hard to see the benefit of the proposed method over multitask training.\n\n**Answer:** We would like to clarify that our method does not require any additional training. In a real-world setting, most of these models would be trained by different people for their use cases and then uploaded to a common hub for public use. Moreover, the inference cost of the merged model is the same as that of the original model. Hence, there is no additional inference cost either.\n___\n**Weakness 1 (Part 3):** Other baseline methods, such as naive weight averaging, Fisher averaging, and regmean, are designed to solve IID (one fine-tuning task instead of multiple different tasks), not OOD. \n\n**Answer:** RegMean was specifically designed for merging different tasks via their objective, and the RegMean paper includes experiments of merging different tasks in section 5.1. Similarly, weight averaging is also used to merge different tasks for better initialization [1]. Moreover, in PDF Table 1, we have provided additional experiments where we show that TIES merging outperforms other methods even when merging multiple checkpoints from the same task. Hence, even in the \"IID\" setting TIES outperforms other methods. For more details please refer to our response to reviewer ioFQ.\n___\n\n**Question 1:** What is the of the parameter \\lambda of task arithmetic in Figure 5? If it is 0.4 as line 210 …\n\n**Answer:** The value of $\\lambda=0.4$ is only used for the experiments without a validation set in Table 1 (three rows); in all other experiments the value of $\\lambda$ is tuned on a validation set. In Figure 5, for each subset of tasks that are merged, we select the best $\\lambda$ on the validation set. Hence, it is not expected for averaging and Task Arithmetic to be close to each other and we see a difference in performances. \n\n[1] Fusing finetuned models for better pretraining, Choshen et. al. 2022\n"}, {"review_id": "qjp7W35MVC", "reviewer": "Reviewer_b9mw", "summary": "This paper delves into the challenge of integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. The authors identified that current methodologies overlook the interference that occurs between parameters of different models. This interference can be attributed to two primary sources: redundant parameter values and conflicts in the sign of a given parameter's values across various models. To counteract these issues, the authors introduced a novel method, TIES, which incorporates three key steps: (1) resetting parameters that underwent minimal changes during fine-tuning, (2) resolving conflicts in sign, and (3) merging only those parameters that align with the final agreed-upon sign. The proposed method demonstrated SOTA performance across different settings. ", "questions": "I don't have further questions for now and I look forward to the authors' enhancements on the weaknesses mentioned above.  ", "limitations": "The author did not mention the limitations of their method and potential future work. I have listed some suggestions for improvement in the weaknesses section. ", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper provides a comprehensive analysis of the sources of interference in existing model merging methods, specifically pinpointing redundancies in model parameters and disagreements between parameter signs. This thorough examination underpins the motivation for the proposed method. \n- The proposed method demonstrates robust performance across a wide array of conditions, including various modalities, domains, task quantities, model sizes, architectures, and fine-tuning settings. This versatility underscores the method's adaptability and broad potential for application\n- The informative and insightful Section 7, which delves into the significance of different components, particularly the estimation of correct signs during the merge process, serves as a valuable resource.", "weaknesses": "- The proposed method is quite heuristic and it would make the paper stronger if the authors can provide more theoretical analysis of the proposed method. \n-  The author did not mention the limitations of their method and potential future work. The authors could further discuss them, which will provide a more balanced view of the method and give readers an idea of the potential directions for future research.\n- The paper could be improved by discussing more real-world applications of the studied problem, integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. Why is this problem important? This would help to demonstrate the practical value of the method and its potential impact in real-world scenarios.\n-  It would be beneficial to include a sensitivity analysis of the hyperparameters used in the method. Although the authors provided a generic recipe of TIES with fixed hyperparameters, it would be good to see how sensitive the method is to different hyperparameters.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Weakness 1:** Theoretical Justification\n\n**Answer:** Our work is supported by several past works that shed light on different components of our method. We discuss each of them below and will add a similar discussion to the updated paper. \n\n**Why Merging Works:** As mentioned in L91-102, model merging and weight interpolation work because models finetuned from the same pre-trained initialization demonstrate mode connectivity [1] which implies that the finetuned checkpoints lie in an approximately convex loss basin in parameter space [2]. The theoretical underpinnings of mode connectivity, therefore, support the merging procedure we develop.\n\n**Existence of Sparse subnetworks:** Past works [3,4] have shown that during model finetuning, most information is encoded in a small and sparse subnetwork inside the full model. Achille et al. [5] describe two phases of training where the first phase discovers the important connections and their topology between layers and the second phase fine-tunes this relatively fixed pattern. In our work, we find that the top-k% parameters form a subnetwork that is responsible for most of the performance and only considers merging them to avoid unnecessary interference from other parameters.\n\n**Relevance of Signs:** In addition, the Lottery Ticket Hypothesis [3] finds subnetworks (winning tickets) by magnitude pruning and shows that by re-training them with static sparsity starting from the initial weights, they reach similar or higher accuracy. However, they demonstrated that random initialization, with the same structure, does not suffice. This implies that the values of the weight are crucial for good performance.  Zhou et al. [4] build upon this and show that one may not need the exact weights at initialization to train these subnetworks (lottery tickets) but the signs may be sufficient. We also find that during merging signs are critical for model performance (see the main paper, Figure 7) and hence we explicitly focus on resolving the sign conflict when merging models.\n___\n**Weakness 2:** Limitation and Future Work\n\n**Answer:** Please refer to the general response Question 3.\n___\n**Weakness 3:** Discussing more real-world applications of the studied problem. Why is this problem important?\n\n**Answer:** As mentioned in L24-28, there are thousands of FT checkpoints released on Huggingface Hub every day and model merging allows us to reuse these checkpoints to create other models with desired properties: (1) Merging provides a cheap way to create and obtain models with specific abilities by merging multiple models. (2) Merging can also be useful in settings where the user cannot explicitly share the data, hence the user cannot share the data but share the model weights which can then be combined from multiple users to create a better model. (3) A process of merging multiple models together also allows for the collaborative development of open-source models where models are created and patched with specific abilities.\n___\n**Weakness 4:** Sensitivity analysis of the hyperparameters used in the method\n\n**Answer:** Please refer to our general response and Figure 1 in the Rebuttal PDF.\n___\n[1] Linear Mode Connectivity and the Lottery Ticket Hypothesis\n\n[2] Loss surfaces, mode connectivity, and fast ensembling of dnns.\n\n[3] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks\n\n[4] Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask\n\n[5] Critical Learning Periods in Deep Networks\n"}, {"review_id": "SNvhDW6pTp", "reviewer": "Reviewer_ioFQ", "summary": "The paper presents a novel method, TIES-MERGING, to merge models in the weight space for multitask learning. It observes an interference problem when linearly interpolating weights, and proposes a simple yet effective two-step solution: parameter trimming for small changes during fine-tuning and sign conflict resolution. The experiments in multitask shows that TIES improves performances, making it a notable (experimental) contribution to the literature of model merging.", "questions": "* What is the impact of task similarity on the number of sign conflicts? Does the number of sign conflicts decrease when two models are fine-tuned on a shared task?\n* Can you visualize the number of sign conflicts for different trimming ratios?\n* Can TIES improves downstream transfer learning performances?\n* Could you plot the curve performance while varying the coefficients given to different tasks: see Pareto curves in \"Patching open-vocabulary models by interpolating weight\", but also in \"Pareto manifold learning: tackling multiple tasks via ensembles of single-task models\", a missing yet important related work.\n* Could you enrich the ablation study to further clarify/highlight the different impacts of the key contributions.\n* How does ensembling of predictions behave in comparison with weight interpolation?", "limitations": "The authors have highlighted in Sections 7.3 and 7.4 that a key limitation is to accurately elect the sign.\nYet, the paper would benefit from a dedicated limitation section. It should ideally include that:\n- weight interpolation lacks proper theoretical understanding,\n- their approach is only (verified) for multitask learning,\n- the averageability relies on a large pretraining and \"good\" hyperparams,\n- TIES may be less impactful in a \"sequential patching\" scenario,\n- they still lag behind the simultaneous multitask learning.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The paper's main strengths lie in its simplicity, as highlighted by its clear description and illustration.\n* The paper successfully builds upon the \"task arithmetic\" task vector approach to report a new interference phenomenon, and then enhances performance with simple yet important modifications. Model merging is an important topic in multitask, and this paper fills a crucial gap in the current literature.\n* The experimental framework is robust, with a focus on significant large-scale tasks across CV and NLP domains using recent architectures and fair benchmarks.", "weaknesses": "Despite its strengths, some areas require attention.\n* The contributions, though valuable, are incremental, and the observed gains in multitask learning are consistent but arguably marginal, and the trimming/sign play only a marginal role in this gain.\n* The experiments focus solely on models trained on different tasks. Yet, weight averaging is also useful to merge models trained on a single target task; on this model soups setup, I speculate that sign interference is less an issue, and that TIES would actually be detrimental as it would increase variance, thus limiting the benefits from combining multiple models, in particular under distribution shifts. As a minimum fix, the title should reflect this specificity, as the current one does not adequately reflect this scope limitation. A (naive) suggestion could be \"Resolving Multitask Interference When Merging Models\".\n* Even whithin the multitask challenge, the experiments do not cover two important scenarios. First, multitask as better pretraining before fine-tuning on a target task (as in \"Fusing finetuned models for better pretraining\"). Second, multitask in the sequential patching scenario (as in \"Patching open-vocabulary models by interpolating weight\").\n* Lack of analysis and clarity of the interpolating coefficient, for TIES and for the baselines. Specifically, without validation samples, could you clarify which values $\\lambda$ is used: it seems that it's $\\lambda=1/|num tasks|$ for weight averaging, $\\lambda=0.4$ for task arithmetic (could you please point where you found this value), but $\\lambda=1$ for TIES. Therefore, is the difference between task arithmetic/weight averaging in Table 1 simply due to the use of different $\\lambda$? Thus (as further suggested from Table 3), scaling is an important factor, these different values of $\\lambda$ hidden the true impact of your core contributions.\n* Similarly, the ablation study in Table 3 could be made clearer. It is not clear whether the ablations are done one at a time or sequentially all together. If the latter is true, then why do we recover 65.5 on T5-base (when all 4 components are removed), while weight averaging performs 65.9? Moreover, what does it mean to remove \"elect\" while keeping \"disjoint mean\"? what does it mean to remove \"scaling\" (is it $\\lambda=1/|num tasks|$ or something different)?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Weakenss 1:** The contributions, though valuable are incremental.\n\n**Answer:** We note that the improvements of TIES-Merging over Task Arithmetic are precisely due to the role of trimming values and electing signs when merging since there are no other differences between the methods. Moreover, the improvements reported in Table 1 (with validation) range from 0.7% to 3.6% over 11 tasks for IA3, 7 tasks for T5, and 8 tasks for vision experiments. Across all tasks that are averaged over, the improvement for IA3 is 27.5%, 4.9% for T5-base, 25.2% for T5-large, 14.4% for ViT-B/32, and 12% for ViT-L/14 across all tasks. Hence, we submit that these improvements are not marginal.\n___\n**Weakness 2 (Part 1), Question 3, and 6:** \"Model soups\" experiments, merging multiple checkpoints from the same task, comparison with ensembling.\n\n**Answer:** We performed additional experiments to merge multiple checkpoints trained on the same task (as done in Model Soups) and also compared with ensembling. We use the experimental setting and code from \"Merging Models with Fisher-Weighted Averaging\", i.e. merging differently fine-tuned BERT models trained on RTE, MRPC, and WNLI. For each of these datasets, we select 10 fine-tuned checkpoints based on bert-base-uncased from the Hugging Face Hub and then merge them using different methods and report the results below.\n\n||RTE|MRPC|WNLI|\n|-|-|-|-|\n|Averaging|59.93|78.19|56.34|\n|Fisher|65.7|81.37|52.11|\n|Ensemble|70.76|86.03|45.07|\n|Task Arithmetic|71.84|86.03|59.15|\n|TIES|72.2|86.76|58.75|\n\nFrom the results presented above (also presented as Table 1 in Rebuttal PDF), we observe that TIES merging works the best in all cases except WNLI, where it only slightly underperforms Task Vectors. Notably, TIES merging provides a dramatic boost over both Fisher Merging and averaging, and **outperforms ensembling in all cases**.\n___\n**Weakness 2 (Part 2) and Question 1:** Sign interference is less of an issue for models trained on a single task.\n\n**Answer:** In rebuttal PDF Figure 2, we show that sign conflicts exist even within models trained on the same task. We plotted the sign conflict between the 10 checkpoints of RTE, MRPC, and WNLI from HF (mentioned above). As the number of checkpoints increases, sign conflict increases. We also compare this with the sign interference when merging different task checkpoints and find a similar degree of interference in all of these cases. Hence, sign conflicts exist even within models trained on the same dataset. We suspect that this is because models are highly overparameterized and hence there are multiple subnetworks (subsets of parameters) that could lead to the same performance where different finetuning runs update the same parameters in different directions.\n___\n**Weakness 3:** multitask as better pretraining before fine-tuning on a target task. \n\n**Answer:** Based on your suggestion, we performed additional experiments following the setting from \"Fusing Finetuned Models for Better Pretraining\", specifically merging the same three GLUE tasks from the previous experiment (RTE, MRPC, WNLI). We take finetuned bert-base-uncased model checkpoints for 8 GLUE tasks (wnli, sst2, rte, qnli, mrpc, cola, mnli, qqp) from HF. When finetuning on a given task, we merge all the other seven checkpoints together (apart from the chosen task) and use that as an initialization for fine-tuning on the chosen task. The results are presented in rebuttal PDF Table 2 and copied below. We find that TIES merging works well in this setting and outperforms all other merging methods by a significant margin (apart from Averaging for WNLI).\n\n|Init Method|RTE|MRPC|WNLI|\n|-|-|-|-|\n|PTM Init|66.42|81.86|56.33|\n|Average|75.81|86.51|56.33|\n|Task Arithmetic|78.33|86.27|50.7|\n|TIES|80.14|87.99|54.92|\n___\n**Weakness 4:** Lack of analysis and clarity of the interpolating coefficient….\n\n**Answer:** Please refer to our general response on how the hyperparameters were selected and their effect on performance. In addition, we note that scaling is indeed an important factor for both Task Vectors and TIES merging because after merging the scale needs to be recalibrated. However, in all the experiments (apart from Table 1, three rows without validation set) for both the methods we fairly tune the $\\lambda$ as a hyperparameter and use the best value. Please refer to the general response for other details. \n___\n**Weakness 5 and Question 5:** Similarly, the ablation study in Table 3 could be made clearer.\n\n**Answer:** As mentioned in L311-314, the ablations are done by removing one component at a time from the full TIES merging method. Removing elect while keeping the disjoint mean refers to taking the mean of values with signs +1 and -1 but not including the 0 values of the trimmed task vectors in the mean. Removing disjoint mean but trimming and electing refers to taking the mean of the values with the elected signs and the 0 for the trimmed values. Removing scaling means using $\\lambda=1$. Removing components one at a time allows us to quantify the effect of each component. If you have any other suggestions for ablations, please feel free to let us know and we will add it to the paper.\n___\n**Question 2:** Can you visualize the number of sign conflicts for different trimming ratios?\n\n**Answer:** In rebuttal PDF Figure 3, we merge the 10 bert-base-uncased models finetuned on for different glue tasks (RTE, MRPC, and WNLI) and plot the sign conflict as a function of K. As we keep more and more parameters, the sign conflict increases and reaches almost 80%. This is also expected as there are many more nonzero parameters that can create conflict even if their magnitude is small. \n___\n**Question 4:** Plot the curve performance while varying the coefficients given to different tasks and missing important related work.\n\n**Answer:** We will cite this in the updated version! For the curves please refer to the general response Question 2.\n"}, {"review_id": "xakWnuREUV", "reviewer": "Reviewer_mRV1", "summary": "This paper identified two sources of performance degradation when merging fine-tuning models, (i) redundant parameter (ii) sing conflict and proposed TIES-MERGING to improve them.\n", "questions": "1. Is there any theoretical support regarding sign conflict?\n2. I needed help understanding how to see Fig. 6. What is \"Parameters With Nonzero Values\" in Fig. 6.a mean?\n3. In Figure 6, why does it look like you assume that a larger mean magnitude will result in better accuracy?\nIn other words, why do you know that a task vector with a small magnitude does not affect accuracy but also that a task vector with a large magnitude improves accuracy?", "limitations": "Yes.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The motivation to merge fine-gunning and performance is compelling.\n2. Experiments were conducted with both NLP and vision, and the proposed method improved accuracy with both.\n3. The proposed method is simple and computationally inexpensive, making it easy to reproduce.\n", "weaknesses": "1. There is no theoretical support for the proposed method.\n2. In particular, the effect of sign conflict on performance is nontrivial and requires theoretical support.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n**Weakness 1, 2 and Question 1:** Theoretical support for the proposed method.\n\n**Answer:** Our work is supported by several past works that shed light on different components of our method. We discuss each of them below and will add a similar discussion to the updated paper. \n\n**Why Merging Works:** As mentioned in L91-102, model merging and weight interpolation work because models finetuned from the same pre-trained initialization demonstrate mode connectivity [1] which implies that the finetuned checkpoints lie in an approximately convex loss basin in parameter space [2]. The theoretical underpinnings of mode connectivity, therefore, support the merging procedure we develop.\n\n**Existence of Sparse subnetworks:** Past works [3,4] have shown that during model finetuning, most information is encoded in a small and sparse subnetwork inside the full model. Achille et al. [5] describe two phases of training where the first phase discovers the important connections and their topology between layers and the second phase fine-tunes this relatively fixed pattern. In our work, we find that the top-k% parameters form a subnetwork that is responsible for most of the performance and only considers merging them to avoid unnecessary interference from other parameters.\n\n**Relevance of Signs:** In addition, the Lottery Ticket Hypothesis [3] finds subnetworks (winning tickets) by magnitude pruning and shows that by re-training them with static sparsity starting from the initial weights, they reach similar or higher accuracy. However, they demonstrated that random initialization, with the same structure, does not suffice. This implies that the values of the weight are crucial for good performance.  Zhou et al. [4] build upon this and show that one may not need the exact weights at initialization to train these subnetworks (lottery tickets) but the signs may be sufficient. We also find that during merging signs are critical for model performance (see the main paper, Figure 7) and hence we explicitly focus on resolving the sign conflict when merging models.\n___\n**Question 2:** I needed help understanding how to see Fig. 6. What is \"Parameters With Nonzero Values\" in Fig. 6.a mean?\n\n**Answer:** In Figure 2,  we provided the intuition of the different types of interference that exist when merging models and how it impacts the parameter values in the merged model. In Figure 6, we quantify the effect of both types of interference for real models that are being merged. As mentioned in L281-283, we bin the parameters of the model into three categories: (1) parameters that are redundant for all tasks (not in top-20% of any of the models); (2) parameters only influential for 1 model; and (3) parameters that are influential to more than one task. These categories help elucidate the impact of interference. For each category, we plot the means of the values in the merged model when using basic mean and trimming + disjoint mean. We know that zeroing out the redundant parameters does not affect the performance of the original task. However, removing redundant parameter values when merging models allows us to retain the change in that parameter's value introduced by another fine-tuned model. In contrast, taking the mean of all the values reduces the magnitude of the parameter in the merged model which might hurt the performance of a task that had a high magnitude in that direction. This is also explained in L49-59. \n___\n**Question 3:** In Figure 6, why does it look like you assume that a larger mean magnitude will result in better accuracy? In other words, why do you know that a task vector with a small magnitude does not affect accuracy but also that a task vector with a large magnitude improves accuracy?\n\n**Answer:** We do not assume that a larger mean magnitude will result in better accuracy. However, in the trimmed task vector if the value of a parameter is high the parameter is influential for the task and we have observed that reducing the values of the parameters results in a degradation in performance. As noted in Figure 3, resetting the top magnitude values results in huge performance drops. Consequently, TIES merging removes all redundant parameters and only merges the influential ones (that lead to a drop in performance if reduced).\n___\n[1] Linear Mode Connectivity and the Lottery Ticket Hypothesis\n\n[2] Loss surfaces, mode connectivity, and fast ensembling of dnns.\n\n[3] The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks\n\n[4] Deconstructing Lottery Tickets: Zeros, Signs, and the Supermask\n\n[5] Critical Learning Periods in Deep Networks\n"}]

Author rebuttal:
We thank all the reviewers for their time and for providing constructive comments for enhancing the paper. We appreciate that reviewers recognized:

- That our paper fills a crucial gap in the current literature (ioFQ).
- Our notable experimental contribution (reviewer ioFQ) via robust, comprehensive, and fair experimental setup (reviewer ioFQ, b9mw, Rfna).
- Simplicity (reviewer mRV1, ioFQ, Rfna, rBCc), clear description (reviewer ioFQ, Rfna), and computational efficiency (reviewer mRV1).
___
In the responses, we include the following additional experiments showing,

1. ("Model soups" experiments) TIES outperform other methods when merging checkpoints trained on the same task (Rebuttal PDF Table 1).
2. ("Fusing for better Finetuning" experiments) TIES provides a better initialization compared to other methods for fine-tuning (Rebuttal PDF Table 2).
3. Effect of hyperparameters on the method’s performance (Rebuttal PDF Figure 1).
4. A significant amount of sign conflicts exist even when merging different checkpoints of the same tasks (Rebuttal PDF Figure 2).
5. Sign conflict increases as we vary the top-k threshold (Rebuttal PDF Figure 3).
___

### **Common Questions.**

**Question 1:** Clarification on the chosen values of lambda for different methods.

**Answer:** As a first point of clarification, in all of our experiments apart from a subset of rows in Table1, $\lambda$ is a hyperparameter and we tune for it over the validation set. All the hyperparameters were chosen in this way for all the methods, including TIES merging and all baselines.

When there is no validation set available, it is not possible to tune hyperparameters in this way, so we use a fixed value of $\lambda = 1$ for TIES (L654-659 and L205-206), and $\lambda=0.4$ for task arithmetic (L210). Weight averaging does not consider $\lambda$ as a hyperparameter and always uses 1/(# tasks) with or without a validation set (L185-186). In Appendix D3 of the task arithmetic paper, they suggested that $\lambda$ values between 0.3-0.5 work best for most cases and we therefore used 0.4 (the midpoint) in our experiments without a validation set.

TIES merging takes the disjoint mean of the task vectors which already normalizes the values by the number of models with non-zero values for a parameter. Hence, we use $\lambda = 1$ for TIES. In contrast, Task Arithmetic adds all the task vectors  ($\theta_m = \theta_{init} + \lambda * \sum \tau_i$), and hence a $\lambda$ value of 1/(# tasks) for task arithmetic is in spirit similar to using $\lambda = 1$ for TIES. 
___
**Question 2:** Effect of hyper-parameters \lambda and k on the performance.

**Answer:** In Rebuttal PDF, Figures 1 (left and middle), we plot the effect of $\lambda$ on the performance when merging T5-base and T5-large models trained on GLUE (Similar to Table-1). For TIES merging, we vary $\lambda$ around the value 1 because TIES takes the mean of task vectors, whereas task arithmetic adds up the task vectors. Hence, a $\lambda$ value of 1 for TIES is similar to using $\lambda$ = 1 / (# tasks) for Task Arithmetic. The range of 0.8-1.8 for TIES was selected based on preliminary experiments on the PEFT setting (as mentioned in L654-659 and L205-206).  We find that TIES-Merging is much less sensitive to changes in $\lambda$ (with an accuracy range of 68-75 across the considered values of \lambda) compared to Task Arithmetic (with an accuracy range of 55-75). For the effect of k ( Rebuttal PDF Figure 1, right), we increment the value of k in steps of 10 and skip k=0 as that would select no parameters. We observe that as K increases the performance drops and then saturates. However, we would like to note that this curve might change based on the distribution of the values in the task vector.
___
**Question 3:** Limitation and future works.

**Answer:** We will add the following discussion to the main paper in the revised version.

As pointed out by reviewer ioFQ, our works share the same general limitations of existing merging methods, like (1) a limited theoretical understanding of why and when weight interpolation works, what are the important underlying factors, and its proper connections with mode connectivity; (2) that merging relies on common initialization and model architecture; and (3) merging individual-task models to create a multitask still lags behind the simultaneous multitask training. Moreover, it is not clear how to select the checkpoints for merging in order to create multitask models useful for specific domains. In addition, while our method provides a way to choose signs when merging task vectors, we still find that using the signs from a multitask model performs better. Some potential future works include figuring out a good way to estimate multitask signs without having access to the multitask model as this has the potential to bridge the gap between multitask merging and multitask training (as demonstrated in Section 7.4).


Meta-review: The paper addresses the task of dataless model merging where one needs to merge the weights of several models into a single multitask model without any data available. The paper presents a novel simple approach to this problem and shows empirically that in can help reduce the gap between fine-tuning and dataless merging. The paper is also easy to read and has a robust experimental framework. As such we believe this paper should be accepted as part of this conference.
Decision: Accept (poster)