Title: TIES-Merging: Resolving Interference When Merging Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: Model Merging

Fusing

Collaborative Training

Robust Fine-tuning

Federated Learning
Abstract: Transfer learning – i.e., further fine-tuning a pre-trained model on a downstream task – can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter’s values across models. To address this, we propose our method, TrIm, Elect Sign & Merge (TIES-Merging), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-Merging outperforms existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, highlight the importance of signs, and show that estimating the signs using the validation data could further improve performance.

TIES-MERGING: Resolving Interference When Merging Models

Abstract

Transfer learning - i.e., further fine-tuning a pre-trained model on a downstream task - can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-MERGING), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-MERGING outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference. 1

1 Introduction

Pre-trained models (PTMs) have become widespread in many real-world applications [91, 6]. Using PTMs typically involves fine-tuning them to specialize on a specific task [69, 12], which can lead to improved performance with less task-specific labeled data. These benefits have resulted in the release of thousands of finetuned checkpoints [81] derived from popular PTMs such as ViT [14] for vision and T5 [58] for language. However, having a separate fine-tuned model for each task has various drawbacks: (1) for each new application, a separate model has to be stored and deployed [17, 89], and (2) models trained in isolation cannot leverage information from related tasks to improve in-domain performance or out-of-domain generalization [66, 58, 75]. Multitask learning [66, 57] could address these concerns but requires costly training and simultaneous access to all tasks [17]. Moreover, it can be complex and resource-intensive to determine how best to mix datasets to ensure that multitask training is beneficial for all tasks [55, 54, 80, 52, 2, 17].

(1) Trim

(2) Elect Sign

Vector

(3) Disjoint Merge

: Influential values

: Redundant values

: Parameter

: Model 1

: Model 2

: Model 3

: Merged Model

Recently, a growing body of research has focused on model merging [40]. One application of merging involves combining multiple task-specific models into a single multitask model without performing additional training. Previous works merge models by summing the individual model weights with different weighting schemes, either via a simple average [9, 28, 83], via more sophisticated means that incorporate parameter importance [45] or account for permutation invariances [1, 31, 70, 74, 42]. Combining fine-tuned models in this way can be seen as adding together task vectors [29] that are computed by subtracting the pre-trained model's parameter values from those of the fine-tuned model.

While weighted averaging of model parameters has proven effective for merging, all of these methods ignore the possibility that values may interfere across models, thereby harming the performance of the merged model. In this paper, we first demonstrate that interference can stem from two major causes (see Fig. 2), both of which can reduce parameter magnitudes in the merged model and eliminate subtle distinctions among values: (1) INTERFERENCE FROM REDUNDANT PARAMETERS: Previous studies on model pruning [25, 76] have shown that during fine-tuning, many model parameters can change over the course of fine-tuning [63] but only have a small impact on performance. However, when merging a parameter that is influential for one model but redundant (i.e. not influential) for other models, the influential value may be obscured by the redundant values, lowering the overall model performance ( in

Original Values

No Interference

Redundant

Sign Conflict

Mean

TIES

Merged Values

Model 1

Model 2

# Fig. 2). (2) INTERFERENCE FROM SIGN DISAGREEMENT: A given parameter might have a positive value for some models and a negative value for others. Consequently, employing simple averaging might compromise the performance on both tasks ( 7 in Fig. 2). In both of these situations, simply aggregating the values lead to interference that shrinks the parameter's value in the merged model. This interference between influential parameters might explain why the performance gap between the merged model and multitask-trained model increases as the number of models increases [31].

To address these sources of interference, we propose TIES-MERGING (TRIM, ELECT SIGN & MERGE) method, a method for merging models by combining task vectors that has three steps (visualized in Fig. 1): First, we trim each task vector to retain only the influential parameter values by setting the redundant values in each task vector to zero (or, equivalently, resetting the fine-tuned

parameter value back to the value from the pre-trained model). After this step, sign conflicts may still persist among influential parameter values, as visualized in Fig. 4. Our second step therefore resolves the sign conflicts between different values and our last step only averages parameters whose sign agrees with the direction of the largest total movement across models.

We demonstrate the effectiveness of our proposed TIES-MERGING method in various setups with: (1) different modalities, including language and vision benchmarks, (2) distinct model sizes and families, such as T5-base and T5-large [58] as well as ViT-B/32 and ViT-L/14 [14], (3) in-domain and out-ofdomain tasks, (4) full finetuning or parameter-efficient finetuning, and (5) in the presence or absence of a validation set for setting merging hyperparameters. We show that TIES-MERGING outperforms other merging methods, such as Task Arithmetic [29], RegMean [31], Fisher Merging [45], and weight averaging [9, 82] across all these experimental settings. Notably, for in-domain evaluation, TIES-MERGING outperforms the strongest baseline by an average of 2.3% and 1.7% absolute in NLP and vision settings (Table 1), respectively. For out-of-domain generalization (Table 2), TIESMERGING outperforms the strongest baseline by 1 . 0% and 4 . 4% absolute for T5-base and T5-large models respectively. In Section 7, we perform ablations on our method components and demonstrate the impact of interference on parameter values. Additionally, we showcase the increased advantage of TIES-MERGING over task arithmetic [29] as the number of tasks increases. Finally, we examine the importance of obtaining the correct sign vector. Our results and analysis establish TIES-MERGING as a powerful and effective method for combining fine-tuned models into a single multi-task model.

2 Related Work

Loss Landscape and Weight Interpolation. While the loss function of a neural network is generally non-convex, recent work has demonstrated that the parameter values from different training runs can sometimes be interpolated without increasing the loss (i.e. they are mode-connected ) [15, 20, 21, 32, 22]. For example, Frankle et al. [19] showed that if a part of the optimization trajectory is shared between two neural networks then they can be interpolated without lowering accuracy. On the other hand, Neyshabur et al. [48] showed that naively interpolating two neural networks with completely disjoint optimization trajectories can result in a catastrophic drop in their accuracies. Entezari et al. [16] hypothesized that if we account for the permutation symmetry of neural networks, then all neural networks of a given architecture trained on the same dataset are linear mode connected. Ainsworth et al. [1], Singh and Jaggi [70], Wang et al. [79] therefore used techniques based on finding permutations [79, 1] and optimal transport [70] to better align neural networks trained from scratch so that they can be interpolated without increasing the loss.

Model Merging and Different Use Cases. Different fine-tuned models initialized from the same pre-trained model effectively share a part of the optimization trajectory, and can therefore often be merged without accounting for permutation symmetry [82, 83, 29, 31]. Therefore, merging fine-tuned models can improve performance on a single target task [30, 23, 82, 9], improving out-of-domain generalization [31, 29, 7, 4, 60, 59], creating a multitask models from different tasks [31, 29, 38], for federated learning [46, 41], compression [39], multimodal merging models [72], continual learning [86, 85], and other settings [38, 13]. The range of applications has led to a proliferation of methods to improve beyond simple parameter averaging. RegMean [31] proposed a closed-form solution for the merged model's parameters by solving a local linear regression problem for each individual linear layer in the model. However, this requires transmitting additional data statistics that are the same size as the model and requires additional inference steps to calculate them. Fisher Merging [45] goes beyond simple averaging to identify the importance of individual parameters using Fisher Information Matrix [18, 3, 34] and uses it to weigh the parameters in each model when merging. However, this shows little gains when merging multiple checkpoints and also requires computing gradients which has a high memory cost. Task Arithmetic [29] presented a method for merging models by generating task vectors and performing arithmetic operations, such as addition, to obtain a multitask checkpoint. A concurrent work by Ortiz-Jiménez et al. [51] provided theoretical insights on model merging based on the weight disentanglement property that arises during pretraining. They showed that finetuning models in their tangent space enhance this property, leading to better-merged models. Our method follows these past works on model merging but additionally takes into account the interference between different parameters during merging.

0

20

40

60

80

100

55

60

65

70

75

Average Perf.

3 Background and Motivation

Problem Setting Given a set of tasks { t 1 , . . . , t n } and a pre-trained model such as T5 [58] or ViT [14], we either finetune the entire model or employ a parameter-efficient finetuning (PEFT) method [43, 26]. In both cases, we denote the trainable parameters as θ , the initialization as θ init, and the finetuned parameters as θ ft. In this paper, we assume access to finetuned model parameters θ ft for multiple tasks and devise a method to merge the weights of these models into a single multitask model proficient on both in-domain and out-of-domain datasets. We follow Ilharco et al. [29] and perform merging with task vectors. Specifically, for a task t , the task-vector τ t ∈ R d is defined as τ t = θ t ft -θ t init . This operation allows us to focus on the changes that happen during the fine-tuning phase of each task-specific model and is equivalent to computing a weighted average of the models' weights with appropriate scaling.

Redundancies in Model Parameters. First, we demonstrate that in a given task vector, many values are redundant (denoted by # in Fig. 2), and removing them does not affect the performance of the task. Specifically, Fig. 3 shows the average performance across eleven task-specific models when "trimming" each task vector to retain only the topk % largest-magnitude values and resetting the rest to their initial value (i.e. setting the corresponding value in the task vector to 0 ). Fig. 3 shows the average performance across varying values of k , demonstrating that keeping only the top20% of values delivers comparable results to retaining all parameters. For additional details and the results on the T5 model, please refer to Appendix C.3. This shows that many parameter changes introduced during fine-tuning are redundant. Hence, disregarding those values during merging could prevent interference with the influential parameters without compromising the task's performance.

Algorithm 1 TIES-MERGING Procedure.

Disagreement between Parameter Signs: Different fine-tuned models might introduce opposing changes to a parameter in their task vectors, causing interference due to conflicting signs (denoted by 7 in Fig. 2). Fig. 4 presents an analysis of the frequency of sign conflicts when merging varying numbers of models. We first trim the task vectors for eleven tasks by keeping only the top 20% of influential parameters. Then, we plot the percentage of parameters that have a sign conflict as we

2

3

4

5

6

7

8

9

10

11

0

0.1

0.2

0.3

Number of Models

Frac. of Params

with Conflict

increase the number of models to be merged from 2 to 11. Notably, sign conflicts occur even when merging only 2 models from different tasks or when merging multiple models from the same task (see Appendix Figure 10), and the likelihood of a sign conflict increases with the number of models being merged. For additional details and the results on the T5 model, please refer to Appendix C.3.

4 TIES-MERGING: TRIM, ELECT SIGN & MERGE

To address the aforementioned issues, we present TIES-MERGING (TRIM, ELECT SIGN & MERGE), which aims to address the kinds of interference mentioned above before performing merging.

4.1 Preliminaries

A task vector τ t ∈ R d represents a direction and the amount of movement required in the d -dimensional parameter space relative to the initialization that leads to a low loss region for the task t . Each entry in τ t (corresponding to a particular parameter) can be thought of as an axis in the d -dimensional space. The sign of a parameter denotes the direction along this axis (positive or negative) that decreases the loss on task t . Hence, a given task-vector τ t can be decomposed into a sign vector γ t ∈ R d and a magnitude vector µ t ∈ R d as τ t = γ t ⊙ µ t , where ⊙ is the elementwise product. Formally, γ t = sgn ( τ t ) , where sgn ( x ) ∗ | x | = x and returns a value of +1 , 0 , or -1 . The magnitude vector µ t is defined as µ t = | τ t | and the value µ i t tells us the movement required in the i -th dimension from the initialization.

4.2 Steps in TIES-MERGING

To merge multiple task-specific models { θ t } n t =1 , we first create corresponding task vectors { τ t } n t =1 . Given these task vectors, TIES-MERGING method follows three steps in order to perform a merge (see Fig. 1 for a diagram and Algorithm 1):

Trim: For each task t , we trim the redundant parameters from the task vector τ t to create ˆ τ t by keeping the topk % values according to their magnitude and trimming the bottom (100 -k )% of the redundant parameters by resetting them to 0. This can be decomposed further as ˆ τ t = ˆ γ t ⊙ ˆ µ t .

Elect: Next, we create an aggregate elected sign vector γ m for the merged model that resolves the disagreements in the sign for each parameter p across different models. To create the elected sign vector, we choose the sign with the highest total magnitude across all relevant models. For each parameter p ∈ { 1 , 2 , . . . , d } , we separate the values { ˆ τ p t } n t =1 based on their sign ( +1 or -1 ) and take their sum to calculate the total mass (i.e., total magnitude) in the positive and the negative direction. We then assign γ p m as the sign with greater total movement. This can be efficiently computed using γ p m = sgn ( ∑ n t =1 ˆ τ p t ) .

Disjoint Merge: Then, for each parameter p , we compute a disjoint mean by only keeping the parameter values from the models whose signs are the same as the aggregated elected sign and calculate their mean. Formally, let A p = { t ∈ [ n ] | ˆ γ p t = γ p m } , then τ p m = 1 |A p | ∑ t ∈A p ˆ τ p t . Note that the disjoint mean always ignores the zero values.

Given the final merged task vector τ m , we scale it and add it to the initial parameter values to obtain the merged model parameters θ m as θ m = θ init + λ ∗ τ m , where λ is a scaling hyperparameter (as used in past work [29]).

5 Experimental Setup

Baseline Methods. We compare TIES-MERGING with four baseline merging methods: (1) Simple Averaging [9, 82] calculates the element-wise mean of all the individual models and can be expressed as θ m = ∑ n t =1 θ t /n . (2) Fisher Merging [45] uses a diagonal approximation of the Fisher Information Matrix ˆ F t [34, 3, 18] to measure the importance of each parameter for task t , where ˆ F t = E x ∼ D t E y ∼ p θ t ( y | x ) ∇ θ t (log p θ t ( y | x t )) 2 . The final merged model is obtained by reweighting each parameter in each fine-tuned model by the corresponding value in the model's approximate Fisher matrix as θ m = ∑ n t =1 ˆ F t θ t / ∑ n t =1 ˆ F t . (3) RegMean [31] computes a closed-form solution to

a least-squares regression problem that aims to minimize the distance between the merged model's activations and the individual models' activations as θ m = ( ∑ n t =1 X T t X t ) -1 ∑ n t =1 ( X T t X t θ t ) , where X t is the input activation of a given layer. (4) Task Artithmetic [29] scales and then adds the task vectors to the initial model to produce the merged model as θ m = θ init + λ ∗ ∑ n t =1 τ t . In addition to these baselines, we present the performance of the individual fine-tuned models involved in the merging process as well as the performance of a multi-task model trained on the concatenation of all tasks' datasets. For more details on compute resources, dataset licenses, and the finetuning procedures, refer to Appendix C.1, C.2, and C.6.

Merging in Absence of the Validation Set. Prior works [29, 45, 82] on model merging assume access to a validation set, which is utilized to compute the Fisher matrix or tune hyper-parameters. To avoid the need for a validation set, RegMean [31] proposed storing and transmitting inner product matrices of the training data for each task that are the same size as the original model. This can quickly become expensive for large models as the storage and transmission scale linearly with model size and the number of tasks.

To consider the setting where no validation set is available, we developed a generic recipe of TIESMERGING with fixed hyperparameters that could be applied in any setting without hyperparameter tuning on a validation set. The recipe keeps the top20% parameters in the task vector resetting the rest to 0 and sets λ = 1 . We chose this recipe based on results in the parameter-efficient fine-tuning (PEFT) setting, so we only apply it to the unseen settings of full model fine-tuning on ViT (vision) and T5 (language) models. We also compare TIES-MERGING with the Task Arithmetic method without a validation set by utilizing the recommended value of λ = 0 . 4 [29]. For further details on how this recipe was created please refer to Appendix C.4.

6 Main Results

Our main goal is to merge multiple task-specific models into a single multitask model that can perform well on both in-domain and out-of-domain scenarios. In this section, we evaluate the performance of TIES-MERGING with other methods across multiple different experimental settings.

Merging PEFT Models. Consider the setting where task vectors are computed based on parameters introduced during parameter-efficient fine-tuning. Specifically, we focus on (IA) 3 [43], a PEFT method that scales the base model activations with learned vectors. We follow Liu et al. [43] and use T0-3B [66] as the base model and finetune (IA) 3 models on the train split of eleven datasets including sentence completion (COPA [61], H-SWAG [88], and Story Cloze [68] datasets), natural language inference (ANLI [49], CB [44], and RTE [11]), coreference resolution (WSC [37] and Winogrande [64]), and word sense disambiguation (WiC [53]). When fine-tuning (IA) 3 parameters added to the T0-3B model, we use prompt templates from the Public Pool of Prompts (P3 [5]) to convert each example in each dataset to a prompted text-to-text format where each label corresponds to a different string. For experiments with (IA) 3 , for each dataset, we report the median score across all templates.

2

3

4

5

6

7

0.7

0.8

0.9

1

1.1

Simple Averaging

Task Arithmetic

TIES

Number of Tasks

Avg. Normalized Perf.

Table 1 using TIES-MERGING to merge models trained with (IA) 3 exceeds the performance of all other merging methods - with a validation set, TIES-MERGING shows an average enhancement of 2 . 5% across 11 tasks compared to the top baseline. For detailed results, refer to Appendix Table 8.

Merging Fully Finetuned Vision Models. For image classification, we adhere to the experimental setting from Ilharco et al. [29, 28]. We employ two variants of the CLIP model [56] with ViT-B/32 and ViT-L/14 models [14] as visual encoders. Subsequently, we finetune the visual encoder on the eight tasks derived from Ilharco et al. [28, 29], Radford et al. [56] while keeping the text encoder fixed. This setting considers a variety of classification domains such as remote sensing, traffic classification, and satellite imagery recognition. Specifically, we work with the following datasets: Cars [35], DTD [10], EuroSAT [24], GTSRB [71], MNIST [36], RESISC45 [8], SUN397 [84], and SVHN [47].

Table 1 shows that using TIES-MERGING to merge fully fine-tuned ViT-B/32 and ViT-L/14 models leads to an average improvement of 1 . 8% and 1 . 5% over 8 tasks, given the availability of a validation set. In the absence of a validation set, TIES-MERGING improves by 6 . 6% and 2 . 7% over other methods for ViT-B/32 and ViT-L/14, respectively. Notably, TIES-MERGING without validation outperforms Task Arithmetic [29] with validation by 2 . 3% and 1 . 5% for ViT-B/32 and ViT-L/14. For more detailed results, refer to Appendix Table 11 and 12.

Merging Fully Finetuned NLP Models. For the NLP domain, we use the T5-base and T5-large [57] models, which are encoder-decoder transformers [77] pretrained via masked language modeling on a large text corpus. We finetune both T5-base and T5-large on seven tasks: question answering (QASC [33], WikiQA [87], and QuaRTz [73]), Paraphrase Identification (PAWS [90]), Sentence Completion (Story Cloze [68]), and Coreference Resolution (Winogrande [64] and WSC [37]).

Table 1 shows that using TIES-MERGING on T5-base and T5-large models with a validation set produces an improvement of 0 . 7% and 3 . 6% respectively over 7 tasks compared to the state-of-the-art. Moreover, for T5-large TIES-MERGING without validation outperforms all baselines (even with a validation set) by 1 . 1% . For more detailed results, refer to Appendix Table 9 and 10.

Out-of-Domain Generalization. In many use-cases, multitask models are used for their ability to generalize better to domain shift. Hence, we use the T5-base and T5-large models merged on the seven in-domain datasets from the previous experiments and evaluate them on six held-out datasets from T0 mixture [65] to measure out-of-domain generalization. Specifically, we report the average performance over the following tasks and datasets: Cosmos QA [27], Social IQA [67], and QuAIL [62] for question answering; WiC [53] for word sense disambiguation; and COPA [61], and H-SWAG [88] for sentence completion. Table 2 shows that TIES-MERGING outperforms the strongest baseline for both T5-base and T5-Large by 1 . 0% and 4 . 4% respectively, demonstrating better out-of-domain generalization. For more elaborate results please refer to Appendix B.6 and Table 13 and 14.

Merging Different Number of Tasks. We evaluate the performance of the merged model on the in-domain tasks as we vary the number of tasks being merged. In Fig. 5, we normalize the accuracy of each task by its fine-tuned model performance and report the average normalized accuracy on the in-domain tasks. We compare with the strongest baseline - Task Arithmetic [29] - as well as

0

1

>1

0

0.02

0.04

0.06

0.08

Mean

Trim + Disjoint Mean

Parameters With Nonzero Values

Mean magnitude

(a) Redundant Parameter Interference.

0.5-0.6

0.6-0.7

0.7-0.8

0.8-0.9

0.9-1.0

0

0.05

0.1

0.15

Mean

Elect + Disjoint Mean

Bins of Sign Agreement

Mean magnitude

(b)

simple averaging [82]. Each data point signifies merging a subset of the tasks, with the solid line representing the mean performance across multiple subsets. For similar results with the T5-base model, please refer to Appendix C.5 and Figure 13.

From Fig. 5, we observe the following: (1) As the number of merged tasks increases, the performance of all methods decreases. (2) When merging two tasks, both TIES-MERGING and Task Arithmetic achieve an average normalized accuracy close to one, indicating negligible performance loss. In contrast, Simple Averaging suffers from a 10% performance drop. (3) As the number of tasks increases, the merging performance of Task Arithmetic declines more rapidly than TIES-MERGING. This suggests that task interference is present when merging multiple tasks and that TIES-MERGING is more effective at mitigating this issue.

Merging Checkpoints of the Same Task For Better Robustness We perform additional experiments to merge multiple checkpoints trained on the same task (as done in ModelSoups [82]) to see if it can improve robustness. Typically, ensembling is used to combine different models on the same task for better generalization. We use the experimental setting and the code from Fisher Merging [45] to merge top-10 fine-tuned base sized BERT models from huggingface for RTE, MRPC, and WNLI datasets from GLUE. From the results presented in Table 3, we observe that TIES-MERGING works the best in all cases except WNLI, where it only slightly underperforms Task Vectors. Notably, TIES-MERGING provides a dramatic boost over both Fisher Merging, averaging, and outperforms ensembling in all cases. Moreover, in Appendix B.4, we show that interference exists even between differently finetuned checkpoints of the same tasks.

Merging Models for Better Initialization. Next, we perform experiments following the setting [9], where we merge checkpoints from different tasks for a better initialization when fine-tuning on a downstream task. We take the finetuned bert-base-uncased checkpoints for 8 GLUE [78] tasks (wnli, sst2, rte, qnli, mrpc, cola, mnli, qqp) from Huggingface [81]. We consider three of these GLUE tasks (RTE, MRPC, WNLI) as our downstream tasks. When fine-tuning on a particular

0.0

0.2

0.4

0.6

0.8

1.0

50

60

70

Top-20

Top-30

Top-50

Bottom-80

Bottom-70

Bottom-50

Probability of flipping the Sign

Average Performance

downstream task (say RTE), we merge all the checkpoints from the other seven tasks together (apart from the chosen task). From Table 4, we find that TIES-MERGING works well in this setting and outperforms all other merging methods by a significant margin (apart from Averaging for WNLI).

7 Additional Results and Analysis

7.1 Types of Interference and Their Effect on Merging

(a) Importance of Removing Redundant Parameters. To better disentangle the effect of redundant parameters on the resulting magnitude of merged parameters, we separate the parameters into three groups: redundant parameters (using a trimming threshold of 20% ), parameters that are influential to exactly one model, and parameters that are influential to more than one model. We then compare the parameter values when they are directly merged versus when they are first trimmed and then (disjointly) merged without electing signs. Specifically, we only take the mean of non-trimmed values. The results are shown for the PEFT setting in Fig. 6a, which demonstrates that redundant parameters cause interference. Specifically, we find that when a parameter is not an influential parameter for any of the task-specific models, the mean value is low, and therefore may be considered noise. However, when only one model sees the parameter as influential, the merged value can still be low since other models assign a small value to this parameter. The merged value is larger when more models see the parameter as influential. When trimming, we see this interference is mostly avoided, and the average size is mostly the same whether one or more models consider a parameter influential. This is because we remove the effect of noisy parameters that unnecessarily decrease the magnitude (see # in Fig. 2). In the Appendix B.5, we bring a more detailed view, including a comparison to applying TIES-MERGING and show reducing interference encourages diversity in specific parameter values (std) together with the similarity of their influence (mean).

(b) Importance of Resolving Sign Interference. To quantify the impact of sign interference, we group the parameters by their sign agreement . A value of 0 . 5 indicates an equal number of positive and negative signs for a given parameter across different models, whereas 1 implies all the parameters have the same sign. We compare the parameter values when those are merged, or when sign disagreement is first resolved by election and then they are (disjointly) merged. The results in the PEFT setting are shown in Fig. 6b, where we demonstrate that the ELECT step preserves the relative parameter magnitudes to avoid sign interference. Specifically, we find that resolving signs increases the overall parameter magnitudes across different ranges of sign agreements. Parameters with low agreement tend to be smaller on average regardless of the interference. One potential cause could be that the sign from noisy parameters pulls the average down, as seen in Fig. 6a. We show in Appendix B.5 that combining both methods indeed reduces some of the difference, but not all, suggesting that a high agreement is correlated with overall influential parameters.

7.2 Relevance of Signs of the Top-k% Parameters

In this experiment, we work with the (IA) 3 models and aim to quantify the importance of the topk % parameters and their directions on a task's performance. For each task vector τ t , we flip the direction of each of the topk % parameters (by magnitude) with a probability p to obtain ˜ τ t . Flipping the direction is done by multiplying the parameters with -1 . Then we add back this direction flipped ˜ τ t to the initialization to get ˜ θ t = θ init +˜ τ t . Finally, we evaluate ˜ θ t and compute the average performance over all tasks t for each value of k and p . As a baseline, we also flip the directions of the bottom (100 -k )% of the parameters. We report the results averaged over three independent runs.

In Fig. 7, we plot the average performance as a function of p , the probability of flipping the direction. A probability of 0 means that the direction of none of the topk % parameters is flipped and a value of 1 means that the direction of all the topk % parameters is flipped. For the top20 / 30% of the parameters (solid lines), we observe that the performance monotonically decreases as we increase the probability of flipping the direction. In contrast, flipping the directions of the bottom80 / 70% of the parameters (dashed lines) has little impact on the performance. These results establish the importance of having the right directions for the parameters with a high magnitude and show the catastrophic performance drop that happens with incorrect directions.

7.3 Ablation of TIES-MERGING Components

We perform ablations on the individual components of TIES-MERGING to assess their importance. In Table 6, we start with TIES-MERGING and remove one component at a time and report the performance on the validation set for full model merging (T5-base) and merging PEFT models ((IA) 3 on T03B). Removing elect while keeping the disjoint mean refers to taking the mean of values with signs +1 and -1 but not including the 0 values of the trimmed task vectors in the mean. Removing disjoint mean but trimming and electing refers to taking the mean

of the values with the elected signs and the 0 for the trimmed values. Removing scaling means using λ = 1 . Table 6 shows that all components of the method are crucial for optimal performance. Specifically, scaling and the disjoint mean emerge as the most crucial, causing performance declines of 2 . 5% and 1 . 9% in T5-base, and 5 . 2% and 3 . 2% in (IA) 3 , respectively.

7.4 Importance of Estimating Correct Signs When Merging Models

Given the importance of sign vectors, we now aim to understand the performance that can be obtained by TIES-MERGING if we can use the oracle sign vector from the multitask model. To test this, we train a multitask (IA) 3 model, θ mult, on the eleven tasks under consideration (as in § 6). We then create the multitask vector τ mult and the multitask sign vector γ mult. Next, while merging models using TIES-MERGING, we assume access to the oracle multitask-sign-vector γ mult. Hence, we skip the conflict resolution step and directly set γ m = γ mult. Surprisingly, from Table 5, we observe that when merging tasks by using the oracle sign vector, we get a performance of 72% compared to 73 . 1% for the multitask trained model. Moreover, on average the merged model performs better task-specific models. This implies that if we can obtain the correction directions for the merged model, then we can closely bridge the gap to multitask models. In Appendix B.1 and Table 7, we attempt to estimate the multitask-sign-vector by using limited data.

8 Conclusion

We introduced TIES-MERGING to address interference when merging models. TIES-MERGING trims low-magnitude changes in fine-tuned model's values and then resolves sign disagreements across the models being merged. We found experimentally that TIES-MERGING enhances the performance of the merged multitask model across various settings and domains, despite being simple with fixed hyperparameters. Our study also sheds light on the impact of different types of interference on model parameters and emphasizes the importance of signs in the merging process. For some discussion on limitations and future directions please refer to Appendix A.

References

S. K. Ainsworth, J. Hayase, and S. Srinivasa. Git re-basin: Merging models modulo permutation symmetries, 2022. https://arxiv.org/abs/2209.04836 .

A. Albalak, C. Raffel, and W. Y. Wang. Improving few-shot generalization by exploring and exploiting auxiliary data. arXiv preprint arXiv:2302.00674 , 2023.

S. Amari. Neural learning in structured parameter spaces - natural riemannian gradient. In NIPS , 1996.

D. Arpit, H. Wang, Y. Zhou, and C. Xiong. Ensemble of averages: Improving model selection and boosting performance in domain generalization. arXiv preprint arXiv:2110.10832 , 2021.

S. H. Bach, V. Sanh, Z.-X. Yong, A. Webson, C. Raffel, N. V. Nayak, A. Sharma, T. Kim, M. S. Bari, T. Févry, et al. PromptSource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01279 , 2022.

R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al. On the opportunities and risks of foundation models, 2021. https://arxiv.org/abs/2108.07258 .

J. Cha, S. Chun, K. Lee, H.-C. Cho, S. Park, Y. Lee, and S. Park. Swad: Domain generalization by seeking flat minima. Advances in Neural Information Processing Systems , 34:22405-22418, 2021.

G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the Institute of Electrical and Electronics Engineers (IEEE) , 2017. https://ieeexplore.ieee.org/abstract/document/7891544 .

L. Choshen, E. Venezian, N. Slonim, and Y. Katz. Fusing finetuned models for better pretraining, 2022. https://arxiv.org/abs/2204.03044 .

M. Cimpoi, S. Maji, I. Kokkinos, S. Mohamed, and A. Vedaldi. Describing textures in the wild. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2014. https://openaccess.thecvf.com/content_cvpr_2014/ html/Cimpoi_Describing_Textures_in_2014_CVPR_paper.html .

I. Dagan, O. Glickman, and B. Magnini. The pascal recognising textual entailment challenge. In Machine Learning Challenges Workshop , 2005. https://link.springer.com/ chapter/10.1007/11736790_9 .

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805 , 2018.

S. Don-Yehiya, E. Venezian, C. Raffel, N. Slonim, Y. Katz, and L. Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, 2022. https://arxiv.org/ abs/2212.01378 .

A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR) , 2021. https://openreview.net/forum?id=YicbFdNTTy .

F. Draxler, K. Veschgini, M. Salmhofer, and F. Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning (ICML) , 2018. https://arxiv.org/abs/1803.00885 .

R. Entezari, H. Sedghi, O. Saukh, and B. Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.06296 , 2021.

C. Fifty, E. Amid, Z. Zhao, T. Yu, R. Anil, and C. Finn. Efficiently identifying task groupings for multi-task learning. Advances in Neural Information Processing Systems , 34:27503-27516, 2021.

R. A. Fisher. On the mathematical foundations of theoretical statistics. Philosophical transactions of the Royal Society of London. Series A, containing papers of a mathematical or physical character , 222(594-604):309-368, 1922.

J. Frankle, G. K. Dziugaite, D. Roy, and M. Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML) , 2020. https://proceedings.mlr.press/v119/frankle20a.html .

C. D. Freeman and J. Bruna. Topology and geometry of half-rectified network optimization. arXiv preprint arXiv:1611.01540 , 2016.

T. Garipov, P. Izmailov, D. Podoprikhin, D. Vetrov, and A. G. Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NeurIPS) , 2018. https://arxiv.org/abs/1802.10026 .

A. Gueta, E. Venezian, C. Raffel, N. Slonim, Y. Katz, and L. Choshen. Knowledge is a region in weight space for fine-tuned language models. arXiv preprint arXiv:2302.04863 , 2023.

V. Gupta, S. A. Serrano, and D. DeCoste. Stochastic weight averaging in parallel: Large-batch training that generalizes well. International Conference on Learning Representations , 2020.

P. Helber, B. Bischke, A. Dengel, and D. Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification. Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 2019. https://arxiv.org/abs/1709.00029 .

T. Hoefler, D. Alistarh, T. Ben-Nun, N. Dryden, and A. Peste. Sparsity in deep learning: Pruning and growth for efficient inference and training in neural networks. The Journal of Machine Learning Research , 22(1):10882-11005, 2021.

E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. ArXiv , abs/2106.09685, 2021.

L. Huang, R. Le Bras, C. Bhagavatula, and Y. Choi. Cosmos qa: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 2391-2401, 2019.

G. Ilharco, M. Wortsman, S. Y. Gadre, S. Song, H. Hajishirzi, S. Kornblith, A. Farhadi, and L. Schmidt. Patching open-vocabulary models by interpolating weights. In Advances in Neural Information Processing Systems (NeurIPS) , 2022. https://arXiv.org/abs/ 2208.05592 .

G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi. Editing models with task arithmetic. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj .

P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson. Averaging weights leads to wider optima and better generalization. In Conference on Uncertainty in Artificial Intelligence (UAI) , 2018. https://arxiv.org/abs/1803.05407 .

X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=FCnohuR6AnM .

K. Jordan, H. Sedghi, O. Saukh, R. Entezari, and B. Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=gU5sJ6ZggcX .

T. Khot, P. Clark, M. Guerquin, P. Jansen, and A. Sabharwal. Qasc: A dataset for question answering via sentence composition. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 34, pages 8082-8090, 2020.

J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS) , 2017. https: //arxiv.org/abs/1612.00796 .

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In International Conference on Computer Vision Workshops (ICML) , 2013. https://www.cv-foundation.org/openaccess/content_iccv_ workshops_2013/W19/html/Krause_3D_Object_Representations_2013_ ICCV_paper.html .

Y. LeCun. The mnist database of handwritten digits, 1998. http://yann.lecun.com/ exdb/mnist/ .

H. Levesque, E. Davis, and L. Morgenstern. The winograd schema challenge. Thirteenth International Conference on the Principles of Knowledge Representation and Reasoning , 2012.

M. Li, S. Gururangan, T. Dettmers, M. Lewis, T. Althoff, N. A. Smith, and L. Zettlemoyer. Branch-train-merge: Embarrassingly parallel training of expert language models, 2022. https: //arxiv.org/abs/2208.03306 .

P. Li, Z. Zhang, P. Yadav, Y .-L. Sung, Y . Cheng, M. Bansal, and T. Chen. Merge, then compress: Demystify efficient smoe with hints from its routing policy, 2023.

W. Li, Y. Peng, M. Zhang, L. Ding, H. Hu, and L. Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698 , 2023.

X. Li, K. Huang, W. Yang, S. Wang, and Z. Zhang. On the convergence of fedavg on non-iid data. In International Conference on Learning Representations , 2019.

Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent learning: Do different neural networks learn the same representations? arXiv preprint arXiv:1511.07543 , 2015.

H. Liu, D. Tam, M. Muqeeth, J. Mohta, T. Huang, M. Bansal, and C. A. Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning. Advances in Neural Information Processing Systems , 35:1950-1965, 2022.

M.-C. d. Marneffe, M. Simons, and J. Tonhauser. The CommitmentBank: Investigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung 23 , 2019.

M. Matena and C. Raffel. Merging models with fisher-weighted averaging. In Advances in Neural Information Processing Systems (NeurIPS) , 2021. https://arxiv.org/abs/ 2111.09832 .

B. McMahan, E. Moore, D. Ramage, S. Hampson, and B. A. y Arcas. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics , pages 1273-1282. PMLR, 2017.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems (NeurIPS) Workshops , 2011. https://storage.googleapis.com/ pub-tools-public-publication-data/pdf/37648.pdf .

B. Neyshabur, H. Sedghi, and C. Zhang. What is being transferred in transfer learning? Advances in neural information processing systems , 33:512-523, 2020.

Y. Nie, A. Williams, E. Dinan, M. Bansal, J. Weston, and D. Kiela. Adversarial NLI: A new benchmark for natural language understanding. arXiv preprint arXiv:1910.14599 , 2019.

H. Orgad, B. Kawar, and Y. Belinkov. Editing implicit assumptions in text-to-image diffusion models. arXiv preprint arXiv:2303.08084 , 2023.

G. Ortiz-Jiménez, A. Favero, and P. Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. NeurIPS , 2023. https://arxiv.org/abs/2305:12827 .

J. Phang, T. Févry, and S. R. Bowman. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088 , 2018.

M. T. Pilehvar and J. Camacho-Collados. WiC: The word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of NAACL-HLT , 2019.

C. Poth, J. Pfeiffer, A. Rücklé, and I. Gurevych. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing , pages 10585-10605, Online and Punta Cana, Dominican Republic, Nov. 2021.

Y. Pruksachatkun, J. Phang, H. Liu, P. M. Htut, X. Zhang, R. Y. Pang, C. Vania, K. Kann, and S. R. Bowman. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics , pages 5231-5247, Online, July 2020.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML) , 2021. https://arxiv.org/abs/2103.00020 .

C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR) , 2020. http://jmlr.org/papers/v21/20-074. html .

C. Raffel, N. M. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv , abs/1910.10683, 2020.

A. Ramé, K. Ahuja, J. Zhang, M. Cord, L. Bottou, and D. Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. arXiv preprint arXiv:2212.10445 , 2022.

A. Ramé, M. Kirchmeyer, T. Rahier, A. Rakotomamonjy, P. Gallinari, and M. Cord. Diverse weight averaging for out-of-distribution generalization. ICML , 2023.

M. Roemmele, C. A. Bejan, and A. S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 2011 AAAI Spring Symposium Series , 2011.

A. Rogers, O. Kovaleva, M. Downey, and A. Rumshisky. Getting closer to AI complete question answering: A set of prerequisite real tasks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020 , pages 8722-8731. AAAI Press, 2020. URL https://aaai.org/ojs/index.php/AAAI/article/view/ 6398 .

S. Ruder. An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 , 2016.

K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi. Winogrande: An adversarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence , 2020.

V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207 , 2021.

V. Sanh, A. Webson, C. Raffel, S. H. Bach, L. Sutawika, Z. Alyafeai, A. Chaffin, A. Stiegler, T. L. Scao, A. Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations (ICLR) , 2021. https://arxiv. org/abs/2110.08207 .

M. Sap, H. Rashkin, D. Chen, R. Le Bras, and Y. Choi. Social iqa: Commonsense reasoning about social interactions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 4463-4473, 2019.

R. Sharma, J. Allen, O. Bakhshandeh, and N. Mostafazadeh. Tackling the story ending biases in the story cloze test. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 752-757, 2018.

E. Shnarch, A. Halfon, A. Gera, M. Danilevsky, Y. Katsis, L. Choshen, M. S. Cooper, D. Epelboim, Z. Zhang, D. Wang, et al. Label sleuth: From unlabeled text to a classifier in a few hours. In Conference on Empirical Methods in Natural Language Processing , 2022.

S. P. Singh and M. Jaggi. Model fusion via optimal transport. Advances in Neural Information Processing Systems , 33:22045-22055, 2020.

J. Stallkamp, M. Schlipsing, J. Salmen, and C. Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In International Joint Conference on Neural Networks (IJCNN) , 2011. https://ieeexplore.ieee.org/document/6033395 .

Y.-L. Sung, L. Li, K. Lin, Z. Gan, M. Bansal, and L. Wang. An empirical study of multimodal model merging. Empirical Methods in Natural Language Processing (Findings) , 2023.

O. Tafjord, M. Gardner, K. Lin, and P. Clark. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) , pages 5941-5946, 2019.

N. Tatro, P.-Y. Chen, P. Das, I. Melnyk, P. Sattigeri, and R. Lai. Optimizing mode connectivity via neuron alignment. Advances in Neural Information Processing Systems , 33:15300-15311, 2020.

Y. Tay, M. Dehghani, V. Q. Tran, X. Garcia, J. Wei, X. Wang, H. W. Chung, D. Bahri, T. Schuster, S. Zheng, et al. Ul2: Unifying language learning paradigms. In The Eleventh International Conference on Learning Representations , 2022.

G. Thimm and E. Fiesler. Evaluating pruning methods. In International Symposium on Artificial Neural Networks , 1995.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS) , 2017. https://arxiv.org/abs/1706.03762 .

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. In International Conference on Learning Representations (ICLR) , 2018. https://arxiv.org/abs/1804.07461 .

H. Wang, M. Yurochkin, Y. Sun, D. Papailiopoulos, and Y. Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations , 2020.

O. Weller, K. Seppi, and M. Gardner. When to use multi-task learning vs intermediate finetuning for pre-trained encoder transfer learning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 272-282, Dublin, Ireland, May 2022.

T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al. Huggingface's transformers: State-of-the-art natural language processing, 2019. https://arxiv.org/abs/1910.03771 .

M. Wortsman, G. Ilharco, S. Y. Gadre, R. Roelofs, R. Gontijo-Lopes, A. S. Morcos, H. Namkoong, A. Farhadi, Y. Carmon, S. Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International Conference on Machine Learning (ICML) , 2022. https://arxiv.org/abs/ 2203.05482 .

M. Wortsman, G. Ilharco, M. Li, J. W. Kim, H. Hajishirzi, A. Farhadi, H. Namkoong, and L. Schmidt. Robust fine-tuning of zero-shot models. In Conference on Computer Vision and Pattern Recognition (CVPR) , 2022. https://arxiv.org/abs/2109.01903 .

J. Xiao, K. A. Ehinger, J. Hays, A. Torralba, and A. Oliva. Sun database: Exploring a large collection of scene categories. International Journal of Computer Vision (IJCV) , 2016. https://link.springer.com/article/10.1007/s11263-014-0748-y .

P. Yadav and M. Bansal. Exclusive supermask subnetwork training for continual learning. In Findings of the Association for Computational Linguistics: ACL 2023 , pages 569-587, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023. findings-acl.36. URL https://aclanthology.org/2023.findings-acl.36 .

P. Yadav, Q. Sun, H. Ding, X. Li, D. Zhang, M. Tan, P. Bhatia, X. Ma, R. Nallapati, M. K. Ramanathan, M. Bansal, and B. Xiang. Exploring continual learning for code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers) , pages 782-792, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-short.68. URL https://aclanthology.org/2023.acl-short.68 .

Y. Yang, W.-t. Yih, and C. Meek. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing , pages 2013-2018, Lisbon, Portugal, Sept. 2015. Association for Computational Linguistics. doi: 10.18653/v1/D15-1237. URL https://aclanthology.org/D15-1237 .

R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi. HellaSwag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830 , 2019.

Y. Zhang and Q. Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering , 34(12):5586-5609, 2021.

Y. Zhang, J. Baldridge, and L. He. PAWS: Paraphrase Adversaries from Word Scrambling. In Proc. of NAACL , 2019.

F. Zhuang, Z. Qi, K. Duan, D. Xi, Y. Zhu, H. Zhu, H. Xiong, and Q. He. A comprehensive survey on transfer learning. Proceedings of the IEEE , 2020. https://arxiv.org/abs/ 1911.02685 .

Appendix for TIES-MERGING

A Limitations and Future Works

Our works share the same general limitations of existing merging methods, like (1) a limited theoretical understanding of why and when weight interpolation works, what are the important underlying factors, and its proper connections with mode connectivity. Recent works like [50] have demonstrated interesting relationships between weight disentanglement and mergingability of models; (2) that merging relies on common initialization and model architecture; and (3) merging individual-task models to create a multitask still lags behind the simultaneous multitask training. Moreover, it is not clear how to select the checkpoints for merging in order to create multitask models useful for specific domains. In addition, while our method provides a way to choose signs when merging task vectors, we still find that using the signs from a multitask model performs better. Some potential future works include figuring out a good way to estimate multitask signs without having access to the multitask model as this has the potential to bridge the gap between multitask merging and multitask training (as demonstrated in Section 7.4).

B Additional Results

B.1 Enhancing Performance by Estimating the Multitask Sign Vector.

Considering the findings, we inquire whether it is possible to efficiently acquire multitask sign vectors without extensive multitask training. Our proposal involves utilizing a limited number of validation samples from each task to cheaply train a multitask model and subsequently derive the relevant sign vector. We create two multitask (IA) 3 models: one developed from scratch and another initialized using the average of task-specific (IA) 3 models intended for merging. We use 32 validation examples from each task to train this model. In Table 5, we notice using the sign vector from the fewshot multitask model initialized with mean yielded a performance increase of 3 . 8% and 1 . 3% compared to Task Arithmetic and TIES-MERGING. Interestingly, training fewshot multitask training from scratch did not yield significant improvements over TIES-MERGING without sign estimation. We believe that exploring this area further may result in improved merging techniques.

B.2 Effect of Hyper-Parameters λ and K on the Performance.

In Figure 8 (left and middle), we plot the effect of λ on the performance when merging T5-base and T5-large models trained on GLUE (Similar to Table-1). For TIES-MERGING, we vary around the value 1 because TIES takes the mean of task vectors, whereas task arithmetic adds up the task vectors. Hence, a value of 1 for TIES is similar to using 1 # tasks for Task Arithmetic [29]. The range of 0.8-1.8 for TIES was selected based on preliminary experiments on the PEFT setting (as mentioned in Section 5). We find that TIES-MERGING is much less sensitive to changes in (with an accuracy range of 68-75% across the considered values of λ ) compared to Task Arithmetic (with an accuracy

Lambda

Avg. Performance

50

55

60

65

70

75

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

T5-Large

TV Merging

Lambda

Avg. Performance

64

66

68

70

72

74

76

0.8

0.9

1.0

1.1

1.2

1.3

1.4

1.5

1.6

1.7

1.8

T5-Large

TIES Merging

Avg. Performance

70

72

74

76

78

80

10

20

30

40

50

60

70

80

90

100

Avg. Performance vs. K (T5-Base)

K

10

20

30

40

50

60

70

80

90

0

0.05

0.1

0.15

0.2

RTE

K (Frac. of parameters kept.)

Frac. of Params

with Conflict

10

20

30

40

50

60

70

80

90

0

0.05

0.1

0.15

MRPC

K (Frac. of parameters kept.)

Frac. of Params

with Conflict

10

20

30

40

50

60

70

80

90

0

0.1

0.2

0.3

WNLI

K (Frac. of parameters kept.)

Frac. of Params

with Conflict

2

3

4

5

6

7

8

9

10

0

0.05

0.1

0.15

0.2

RTE

Number of Models

Frac. of Params

with Conflict

2

3

4

5

6

7

8

9

10

0

0.05

0.1

0.15

MRPC

Number of Models

Frac. of Params

with Conflict

2

3

4

5

6

7

8

9

10

0

0.1

0.2

0.3

WNLI

Number of Models

Frac. of Params

with Conflict

2

3

4

5

6

7

8

9

10

11

0

0.1

0.2

0.3

Number of Models

Frac. of Params

with Conflict

range of 55-75). Figure 8 (right) demonstrates the effect of k as we increment the value of k in steps of 10 and skip k = 0 as that would select no parameters. We observe that as k increases the performance drops and then saturates. However, we would like to note that this curve might change based on the distribution of the parameter values in the task vector.

B.3 Sign Conflict Increases as We Trim Less Parameters

In Figure 9, we merge 10 bert-base-uncased checkpoints from huggingface finetuned on for three different glue tasks (RTE, MRPC, and WNLI) and plot the sign conflict as a function of k . As we keep more and more parameters, the sign conflict increases and reaches almost 80%. This is also expected as there are many more nonzero parameters that can create conflict even if their magnitude is small.

B.4 Sign Conflicts Exists Between Different Checkpoints for the Same Task

In Figure 10, we show that sign conflicts exist even within models trained on the same task. We plotted the sign conflict (similar to Figure 4) between the 10 checkpoints of RTE, MRPC, and WNLI from Huggingface. As the number of checkpoints increases, sign conflict increases. We also compare this with the sign interference when merging different task checkpoints and find a similar degree of interference in all of these cases. Hence, sign conflicts exist even within models trained on the same dataset. We suspect that this is because models are highly overparameterized and hence there are multiple subnetworks (subsets of parameters) that could lead to the same performance where different finetuning runs update the same parameters in different directions.

0

1

>1

0

0.05

0.1

0.15

Mean

Trim + Dis. Mean

TIES

Parameters with nonzero values

Mean magnitude

(a) Redundant Parameter Interference for (IA) 3 with STD.

0.5-0.6

0.6-0.7

0.7-0.8

0.8-0.9

0.9-1.0

0

0.05

0.1

0.15

0.2

0.25

Mean

Elect + Dis. Mean

TIES

Bins of Sign Agreement

Mean magnitude

(b)

3

with STD.

2

3

4

5

6

7

0

0.05

0.1

0.15

0.2

Number of Models

Frac. of Params

with Conflict

(a) Fraction of Parameters with Sign conflicts for T5-Base model versus number of models.

0

1

>1

0

0.002

0.004

0.006

Mean

Trim + Dis. Mean

TIES

Parameters with nonzero values

Mean magnitude

(b) Redundant Parameter Interfer- ence for T5-Base with STD.

0.5-0.6 0.6-0.7 0.7-0.8 0.8-0.9 0.9-1.0

0

0.002

0.004

0.006

Mean

Elect + Dis. Mean

TIES

Bins of Sign Agreement

Mean magnitude

(c) Sign Interference for T5-Base model with STD.

B.5 Detailed Results for Types of Interference and Their Effect on Merging

In Section 7.1 and Figure 6, we showed the effect of redundant parameters and sign conflicts on parameter magnitudes when comparing simple averaging vs disjoint mean after either trimming or electing and showed that performing these operations helps with the parameter magnitudes. In Figure 11, we additionally compare with TIES-MERGING and show that performing both trimming and electing usually results in higher magnitude and also higher standard deviation in parameter magnitudes. Higher std denotes that all parameter values in the merged model are the same and that there is a significant variation in the magnitude which is in contrast to simple averaging as it decreases the magnitude of not redundant parameters and reduces the magnitude of the influential parameters in the merged model. Similar plots for the T5-base model are provided in Figure 12.

B.6 Comprehensive Task-Level Results

We provide the task level for all the in-domain evaluation experiments in the main Table 1. Table 8, 9, 10, 11, and 12 provide the task level results IA3 [43], T5-Base, T5-Large [58], ViT-B/32, and ViT-L/14 [14] respectively. The task level results of the out-of-domain experiments for T5-Base and T5-Large can be found in Table 13, and 14. Lastly, Figure 13, shows the scaling of the T5-Base model as we merge different numbers of tasks.

C Implementation Details

C.1 Compute Resources Used and Runtimes

We executed all our experiments on Nvidia A6000 GPUs equipped with 48GB RAM. Single-task (IA) 3 models for eleven tasks required 1-2 hours per model, while the multitask vector took around 24 hours on four GPUs. The T5-Base and T5-Large models, based on dataset size, needed between 15 minutes and 2 hours per task, and approximately eight hours for the multitask checkpoint. Vision

models ViT-B/32 and ViT-L/14 were utilized, as supplied by Ilharco et al. [29]. 2 Merge experiments were efficient, with evaluations consuming less than 2 minutes for the T5-Base, T5-Large, ViTB/32, and ViT-L/14 experiments. The assessment of (IA) 3 models, due to the necessity of using multiple templates from prompt sources and median result calculations across all templates, required approximately one hour per 11 dataset evaluation.

C.2 Employed Datasets and Associated Licences

We use the following datasets in the paper with the following licenses. ANLI [49], WiC [53], WSC [37], and Story Cloze [68], QuaRTz [73], Cars [35], GTSRB [71] are under Creative Commons License. Winogrande [64], QASC [33] are under Apache license. COPA [61] is under a BSD-2 Clause license. H-SWAG [88], EuroSAT [24], is under MIT Licence. MNIST [36] is under Gnu General Public License. We could not find the licences of DTD [10], RESISC45 [8], SUN397 [84], SVHN [47], CB [44], RTE [11]), and PAWS [90] but they are publically for research use.

C.3 Details of the Motivation Experiments

For both Figure 3, and 4 in Section 3, we perform experiment on the eleven (IA) 3 models used in our PEFT merging experiments (§ 6). For a Figure similar to Fig. 4 demonstrating the fraction of parameters with a sign conflict for T5-base model, please refer to Fig. 12a.

2

3

4

5

6

7

0.7

0.8

0.9

1

1.1

Simple Averaging

Task Arithmetic

TIES

Number of Tasks

Average Normalized Perf.

C.4 Merging in the absence of the Validation Set

In our investigation into scenarios where a validation set is not available, we have devised a recipe and identified the optimal hyperparameters, employing the PEFT experimental procedure detailed in Section 6. This approach was applied to the eleven task-specific models presented in the same section, utilizing the TIES-MERGING method for tuning the hyperparameters. Our preliminary estimates for the hyperparameters were k = 20 and λ close to 1. The hyperparameter search was conducted using the eleven task-specific (IA) 3 models, with k ∈ { 10 , 20 , 30 } , and λ spanning from 0.8 to 3.0, in increments of 0.1. The results of this comprehensive search indicated an optimal value of k = 20 , with values of λ = 0 . 9 , λ = 1 . 0 , and λ = 1 . 1 demonstrating equivalent performance. To maintain simplicity in our model, we chose a λ value of 1. Thus, the final selection of parameters for TIES-MERGING is k = 20 , signs based on mass, the disjoint mean, and a λ value of 1.

C.5 Merging Different Number of Tasks

Here we provide some additional details on the experiments when merging different numbers of tasks. In Fig. 5, we perform the experiment with T5-Large when merging the seven tasks considered in Tab. 1 and described in Sec. 6. The x-axis shows the different number of tasks being merged. Note that when merging T tasks, we have a total of ( 7 T ) combinations. However, in our experiment, we sample at most 10 distinct combinations for each value of T . A similar plot for the T5-Base model is shown in Fig. 13.

In Figure 5, for each number of tasks we take at most 10 random subsets of the 8 tasks we were considering. The solid line is the average of the merged model's performance from these different runs. Below we provide the optimal λ values for the different subsets of tasks we merged for both TIES-MERGING and Task Arithmetic, note that for averaging the λ = 1 # tasks always. Each entry in the list is the optimal λ for a particular subset of tasks selected on the validation set.

C.6 Training Details

In our research, we utilized two variants of the T5 model, specifically the T5-base and T5-large models, which were trained to a maximum of 75,000 steps. An effective training batch size of 1024 was implemented, alongside a learning rate (lr) of 0.0001. We instituted an early stopping mechanism with a patience threshold of 5 to prevent overfitting. During the training process, bfloat16 was adopted to curtail GPU memory expenditure, and the maximum sequence length was set at 128. In contrast, for the PEFT configuration of the (IA) 3 approach on the T0-3B model, we modified our parameters. An effective training batch size of 16 was deployed along with an evaluation batch size of 32, while maintaining the learning rate at 0.0001. To accommodate the model's complexity, the early stopping patience was augmented to 10. We do not use any lr scheduler and weight decay for any of our model training.

For the purpose of evaluation, we perform rank classification . In this method, the model's log probabilities for all potential label strings are ranked. The model's prediction is deemed accurate if the choice ranked highest aligns with the correct answer. It should be noted that rank classification evaluation can accommodate both classification tasks and multiple-choice tasks.

Peer reviews:
comments: SUMMARY:
This paper identified two sources of performance degradation when merging fine-tuning models, (i) redundant parameter (ii) sing conflict and proposed TIES-MERGING to improve them.

STRENGTHS:
1. The motivation to merge fine-gunning and performance is compelling.
2. Experiments were conducted with both NLP and vision, and the proposed method improved accuracy with both.
3. The proposed method is simple and computationally inexpensive, making it easy to reproduce.

WEAKNESSES:
1. There is no theoretical support for the proposed method.
2. In particular, the effect of sign conflict on performance is nontrivial and requires theoretical support.

QUESTIONS:
1. Is there any theoretical support regarding sign conflict?
2. I needed help understanding how to see Fig. 6. What is "Parameters With Nonzero Values" in Fig. 6.a mean?
3. In Figure 6, why does it look like you assume that a larger mean magnitude will result in better accuracy?
In other words, why do you know that a task vector with a small magnitude does not affect accuracy but also that a task vector with a large magnitude improves accuracy?

LIMITATIONS:
Yes.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel method, TIES-MERGING, to merge models in the weight space for multitask learning. It observes an interference problem when linearly interpolating weights, and proposes a simple yet effective two-step solution: parameter trimming for small changes during fine-tuning and sign conflict resolution. The experiments in multitask shows that TIES improves performances, making it a notable (experimental) contribution to the literature of model merging.\n\nSTRENGTHS:\n* The paper's main strengths lie in its simplicity, as highlighted by its clear description and illustration.\n* The paper successfully builds upon the \"task arithmetic\" task vector approach to report a new interference phenomenon, and then enhances performance with simple yet important modifications. Model merging is an important topic in multitask, and this paper fills a crucial gap in the current literature.\n* The experimental framework is robust, with a focus on significant large-scale tasks across CV and NLP domains using recent architectures and fair benchmarks.\n\nWEAKNESSES:\nDespite its strengths, some areas require attention.\n* The contributions, though valuable, are incremental, and the observed gains in multitask learning are consistent but arguably marginal, and the trimming/sign play only a marginal role in this gain.\n* The experiments focus solely on models trained on different tasks. Yet, weight averaging is also useful to merge models trained on a single target task; on this model soups setup, I speculate that sign interference is less an issue, and that TIES would actually be detrimental as it would increase variance, thus limiting the benefits from combining multiple models, in particular under distribution shifts. As a minimum fix, the title should reflect this specificity, as the current one does not adequately reflect this scope limitation. A (naive) suggestion could be \"Resolving Multitask Interference When Merging Models\".\n* Even whithin the multitask challenge, the experiments do not cover two important scenarios. First, multitask as better pretraining before fine-tuning on a target task (as in \"Fusing finetuned models for better pretraining\"). Second, multitask in the sequential patching scenario (as in \"Patching open-vocabulary models by interpolating weight\").\n* Lack of analysis and clarity of the interpolating coefficient, for TIES and for the baselines. Specifically, without validation samples, could you clarify which values $\\lambda$ is used: it seems that it's $\\lambda=1/|num tasks|$ for weight averaging, $\\lambda=0.4$ for task arithmetic (could you please point where you found this value), but $\\lambda=1$ for TIES. Therefore, is the difference between task arithmetic/weight averaging in Table 1 simply due to the use of different $\\lambda$? Thus (as further suggested from Table 3), scaling is an important factor, these different values of $\\lambda$ hidden the true impact of your core contributions.\n* Similarly, the ablation study in Table 3 could be made clearer. It is not clear whether the ablations are done one at a time or sequentially all together. If the latter is true, then why do we recover 65.5 on T5-base (when all 4 components are removed), while weight averaging performs 65.9? Moreover, what does it mean to remove \"elect\" while keeping \"disjoint mean\"? what does it mean to remove \"scaling\" (is it $\\lambda=1/|num tasks|$ or something different)?\n\nQUESTIONS:\n* What is the impact of task similarity on the number of sign conflicts? Does the number of sign conflicts decrease when two models are fine-tuned on a shared task?\n* Can you visualize the number of sign conflicts for different trimming ratios?\n* Can TIES improves downstream transfer learning performances?\n* Could you plot the curve performance while varying the coefficients given to different tasks: see Pareto curves in \"Patching open-vocabulary models by interpolating weight\", but also in \"Pareto manifold learning: tackling multiple tasks via ensembles of single-task models\", a missing yet important related work.\n* Could you enrich the ablation study to further clarify/highlight the different impacts of the key contributions.\n* How does ensembling of predictions behave in comparison with weight interpolation?\n\nLIMITATIONS:\nThe authors have highlighted in Sections 7.3 and 7.4 that a key limitation is to accurately elect the sign.\nYet, the paper would benefit from a dedicated limitation section. It should ideally include that:\n- weight interpolation lacks proper theoretical understanding,\n- their approach is only (verified) for multitask learning,\n- the averageability relies on a large pretraining and \"good\" hyperparams,\n- TIES may be less impactful in a \"sequential patching\" scenario,\n- they still lag behind the simultaneous multitask learning.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper delves into the challenge of integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. The authors identified that current methodologies overlook the interference that occurs between parameters of different models. This interference can be attributed to two primary sources: redundant parameter values and conflicts in the sign of a given parameter's values across various models. To counteract these issues, the authors introduced a novel method, TIES, which incorporates three key steps: (1) resetting parameters that underwent minimal changes during fine-tuning, (2) resolving conflicts in sign, and (3) merging only those parameters that align with the final agreed-upon sign. The proposed method demonstrated SOTA performance across different settings.\n\nSTRENGTHS:\n- The paper provides a comprehensive analysis of the sources of interference in existing model merging methods, specifically pinpointing redundancies in model parameters and disagreements between parameter signs. This thorough examination underpins the motivation for the proposed method. \n- The proposed method demonstrates robust performance across a wide array of conditions, including various modalities, domains, task quantities, model sizes, architectures, and fine-tuning settings. This versatility underscores the method's adaptability and broad potential for application\n- The informative and insightful Section 7, which delves into the significance of different components, particularly the estimation of correct signs during the merge process, serves as a valuable resource.\n\nWEAKNESSES:\n- The proposed method is quite heuristic and it would make the paper stronger if the authors can provide more theoretical analysis of the proposed method. \n-  The author did not mention the limitations of their method and potential future work. The authors could further discuss them, which will provide a more balanced view of the method and give readers an idea of the potential directions for future research.\n- The paper could be improved by discussing more real-world applications of the studied problem, integrating multiple task-specific fine-tuned models into a singular, multitask model, without necessitating additional training. Why is this problem important? This would help to demonstrate the practical value of the method and its potential impact in real-world scenarios.\n-  It would be beneficial to include a sensitivity analysis of the hyperparameters used in the method. Although the authors provided a generic recipe of TIES with fixed hyperparameters, it would be good to see how sensitive the method is to different hyperparameters.\n\nQUESTIONS:\nI don't have further questions for now and I look forward to the authors' enhancements on the weaknesses mentioned above.\n\nLIMITATIONS:\nThe author did not mention the limitations of their method and potential future work. I have listed some suggestions for improvement in the weaknesses section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMerge mutiple **fine-tuned** neural networks, each from an unique task (dataset), into one neural network by weight averaging. Denate the weights of pretrained network as $\\theta_0$, fine-tuned network as $\\theta_{\\tau}$, the weight updating direction $\\tau = \\theta_{\\tau} - \\theta_0$ \n\nThe paper introduces two tricks 1) for each $\\tau$ only keep the top $k$% elements in $\\tau$ according to their magnitude. (keep the others as zero), 2) during the weight averaging, only retain the dominate direction (+1 or -1) for each elements. \n\nThe experiments shows the improvements of the proposed Ties-Merging method over other weight averaging methods, such as naive weight averaging, task arithmetic, and regmean. But Ties-Merging is still significantly lagging behind MultiTask training.\n\nSTRENGTHS:\n- The writing is clear and easy to read. \n- The introduced two tricks are resonable and easy to understand. \n- Comparing with naive weight averaging, task arithmetic, Fisher averaging, and regmean, the proposed ties-merging performs very well.\n\nWEAKNESSES:\n- Even though the proposed Weight Averaging method out-perform other weight averaging methods, it still legs behind MultiTask training. \nFurthermore, considering both the training cost and inference time, it is hard to see the benefit of the proposed method over mutitask training. As to other baseline methods, such as naive weight averaging, Fisher averaging and regmean, they are designed to solve IID (one fine-tuning task instead of mutiple different tasks) not OOD. These two facts reduce the strongness of the experiments.  (But it is fine. Because the two introduced tricks are good enough).\n\nQUESTIONS:\n- Simple averaging is $1/n \\sum_{\\tau} \\theta_{\\tau}  = \\theta_0 + 1/n \\sum_{\\tau}\\tau$\n- Task Arithmetic  is $ \\theta_0 + \\lambda \\sum_{\\tau}\\tau$\n\nWhat is the parameter $\\lambda$ of task arithmetic in Figure 5? If it is $0.4$ as line 210, why there is a big gap between simple averaging and task arithmetic in figure 5, ( given that $1/2=0.5, 1/3=0.33$)\n\nLIMITATIONS:\nno negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes to resolve the interference of model merging, a solution to combine multiple task-specific models into a single multitask model. It demonstrates two major sources of interference, including redundant parameter values and sign conflict and proposes solutions to resolve the interference.

STRENGTHS:
The paper demonstrates some interesting research insights and the solutions are simple and clear.

WEAKNESSES:
1: I question the necessity of Section 7.3.  According to the network pruning literature, a high magnitude is always an implication of importance/sensitivity, and a widely used pruning metric. Flipping the sign of or pruning away the Top-k% parameters can cause significant performance drop is well-known to the literature. It seems to have weak connection with this paper’s main argument. 

2: I did not find the discussions regarding limitations and future work.

3: More analysis on the experimental results are required. For example, what are the effects of the proposed on each task respectively? Why do some tasks perform better than others?

QUESTIONS:
Refer to questions.

LIMITATIONS:
The authors do NOT adequately address the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-25