SlowGuess/NeurIPS_2023_OCR huggingface dataset:
TIES-MERGING: Resolving Interference When Merging Models
Prateek Yadav $^{1}$  Derek Tam $^{1}$
Leshem Choshen $^{2,3}$  Colin Raffel $^{1}$  Mohit Bansal $^{1}$
<sup>1</sup> University of North Carolina at Chapel Hill 2 IBM Research 3 MIT
leshem.choshen@ibm.com
{praty,dtredsox,craffel,mbansal}@cs.unc.edu
Transfer learning – i.e., further fine-tuning a pre-trained model on a downstream task – can confer significant advantages, including improved downstream performance, faster convergence, and better sample efficiency. These advantages have led to a proliferation of task-specific fine-tuned models, which typically can only perform a single task and do not benefit from one another. Recently, model merging techniques have emerged as a solution to combine multiple task-specific models into a single multitask model without performing additional training. However, existing merging methods often ignore the interference between parameters of different models, resulting in large performance drops when merging multiple models. In this paper, we demonstrate that prior merging techniques inadvertently lose valuable information due to two major sources of interference: (a) interference due to redundant parameter values and (b) disagreement on the sign of a given parameter's values across models. To address this, we propose our method, TRIM, ELECT SIGN & MERGE (TIES-MERGING), which introduces three novel steps when merging models: (1) resetting parameters that only changed a small amount during fine-tuning, (2) resolving sign conflicts, and (3) merging only the parameters that are in alignment with the final agreed-upon sign. We find that TIES-MERGING outperforms several existing methods in diverse settings covering a range of modalities, domains, number of tasks, model sizes, architectures, and fine-tuning settings. We further analyze the impact of different types of interference on model parameters, and highlight the importance of resolving sign interference.<sup>1</sup>
Pre-trained models (PTMs) have become widespread in many real-world applications [91, 6]. Using PTMs typically involves fine-tuning them to specialize on a specific task [69, 12], which can lead to improved performance with less task-specific labeled data. These benefits have resulted in the release of thousands of finetuned checkpoints [81] derived from popular PTMs such as ViT [14] for vision and T5 [58] for language. However, having a separate fine-tuned model for each task has various drawbacks: (1) for each new application, a separate model has to be stored and deployed [17, 89], and (2) models trained in isolation cannot leverage information from related tasks to improve in-domain performance or out-of-domain generalization [66, 58, 75]. Multitask learning [66, 57] could address these concerns but requires costly training and simultaneous access to all tasks [17]. Moreover, it can be complex and resource-intensive to determine how best to mix datasets to ensure that multitask training is beneficial for all tasks [55, 54, 80, 52, 2, 17].
<sup>1</sup>Our code is available at https://github.com/prateeky2806/ties-merging
Figure 1: A depiction of the steps involved in TIES-MERGING. We visualize each parameter in a model as a square. The arrows depict the update (task vector,  $\tau$ ) to a parameter produced by fine-tuning on different tasks (coded by colors), with direction denoting sign and length denoting magnitude. We first trim the task vector values based on their magnitude, then we elect the sign for each parameter ( $\gamma_{m}$ , green vector containing  $+1$  or  $-1$ ) by resolving sign conflicts. Finally, we pick only the values that align with the elected sign and take their mean as the final parameter value.
Recently, a growing body of research has focused on model merging [40]. One application of merging involves combining multiple task-specific models into a single multitask model without performing additional training. Previous works merge models by summing the individual model weights with different weighting schemes, either via a simple average [9, 28, 83], via more sophisticated means that incorporate parameter importance [45] or account for permutation invariances [1, 31, 70, 74, 42]. Combining fine-tuned models in this way can be seen as adding together task vectors [29] that are computed by subtracting the pre-trained model's parameter values from those of the fine-tuned model.
While weighted averaging of model parameters has proven effective for merging, all of these methods ignore the possibility that values may interfere across models, thereby harming the performance of the merged model. In this paper, we first demonstrate that interference can stem from two major causes (see Fig. 2), both of which can reduce parameter magnitudes in the merged model and eliminate subtle distinctions among values: (1) INTERFERENCE FROM REDUNDANT PARAMETERS: Previous studies on model pruning [25, 76] have shown that during fine-tuning, many model parameters can change over the course of fine-tuning [63] but only have a small impact on performance. However, when merging a parameter that is influential for one model but redundant (i.e. not influential) for other models, the influential value may be obscured by the redundant values, lowering the overall model performance  $(\bigcirc$  in
Figure 2: Different types of conflict and merged outputs produced by either averaging or TIES-MERGING. The parameters causing interference are denoted by dotted arrows.
Fig. 2). (2) INTERFERENCE FROM SIGN DISAGREEMENT: A given parameter might have a positive value for some models and a negative value for others. Consequently, employing simple averaging might compromise the performance on both tasks (  $\bigcirc$  in Fig. 2). In both of these situations, simply aggregating the values lead to interference that shrinks the parameter's value in the merged model. This interference between influential parameters might explain why the performance gap between the merged model and multitask-trained model increases as the number of models increases [31].
To address these sources of interference, we propose TIES-MERGING (TRIM, ELECT SIGN & MERGE) method, a method for merging models by combining task vectors that has three steps (visualized in Fig. 1): First, we trim each task vector to retain only the influential parameter values by setting the redundant values in each task vector to zero (or, equivalently, resetting the fine-tuned
parameter value back to the value from the pre-trained model). After this step, sign conflicts may still persist among influential parameter values, as visualized in Fig. 4. Our second step therefore resolves the sign conflicts between different values and our last step only averages parameters whose sign agrees with the direction of the largest total movement across models.
We demonstrate the effectiveness of our proposed TIES-MERGING method in various setups with: (1) different modalities, including language and vision benchmarks, (2) distinct model sizes and families, such as T5-base and T5-large [58] as well as ViT-B/32 and ViT-L/14 [14], (3) in-domain and out-of-domain tasks, (4) full finetuning or parameter-efficient finetuning, and (5) in the presence or absence of a validation set for setting merging hyperparameters. We show that TIES-MERGING outperforms other merging methods, such as Task Arithmetic [29], RegMean [31], Fisher Merging [45], and weight averaging [9, 82] across all these experimental settings. Notably, for in-domain evaluation, TIES-MERGING outperforms the strongest baseline by an average of  $2.3\%$  and  $1.7\%$  absolute in NLP and vision settings (Table 1), respectively. For out-of-domain generalization (Table 2), TIES-MERGING outperforms the strongest baseline by  $1.0\%$  and  $4.4\%$  absolute for T5-base and T5-large models respectively. In Section 7, we perform ablations on our method components and demonstrate the impact of interference on parameter values. Additionally, we showcase the increased advantage of TIES-MERGING over task arithmetic [29] as the number of tasks increases. Finally, we examine the importance of obtaining the correct sign vector. Our results and analysis establish TIES-MERGING as a powerful and effective method for combining fine-tuned models into a single multi-task model.
Loss Landscape and Weight Interpolation. While the loss function of a neural network is generally non-convex, recent work has demonstrated that the parameter values from different training runs can sometimes be interpolated without increasing the loss (i.e. they are mode-connected) [15, 20, 21, 32, 22]. For example, Frankle et al. [19] showed that if a part of the optimization trajectory is shared between two neural networks then they can be interpolated without lowering accuracy. On the other hand, Neyshabur et al. [48] showed that naively interpolating two neural networks with completely disjoint optimization trajectories can result in a catastrophic drop in their accuracies. Entezari et al. [16] hypothesized that if we account for the permutation symmetry of neural networks, then all neural networks of a given architecture trained on the same dataset are linear mode connected. Ainsworth et al. [1], Singh and Jaggi [70], Wang et al. [79] therefore used techniques based on finding permutations [79, 1] and optimal transport [70] to better align neural networks trained from scratch so that they can be interpolated without increasing the loss.
Model Merging and Different Use Cases. Different fine-tuned models initialized from the same pre-trained model effectively share a part of the optimization trajectory, and can therefore often be merged without accounting for permutation symmetry [82, 83, 29, 31]. Therefore, merging fine-tuned models can improve performance on a single target task [30, 23, 82, 9], improving out-of-domain generalization [31, 29, 7, 4, 60, 59], creating a multitask models from different tasks [31, 29, 38], for federated learning [46, 41], compression [39], multimodal merging models [72], continual learning [86, 85], and other settings [38, 13]. The range of applications has led to a proliferation of methods to improve beyond simple parameter averaging. RegMean [31] proposed a closed-form solution for the merged model's parameters by solving a local linear regression problem for each individual linear layer in the model. However, this requires transmitting additional data statistics that are the same size as the model and requires additional inference steps to calculate them. Fisher Merging [45] goes beyond simple averaging to identify the importance of individual parameters using Fisher Information Matrix [18, 3, 34] and uses it to weigh the parameters in each model when merging. However, this shows little gains when merging multiple checkpoints and also requires computing gradients which has a high memory cost. Task Arithmetic [29] presented a method for merging models by generating task vectors and performing arithmetic operations, such as addition, to obtain a multitask checkpoint. A concurrent work by Ortiz-Jiménez et al. [51] provided theoretical insights on model merging based on the weight disentanglement property that arises during pretraining. They showed that finetuning models in their tangent space enhance this property, leading to better-merged models. Our method follows these past works on model merging but additionally takes into account the interference between different parameters during merging.
Figure 3: Performance depends on a small fraction of high-magnitude parameters. For each task vector, we keep only the largest - top-  $k\%$  parameters and plot the average performance across eleven tasks. Keeping only the top-20% of the parameter does not degrade the performance.
Figure 4: Sign conflicts occur even after trimming and increase with the number of models. We plot the fraction of parameters that have a sign conflict after trimming versus the number of models being merged.
Problem Setting Given a set of tasks  $\{t_1,\dots ,t_n\}$  and a pre-trained model such as T5 [58] or ViT [14], we either finetune the entire model or employ a parameter-efficient finetuning (PEFT) method [43, 26]. In both cases, we denote the trainable parameters as  $\theta$ , the initialization as  $\theta_{\mathrm{init}}$ , and the finetuned parameters as  $\theta_{\mathrm{ft}}$ . In this paper, we assume access to finetuned model parameters  $\theta_{\mathrm{ft}}$  for multiple tasks and devise a method to merge the weights of these models into a single multitask model proficient on both in-domain and out-of-domain datasets. We follow Iharco et al. [29] and perform merging with task vectors. Specifically, for a task  $t$ , the task-vector  $\tau_t\in \mathbb{R}^d$  is defined as  $\tau_{t} = \theta_{\mathrm{ft}}^{t} - \theta_{\mathrm{init}}^{t}$ . This operation allows us to focus on the changes that happen during the fine-tuning phase of each task-specific model and is equivalent to computing a weighted average of the models' weights with appropriate scaling.
Redundancies in Model Parameters. First, we demonstrate that in a given task vector, many values are redundant (denoted by  $\bigcirc$  in Fig. 2), and removing them does not affect the performance of the task. Specifically, Fig. 3 shows the average performance across eleven task-specific models when "trimming" each task vector to retain only the top- $k\%$  largest-magnitude values and resetting the rest to their initial value (i.e. setting the corresponding value in the task vector to 0). Fig. 3 shows the average performance across varying values of  $k$ , demonstrating that keeping only the top- $20\%$  of values delivers comparable results to retaining all parameters. For additional details and the results on the T5 model, please refer to Appendix C.3. This shows that many parameter changes introduced during fine-tuning are redundant. Hence, disregarding those values during merging could prevent interference with the influential parameters without compromising the task's performance.
Algorithm 1 TIES-MERGING Procedure.
Input: Fine-tuned models  $\{\theta_t\}_{t=1}^n$ , Initialization  $\theta_{\mathrm{init}}$ ,  $k$ , and  $\lambda$ .  
Output: Merged Model  $\theta_m$   
for all  $t \in 1, \dots, n$  do  
 $\triangleright$  Create task vectors.  
 $\tau_t = \theta_t - \theta_{\mathrm{init}}$ $\triangleright$  Step 1: Trim redundant parameters.  
 $\hat{\tau}_t \gets$  keep_top_k_reset/rest_to_zero  $(\tau_t, k)$ $\hat{\gamma}_t \gets$  sgn  $(\hat{\tau}_t)$ $\hat{\mu}_t \gets |\hat{\tau}_t|$   
end  
 $\triangleright$  Step 2: Elect Final Signs.  
 $\gamma_m = \operatorname{sgn}(\sum_{t=1}^n \hat{\tau}_t)$ $\triangleright$  Step 3: Disjoint Merge.  
forall  $p \in 1, \dots, d$  do  
 $\mathcal{A}^p = \{t \in [n] | \hat{\gamma}_t^p = \gamma_m^p\}$ $\tau_m^p = \frac{1}{|\mathcal{A}^p|} \sum_{t \in \mathcal{A}^p} \hat{\tau}_t^p$   
end  
 $\triangleright$  Obtain merged checkpoint  
 $\theta_m \gets \theta_{\mathrm{init}} + \lambda * \tau_m$   
return  $\theta_m$
Disagreement between Parameter Signs: Different fine-tuned models might introduce opposing changes to a parameter in their task vectors, causing interference due to conflicting signs (denoted by  $\square$  in Fig. 2). Fig. 4 presents an analysis of the frequency of sign conflicts when merging varying numbers of models. We first trim the task vectors for eleven tasks by keeping only the top  $20\%$  of influential parameters. Then, we plot the percentage of parameters that have a sign conflict as we
increase the number of models to be merged from 2 to 11. Notably, sign conflicts occur even when merging only 2 models from different tasks or when merging multiple models from the same task (see Appendix Figure 10), and the likelihood of a sign conflict increases with the number of models being merged. For additional details and the results on the T5 model, please refer to Appendix C.3.
4 TIES-MERGING: TRIM, ELECT SIGN & MERGE
To address the aforementioned issues, we present TIES-MERGING (TRIM, ELECT SIGN & MERGE), which aims to address the kinds of interference mentioned above before performing merging.
A task vector  $\tau_t \in \mathbb{R}^d$  represents a direction and the amount of movement required in the  $d$ -dimensional parameter space relative to the initialization that leads to a low loss region for the task  $t$ . Each entry in  $\tau_t$  (corresponding to a particular parameter) can be thought of as an axis in the  $d$ -dimensional space. The sign of a parameter denotes the direction along this axis (positive or negative) that decreases the loss on task  $t$ . Hence, a given task-vector  $\tau_t$  can be decomposed into a sign vector  $\gamma_t \in \mathbb{R}^d$  and a magnitude vector  $\mu_t \in \mathbb{R}^d$  as  $\tau_t = \gamma_t \odot \mu_t$ , where  $\odot$  is the elementwise product. Formally,  $\gamma_t = \operatorname{sgn}(\tau_t)$ , where  $\operatorname{sgn}(x) * |x| = x$  and returns a value of  $+1, 0$ , or  $-1$ . The magnitude vector  $\mu_t$  is defined as  $\mu_t = |\tau_t|$  and the value  $\mu_t^i$  tells us the movement required in the  $i$ -th dimension from the initialization.
4.2 Steps in TIES-MERGING
To merge multiple task-specific models  $\{\theta_t\}_{t=1}^n$ , we first create corresponding task vectors  $\{\tau_t\}_{t=1}^n$ . Given these task vectors, TIES-MERGING method follows three steps in order to perform a merge (see Fig. 1 for a diagram and Algorithm 1):
1. Trim: For each task  $t$ , we trim the redundant parameters from the task vector  $\tau_t$  to create  $\hat{\tau}_t$  by keeping the top- $k\%$  values according to their magnitude and trimming the bottom  $(100 - k)\%$  of the redundant parameters by resetting them to 0. This can be decomposed further as  $\hat{\tau}_t = \hat{\gamma}_t \odot \hat{\mu}_t$ .
2. Elect: Next, we create an aggregate elected sign vector  $\gamma_{m}$  for the merged model that resolves the disagreements in the sign for each parameter  $p$  across different models. To create the elected sign vector, we choose the sign with the highest total magnitude across all relevant models. For each parameter  $p\in \{1,2,\dots ,d\}$ , we separate the values  $\{\hat{\tau}_t^p\}_{t = 1}^n$  based on their sign (+1 or -1) and take their sum to calculate the total mass (i.e., total magnitude) in the positive and the negative direction. We then assign  $\gamma_{m}^{p}$  as the sign with greater total movement. This can be efficiently computed using  $\gamma_{m}^{p} = \mathrm{sgn}(\sum_{t = 1}^{n}\hat{\tau}_{t}^{p})$ .
3. Disjoint Merge: Then, for each parameter  $p$ , we compute a disjoint mean by only keeping the parameter values from the models whose signs are the same as the aggregated elected sign and calculate their mean. Formally, let  $\mathcal{A}^p = \{t \in [n] \mid \hat{\gamma}_t^p = \gamma_m^p\}$ , then  $\tau_m^p = \frac{1}{|\mathcal{A}^p|} \sum_{t \in \mathcal{A}^p} \hat{\tau}_t^p$ . Note that the disjoint mean always ignores the zero values.
Given the final merged task vector  $\tau_{m}$ , we scale it and add it to the initial parameter values to obtain the merged model parameters  $\theta_{m}$  as  $\theta_{m} = \theta_{\mathrm{init}} + \lambda * \tau_{m}$ , where  $\lambda$  is a scaling hyperparameter (as used in past work [29]).
Baseline Methods. We compare TIES-MERGING with four baseline merging methods: (1) Simple Averaging [9, 82] calculates the element-wise mean of all the individual models and can be expressed as  $\theta_{m} = \sum_{t=1}^{n} \theta_{t} / n$ . (2) Fisher Merging [45] uses a diagonal approximation of the Fisher Information Matrix  $\hat{F}_{t}$  [34, 3, 18] to measure the importance of each parameter for task  $t$ , where  $\hat{F}_{t} = \mathbb{E}_{x \sim D_{t}} \mathbb{E}_{y \sim p_{\theta_{t}}(y|x)} \nabla_{\theta_{t}} (\log p_{\theta_{t}}(y|x_{t}))^{2}$ . The final merged model is obtained by reweighting each parameter in each fine-tuned model by the corresponding value in the model's approximate Fisher matrix as  $\theta_{m} = \sum_{t=1}^{n} \hat{F}_{t} \theta_{t} / \sum_{t=1}^{n} \hat{F}_{t}$ . (3) RegMean [31] computes a closed-form solution to
Method (↓)	Validation	PEFT	Full Finetuning
Model (→)	(IA)3	T5-Base	T5-Large	ViT-B/32	ViT-L/14
FINE-TUNED	-	71.4	82.8	88.8	90.5	94.2
MULTITASK	-	73.1	83.6	88.1	88.9	93.5
AVERAGING [82, 9]	✗	-	65.9	59.6	65.8	79.6
TASK ARITHMETIC [29]	✗	-	73.2	73.5	60.4	83.3
TIES-MERGING	✗	-	69.7 [-3.2]	74.4 [+0.9]	72.4 [+6.6]	86.0 [+2.7]
FISHER MERGING [45]	✓	62.2	68.9	64.6	68.3	82.2
REGMEAN [31]	✓	58.0	71.2	73.2	71.8	83.7
TASK ARITHMETIC [29]	✓	63.9	73.2	73.3	70.1	84.5
TIES-MERGING	✓	66.4 [+2.5]	73.9 [+0.7]	76.9 [+3.6]	73.6 [+1.8]	86.0 [+1.5]
Table 1: Comparing model merging methods across multiple fine-tuning settings and modalities (NLP and Vision) with and without the availability of a validation set.
a least-squares regression problem that aims to minimize the distance between the merged model's activations and the individual models' activations as  $\theta_{m} = (\sum_{t=1}^{n} X_{t}^{T} X_{t})^{-1} \sum_{t=1}^{n} (X_{t}^{T} X_{t} \theta_{t})$ , where  $X_{t}$  is the input activation of a given layer. (4) Task Artithmetic [29] scales and then adds the task vectors to the initial model to produce the merged model as  $\theta_{m} = \theta_{\mathrm{init}} + \lambda * \sum_{t=1}^{n} \tau_{t}$ . In addition to these baselines, we present the performance of the individual fine-tuned models involved in the merging process as well as the performance of a multi-task model trained on the concatenation of all tasks' datasets. For more details on compute resources, dataset licenses, and the finetuning procedures, refer to Appendix C.1, C.2, and C.6.
Merging in Absence of the Validation Set. Prior works [29, 45, 82] on model merging assume access to a validation set, which is utilized to compute the Fisher matrix or tune hyper-parameters. To avoid the need for a validation set, RegMean [31] proposed storing and transmitting inner product matrices of the training data for each task that are the same size as the original model. This can quickly become expensive for large models as the storage and transmission scale linearly with model size and the number of tasks.
To consider the setting where no validation set is available, we developed a generic recipe of TIES-MERGING with fixed hyperparameters that could be applied in any setting without hyperparameter tuning on a validation set. The recipe keeps the top-20% parameters in the task vector resetting the rest to 0 and sets  $\lambda = 1$ . We chose this recipe based on results in the parameter-efficient fine-tuning (PEFT) setting, so we only apply it to the unseen settings of full model fine-tuning on ViT (vision) and T5 (language) models. We also compare TIES-MERGING with the Task Arithmetic method without a validation set by utilizing the recommended value of  $\lambda = 0.4$  [29]. For further details on how this recipe was created please refer to Appendix C.4.
6 Main Results
Our main goal is to merge multiple task-specific models into a single multitask model that can perform well on both in-domain and out-of-domain scenarios. In this section, we evaluate the performance of TIES-MERGING with other methods across multiple different experimental settings.
Merging PEFT Models. Consider the setting where task vectors are computed based on parameters introduced during parameter-efficient fine-tuning. Specifically, we focus on  $(\mathrm{IA})^3$  [43], a PEFT method that scales the base model activations with learned vectors. We follow Liu et al. [43] and use T0-3B [66] as the base model and finetune  $(\mathrm{IA})^3$  models on the train split of eleven datasets including sentence completion (COPA [61], H-SWAG [88], and Story Cloze [68] datasets), natural language inference (ANLI [49], CB [44], and RTE [11]), coreference resolution (WSC [37] and Winogrande [64]), and word sense disambiguation (WiC [53]). When fine-tuning  $(\mathrm{IA})^3$  parameters added to the T0-3B model, we use prompt templates from the Public Pool of Prompts (P3 [5]) to convert each example in each dataset to a prompted text-to-text format where each label corresponds to a different string. For experiments with  $(\mathrm{IA})^3$ , for each dataset, we report the median score across all templates.