What Matters for Model Merging at Scale?
Authors: Prateek Yadav
Tu Vu
Jonathan Lai
Alexandra Chronopoulou
Manaal Faruqui
Mohit Bansal
Tsendsuren Munkhdalai
Venue: Accepted by TMLR ()

Abstract: Model merging aims to combine multiple expert models into a more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for decentralized model development. Despite its promise, previous studies have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how it interplays with other key factors—like the base model quality and number of expert models— to affect the merged model’s performance. This work systematically evaluates the utility of model merging at scale for transformer based models to examine the impact of these different factors. We experiment with merging fully fine-tuned models using four popular merging methods—Averaging, Task Arithmetic, Dare-TIES, and TIES-Merging—across model sizes ranging from 1B to 64B parameters and merging up to 8 different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert’s training tasks, and zero-shot generalization to unseen held-out tasks. Our wide range of experiments provide several new insights about merging transformer based language models at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e., models with good zero-shot performance, compared to pre-trained ones. Second, larger models perform better when merged. Third merging consistently improves generalization capabilities. Notably, when merging eight large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, our findings shed light on some interesting properties of model merging while also highlighting some limitations.

Published in Transactions on Machine Learning Research (07/2025)
What Matters for Model Merging at Scale?
Pratcck Yaday' The University of North Carolina at Chapel Hill Google Mind
praty@cc.unc.edu
Tu Vu
tuvu@vt.edu
Virginia Tech
Google Deep
Jonathan Lai
jhflai @google.com
Google Mind
Alexandra Chronopoulou Google Meep
alezandtrachroon@ggggleecom
Manaal Faruqui" Google Mind
faaruuui @google.cm
Mohit Bansal
abaasal@cs..ncee
The University of North Carolina Chapel Hill
Tsendsuren Munkhdalai
teendsuren@googlee.com
Google Mind
Reviewed OpenReview: https: oppnrreveew. net/ forum? id= 9sbe tmuNppy
Abstract
Model merging aims to combine multiple expert models into more capable single model, offering benefits such as reduced storage and serving costs, improved generalization, and support for docentralized model development. Despite its promise, previous studics have primarily focused on merging a few small models. This leaves many unanswered questions about the effect of scaling model size and how interplays with other key factors like the base model quality and number of expert models to affect the merged model's performance. This work systematically evaluates the utility of model merging at scale for transformer based models to examine the impact of these different factors. We experiment with merging fully fine-tuned models using four popular merging Dare-TIES, and TIES-Merging across model sizes ranging from 1B to 64B parameters and merging up to different expert models. We evaluate the merged models on both held-in tasks, i.e., the expert's training tasks, and zero-shot generalization to unseen held-out tasks. Our wide range of experiments provide several new insights about merging transformer based models at scale and the interplay between different factors. First, we find that merging is more effective when experts are created from strong base models, i.e. models with good zero-shot performance, compared to pre-trained ones. Second, larger models perform better when merged. Third merging consistently improves generalization capabilities. Notably, when merging eight large expert models, the merged models often generalize better compared to the multitask trained models. Fourth, we can better merge more expert models when working with larger models. Fifth, different merging methods behave very similarly at larger scales. Overall, findings shed light some interesting properties of model merging while also highlighting some limitations.
Averaging, Task Ari thmetic.
Work done while Google Deed

---

Published in Transactions on Machine Learning Research (07/2025)
Model PalM
Average
Size
Base Modal
Average
Size
Multitask
Task Arithmetic
-18
FRLM
Task Artthmetic
-B
2-1M-2-77
TIES
48
PALM 2-1
-0
Dare-TIES
248
1.0
Dare-TIES
245
Dare-TIES
4-10
Dare-TIES
64B
18
1B
TIES
TIES
Task Arithmetic
Task Arithmetic
0.4
0.4
Multitask
Multitask
Average
Average
Average
Average
Multitask
Multitask
Task Arithmetic
Task Arithmetic
TIES
TIES
248
BB
2
8B Dare-TIES
Dare-TIES
Dare-TIEs
Dare-TIES
TIES
TIES
Task Arithmetic
Multitask
Task Arithmetic
Multitask
Average
Average
(a) Merging experts, Held-In.
(b) Merging 8 experts, Held-In.
Figure 1: Held-In performance results from our large scale model merging experiments conducted over keys factors like base models, model sizes, merging methods, and number of experts being merged. We present results for two base models, PaLM-2 and an instruction tuned version of it, PaLM-2-IT, four different models sizes (1B, 8B, 24B, 64B), four merging methods (Averaging, Task Arithmetic, Dare- TIES, and TIES-Merging), when merging either with the oracle expert's performance which is denoted by the bold black circle of radius 1. We also present the performance of multilask baseline train on the held-in tasks. We find merging expert models created from the instruction tuned PaLM-2-IT model always performs better than merging PaLM-2 based experts. Morcover, the gap between these model increase when we merge more experts. Larger experts (64B) merge better and show the best held-in performance.
8 expert models. We report the performance normalized
1 Introduction
Model merging (Raffel, 2021) refers the process of combining two or more constituent (expert) models to produce new, and potentially more powerful model. The appeal of this technique is rooted in several benefits it can confer: first, it dramatically reduces storage and serving costs by reusing single model across tasks; second, it enables compositional combination of capabilities from expert models, which can improve generalization to novel tasks; and third, merging supports decentralized and modular model development by allowing multiple contributors to independently build models and later combine them together.
These characteristics have led to a great deal of recent efforts in developing cost-effective model merging methods (Matena & Raffel, 2022b; Ilharco al., 2022; Jin al., 2022; Yadav et al., 2024b; Yang al., 2023; Yu et al., 2024d; Shah et al., 2023; Tam et al., 2023; Zhao et al., 2024), often using simple arithmetic operations, such averaging the parameters of the constituent models. However, most of these studies are limited to small-scale experiments with relatively small models (lypically < 7B parameters) and merging 2 or 3 experts (Yu et al., 2024a;c), and mainly focus on improving benchmark performance on held-in tasks that the expert models were trained on (Yu et al., 2024a; Yadav et al., 2024b). Despite the promiscs that model merging holds, the research community still lacks comprehensive study to evaluate ils effectiveness as we scale the model size. Moreover, it not clear how scale interplays with other factors like number of expert models and base model quality to affect the merged model's held-in performance and zero-shot generalization. This is of paramount importance, models are rapidly growing in size, and more open-weight models and datasets are becoming available," driving the need for practical and scalable merging methods.
Our primary goal in this paper is to provide insight into the scalability of model merging for transformer based models. Although some studies have explored merging at the 13B parameter scale (Huang et al., 2024a; Yu et al., 2024d:b), they primarily leverage increased model size and combine only 2-3 models to attain better performance on held-in tasks. As such, the interplay of factors like model size, base model quality,
As of writing Hugging Face hosts a hora commanity-contributed resources, with models and 200K datasets.
2

---

Published in Transactions on Machine Learning Research (07 (07/2025)
PaLM-2-IT, 1B
PaLM-2-IT, 24B
PaLM-2-IT, 64B
Held-Out Bald-Out Performance
Merging Method
Task Arithmetic
1.1
1.1
1.1
TIES
Multitask
PaLM-2-IT
PaLM-2-IT
PaLM-2-IT
6
8
6
8
6
8
# of Experts
Figure Merged experts created from big and strong base models generalize better than multitask models. We find that for strong base models as we merge more experts (x-axis, +), the merged model's generalization performance (y-axis, t) monolonically increases to approach and eventually surpasses multitask baseline. (yellow line). More details in Section 4.3.
number of constituent models and their effect both held-in and zero-shot generalization performance remamis largely uncrplored. Hence, we aim to address the following four research questions (RQ):
RQ1: What is the effect of using pretrained vs. instruction-tan base models for creating expert models for merging?
RQ2: Does model merging perform better or worse as the model size increases?
RQ3: How does merging affect zero-shot generalization to held-out tasks, and how this influenced by model size?
RQ4: How many expert models can be merged without performance loss, and how does this depend on model size? size?
To answer these question, we systematically evaluate the effectiveness of current Shate-of-the-ary merging methods through empirical experiments. Specifically, we utilize the PaLM-2 model (Anil et al., 2023) and its isstruttion-uune variant, PaLM-2-IT, while scaling the model sizes up to 64B parameters. We experiment with four popular merging methods, namely, Averaging (Wortsman et al., 2022a; Choshen et al., 2022b), Task Arithmetic (Ilharco et al., 2022), TIES-Merging (Yadav et al., 2024b), and Dare-TIES (Yu et al., 2024d). We conduct a series of sensitivity and ablation experiments to understand the relative importance of several factors like model size (1B, 8B, 24B, 64B parameters), base model quality (pretrained vs. instruction-tunele and number of constituent models (2, 4, 6, being merged. Additionally, we consider two axes evaluation using the TO data collection (Sanh et al., 2021a): held-in evaluation with tasks the expert models were trained on, and held-out, for zero-shot generalization to unseen tasks.
Our experiment results shed light the promises of model merging and reveal interesting insights into the behaviors of different. factors at scale. Firsl, find that the model initialization plays a crucial role in enhancing the performance of the merged model. Specifically, across all evaluation settings, using strong zero-shot instruction- tuned base models to create expert models leads to improved performance compared to using pretrained models (see $4.1). Second, larger models perform better when merged. This holds true regardless of the base model used (instruction-tumed or not), number of models merged, or merging method (see $1.2). Third, our results demonstrate that merging significantly enhances zero-shot generalization, consistently improving the ability to adapt to new tasks. Notobly, when using strong base models as the number of merged experts increases, the merged model either matches exceeds the performance of a strong multi-task training bascline (see $4.3). Fourth, larger models are better at merging a larger number of expert models (see $4.4). Finally, our numerous experiments identify specific sellings where we expect model merging to be much more useful. From this we provide general recommendations for practitioners (see $4.7). Taken as a whole, our findings are powerful testament to the potential of model merging at scale for creating highly generalizable language models, which we hope will spur more fundamental rescarch into the development of practical and scalable merging methods.
3

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
2
Background
Model merging has emerged as cost-cffective method for developing improved models. Two common use cases of merging are: (1) combining model checkpoints from different data versions, hyperparameters, or training stages to enhance distributional robustness (Team et al., 2024; Dubey et al., 2024), and (2) combining multiple expert models trained on different datasets to leverage their complementary capabilities. In both scenarios, the expert models generally share a common architecture and a base model from which the expert models are created via fine-tuning.
This work focuses on merging specialized, fine-tuned versions (experts) of single base model to enhance its capabilities. Each expert model is trained on distinct datasets covering different tasks, domains, and/or capabilities. We refer to the tasks/dataser used for training the expert models as "held-in", while those that are new and unseen are called "held-out". Our goal is to create a unified model that retains the individual expert models' capabilities on held-in tasks while improving zero-shot generalization on held-out tasks. This merging approach provides a flexible, modular method for post- post-training large language models, facilitating the addition of new features and capabilities to top-performing models.
2.1 Model Merging Methods
We denote the set of N expert tasks as t....., , by and the base model weights, representing the common ancestor of all expert models as Ouasa- The weights of the corresponding specialized expert models, each obtained by fully fine-tuning the base model on specific expert task, are denoted as 01. ... 8x. respectively. We focus on "open vocabulary' models which utilize natural language as input and output for both classification and generation tasks, eliminating the need for task-specific classification heads making the merging process simpler. Given this, model merging methods can be defined as a function MM.). This function takes as input the base model, the set of N expert models, and potentially additional information, denoted by t. This additional information may include activation statistics, Fisher matrices, or other method- specific data. The oulput of the function is the merged model, represented by its parameters Oz. Formally, 0g =M Oreee: L) where & is method specific data.
Given our focus on studying model merging with large models, we select four merging methods based on their popularity and simplicity. We only study merging methods that can scale to tens of billions of model weight parameters and do not require any additional information to perform merging, i.e., & = O: as these techniques are efficient for even larger models. Other more complex methods that require computing fisher matrices (Matena & Raffel, 2022a), backward passes (Yang et al., 2023), or additional information like model activation (Jin et al., 2023) are skipped because of their computational complexities for large scale model merging that we focus in this work. Next, we describe the four selected model merging methods in detail.
2.1.1 Averaging
Parameter averaging (Choshen et al., 2022b; Wortsman et al., 2022a) is well-established technique in federated learning (McMahan et al., 2017) and recent applications extend its utility to merge models for enhancing model robustness against -ut-of-distribuutin data (Wortsman et al., 2022b; Ramé et al., 2022a), refine pre-trained models (Yu et al., 2024a), develop multimodal models (Sung et al., 2023), and create multitask models by combining capabilities (Yadav et al., 2024b; Ilharco et al., 2022). Parameter averaging is achioved by taking a mean of all the expert model weights together without using the base model which can be formally described as, M({f1]H Srase) = E 0g.
2.1.2 Task Arithmetic
Task Arithmetic (Ilharco et al., 2022) introduces a novel concept of "task vectors' for model merging. For task t,, the lask vector is denoted as T, = 0, - Orasa which capt ures lask-specific knowledge by quantifying the difference between the fine-tuned expert parameters (0z) and the original base model parameters (eose)) A scaling hyperparamcter controls the contribution of the aggregated task-spocific knowledge to the final model. The merged model is then constructed by linearly combining the base model parameters with scaled sum of all task vectors. Formally, task arithmetic can be described as, M([0 |1=1> Orase; X) Orase + A D" (0, Obasa).
4

---

Published in Transactions on Machine Learning Research (07/ 2205))
2.1.3 TIES Merging
TIES-Merging (Yadav et al., 2024b) identifies two main challenges with model merging: 1 during finetuning expert models accumulate a lot of noise in the parameters, and @ different experts might want to change the same parameter in different directions leading to interference/ cnaeferrence/oooffittt /confict between the expert models. They demonstrate that both of these factors hurt model merging and propose three steps process to remove redundant, parameters, followed by resolving sign conflicts, and finally aggregating only the parameters that. are not conflicting. Specifically, in TIES Merging they first zero out the values in each task vector that have low magnitudes to obtain the trimmed task vector é for each task. Next, they chose the aggregate sign (7)) for each parameter based on whether the parameter has a higher total magnitude in the positive or the negative direction across all trimmed task vector, formally, % = sgn( ft). Next, for each parameters p the models whose sign matches the aggregate sign are averaged to obtain the merged task vector. Finally, the merged model obtained by scaling the merged task vector using a hyperparameter À and then added back to the base model as, 0o Guecr + x 1dy EiEAp f.. where AP {i E [N] 1 = =
2.1.4 Dare Merging
Dare (Yu et al., 2024a) extends the idea of TIES merging by proposing to use dropout-like pruning stage to remove noise before merging. Specifically, a Bernoulli mask M2 with drop probability is applied to each task vector to obtain the pruned task vector A = (1 - M2) O / - -p)- This stochastic process randomly zeroes out elements within the task vector while preserving its expected value. These pruned task vectors are then used along with either TIES Merging or Task Arithmetic. Due to the popularity of the Dare variant that uses TIES Merging, we use that to represent. the Dare method and call it Dare-TIES.
2.2 Challenges/Limitations
Model Merging has been utilized at a growing rate in practice as has recently been applied to building modern language models like Llama-3 (Dubey et al., 2024) and Gemma-2 (Team et al., 2024). However, most. formal studies on model merging have been performed with relatively small models. There are few studies that look at larger models with 7B and 13B parameters. However, those studies mostly focus on merging 2-3 models to improve benchmark numbers as opposed to better underst aaddng how the size of the model affects the model merging process and the resultant model. To motivate our work, we present some of the limitations of the existing studies and highlight their difference with our work.
Most Studies on Small Models (< 7B parameters): Most existing model merging papers rarely use large models (> 7B). For example past works (He et al., 2024; Ortiz-Jimener et al., 2024; Jang et al., 2024), including methods like ModelSoup (Wortsman et al., 2022a), Task Arithmetic (Ilharco et al., 2023) and TIES-Merging (Yadav et al., 2024b), Ada-Merging (Yang et al., 2023), MatS (Tam et al., 2024) perform experiments with model families like CLIP (Radford al., 2021), ViT (Dosovilskiy et al., 2021). 1; T5 (Raffel et al., 2020a), DeBERTa (He et al., 2021), Roberta (Liu al., 2019), BERT (Devlin et al., 2018) with less than 1B parameters. Hence, it is unclear how well model merging works for large models, what factors play important role, the effect of model size, number of tasks being merged, and its effect both held-in performance and generalization of the model. Some studies hypothesize that bigger models perform better when merged, however there are no concrete large scale studies to thoroughly assess such claims at large scale.
Model Merging Studies with Large Models are Shallow: Some recent works like DARE (Yu et al., 2024a), WIDEN (Yu et al, 2024c), Chat-Vector (Huang et al., 2024b) demonstrate merging results for larger models with up to 13B parameters, however these studies have a few limitations: ! They primarily focus using model merging to improve model quality and hence their experiments do not provide concrete insights on how model size interplays with merging, They only merge a maximum of two or three models at once, O They primarily focus on held-in tasks and do not provide any insights on the effect of merging on a model's generalization abilitics. Other works like RewardSoup (Rame et al., 2024), WARM (Rame et al., 2024), FuseLLM (Wan al., 2024) also work with 7B sized models and focus on specific applications of model merging without providing any deeper insight about how merging performance changes for large models.
5

---

Published in Transactions on Machine Learning Research (07/2025)
Varied Evaluation Setups: Most previous works rarely share their experimental setup where both the expert datasets and the objective vary. For example, Task Arithmetic (Πharoo et. al., 2023), TIES (Yadav et al., 2024b), MaTS (Tam et al., 2024) uses GLUE tasks (Wang et al., 2018), Vision tasks, TO held-out, and TO held-in (Sanh et al., 2021b) tasks respectively. Morcover, different works evaluate for different use cases like intermediate lask training in Fisher merging (Matena & Raffel, 2022a), robustness in modelsoups (Wortsman et al., 2022a); and held-in performance for Dare (Yu et al., 2024a), both held-in and held-out performance in TIES Merging (Yadav et al., 2024b). Given our focus on combining model capabilities in the post training phase, focus evaluating on both held-in tasks and generalization to unseen held-out tasks.
3 Large Scale Evaluation of Model Merging
In this work, we address the limitations mentioned above by systematically undorstanding the effect of various factors like model size, base model quality, merging method, and the number of models being merged on both the held-in and generalization performance of the merged model. Next, we describe our experimental design.
Data: Sanh el al. (2021a) found that explicit multitask training of T5 (Raffel et al., 2020b) on a collection of prompted datasets produces a model with strong zero-shot performance on unseen tasks. This has become a common experimental setting for benchmarking zero-shot generalization (e.g. (Longpre et al., 2023; Jang et al., 2023; Zhou et al., 2022; Chung al., 2024; Muqeeth et al., 2024). Hence, we adopt the experimental setting from the TO mixture (Sanh et al., 2021a) which contains 8 held-in and 4 held-out, task categories. For each of these categories there are multiple datasets in the TO mixture (Sanh et al., 2021b) and hence to reduce evaluation costs, we select 2 datasels from each category based on the popularity and the train dataset size. Specifically, the 8 held-in task categories (with a total of 16 datasets) include Multiple-choice QA, Extractive QA, Closed-Book QA, Sentiment Analysis, Topic Classification, Structure-to-text, Summarization, and Paraphrase Identification. Similary, the 4 held-out task categories (with a total of 7 datasets) are Sentence Completion, Natural Language Inference, Coreference Resolution, and Word Sense Disambiguation. For more details see Section A.
Recognizing the significance of post-training for LLMs where models are typically fully fine-tuned, we perform full fine-tuning to create our expert models to better mimic the post-training sctting. Moreover, in post-training phases it is common to first perform Instruction Tuming (IT) on the model before moving on to other steps. Hence, we examine the effect of using strong instruction-tunnd base models on the process and outcome of model merging. Given this, utilize the transformer based PaLM-2 models (Anil et al., 2023) with sizes 1B, 8B, 24B, and 64B as our base models (Dbase). To obtain the instruction tuned base model, we further fine-tuned the PaLM-2 models on the FLAN-v2 dataset (Longpre et al., 2023) while excluding the T0-mixture tasks (Sanh et al., 2021a). These instruction-tumed variants are denoted as PaLM-2-IT. For each of the 2 base model types (non-IT vs IT) and 4 model sizes, we perform full fine-tuning on the 8 held-in task categorics resulting 64 specializod experts models which are then used further in our experiments. Comprehensive details regarding hyper parameters and computational requirements are provided in Appendix B.
Expert Model Creation:
Experimental Setting: Given our collection of expert models, for each merging experiment we select a subset of expert models which we call the constituent models. We create a large merging experiment grid with base models (PaLM-2 and PaLM-2-IT), 4 model sizes (1B, 8B, 24B, 64B), Merging methods (Averaging, Task Arithmetic, TIES, and TIES), the number of constituent models (2, 4, 6, 8), and 3 seeds to randomly select the constituent tasks for the experiment resulting in total of 384 merging experiments. These seeds are shared across different experimental settings and control the different combinations of models that are selected for merging. They ensure that the same lasks are selected across base models, model sizes and merging methods to ensure fair comparison. For example, in an experiment where we merged 2 expert models, derived from the 64B PaLM-2 base model with the constituent models being MCQ and Summarization experts while the same experiment with a different seed resulted in Closed Book QA and Sentiment Analysis experts the constituent models.
6

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Experts
Experts
Experts
Experts
Performanee
1.1
1.0
Expert Models
1.1
Expert Models
1.0
1.2
Expert Models
1.2
Expert Models
PaLM
PaLM-2-IT
Models
0.3
0.8
0.8
Merging Methods
Average
0.6
Task Arithmetic
0.5
0.4
0.4
Dare-TIES
Model Size
Figure 3: Instruction-tuned models perform better when merged. PaLM-2-IT (+) consistently outperforms PaLM-2 (+) as shown by the huge gap between the green point (+) being higher than red points (+), across various merging methods, model sizes, and numbers of constituent models, indicating that stronger instruction-tuned base models enhance the performance of merged models. The dashed lines denoted the performance of the experts trained the held-in tasks defined in 3. For more details Section 4.1.
Evaluation: For cach of the experiments above, we assess the mergod model's performance by evaluating it both the held-in lasks i.e., the training lasks of the constituent expert models and all 4 held-out task categories. For example, if the constituent models are MCQ and Summarization experts, then for held-in tasks we evaluate on the MCQ datasets (DREAM and Cosmos QA) and Summarization datasets (CNN Daily Mail and XSum) resulting a total of held-in evaluation datasets. Moreover, all merging experiments are also evaluated on the 4 held-out tasks categories consisting of 7 datasets listed in Appendix A. There we perform approxinately ~ 9000 model evaluations across all of our experiments.
Metric: Given that different datasets use different metrics, we normalize the performance metrics to make them unitless so that they can be aggregated. Similar to Task Arithmetic (Ilharco et al., 2022) and TIES- Merging (Yadav al., 2024b), for held-in tasks, the merged model's performance is normalized against the corresponding task expert model's performance. However, for held-out tasks, the normalization was performed relative to the base model's performance.
For example, say we have two held-in tasks f1, to and two held-out tasks t2 and t4. Now we train an expert model on Ej and Es on the tasks é,, i2 respectively. Next, we merge experts En and Es to obtain the merged model, M. Now, to evaluated M on the held-in tasks, we report the average normalized performance on a.n()
acc[M. and ecc(Es,fa) where acc(E, t) means the accuracy its held-in tasks (t) and f2) as the average of acc(Ei,4)) of expert model E on task t. For held-out tasks, the denominator is the performance of the model from which the experts are created. Concretely, the held-out performance is the average of acc(M.fa) acc acc(bese.fs) acc(base,f4))
where base is the model from which exports are created.
We denote this metric as normalized performance throughout the paper. Importantly, want to emphasise that this metric is relative, with value of 1 indicating performance comparable to the reference model. Hence, for held-in lasks a value of 1 means performance similar to the domain expert model while for held-out tasks it means performance is similar to the base model. We mark this line in most of our figures and specify the models that are used for normalization. Finally, to generate aggregated results, we compute the mean of normalized performance across all datasets within each category, then across all categories and then over the three seeds.
4 Experimental Results
In this section, we explore the interplay between model size and key factors such as base model quality, merging method, and the number of constit (expert) model, along with their effect on both held-in and zero-shot generalization (held-out) performance. Our findings are: Merging is more effective when the constituent models are derived from instruction-tuned base models rather than pretrained ones (scc $4.1); Larger models perform better when merged ($4.2); ? Merging significantly improves zero-shot generalization, with instruction-tamed models benefiting from increased constituent models, and larger model
7

---

Published in Transactions on Machine Learning Research (07 2025)
Held-In Performance
PaLM-2, Experts
PaLM-2-IT, 2 Experts
PaLM-2-IT, Experts
Expert Models
Expert Models
Expert Models
1.0
1.0
1.0
Size
1B
0.8
0.8
0.8
248
64B
0.6
0.6
0.6
Average
Dare-TIES TIES
Task Arith.
Average
Dare-TIES TIES
TIES
Task Arith.
Average
Dare-TIES TIES
Task Arith.
Merging Method
Figure 4: Bigger models merge better. On Held-In evaluations, we find that bigger models always perform better compared lo smaller models, barring a few outliers. We find that large instruction tuned models like 64B PaLM-2-IT perform best when merged. For more details see Section 4.2.
sizes allowing the merged model to match or exceed multi-task training ($4.3); We can merge more models effectively when using larger models ($4.4); and @ Different merging methods perform similarly when applied large-scale instruction-tuned models. Below, we outline the experimental setup and discuss these findings in detail. Our full results along with the standard deviations can be found in Appendir C and Table 2, 3,4,5.
4.1 Instruction- Tuned Models Facilitate Perform Better When Merged
Experimental Setup: Prior works suggests connection between good zero-shot models and effective model merging. Wortsman et al. (2022a) demonstrate that averaging strong zero-shot models improves out-of-distribution robustness. Ortiz-Jimenez et al. (2024) indicate that effective pretraining allows weight disentanglement, and thus enhances merging. Yadav et al. (2024b); Ilharco et al. (2023) propose that instruction tuned base models could aid model merging, though this hypothesis remains largely untested.
To assess how base model quality affects the held-in performance of merged models, perform merging exper- iments with fully fine-tuned experts from PaLM-2 and PaLM-2-IT. We vary model sizes in {1B, 8B, 24B, 64B) and the number of constituent models in 2, Held-in performance is mcasured over three trials in which different combinations of models are selected in order to minimize the impact of selected expert models and their data distributions on performance trends. A consistent seed is used to select the tasks across different base models, model sizes, and merging methods to ensure fair task comparisons. We evaluate four merging methods: averaging, task arithmetic, TIES, and Daro-TIEs and also compare against the performance of lask-specific expert models.
Findings: Our results, presented in Figure 3, indicate that PaLM-2-IT models denoted by green color (+), consistently outperforms PaLM-2 models (+) across various merging methods (+, *). model sizes (x-axis +), and numbers of constituent models (subplots). This supports our hypothesis that for transformer based LLMs stronger instruction-tuned base models enhance the performance of merged models. Similar to the findings of Orliz-Jimenez el. al. (2024), we believe that for transformer based LLMss large-scale instruction tuning further disentangles model weights, facilitating effective model merging and improving the base model's zero-shot performance.
4.2 Model Merging Performs Better With Bigger Models
In this section, we explore the effect of model size on the held-in performance of merged models. We run experiments using different model sizes, base models, merging methods, and numbers of constituent models. As in the previous experiment, we report the average results over three random seeds and compare the performance of the merged models to that of the task-specific expert models.
Experimental Setup:
8

---

Published in Transactions on Machine Learning Research (07/2025)
PaLM-2, 1B
PaLM-2, 24B
PaLM-2, 64B
Held- Out Performance
1.8
1.8
1.8
1.6
1.6
1.6
Merging Method
Task Arithmetic
1.4
1.4
1.4
TIES
Multitask
1.2
1.2
1.2
1.0
1.0
1.0
PaLM-2
PaLM-2
PaLM-2
8
4
6
of Experts
Figure 5: Merged models at scale generalize better. We plot the held-out, generalization of the merged model for two merging methods. We also include the performance of base model (dashed line) and the multitask baseline (yellow line) which trained constituent expert models (x-axis, -) had little effect on zero-shot generalization as shown in the left and center plots. However, increasing model size significantly to 64B improved the merged model's performance over the base model (right plot). For more details see Section 4.3.
mixture of held-in tasks. We find that the number of
Findings: Figure illustrates how incrcasing base model size impacts merging effectiveness. As model size grows (denoted by colors, trend is consistent across all base models (different subplots), merging methods (x-axis +), and numbers of constituent models (subplots). For large instruction-tuned PaLM-2-IT models, the merged models perform nearly as well as task-specific expert models denoted by dashed line. These results demonstrate that for transformer based LLMS larger models facilitate merging. This suggests promising approach for developing adaptive, modular post-training recipes. If the remaining performance gap can be further reduced, model merging could become cost-effective alternative to multitask training. Our full results across all settings with standard deviations are available in the Appendix C.
), merged model performance generally improves. This positive
4.3 Merged Models at Scale Generalize Better
Experimental Setup: Expert models created by fine-tuning our base model on specialized tasks, which can lead to decrease in its generalization capabilities. This raises the question: How well, if at all, the merged model generalise to held-out tasks? Ideally, the merged model should perform at least as well as the base model on these tasks. To explore this, we evaluate the merged model's performance on unseen tasks across various model sizes, merging methods, and numbers of constituent models. Additionally, we compare these merging approach to a traditional multitask baseline, where single model is trained on a mixture of all eight held-in task categories. As detailed in Section 3, we normalize the performance of both the merged and multitask model against the base model to assess relative gains losses in generalization abilitics.
Findings: Figure 2 and Figure 5 show the zero-shot generalization performance of the merged model using PaLM-2-IT and PaLM-2, respectively. Overall, we find that: The merged models outperform their corresponding base models in zero-shot generalization to held-out tasks, as indicated by performance values greater than 1 most cases; This improvement consistent across various model sizes (denoted by subplot), base models (different figures), merging methods (different colors
and numbers of constituent. models (on x-axis +), suggesting that merging for transformer based LLMs generally improves generalization; @ For weak base models (i.e., PaLM-2) illustrated in Figure 5, the number of constituent expert models had little effect zero-shot generalization (Left and Center plots). However, increasing model size significantly improved the merged model's performance over the base model (Right plot); θ In contrast, strong basc models (PaLM-2-IT) show a different. (rend, zero-shot generalization monolonically improves with the addition of more expert models as shown in Figure 2. We hypothesize this positive correlation arises as the noise learned during finetuning canceled out when merging different expert models, resulting better generalization; and Notably, for transformer based LLMs the merged model outperforms the multitask baseline when combining more than 6 large instruction-tuned expert models (over 24B). This indicates that transformer
9

---

Published in Transactions on Machine Learning Research (07/2025)
Normalized Performance
PaLM-2, Held-In
PaLM-2-IT, Held-In
PaLM-2, Held-Out
PaLM-2-IT, Held-Out
Expert Model
Expert Model
1.4
1.4
1.0
1.0
1.3
1.3
Size
0.8
1.2
1.2
18
8B
1.1
248
0.6
0.6
1.0
1.0
0.4
PaLM-2
PaLM-2-IT
0.4
0.9
0.9
2
4
6
2
4
6
2
4
6
2
4
6
of Experts
Figure 6: Bigger model sizes can merge more experts. We merge experts of various sizes created from PaLM-2 and PaLM-2-IT models and plot the held-in (left) and held-out (right) performance merged models. While PaLM-2's held-in performance degrades with more experts, PaLM-2-IT' performance stabilizes much higher level. Both PaLM-2 and PaLM-2-IT models consistently improve held-out, generalization, particularly at. 24B and 64B scales with increasing expert count. For more details see Section 4.4.
based LLMs models developed through merging can generalize even better than those trained on a multitask mixture, offering promising approach for developing highly capable language models. Our full results on other merging methods and model size available in Appendix C.
4.4 Bigger Model Sizes Can Merge More Experts
Experimental Setup: When creating multitask models, datasets for different tasks domains typically combined. In contrast, model merging involves developing separate expert models for each task or domain before combining them. Previous work has shown that merging multiple models can reduce the quality of the resulting model (Yadav et al., 2024b; Ilharco et al., 2022). In this study, we experiment with merging up to expert models from various base models, model sizes, and merging methods to assess their impact on successful merges.
Findings: Figure 6 shows the held-in and held-out performance of the merged models using Task Arithmetic as the number of constiluent models increases shown on x-axis. Results for other methods can be found in Appendix C. Overall, we observe that: Unlike merging with PaLM-2, where held-in performance typically declines with more model merges, merging with stronger zero-shot PaLM-2-IT initially drops slightly performance before stabilizing as number of constituent models increase. For example, merging eight 8B PaLM-2 models decreases performance from 0.66 to 0.39 when increasing the number of experts from 2 8, whereas PaLM-2-IT's performance only slightly drops from 0.91 to 0.86; @ In the held-out evaluations, the merged experts based on PaLM-2 models generally outperform the base PALM-2 models by small margin. However, with larger model sizes (64B), the performance improvement increases significantly, achieving about 30 percentage relative improvement. We attribute this substantial gain to the base PaLM-2 model's weak zero-shot performance; and The merged models based on PaLM-2-IT show improved generalization lization over PaLM-2-IT across all settings. Additionally, for the 24B and 64B models, we observe consistent increase in generalization capabilities with the addition of more constituent. expert models.
4.5 Merging Llama Based Models
Given the compute constraints, we focused on single transformers based model family PaLM-2. However, to assess the generality of some of our claims performed experiments using the Llama-2 (Touvron et al., 2023) based models. We use existing Llama-2 based fully finetuned models, specifically we work with the WizardMath (Luo et al., 2023) and Code-Llama (Roziere et al., 2023a) models with 7B, 13B, and 70B parameters available on huggingface. We merge the Code-llama and Wizardmath models using all the merging methods and present the results. The results in Table 1 show similar trends to what we observed earlier in the paper: (1) As the model size increases the performance of the merged models gets closer to the expert models held-in tasks. (2) For large models the performance, different merging met hods perform
10

---

Published in Transactions on Machine Learning Research (07/2025)
very similar to each other. (3) Task Arithmetic, TIES, DARE-Ties typically perform better than averaging for smaller scales. Given this we believe our claims generalize to most transformer based language models.
Table Merging a coding and math expert models created from Llama-2 by performing full finetuning.
Size(U)
Model (4)
Un Un-Normalized
Normalized
GSM8k
MATH
HumanEval
GSM8k
MATH
HumanEval
Llama-2
14.6
2.5
12.2
WizardMath
84.1
43.5
Bodellama-Instruct
34.8
7B
Average
76.5
39.6
31.5
0.91
0.91
0.91
Task Arithmetic
79.1
40.4
32.7
0.94
0.93
0.94
TIES
80.3
41.2
33.1
0.95
0.95
0.95
DARE-TIES
79.7
40.8
33.2
0.95
0.94
0.95
Llama-2
28.7
3.9
20.1
WizardMath
89.7
50.6
Codellama-Instruc
42.7
13B
Average
84.5
48.1
40.1
0.94
0.95
0.94
Task Arithmetic
88.4
48.6
41.4
0.99
0.96
0.97
TIES
87.8
49.4
41.9
0.98
0.98
0.98
DARE-TIES
87.3
49.1
42.1
0.97
0.97
0.99
Llama-2
56.8
13.5
30.5
WizardMath
92.8
58.6
Codell Iieellam--Instruc
67.2
70B
Average
94.2
60.1
69.1
1.02
1.03
1.03
Task Arithmetic
94.4
60.2
69.5
1.02
1.03
1.03
TIES
94.4
60.3
69.4
1.02
1.03
1.03
DARE-TIES
94.3
60.2
69.4
1.02
1.03
1.03
4.6 Merging Methods Become Similar at Scale
We find that all merging methods exhibit similar performance when merging large instruction-tuned transformer based models. This suggests that sim- pler methods like Averaging, can be sufficient for merging large strong expert models. Figure 7 shows the held-in and held-out performance of the 64B ex-
Held-In, 64B
Held-Out, 64B
1.10
Average
Tack Arithmatic
Normalized Metric
1.05
1.10
Expert Model
1.05
PaLM-2-IT
1.00
1.00
0.95
perts derived from PaLM-2-IT. All merging methods yield comparable results on both held-in and held-out tasks for any number of constituent models (shown on x-axis). We hypothesize that for transformer based LLMs as the model size increases, expert models are highly over- prrapaaraeteri due to limited train- ing data. Consequently, the subtle advantages of certain merging techniques - such as highlighting information via task vectors, resolving interference, or pruning which benefit smaller models, bocome less relevant. This indicates need for more practical and scalable methods improve merging at scale.
0.95
0.90
0.90
0.85
0.80
0.80
8
# of Experts
Figure 7: Different merging methods become similar at scale. We plot the held-in and held-out performances of merged 64B PaLM-2-IT models across different merging methods and numbers of constituent models. For more details see Section 4.6.
11

---

Published in Transactions on Machine Learning Research (07/ 2205))
4.7 Discussion and Takeaways
In this section, summarize key insights from study and provide practical recommendations for model merging practitioners intending to use it for transformer based LLMs. Overall, we find that: O Creating expert models from the best available base model is always beneficial. The quality of the basc model can be gauged by its zero-shot generalization capabilities. For transformer based language models, we hypothesize that better generalization leads to improved weight disentanglement (Ortiz-Jimenez et al., 2024) and a flatter loss landscape, enhancing lincar mode connoctivity and facilitating model merging: @ Mergod models often underperform compared to task-specific expert models, indicating a potential loss in performance. Despite this, specialized expert models generally outperform general-purpose multitask models (Liu et al., 2022; Roziere et al., 2023b; Luo et al., 2023), suggesting that for transformer based LLMs the performance loss may not be significant when compared to multitask models trained on specific tasks; and @ Our findings indicate that for P models large-scale merging can accommodate more models and significantly improve generalization, outperforming multitask training when a powerful zero-shot base model is employed. ! Surprisingly, we find that when working with transformer based large instruction tuned models, different. merging methods perform very similary. This implies that using simple merging methods like averaging will result in models that comparable in quality with the models obtained from more advanced merging method. We hope our rescarch inspires further fundamental studies developing more practical and scalable merging methods.
5
Related Work
5.1 Loss Landscape and Weight Interpolation
While the loss function of a neural network is generally non-convex, recent work (Draxler et al., 2018; Freeman & Bruna, 2016; Garipov et al., 2018; Jordan et al., 2023; Gueta et al., 2023) has demonstrated that the parameter values from different training runs can sometimes be interpolated without increasing the loss (i.c. they are mode-connected). Many methods (Kuditipudi et al., 2019; Tatro et al., 2020; Benton al., 2021) have explored finding these low-loss paths between models, focusing on simple (not necessarily linear) interpolations. For example, Frankle et al. (2020) showed that if part of the optimization trajectory is shared between two neural networks then they can be interpolated without lowering accuracy. On the other hand, Neyshabur et al. (2020) showed that naively interpolating two neural networks with completely disjoint optimization trajectories can result in a catastrophic drop in their accuracios. Entezari et al. (2021) hypothesized that if we account for the permutation symmetry of noural networks, then all neural networks of a given architecture trained on the same dataset are linear mode connected. This assumption of the existence of low-loss "basin' in parameter space encompassing the models is critical for model merging (Ilharco et al., 2023). Ainsworth et al. (2022); Singh & Jaggi (2020); Wang et al. (2020); Jordan ct al. (2022); Peña et al. (2023) therefore used techniques based on finding permulations (Wang et al., 2020; Ainsworth el al., 2022) and optimal transport (Singh & Jaggi, 2020) to better align neural networks trained from scratch so that they can be merged or interpolated without increasing the loss.
5.2 Model Merging
Section 2.1 discusses the merging methods that we use for our experiments, however, the popularity of model merging has led to a ever-growing number of methods and applications of model merging (He et al., 2024; Daheim al., 2023; Yadav et al., 2023a;b; 2024b; Matena & Raffel, 2022a; Jin et al., 2023). Next, we discuss some of these methods which were omitted due to large scale practical considerations. Tangent Task Arithmetic (Ortiz-Jimenes et al., 2024) fine-tune models in the tangent space for better weight disentanglement when using Task Arithmetic. Akiba et al. (2024) explore using cvolutionary algorithms to choose which layers to merge. SLERP (Shoemake, 1985) and Model Stock (Jang el al., 2024) consider the geometric properties in weight space where SLERP performs spherical interpolation of model weights while Model Stock approximates a center-close weight based on several FT models, utilizing their backbone as an anchor point. Tang al. (2023) train a mask that learns which parameters are important for the merged model. Ye et al. (2023) train a gating network to predict weight that is then used to compute weighted average of
12

---

Published in Transactions on Machine Learning Research (07 (07/2025)
examples during inference. Yadav et al. (2024a) provides comprehensive survey of methods that train router to route between the different models to merge. Moreover, other applications of model merging include intermediate-task training (Ramé et al., 2022b; Choshen et al., 2022a;b), continual learning (Don-Yet) Yehiya et al., 2022), model alignment (Rame et al., 2024; Ramé et al., 2024), merging pretrained models Yu et al. (2024e), or merging models in different modalities (Sung al., 2023).
6 Conclusions
This study conducted systematic, large-scale empirical investigation of model merging for transformer based language models like PaLM-2 and PaLM-2-IT, addressing the limitations of previous research confined to small-scale models and limited merging scenarios. Through extensive experiments with PaLM-2 and PaLM-2-II models ranging from 1B to 64B parameters, we analyzed the impact model size, base model quality, merging method, and number of experts on both in-domain and out-of-domain generalization performance, Our findings demonstrate that for these models, model merging effectively combines diverse expert. knowledge particularly with increasing model size and with insruction-tunee base models. We found that larger models consistently perform better when merged and can merge more models with less performance degradation. Importantly, for transformer based LLMs model merging led to enhanced generalization capabilities, with large merged models surpassing the performance of multitask models on held-out tasks. These results show that we can develop models that generalise well a decentralized and modular manner.
7 Broader Impact
We believe that this work has no broader implications of its own; however, large language models in general affects user applications and behaviors. Hence, all the implication of LLMs are also aplicable to our work.
References
David Ifeoluwa Adelani, Jade Abbott, Graham Neubig, Daniel D'souza, Julia Kreutzer, Constantine Lignos, Chester Palen-Michel, Happy Buzaaba, Shruti Rijhwani, Sebastian Ruder, et al. Masakhaner: Named entity recognition for african languages. Transactions of the Association for Computational Länguistics, 9: 1116-1131, 2021.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries, 2022. https: //arxiv...ivv.a/aa wwtrrxiv.orr/aaae arg/abs/2209 248222.99.........2..2
Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiu preprint ar arXiv:2403.13188 2024.
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailcy, Zhifeng Chen, et al. Palm 2 technical report. arXiv preprint 230:2305.10000 2023.
Gregory Benton, Wesley Maddox, Sanae Lotfi, and Andrew Gordon Gordon Wilson. Loss surface simplexes for mode connecting volumes and fast ensembling. In International Conference on Machine Learning (ICML), 2021.
13042.
Leshem Choshen, Elad Venezian, Shachar Roh-Yehia, Noam Slonim, and Yoav Katz. Where to start? analyzing the potential value of intermediate models, 2022a. https:// http::///awww..o/aaaa/////// a2g/21bs2221... //////22...0111 00107.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. arXiv preprint arXiv:2204.032 03044, 2022b.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fodus, Yunxuan Li, Xuczhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finchine language models. Journal of Machine Learning Research, 25(70):1-533 2024.
13

---

Published in Transactions on Machine Learning Research (07/2025)
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment challenge. In Ma- chine Learning Challenges Workshop, 2005. https://link springer.rnne.rerrerr cttpeerrrccmoooooeeeeeeeeeeeeeee wwt. cctm//ccaapeer/0..00///7777333 9.
Nico Daheim, Thomas Möllenhoff, Edoardo Maria Ponti, Tryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching, arXiv preprint 0rXiv:2310.12800 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Loe, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint XXix:1810.04805, 2018.
William B. Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In International Workshop on Paraphrasing, 2005. https://aclanthoology
Shachar Don-Ychiya, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Cold fusion: Collaborative descent for distributed multitask finetuning, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021. https: https://pppervveeeee wttp:/oormm..dddeccooooo...
Felix Draxler, Kambis Veschgini, Manfred Salmhofer, and Fred Hamprecht. Essentially no barriers in neural network energy landscape. In International Conference on Machine Learning (ICML), 2018.
https://arxiv, wwtpr/ivvvorggabaa/eeeee wwwp/b//////aaa//////////// 00885.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. orXiv preprinl WxXiv:240.2118.3 2024.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. arXiv preprint arXiv:2110.066 06296, 2021.
Jonathan Frankle, Gintare Karolina. Dziugaite, Daniel Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In International Conference on Machine Learning (ICML), 2020. https://proceeddnngs mlr. mlr.
html...
C Daniel Freeman and Joan Bruna. Topology and geometry of half-rectified network optimization, arXic preprint arXie:1611.015400 2016.
Timur Garipov, Pavel Izmailov, Dmitrii Podoprikhin, Dmitry Vetrov, and Andrew Gordon Wilson. Loss surfaces, mode connectivity, and fast ensembling of dnns. In Advances in Neural Information Processing Systems (NewrIPS), 2018. https://arxiv, http://aawwvvvo/aa////a/aa//////
Almog Gueta, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. Knowledge is region in weight space for fine-tuned language models. arXio preprint ar Xiv:2300 04863, 2023.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with
disentangled attention. In International Conference on Learning Representations, 2021. URL https: //openreview //openreriew..nt///oouuee wtt.:e/toouuiiid-///ooouoa
Yifei He, Yuzheng Hu, Yong Lin, Tong Zhang, and Han Zhao. Localize-and-stitchh: Efficient model merging via sparse task arithmetic. arXiv preprint arXiv:2408 13656, 2024.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos qa: Machine reading comprehen- sion with contextual commonsense reasoning. arXio preprint arXiv: 10090077,, 2019.
Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang- Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lec. Chat vector: A simple approach equíp LLMs with instruction following and model aligmment in new languages. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (ACL 2024), pp. 10943 10959, 2024a. URL https: https://aclantthll.g wwwpchhooogo..g//2244444441 acl-lo4..llloo...... 590.
14

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Shih-Cheng Huang, Pin-Zu Li, Yu-chi Hsu, Kuang-Ming Chen, Yu Tung Lin, Shih-Kai Hsiao, Richard Tsai, and Hung-yi Lee. Chat vector: A simple approach to equip LLMs with instruction following and model alignment in new languages. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 10943 109599, Bangkok, Thailand, August 2024b. Association for Compulational Linguistics. doi: 10.18653/v /000663//112224..2 20w...ac--ooo..5000 URL https: ptttps://aclantholoogy wwwpannhhooooooooo2222ee2e org/2024. 222044.cclloom 1ong.590.
Gabriel Ihharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. ar preprint orXi:22122.44088 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In The Eleventh International Conference Learning Representations, 2023. URL https://openrerveew wwtp::/eeeeefoorreeeeeeee nett/torum/id-6tt0=fff-----
Shankar Iyer, Nikhil Dandekar, and Kornel Csernai. First quora dataset release: Question pairs, 2017. URL Daraas-aatt------tt------- -uweeasse-uussionnP-aee
attps://datas http://aaaaauur...a.aorr
Dong-Hwan Hwan Jang, Sangdoo Yun, and Dongyoon Han. Model stock: All we need is just few fine-tuned models, 2024. URL https://arxiv .rgg arg/abb/2403.. 19522.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. arXiv preprint ar 03202, 2023.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. orXiv preprint ar xrxim:221 09849, 2022.
Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In The Eleventh International Conference on Learning Representations, 2023. URL
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. Repair: Renormalizing permuted activations for interpolation repair. arXiv preprint rxXi:22211084033 2022.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: REnormalizing permuted activations for interpolation repair. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreviee
Rohith Kuditipudi, Xiang Wang, Holden Lee, Yi Zhang, Zhiyuan Li, Wei Hu, Rong Ge, and Sanjeev Arora. Explaining landscape connectivity of low-cost solutions for multilayer nets. Advances in Neurol Information Processing Systems (NeurIPS), 2019. https://arxiv.org httg///bs/190... 06247.
Rémi Lebret, David Grangier, and Michael Auli. Neural text generation from structured data with application to the biography domain. arXiv preprint arXiv:1603.07771, 2016.
Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morscy, Patrick Van Kloef, Sören Aucr, et al. Dbpodia-a large-scale, multilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167 195, 2015.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning, 2012a.
Hector Levesque, Ernest. Davis, and Leora Morgenstern. The winograd schema challenge. Thirleenth International Conference on the Principles of Knowledge Representation and Reasoning, 2012b.
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguisties: EMNLP, 2020. https://www http:///ww.ccweeborrg anthology/2020. findings-emnlp.r 165.
15

---

Published in Transactions on Machine Learning Research (07/2025)
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. Reasoning over paragraph effects in situations. arxiv preprint ar 10:1900 1908.0585222 05852, 2019.
Haokun Liu, Derek Tam, Mohammed Mugeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin A Raffel. Few-shot parameter-cfficien fine-tuning is better and cheaper than in-context learning. Advances in Newral Information Processing Systems, 35:1950 1965, 2022.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyor, and Veselin Stoyanov. Roberta: robustly optimized bert pretraining approach, 2019.
htttss///axiii http:///axxvvv...aabbee wrggaba/////eaoooooooooooeeeeaaa /190///990...0 11692.
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction tuning. arXiv preprint ar arXtv:2301113688 2023.
Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical reasoning for large language models via reinforced evol-instruct. arXiv preprint arXiv:2308.098 09583, 2023.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeling of the associalion for 2011.
computational linguistics: Human language technologies, pp.
Michael S Matena and Colin A Raffel. Merging models with weighte averaging. Advances in Neural Information Processing Systems, 35:17703 17716, 2022a.
Michael S Matena and Colin A Raffel Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703 11117703-1771,, 2022b.
Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communccttion efficient. learning of deep nelworks from decentralized data. In Artificial intelligence und statislics, pp- 1273 1282. PMLR, 2017.
Mohammed Muqocth, Haokun Liu, Yufan Liu, and Colin Raffel. Learning to route among specialized experts for zero-shot generalization. In Ruslan Salakhutdinov, Zico Kolter, Katherine Heller, Adrian Weller, Nuria Oliver, Jonathan Scarlett, and Felix Berkenkamp (eds.), Proceedings of the 41st International Conference Machine Learning, volume 235 of Proceedings of Machine Learning Research, pp. 36829 36846. PMLR, 21-27 Jul 2024. URL https: http/c/edoocegsss./eeeeee////.ee mlr vrressvv35/// 22335/mqqeeth2a. html.
Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. arXiv preprint arXiv:1808.0877 08745, 2018
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? Advances in neural information processing systems, 33:512-523, 2020.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversarial nli: A new benchmark for natural language understanding. arXiv preprint arXiv:1910. 145599990.1499,, 2019.
Guillermo Ortiz-Jimencz, Alessandro Favero, and Pascal Frossard. Task arithmotic the tangent space:
Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36, 2024.
Fidel A Guerrero Peña, Heitor Rapela Medeiros, Thomas Dubail, Masih Aminbeidokhti, Eric Granger, and Marco Pedersoli. Re-basin via implicit sinkhorn differentiation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20237 20246, 2023.
Mohammad Taher Pilehvar and Jose Kamacho-Colladee Wic: the word-in-context dataset for evaluating context-sensitive meaning representations. arXiv preprint ar Xiv: 1008 09121, 2018.
16

---

Published in Transactions on Machine Learning Research (07 (225))
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Colin Raffel, A call to build models like we build open-source software, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research (JMLR), 2020a. http://jmla wwtpppppprs//11100077777 html.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to-text transformer. ArXie, 1bs/1910..ee 10683, 2020b.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. arviv preprint 0rXiv:2212.10445, 2022a.
Alexandre Ramé, Kartik Ahuja, Jianyu Zhang, Matthicu Cord, Léon Bottou, and David Lopez-Paz. Model ratatouille: Recycling diverse models for out-of-distribution generalization. Xiv preprint ar 2rv: 10445, 2022b.
Alexandre Rame, Guillaume Couairon, Corentin Dancette, Caan-Baptiste Gaya, Mustafa Shukor, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. Advances in Neural Information Processing Systems, 36, 2024.
Alexandre Ramé, Johan Ferret, Nino Vieillard, Robert Dadashi, Léonard Hussenot, Pierre-Louis Cedoz, Pier Giuseppe Sessa, Sertan Girgin, Arthur Douillard, and Olivier Bachem. Warp: On the benefits of weight averaged rewarded policies. arXiv preprint 2rv::22406. 16768, 2024.
Alexandre Rame, Nino Vieillard, Leonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, and Johan Ferret. Warm: On the benefits of weight averaged reward models. In Forty-firs International Conference on Machine Learning, 2024.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. In 2011 AAAI spring symposium series, 2011.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprinl ar Xiv:2330 12950, 2023a.
Baptiste Roziere, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, et al. Code llama: Open foundation models for code. arXiv preprint Xrx:::2308. 12950, 2023b.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnand Stieglor, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. Xiv preprint ar Xiv:21 0ww::21100080000 10.08207, 2021a.
Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations (ICLR), 2021b. https: wwaaxii.org/ass///... 082000
Abigail See, Peter J Liu, and Christopher D Manning. Get to the point: Summarization with pointer-generator networks. arXiv preprint arXie:1704.04360 2017.
Viraj Shah, Nataniel Ruiz, Forrester Cole, Erika Lu, Svetlana Lazebnik, Yuanzhen Li, and Varun Jampani. Ziplora: Any subject in any style by effectively merging loras. arXiv preprint arXiv:231 1.13600, 2023.
17

---

Published in Transactions on Machine Learning Research (07/2025)
Noam Shazeer and Mitchell Stern. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning. PMLR, 2018.
Ken Shoemake. Animating rotation with quaternion curves. In Proceedings of the 12th annual conference on Computer graphics and interactive techniques, pp. 245 254, 1985.
Sidak Pal Singh and Martin Jaggi. Model fusion via optimal transport. Aduances in Neural Information Processing Systems, 33:22045 22055, 2020.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. Dream: A challenge data set and models for dialogue-based based reading comprehension. Transactions of the Association for Computational Linguistics, 7:217 231, 2019.
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. Empirical Methods in Natural Language Processing (Findings), 2023.
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task subspaces. orXiv preprinl WrXiv2332.043388 2023.
Derek Tam, Mohit Bansal, and Colin Raffel. Merging by matching models in task parameter subspaces. Transactions on Machine Learning Research, 2024.
Anke Tang, Li Shen, Yong Luo, Liang Ding, Han Hu, Bo Du, and Dacheng Tao. Concrete subspace learning based interference elimination for multi-task model fusion. arXiv preprint arXiv:2312.06177 2023.
Norman Tatro, Pin-Ya Yu Chen, Payel Das, Igor Melnyk, Prasanna Sattigeri, and Rongjie Lai. Optimizing mode connectivity via neuron alignment. Advonces in Neural Information Processing Systems, 33:15300-15331 2020.
Gemma Team, Morgane Rivicre, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Téonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma Improving open language models at a practical size. arXiv preprint arxiv:2408. 000:200000018, 2024.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXie preprint orXiv:2307.0922: 099888 2023.
Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowledge fusion of large language models. arXiv preprint 2rXiv:2401.10499 2024.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. Glue: A multi- lask benchmark and analysis platform for natural language understanding, In International Conference on Learning Representations (ICLR), 2018.
wttpsiiii.rr///////0...... 07461.
Hongyi Wang, Mikhail Yurochkin, Yuekai Sun, Dimitris Papailiopoulos, and Yasaman Khazaeni. Federated learning with matched averaging. In International Conference on Learning Representations, 2020.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. Tn Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 23965 23998. PMLR, 2022a. URL https://prooceedings mlr press/ mal.geesss.srers/////eeee.e v162/wortsman22a. html.
Mitchell Wortsman, Gabriel Ilharco, Jong Wook Kim, Mike Li, Simon Kornblith, Rebecca Roelofs, Raphael Gontijo Lopes, Hannanch Hajishirzi, Ali Farhadi, Hongscok Namkoong, et al. Robust fine- tuning of zero-shot models. In Proceedings of the IEEE/CVF conference on computer vision and pottern recognition, pp. 7959-79711 7971, 2022b.
18

---

Published in Transactions on Machine Learning Research (07/2025)
Prateek Yadav, Leshem Choshen, Colin Raffel, and Mohit Bansal. Compeft: Compression for communcating parameter efficient updates via sparsification and quantization, 2023a.
Prateck Yadav, Qing Sun, Hantian Ding, Xiaopeng Li, Dejiao Zhang, Ming Tan, Parminder Bhatia, Xiaofei Ma, Ramesh Nallapati, Murali Krishna Ramanathan, Mohit Bansal, and Bing Xiang. Exploring continual learning for code generation models. In Proceedings of the 61st Annual Meeting of the Association for Computational Lónguistics (Volume 2: Short Papers), pp. 782 792, Toronto, Canada, July 2023b. Association for Computational Linguistics. doi: (0.18633/v112022. aacaaaclsoooo aco-ssoor..... URL attps://aclaanthooogy org/ 2023. wtl..acl-ssor...0 68.
Prateek Yadav, Colin Rasfel, Mohammed Muqeeth, Lucas Caccia, Haokun Liu, Tianlong Chen, Mohit Bansal, Leshem Choshen, and Alessandro Sordoni. A survey model moerging: Recycling and routing among specialized experts for collaborative learning, 2024a. URL https: / htt///rxvvv.o///// orgg htsp//bb/2408.. 07057.
Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-me Resolving interference when merging models. Advances in Neural Information Processing Systems, 36, 2024b.
Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. orXiv preprint (ttt:221000257, 2023.
Yi Yang, Wen-tau Yih, and Christopher Meek. WikiQA: A challenge dataset for open-domain question answering. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 2013-2018, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 18653/v1/D15-1237 URL https://acllanthologgy wtwp:////o...////1227...
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable molti-hop question answering. In Conference on Empiricul Methods in Natwral Language Processing (EMNLP), 2018.
Peng Ye, Chenyu Huang, Mingzhu Shen, Tao Chen, Yongqi Huang, Yuning Zhang, and Wanli Ouyang. Merging vision transformers from different tasks and domains. arXiv preprint Xii:22312112240, 2023.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-fors International Conference on Machine Learning, 2024a.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Extend model merging from finc-tuned to pre-trained large language models via weight disentanglement. arXiv preprint urXin:2408. 0030090,,000.0.. 2024b.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement. arXiv preprint arXiv:2408. 03092, 2024c.
Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forby-frst International Conference on Machine Learning, 2024d.
Le Yu, Bowen Yu, Haiyang Yu, Fci Huang, and Yongbin Li. Extend model merging from fine-tuned to pre-trained large language models via weight disentanglement. arXiv preprint ar arXin:2408. 00000090,, 2024e.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXio:1905.00 07830, 2019.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Advances in Newral Information Processing Systems (NewrIPS), 2015. https://proceeddnng iewpeeedddnnnnnnnnii..r.ee pdf.
Yiran Zhao, Wenxuan Zhang, Huiming Wang, Kenji Kawaguchi, and Lidong Bing. Adamergex: Cross-lingual transfer with large language models via adaptive adapter merging. arXie preprint arXiu:2402. 18913, 2024.
Jing Zhou, Zongyu Lin, Yanan Zheng, Jian Li, and Zhilin Yang. Not all tasks are born equal: Understanding zero-shot generalization. In The Eleventh International Conference on Learning Representations, 2022.
19

---

Published in Transactions on Machine Learning Research (07/ (07/2025)
A Detailed Task Descriptions.
We adopt the experimental setting from the TO mixture (Sanh et al., 2021a) which contains 8 held-in and 4 held-out task categories Specifically, the 8 held-in task categories include Multiple-choice QA (with selected datasets DREAM (Sun et al., 2019), Cosmos QA (Huang et al., 2019)), Patractive Qa (Adversarial QA (Adelani et al., 2021), ROPES (Lin et al., 2019)), Closed- Book QA (Hotpot QA (Yang et al., 2018), Wiki QA (Yang et al., 2015)), Sentiment Analysis (App Reviews (), IMDB (Maas et al., 2011)), Topic Classification (AG News (Zhang et al., 2015), DBPedia (Lehmann et al., 2015)), Structure-lo-tenl (Common Gen (Lin et al., 2020), Wiki Bio (Lebret et al., 2016)); Summarization (CNN Daily Mail (See et al., 2017), XSum (Narayan et al., 2018)) and Paraphrase Identification (MRPC (Dolan & Brockett, 2005), QQP (Iyer et al., 2017)). Similary, the 4 held-out task categories are Sentence Completion (with selected dataset COPA (Roemmele et al., 2011), HellaSwag (Zellers et al., 2019)), Natural Language Inference (ANLI (Nie et al., 2019), RTE (Dagan et al., 2005)), Coreference Resolution (WSC (Levesque et al., 2012b), Winogrande (Levesque et al., 2012a)) and Word Sense Disambigualion (WiC (Pilehvar & Camacho-Colladoss 2018)).
B Expert Training Details
In our research, we utilized two base models, namely PaLM-2 and PaLM-2-IT to create specialized expert models. We train the PaLM-2model for an additional 60000 steps on the Flan-v2 dataset (Longpre al., 2023) to obtain the PaLM-2-IT model. We removed the TO tasks from the flan mixture in order to training experts on them in future. Many of these training jobs were early stopped due to convergence. We used Sharded Adafactor (Shazeer & Stern, 2018) optimizer along with cosine decay and a learning rate of le-4 for IB, 24B, and 64B model sizes and 3e-5 for 8B model. We use a dropout value of 0.05. Following Chung et al. (2024), we used an input length of 2048 and output length of 512. To create expert models we perform full finetuning with the following hyperparameters. For training the experts model, for all model size, we train by default for 2000 steps with a learning rate of 30-5 and dropout of 0.05. For some task we adjust the number of steps depending upon the convergence. For the purpose of evaluating classification tasks (Raffel el al., 2020b), we perform rank classification. Tn this method, the model's log probabilities for all potential label strings are ranked. The model's prediction is deemed accurate if the choice ranked highest aligns with the correct answer. It should be noted that rank classification evaluation can accommodate both classification tasks and multiple-choice tasks. for Task Arithmetic, TIES and DARE methods, we tested values between 0 and 1, in steps 0.1. For TIES and DARE, we pruned 80% and 90% of the values. For TIES we pruned the bottom x% values in the lask vector by magnitude, while for DARE we pruned randomly. These pruning methods are from their original papers.
C Full Result Tables
In this section, we provide the result for the full grid of experiments that we performed. The results contain information about any of the plots that are not provided in the main paper. Table 4 and present the held-in and held-out performance of PaLM-2 model across all model sizes, base models, merging methods, and the number of experts being merged. Similarly, Table 2 and 3 present the held-in and held-out performance of PaLM-2-IT model.
Table 2: The table reports the mean (std) of the normalized performance for the held-in tasks when merging experts created from PaLM-2- IT base models.
Horging Nosted (1)
24B
648
Espartx (-)
Average
0.5
0.
0.8
0.53p)
0,52
0.52
0.55un
0.84pax:
0.77m)
0.97
0.01
0.50
0.50gm
Taok Arithmetic
0.90p
0.96 em
0.58 su
0.55
0.88mm
08)cm)
4.9gg)
10,30
0.91
0.8pm0)
0.55
Baxe-TIE8
6.8030
08
0.86p
0.506
0.56
0.580
1944000)
08%
08fice)
0850
0.91.am
0,80
0.53km
TIE8
089uus;
08
0.85m
0.80gmm
0.3800
08800m;
0.16
0.9rue)
0.80
0.885m
Haltisank
0.97
mm00
0.00
0.00pm
0.30
0.36
0.56un
0.99
6,57
0.58.m
0.95p)
0,00
0.98p
n,95,0
0.50
20

---

Published in Transactions on Machine Learning Research (07/202) 2025)
Table 3:
The table reports mean (std) of the normalized performance the held-out tasks when merging experts created from PaLM-2-IT base models.
Worging Mathod II:
13
8B
24B
648
# of Exparta (-4)
Average
LMMar:
105
100
182o6
1000urn
1.10qum::
1.16p0
1ao
1.00
1.06qua0
1 1,00g0
Taak Arsthmetic
1055mm
188
LMaae:
105m
106mmn
1.6
1.6ppm
105m
10
Linas
1.15pm
1.38m
km0
1.00
106gmm
1.00g
Dare-TTR3
10
108
1.8ga::
105g:
105pau
168g2
1.0kpm
105g
1/0000
1.17
1.37m0
100
1400
14houa))
1 1000g
1138
102gan)
L6Ruan
13kgi:
105ac
108gua
160g
108ppx
10kac
18kgnn
100gx))
1.1gaa0
1.16p0
100
1,00g14)
1.6qua00
1.1g
Noleitank
1.3gg
1.1g
1.0m
1.1g
L.2gg
115g
1.2g
112m
1.5g
1.8m
1.8g
1.18
1.65p
1.05m
146
105g
Table 4: The table reports mean (std) of the normalized performance on the held-in tasks when merging experts created from PaLM-2 base models.
Harging Nethod (1)
Experts (-9)
Average
0:44)
0.36
0.30
0.
0.5kaan
00
0.300
0Taas
00)nas)
0.37a)
0/5pa
0.T4jam
0.00
0.87mn
Arithmetic
0.6
6:02
0.44x1))
0.50gs
0.5gya
0.42m
0.72mm0
0.6pmm
0.46r)
0.pmm
0.74
0.00
0.67
0.57g)
0.
0.amm
0.30m
007mmr)
05)pa))
4270)
05pmm
0.74px4)
0.00pmm)
0.67m
006
6.5pa0
6:11pm)
0.5k
0.57
0,52
0.88415)
0.29m
0.5..0
0.3
0.5.am
027g
05
0.71gam0
0.6
0.50
Helcisaok
0.8
0.5
1,06gon
1.04ys0
1/8aa
1.0%g
125
1.35am
1.5ga
1/8g
0.66
0.96
0.96
Table 5: The table reports mean (std) of the normalized performance on the held-out tasks when merging experts created from PaLM-2 base models.
Nostod
Esparts
Average
0.95
1002
1,04g)
1.01gs
0.97
0.58pn
0.95p
0.83p)
1.20
1.24g
1.29
L.30gp
Taak Art theeteei
100
1000000
1.00m
1.00pam0
1,08pma
1.55000
10500s)
1.1aas
1.08g)
1.20/
1.28
1.50gam0
1.35pn
Dara TIBS
099
100gpe
1.4g00
1.06m
1.22amm
1.0m
08%oo
09%
1.28
1.20gm0
1.28
1.30
14m
L54m
1.5km
21


Reviews:
[{"review_id": "NjBYoyrdek", "reviewer": "Reviewer_TVtf", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The submission has some strong points, in that some of the findings are quite interesting:\n* The observation that stronger base models correlate well with the utility of model merging is interesting. Can the authors comment on whether there are some tasks where the base models are expected to outperform the instruction tuned models, providing an indication of whether it is the instruction tuning or improved zero-shot performance that matters?\n* The finding that model merging methods all become quite similar as the size of the base model is scaled is quite interesting.\n* The finding are shown to be replicated across model families; experiments are mainly conducted on the PALM fmaily of models, but some additional confirmatory experiments are run on Llama-2 models.\n\nThe main weakness is the lack of rigorous motivation for the particular normalised performance metric. There are a wealth of tools in non-parametric statistics that are designed to solve the problem the ad hoc normalisation approach in this paper attempts to solve. In particular, comparing *ranks* of methods rather than accuracies is a common approach for dealing with accuracies measured on different data distributions/datasets. The ensures that skews (and higher order moments) in the distribution of performances do not unfairly impact the comparison. In contrast, the proposed normalisation will be affected by higher order moments of the performance distribution. A very common approach used in machine learning for comparing multiple methods on multiple datasets is given by Demsar [1]. A more modern take on this is provided by Jansen et al [2]. I would strongly recommend the authors use average ranks across datasets, rather than normalised accuracies.\n\nSome minor issues persist in this revision:\n* The addition of standard deviations in the appendix is appreciated, but it is not clear what observational units are used to compute these.\n* Some clarity issues, which are described below.\n\n[1] J Demsar. \"Statistical Comparisons of Classifiers over Multiple Data Sets\". In JMLR, 2006.\n\n[2] C Jansen, M Nalenz, G Schollmeyer, T Augustin. \"Statistical Comparisons of Classifiers by Generalized Stochastic Dominance\". In JMLR, 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The submission undertakes an experimental analysis of model merging in LLMs across PALM and Llama-2 model families with different parameters sizes. Four model merging techniques of various complexity are considered, and models are evaluated on both held-in and held-out tasks. The submission makes several claims: model merging is more effective for base models with larger zero-shot performance; larger models facilitate \"easier\" merging; model merging can outperform multitask learning; and for larger models, the performance of different merging techniques is more similar.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* The biggest change I would like to see would be the use of average ranks (or a similar non-parametric quantity) used as the metric for comparing multiple models on multiple datasets. Alternatively, a much stronger justification that the proposed metric is suitable in the context of this paper would also work.\n* It would be nice if the standard deviations (or resulting standard errors) could be incorporated into the plots somehow. More details should be added about what precisely is different between each trial---these details could go in Appendix C with the full results tables.\n* In Section 4.3 the paper refers to \"our\" merging approach. What is meant by this?\n* Later on in Section 4.3, the phrase \"model noise\" is used; what is this?", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "JAJzQer9Bz", "reviewer": "Reviewer_MXWs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n    - The paper is generally well-written, and the experiments are carefully designed to ensure fair comparisons.\n    - It explores different merging factors and provides clear, conclusive findings for each factor, addressing gaps not adequately measured in previous works.\n- Weakness\n    - The authors addressed most of my requested issues. Although the Llama experiment did not perform on held-out tasks, it shows the similar trend as PaLM-2 on held-in tasks and different merging methods.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- This work conducts a comprehensive and systematic study on evaluating various factors influencing model merging, including model size, merging methods, the number of constituent models, and performance on held-in and held-out tasks.\n- The paper presents several insightful findings derived from exhaustive empirical observations, offering valuable guidance for future model merging research.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Section 4.2 title is grammatically wrong, please consider to change it as “Model merging performs better with bigger model” or “Model merging becomes more effective with bigger models”.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wIYttqGnDC", "reviewer": "Reviewer_Lys5", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n- This paper provides a wide range of experiments regarding model merging and interesting insight into the effect of merge configurations.\n- The experimental findings seem reasonable, and actually verifying them is valuable for the community.\n- The paper is well-written and easy to understand.\n- Although I am not the reviewer for the previous submission, the author's statements about the changes since the last submission are reasonable to me. Therefore, I think the paper has been improved based on the previous review.\n\n[Weaknesses]\n- As the contribution of this paper fully depends on experimental evaluation, the generalization of the result might be limited. For example, it might be difficult to generalize the result to other models than transformer-based large language models.\n- The hyperparameter tuning method for the merging methods is not explained.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper experimentally investigates the effects of scaling model size, the base model quality, and the number of expert models to be merged in model merging. In the experiments, the authors target four relatively simple merging methods and compare their performance with different model sizes and the number of expert models. The experimental result indicates that the merging expert models derived from the instruction-tuned model exhibit better performance, and larger models perform better in model merging and are not sensitive against merging methods. In addition, when merging eight large expert models, the merged models outperform the multitask-trained models. These experimental findings will provide insight for further development and practitioners of model merging.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Some model merging methods, such as task arithmetic and TIES merging, have hyperparameters. In general, tuning such hyperparameters affects the merged model performance. How do the authors decide the hyperparameters in merging methods in the experiment? It would be better to clarify this point.\n\nInvestigating the effect and sensitivity of such hyperparameters when merging large models would be interesting. However, this extended direction is out of the scope of this paper and will be a future work.", "broader_impact_concerns": "I do not have concerns about the ethical implications of this paper. However, it might be better to add a broader impact statement to the paper because the large-scale merged model may have an impact on various areas.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)