LM-Cocktail: Resilient Tuning of Language Models via Model Merging (2024)

Abstract: The pre-trained language models are continually fine-tuned to better support downstream applications. However, this operation may result in significant performance degeneration on general tasks beyond the targeted domain. To overcome this problem, we propose LM-Cocktail which enables the fine-tuned model to stay resilient in general perspectives. Our method is conducted in the form of model merging, where the fine-tuned language model is merged with the pre-trained base model or the peer models from other domains through weighted average. Despite simplicity, LM-Cocktail is surprisingly effective: the resulted model is able to achieve a strong empirical performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. We conduct comprehensive experiments with LLama and BGE models on popular benchmarks, including FLAN, MMLU, MTEB, whose results validate the efficacy of our proposed method.

heading: 1 Introduction
text: Language models (LM) are fundamental pillars of artificial intelligence and natural language processing. Thanks to the considerable expansion of training scale and model size (Devlin et al., 2018; Liu et al., 2019; Raffel et al., 2020; Radford et al., 2019; Brown et al., 2020), language models have made remarkable breakthroughs on a wide variety of NLP tasks, including representation, understanding, reasoning, and generation. In many of the applications, language models are frequently used via the “pretraining and fine-tuning” paradigm. Particularly, a generalist LM is pre-trained in the first place through an unsupervised or general-purpose supervised learning process (Brown et al., 2020; Touvron et al., 2023; Wei et al., 2022, 2021; Ouyang et al., 2022); then, the pre-trained generalist model is finetuned to be a specialist model for a down-stream task on top of certain in-domain data.
∗Correspondence author
Despite the improved performance in each particular application, the fine-tuning operation could lead to severe degeneration of LM’s general capabilities beyond the targeted domain. Such a phenomenon is commonly referred as catastrophic forgetting (Goodfellow et al., 2013; Kirkpatrick et al., 2017; Thompson et al., 2019; Chen et al., 2020). As shown in Figure 1, fine-tuning Llama model on the target task can significantly improve its performance on the target task, but decrease its performance on other unrelated tasks. In many real-world scenarios, catastrophic forgetting is unwelcome because language models need to exhibit both specialist and generalist characteristics simultaneously(Roziere et al., 2023; Chen et al., 2021; Singhal et al., 2022).
The combat against catastrophic forgetting represents a sustained campaign within the machine learning communities, where numerous approaches have been continually proposed in recent years. There are two representative strategies which are widely adopted as the designing logic by many existing methods. One strategy is to rely on experience replay, where the model is learned with the mixed training data from both the new task and the previous tasks (Rolnick et al., 2019; Shin et al., 2017). The other strategy is to leverage regularization, where the changes in predictions or weights are regularized between the newly fine-tuned model and the historical pre-trained one (Kirkpatrick et al., 2017; Li and Hoiem, 2017; Rannen et al., 2017). However, it remains to explore more effective methods in the context of fine-tuned language models given the practical constraints of the existing methods. On one hand, it is infeasible to fully collect the training samples for all previous tasks, and have the model trained over again on the historical data once a new task is presented. On the other hand, the regularization may result in major changes to the existing fine-tuning operations, which could be incompatible with the well-established fine-tuning
2474
pipeline. In this work, we aim to design an effective framework to confront catastrophic forgetting, which will enable the fine-tuned language models to stay resilient in general tasks. Besides, we also expect the new framework to be more practical, which means it must be simple to conduct and fully compatible with the common model training workflow.
With these considerations, we propose a new approach, called LM-Cocktail, which continually adapts well-fine-tuned language models on top of model merging (Wortsman et al., 2022a). LMCocktail is a general paradigm, which can work under several different conditions. In the simplest form, it directly merges the fine-tuned model with the pre-trained base model to improve the general capabilities of the fine-tuned model. It can further accommodate more peer models fine-tuned for other general domains, and result in stronger empirical performances on top of merging weights estimated by few-shot validation examples. Finally, even at the absence of fine-tuning data, the merging strategy can be still applied to the remaining pretrained base model and the fine-tuned models in other general domains for a competitive resilience.
Our proposed method leads to a couple of immediate advantages given its working mechanism. First of all, LM-Cocktail is extremely simple: the mixing weights can be directly derived from validation samples where no expensive training operations are needed. Secondly, LM-Cocktail is fully compatible with the existing training pipeline, knowing that it simply works as a post-refinement step following the fine-tuning process. Above all, LM-Cocktail is empirically competitive. According to our evaluations on three representative benchmarks, including FLAN (Wei et al., 2021), MMLU (Hendrycks et al., 2020), and MTEB (Muennighoff et al., 2022), LM-Cocktail achieves a strong re-
silience in general domain tasks while preserving a superior fine-tuning performance on its targeted domain. Finally, LM-Cocktail turns out to be universally applicable: it can substantially contribute to both the decoder-based LM in language generation tasks and the encoder-based LM in language representation tasks.

heading: 2 LM-Cocktail

heading: 2.1 General Paradigm
text: As a prerequisite condition, we are given a base language model, denoted asMb, which are well pre-trained for general applications. The base LM is continually fine-tuned to support one targeted down-stream task (t) with domain-specific training samples Xt, which results in the fine-tuned model for the corresponding task:Mt.
However, the fine-tuned modelMt is prone to degenerate empirical performances (catastrophic forgetting) on other general domains beyond the targeted domain t. The goal of LM-Cocktail is to maintain the general capabilities when fine-tuning on the target task. The core of LM-Cocktail is combining multiple models (with the same architecture but different weights) into a unified one by aggregating the weights from different models. In this way, the resilient fine-tuned model can integrate the strengths from multiple individual models.
To derive the appropriate model merging strategy for LM-Cocktail, there are two fundamental problems to solve: 1) which group of candidate models to merge, 2) how to determine the merging weights. Knowing that the resilient fine-tuned LM is to restore the degenerated performances in general domains, there are two sources of candidate models to consider. One source is the pre-trained base modelMb, the other source is the entire group of fine-tuned models in other domains ({Md}D).
Without loss of generality, we derive the following form of merging function:
Mr ← αMt + (1− α) ∑
Mb,{Md}D wi ∗Mi, (1)
whereMr is the resilient-tuned model, α is a hyperparameter whose default value is 0.5, and wi indicates the merging weight which has been normalized: ∑ iwi = 1. For our case, we require the resilient-tuned model to preserve strong capacity as the directly fine-tuned model in its targeted domain while improving the general domain performance. Therefore, the candidate models’ performances in the targeted domain are the critical indicators of merging weights. Based on this intuition, we introduce the following form of weight computation:
wi ← softmax(−L(Mi, Et)/τ). (2) In this function, L(Mi, Et) stands for the prediction loss of candidate modelMi on the few-shot examples Et from the targeted domain t, τ is the temperature to control the smoothness. That is to say, the larger loss on the targeted domain, the smaller weight is allocated to the candidate model. So we can give lower coefficients to models that perform very badly in the target task. The few-shot examples are a tiny group of hold-back samples from the targeted domain. According to our empirical study, 5-shot examples have been sufficiently competitive throughout different settings.

heading: 2.2 Variations
text: The general form of LM-Cocktail in Eq 1 requires the presence of three elements: the base modelMb, the fine-tuned model for the targeted domainMt, and the fine-tuned models in other general domains {Md}D. Nevertheless, the general requirement can be largely relaxed to accommodate different real-world settings. Here, we introduce two common variational forms to confront the situations where either diverse general-domain specialists or targeted domain fine-tuning is not available. •Mono-Specialist. When the diverse fine-tuned models in general domains are absent, the merging function is simplified as the combination of base modelMb and the mono-specialist model from the targeted domainMt:
Mr ← αMt + (1− α)Mb. (3) Given that fine-tuned modelMt typically exhibits significantly lower loss compared to other models, we did not employ Eqn 2 to calculate weights;
instead, we introduce a hyperparameter α. Experimental results demonstrate that simply setting α to 0.5 yields promising outcomes. •Without Fine-tuning. The fine-tuning in the targeted domain can be constrained due to the absence of domain-specific data or computation resources. In this situation, the merging function is transformed into the combination of base model and fine-tuned model from general domains:
Mr ← ∑
Mb,{Md}D wi ∗Mi. (4)
In this place, we assume the few-shot examples Et for merging weights (Eq. 2) are always available, which is a very moderate condition in practice. In this manner, we obviate the need for training any new models; instead, by incurring minimal costs, we can seamlessly integrate existing models to obtain a model tailored for downstream tasks.

heading: 3 Experimental setup
text: We conducted experiments with two types of models: decoder-based LM and encoder-based LM. We fine-tuned 9 encoder-based models and 9 decoderbased models separately, and then evaluated the performance of fine-tuned models and resilient-tuned models.

heading: 3.1 Decoder-based LM
text: • Base Model. We use Llama-2-chat-7b 1 (Touvron et al., 2023) as the base model, which has an impressive zero-shot ability on various tasks. • Fine-tune. We use the datasets collected by (Cheng et al., 2023; Wang et al., 2023), which consist of 30 tasks from FLAN (Wei et al., 2022). We select 9 different tasks from it to fine-tune the base model, including NQ, SQuAD, Hellaswag, SST2, Winogrande, CommonGen, MRPC, AG News, and MNLI. For more information of training data please refer to Appendix A. The fine-tuned code is based on FastChat package2. The learning rate is 2e-5, the batch size is 128, and the max number of epochs is 3. • Evaluation. We evaluate the performance on the test set of 30 tasks collected by (Cheng et al., 2023; Wang et al., 2023). The test data for fine-tuning tasks (NQ, SQuAD, Hellaswag, SST2, Winogrande, CommonGen, MRPC, AG News, and MNLI) are also included in this collection. The
1https://huggingface.co/meta-llama/Llama-2-7b-chat-hf 2https://github.com/lm-sys/FastChat
detailed metric for each task can refer to (Wang et al., 2023). Besides, we also conduct experiments on additional tasks from MMLU datasets, which is a widely used benchmark for LLMs.

heading: 3.2 Encoder-based LM
text: • Base Model. We choose the bge-base-v1.5 embedding model (Xiao et al., 2023) as the base model for embedding tasks, which can map text into embedding representation. • Fine-tune. We select 9 datasets from sentence transformers repo3, including GooAQ, Yahoo Answers, MSMarco, Stack Exchange, ELI5, SQuAD, AmazonQA, Quora, HotpotQA. Appendix A shows the details of training data. We fine-tune BGE model on these datasets with FlagEmbedding tool4. We use the AdamW optimizer with a learning rate of 2e-5. The batch size is 256, and the temperature for contrastive learning is 0.02. • Evaluation. We evaluate the models with the 15 retrieval tasks in mteb benchmark (Muennighoff et al., 2022), and use NDCG@10 as the evaluation metric. For the purpose of facilitating training and testing across various tasks, we don’t add the
3https://huggingface.co/datasets/sentencetransformers/embedding-training-data
4https://github.com/FlagOpen/FlagEmbedding
default query instruction from (Xiao et al., 2023).

heading: 4 Experimental Results
text: In this section, we show the experimental results and represent the key findings. Firstly, we compare the performance of fine-tuned models and resilienttuned models. Next, we evaluate the performance of LM-Cocktail when fine-tuning on target task is unavailable. Finally, we investigate the impact of weight α and the number of examples.
We also investigate the effectiveness of LMCocktail under different fine-tuning strategies and base models. Additionally, we compare the effects of various merging methods. These results are presented in the Appendix B.

heading: 4.1 Overall Comparison
text: Our experiments compare the performance of base models, corresponding fine-tuned models, and models resilient-tuned via LM-Cocktail. For each fine-tuned model, we measure its performance on the specific target task as well as its performance on other tasks. We also tested models resilient-tuned using our method, which include two variants: (1) LM-Cocktail2: merge the fine-tuned model with the base model; (2) LM-Cocktail10: merge 10 models, including model fine-tuned on target task,
base model, and eight models fine-tuned on other tasks from section 3.1. We calculate weights for each model using five examples from the training set. Since there are some dataset in the retrieval tasks without a training set, for those (e.g., ArguAna), we randomly select five examples from the test set.
We have summarized the results in Table 1 and 2. For detailed results for each test task please refer to Appendix B.4.

heading: 4.1.1 Analysis on decoder-based LM
text: From Table 1, we have following observations: (1) the fine-tuned model demonstrates significant improvement over the base model in the corresponding task. For example, the model fine-tuned on AG News achieves an accuracy of 94.42% in the corresponding task, whereas the base model only achieves 40.9% accuracy on the same task. (2) However, this gain comes at a cost: in other tasks, the fine-tuned model often lags behind the performance of the base model. For example, the accuracy of the fine-tuned model on other tasks is only 38.58%, substantially lower than the 46.8% accuracy of the base model. (3) In contrast, LMCocktail2 maintains effectiveness in its corresponding task (94.46% in AG News task) while also demonstrating competitive performance in other tasks (47.73%). And LM-Cocktail10 further enhances the performance of other tasks (the accuracy increases from 38.58% to 48.32% after merging). In most of the cases, LM-Cocktail2 and LM-Cocktail10 even outperform the base model on other tasks. This finding demonstrates that our approach can integrate the strengths of models to be merged, and even surpass them in performance. (4) Besides, fine-tuning on some tasks (e.g., NQ) can enhance performance not only on the corresponding task but also on other tasks; our proposed method remains effective on these tasks: LM-
Cocktail achieves higher accuracy both in target task and other tasks. These findings demonstrate the versatility of our approach.

heading: 4.1.2 Analysis on encoder-based LM
text: The results of encoder models are shown in Table 2. We can observe the same trend in the section 4.1.1: LM-Cocktail2 significantly enhances performance in downstream tasks while maintaining performance in other unrelated tasks. LM-Cocktail10 further improves the general ability by merging the models fine-tuned on different tasks. These results show the applicability of LM-Cocktail for both generative models and representation models, validating the universality of our proposed methodology.

heading: 4.2 LM-Cocktail without Fine-tuning
text: In many scenarios, fine-tuning on the target domain is not always available. Besides, fine-tuning a separate model for each task is costly and inflexible. LM-Cocktail can handle this situation well: it can perform new tasks wihtout fine-tuning by merging existing models. We report the performance of LM-Cocktail without fine-tuning in Table 3 and 4

heading: 4.2.1 Analysis on Decoder-based LM
text: To evaluate the performance on tasks which haven’t been seen in fine-tuning, we introduce additional tasks from MMLU benchmark. There are 57 tasks in MMLU, which are different from the fine-tuning tasks in section 3.1. We use the evaluation script and five-shot examples from the widely used framework EleutherAI 5.
The results are summarized in Table 3. “LlamaICL” indicates the results using in-context learning with five examples. For Multitask-learning, we merge all training data from 9 fine-tune tasks (see section 3.1) and fine-tune the Llama on this
5https://github.com/EleutherAI/lm-evaluation-harness
Dataset Llama Llama-ICL Multitask-learning LM-Cocktailblackbox LM-Cocktail LM-Cocktail’ublackbox LM-Cocktail u
Avg 45.87 46.65 32.88 42.28 48.01 47.46 48.21 abstract-algebra 28.0 30.0 21.0 29.0 35.0 33.0 34.0
anatomy 42.96 42.22 34.07 45.19 46.67 46.67 48.15 astronomy 44.08 48.03 34.21 46.05 46.05 44.08 47.37 business-ethics 42.0 42.0 41.0 50.0 46.0 52.0 48.0 clinical-knowledge 50.57 51.32 39.62 47.92 51.32 51.7 51.32
college-biology 50.0 52.78 27.08 41.67 52.08 49.31 51.39 college-chemistry 23.0 26.0 31.0 19.0 29.0 29.0 29.0 college-computer-science 29.0 37.0 37.0 33.0 46.0 43.0 45.0 college-mathematics 29.0 33.0 36.0 29.0 31.0 35.0 31.0
college-medicine 38.15 40.46 31.79 28.32 40.46 39.31 40.46 college-physics 21.57 24.51 20.59 21.57 19.61 19.61 19.61
computer-security 59.0 54.0 40.0 59.0 55.0 49.0 57.0 conceptual-physics 38.3 38.72 29.79 38.72 39.57 39.57 40.0
econometrics 28.95 33.33 22.81 28.07 26.32 35.09 28.07 electrical-engineering 42.76 43.45 34.48 33.1 46.9 44.14 47.59 elementary-mathematics 27.25 28.04 21.16 28.84 26.46 26.72 26.72 formal-logic 22.22 25.4 35.71 28.57 25.4 24.6 25.4 global-facts 41.0 31.0 26.0 37.0 35.0 33.0 35.0
high-school-biology 46.45 52.58 33.23 35.16 53.87 53.55 53.87 high-school-chemistry 30.54 33.99 21.67 30.05 32.02 32.51 31.53 high-school-computer-science 39.0 46.0 30.0 37.0 40.0 41.0 40.0 high-school-european-history 60.61 56.97 32.12 51.52 63.03 63.64 64.85
high-school-geography 57.07 59.6 33.33 55.56 60.1 61.62 59.09 high-school-government-and-politics 70.47 67.36 38.34 39.38 70.98 66.84 70.98
high-school-macroeconomics 39.23 41.54 31.79 38.21 45.64 44.1 44.87 high-school-mathematics 26.3 23.7 22.96 25.19 24.81 24.44 24.81 high-school-microeconomics 36.55 43.7 31.93 36.55 41.6 40.34 41.6 high-school-physics 25.83 28.48 29.8 27.81 29.14 30.46 29.14
high-school-psychology 59.63 64.04 33.94 60.0 65.32 65.87 64.77 high-school-statistics 23.61 31.48 36.57 25.46 28.24 30.09 28.7
high-school-us-history 65.2 66.18 38.24 65.2 65.69 66.67 64.71 high-school-world-history 61.18 66.24 35.02 57.81 66.24 65.82 65.4
human-aging 58.74 57.4 36.77 55.61 58.74 54.26 58.74 human-sexuality 54.96 48.09 38.17 29.01 57.25 55.73 56.49 international-law 59.5 57.02 42.15 48.76 61.16 58.68 60.33
jurisprudence 55.56 57.41 34.26 52.78 49.07 50.93 50.93 logical-fallacies 58.28 53.99 29.45 57.67 55.21 55.83 54.6 machine-learning 36.61 35.71 28.57 33.04 39.29 41.07 41.96 management 64.08 67.96 43.69 61.17 68.93 66.99 68.93
marketing 73.5 74.36 48.29 73.5 76.5 70.94 76.07 medical-genetics 46.0 53.0 32.0 49.0 50.0 51.0 49.0
miscellaneous 66.16 66.54 38.19 65.13 68.58 65.52 69.6 moral-disputes 50.87 52.89 28.32 44.22 49.42 49.13 50.0 moral-scenarios 24.25 21.34 24.8 23.35 24.25 24.25 24.25
nutrition 50.0 51.96 40.85 47.71 54.58 50.0 54.9 philosophy 51.77 56.91 30.87 47.59 54.02 53.7 53.05 prehistory 51.85 56.79 31.79 51.85 51.85 49.38 51.23 professional-accounting 34.75 35.46 29.08 34.75 37.23 36.52 37.94 professional-law 34.55 33.31 27.71 31.03 36.11 36.31 36.05 professional-medicine 40.44 34.19 29.41 25.37 43.01 44.85 43.75 professional-psychology 44.93 47.55 29.74 42.32 45.92 44.61 45.59
public-relations 53.64 51.82 30.0 38.18 56.36 54.55 56.36 security-studies 49.8 45.71 27.76 48.57 55.1 57.14 56.33
sociology 71.14 57.21 46.27 47.76 71.64 74.63 72.14 us-foreign-policy 73.0 68.0 42.0 67.0 71.0 67.0 73.0
virology 45.78 43.37 34.34 43.98 46.39 42.77 46.99 world-religions 64.91 67.84 37.43 61.99 70.18 67.84 70.18
Table 3: Results of merging decoder models from other tasks.
multitask datasets. For LM-Cocktial, we use the official 5 examples to compute weights and tune a new model for each task by merging 9 finetuned models and the base model. Inspired by LoraHub (Huang et al., 2023), we also compare an alternative method to compute weight: using black-box optimization in (Huang et al., 2023) to find the optimal weight assignment. We use LMCocktailblackbox to denote this variant. Besides, we aggregate all examples from each task to compute merging weights, and produce a unified model named LM-Cocktailu for all tasks, rather than generate a separate model for each task.
There are some key findings: • The performance of multitask-learning is inferior to the original llama model. This shows fine-tuning will compromise the overall generality of the original model, and also indicates there is no direct correlation between these fine-tuning
datasets and the tasks listed on the MMLU. • LM-Cocktail achieves higher accuracy than the Llama and Llama-ICL. Given five examples for each task, our approach computes the weights for existing models and merges them based on the computed weights, demonstrating a significant performance improvement. LM-Cocktail merely involves recombining existing models without the need for additional model training. Furthermore, it does not introduce any latency to the inference process, while the Llama-ICL needs to process more tokens because of the added few-shot prompt. • In comparison to black-box optimization, our method to compute weight is simpler yet highly effective. We observed that black-box optimization methods struggle to ensure the sum of weights equals 1, leading to suboptimal performance. • The unified model LM-Cocktailu also shows superior performance, which demonstrates the pro-
posed method is capable of simultaneously handling multiple new tasks. Besides, we further investigate the impact of the number of examples in section 4.4.

heading: 4.2.2 Analysis on Encoder-based LM
text: Following the setting in section 4.2.1, we compare the performance of the original BGE model, multitask-learning model, and LM-Cocktail with different methods to mix models. We collect 9 fine-tuned models from section 3.2. For evaluation, we excluded tasks which has been fine-tuned in section 3.2 (i.e., HotpotQA, MSMATCO, and Quora). As reported in Table 4, LM-Cocktail achieves higher accuracy than other models. It demonstrates that we can improve the accuracy of the new task by only mixing existing language models.

heading: 4.3 Impact of Weight α
text: In this section, we conduct a performance comparison under various weights α. To eliminate the influence of other factors, we conducted experiments in the simplest configuration: merging the fine-tuned model and base model based on the weight α.
The results of decoders are shown in Figure 2, and the results of encoder models can be seen in Appendix B.5. We incrementally varied the hyperparameter α from 0 to 1, and evaluated the model’s performance on the target task as well as other unrelated tasks. It can be observed that by changing the weights of the fine-tuned model, we can significantly improve the accuracy on other tasks, even surpassing that of the base model, while ensuring that the accuracy on the target task does not decline.

heading: 4.4 Analysis on Number of Examples
text: Additionally, we investigate the effect of the number of examples. Given some specialist models from other tasks, LM-Cocktail needs a few examples for new task to compute the merging weights,
and merge these specialist models via weighted sum. Then the merged model can be used to enhance the model fine-tuned on new task or directly perform the new task. In this section, we evaluate the performance of merged models on new tasks following the setting in section 4.2. For decoderbased model, a total of 285 examples are provided in MMLU datasets, and we randomly sample 5, 50, and 100 examples from the entire set to merge the specialist models, and test their performance. For encoder-based model, there are a total of 115 examples, and we also sample a subset to evaluate its performance. The average metric is reported in Table 5.
As shown in Table 5, Our approach achieves satisfactory performance using only five examples, and performance further improves with an increase in the number of examples. However, beyond fifty examples, the performance improvement becomes significantly limited.

heading: 5 Related Work

heading: 5.1 Fine-tuning of Language Model
text: Catastrophic forgetting problem generally exists in the continual fine-tuning of different language models (Luo et al., 2023): Fine-tuning can improve the performance of the target domain, but significantly undermine the language models’ general capabilities beyond their target domain. One solution is to add the data from previous tasks to maintain the previous abilities (Rolnick et al., 2019; Shin et al., 2017; Rebuffi et al., 2017). Some regularizationbased methods also have been proposed to alleviate this problem, where the updating of model parameters is regularized to preserve the general capability of the pre-trained model(Kirkpatrick et al., 2017; Li and Hoiem, 2017; Rannen et al., 2017). Different from adding previous data, our method has no additional costs for training. Moreover, unlike regularization-based methods, our proposed method requires no modification to the standard fine-tuning process.

heading: 5.2 Model Merging
text: Model merging averages the weights of multiple models to improve the performance of a single model (Wortsman et al., 2022a; Ilharco et al., 2022a). Wortsman et al. (Wortsman et al., 2022a) find that averaging the weights of models finetuned with different hyper-parameter configurations can often improve accuracy. Some researchers propose more complex methods to align the parameters of different models and merge them (Nguyen et al., 2021; Matena and Raffel, 2022; Jin et al., 2022). Prateek et.al and Yu et.al propose to delete the redundant values in the delta parameter before model merging (Yadav et al., 2023; Yu et al., 2023). Unlike existing methods that focus on how to select useful parameters from a fine-tuned model, we concentrate on selecting an appropriate checkpoint. These complex methods are orthogonal to our work and may potentially be beneficial to our approach. We leave the exploration of integrating these methods in LM-Cocltail for future work.
A related direction is utilizing model merging to do cross-task generalization. Most of these methods focus on merging parameter-efficient modules (e.g., LoRA (Hu et al., 2021), soft prompt(Lester et al., 2021)). For the target task, some researchers (Ponti et al., 2023; Wu et al., 2023; Lv et al., 2023) propose to aggregate the parameters
of lightweight task-specific experts that are learned from similar tasks to enhance the accuracy. Some works also have been proposed to merge prompt embeddings from lots of source tasks to the target domain (Vu et al., 2021; Poth et al., 2021; Sun et al., 2023). The latest work is LoRAHub (Huang et al., 2023). Given a few examples, it uses the black-box optimization tool Shiwa (Liu et al., 2020) to combine multiple existing fine-tuned LoRA modules and generate a new LoRA module for the target task. In contrast to the above methods, we merge fine-tuned models and the base model with the aim of maintaining the general capabilities after finetuning. Meanwhile, to ensure performance on the target task, we employ losses from a small set of examples to filter out models that perform poorly on the target task. Besides, we merge the entire model instead of the parameter-efficient modules.
The other direction relevant to our work is applying model merging in robust fine-tuning. Wortsman et al. (Wortsman et al., 2022b) and Ilharco (Ilharco et al., 2022b) both find that the fine-tuned clip (Radford et al., 2021) model substantially improves accuracy on a given target distribution but reduces robustness to distribution shifts. To address this problem, they use a manually set coefficient to merge the fine-tuned model and the base model. Unlike these methods, we propose to utilize not only the base pre-trained model but also the specialist models from other tasks. In this way, our proposed method further improves the general capabilities and even can function in situations where fine-tuning is not feasible. Besides, we utilize a simple method to compute the merging weights for different models automatically.

heading: 6 Conclusion
text: In this work, we introduce the LM-Cocktail, a simple method to improve performance on target tasks without decreasing accuracy on other unrelated tasks. LM-Cocktail produces a resilient-tuned model by weighted averaging the parameters from different models: the model fine-tuned on the target task, the pre-trained base model, and the peer models from other domains. The empirical results on both decoder and encoder models demonstrate that LM-Cocktail can achieve strong performance in the whole scope of general tasks while preserving a superior capacity in its targeted domain. We further demonstrated the effectiveness of LM-Cocktail when unable to fine-tune on domain-specific data.

heading: 7 Limitations
text: Different from existing methods (Yadav et al., 2023; Yu et al., 2023), our approach can select appropriate checkpoints by computing weights. However, it still suffers from similar constraints: it relies on the same initialization and model structure, making it inapplicable to models with different architectures. In addition, the calculation of weights still requires a certain amount of computation, and it scales linearly with the number of models. If there are too many models on the platform, considerable computational costs will still be required. Conducting a simple filtering based on the nature of the target task beforehand is a feasible approach.

heading: 8 Ethical Considerations
text: LM-Cocktail relies on the open-source language models. Consequently, it inherits similar ethical and social risks, such as bias, discrimination, and toxicity. Besides, open-source models may involve the incorporation of private or contentious data during the training phase.

heading: Acknowledgements
text: This research is supported by National Science and Technology Major Project(2023ZD0121504).

heading: A Datasets used in fine-tuning
text: We fine-tune the Llama model on 9 different datasets, whose details are shown in Table 6. The details of datasets used to fine-tune BGE are shown in Table 7.

heading: B More Experimental Results
text: B.1 LM-Cocktail for Parameter-Efficient fine-Tuning
LoRA (Hu et al., 2021) is a widely used ParameterEfficient Fine-Tuning approach to reduce the cost of training. We also investigated the effects of our method for LoRA fine-tuning, and the results are summarized in Table 8.
As shown in Table 8, LoRA demonstrates improved mitigation of catastrophic forgetting by finetuning only a small subset of parameters, with minimal degradation in performance on other tasks. However, its performance on the target task(AG News) noticeably lags behind that of full-parameter fine-tuning. With LM-cocktail, performance enhancements across other tasks are achieved while maintaining the performance on the target task.
B.2 LM-Cocktail for different base models
The experimental results of LM-Cocktail for different base models are shown in Table 9. We con-
ducted experiments on a 13B model: Llama-13B6
6https://huggingface.co/meta-llama/Llama-2-13b-chat-hf
and a different model: Mistral7. The experimental results are consistent with previous results of Llama-7B, indicating that finetuning for the target task may decrease performance on other tasks. Our approach enables the enhancement of performance on other tasks while preserving the accuracy of the target task, thereby improving the generalizability of the fine-tuned model.
B.3 Comparison with other merging methods To investigate the effectiveness of our model merging approach, we employed various methods to merge 10 models: the base model and 9 models fine-tuned on 9 different tasks(see Section 3.1). Baselines for comparison include Mean(average the weight from all models) and several recently proposed methods: LoraHub (Huang et al., 2023), DARE (Yu et al., 2023), and TIES Merging(Yadav et al., 2023). We use the MergeKit8 tool to implement DARE and TIES Merging methods. For our method, we merge these 10 models following Eqn. 4. The results are shown in Table 10. We observe that achieving satisfactory results can be attained simply by averaging parameters. The DARE method achieves competitive results by retaining only the most crucial parameters. Our approach excels in parameter fusion selection, yielding optimal outcomes.
B.4 Detailed results for each task The detailed results for each task are reported in
Table 11 and 12.
B.5 Impact of α for encoder-based LM The performance of encoder-based LMs with different merging weight are shown in Figure 3.
7https://huggingface.co/mistralai/Mistral-7B-Instructv0.2
8https://github.com/arcee-ai/mergekit
Ta bl
e 11
: T
he de
ta ile
d re
su lts
of ba
se m
od el
,fi ne
-t un
ed m
od el
,a nd
re si
lie nt
-t un
ed m
od el
vi a
L M
-C oc
kt ai
l. W
e us
e th
e na
m e
of ta
sk to
de no
te th
e m
od el
fin e-
tu ne
d on
th is
ta sk
(e .g
., A
G N
ew s
in th
e fir
st ro
w re
pr es
en tt
he m
od el
fin e-
tu ne
d on
A G
N ew
s da
ta se
t) .L
M -C
oc kt
ai l 2
is pr
od uc
ed by
m er
gi ng
th e
ba se
m od
el an
d fin
etu
ne d
m od
el ,w
hi le
L M
-C oc
kt ai
l 1 0
m er
ge s
fin e-
tu ne
d m
od el
,b as
e m
od el
,a nd
ot he
rm od
el s
fin e-
tu ne
d on
8 di
ff er
en tt
as ks
.
D at
as et
L la
m a
A G
N ew
s L
M -C
oc kt
ai l 2
L M
-C oc
kt ai
l 1 0
C om
m en
G en
L M
-C oc
kt ai
l 2 L
M -C
oc kt
ai l 1 0
SQ uA
D L
M -C
oc kt
ai l 2
L M
-C oc
kt ai
l 1 0
N Q
L M
-C oc
kt ai
l 2 L
M -C
oc kt
ai l 1 0
W in
og ra
nd e
L M
-C oc
kt ai
l 2 L
M -C
oc kt
ai l 1 0
H el
la sw
ag L
M -C
oc kt
ai l 2
L M
-C oc
kt ai
l 1 0
M N
L I
L M
-C oc
kt ai
l 2 L
M -C
oc kt
ai l 1 0
M R
PC L
M -C
oc kt
ai l 2
L M
-C oc
kt ai
l 1 0
SS T
2 L
M -C
oc kt
ai l 2
L M
-C oc
kt ai
l 1 0
ar c-
c 36
.0 5
34 .6
8 36
.9 1
38 .3
7 37
.6 8
37 .7
7 42
.1 5
36 .0
5 42
.3 2
43 .4
3 41
.4 6
43 .7
8 43
.5 2
35 .8
8 38
.8 41
.7 2
38 .5
4 37
.6 8
41 .1
2 33
.4 8
35 .7
9 37
.1 7
30 .7
3 32
.1 35
.7 9
32 .9
6 37
.6 8
38 .2
ar c-
e 51
.5 9
41 .6
5 51
.0 8
54 .1
6 60
.2 1
60 .4
7 66
.5 1
58 .3
9 65
.5 8
68 .9
6 68
.8 4
68 .9
2 71
.7 1
55 .5
6 60
.8 66
.0 5
60 .0
59 .8
7 65
.3 7
40 .2
1 49
.1 8
52 .5
2 34
.9 7
42 .1
1 48
.3 7
45 .4
1 50
.0 6
52 .1 8 na tu ra lqu es tio ns 0. 0 0. 0 0. 03 0. 03 0. 0 0. 11 6. 84 14 .0 7 14 .7 9 16 .1 2 29 .0 9 29 .2 5 29 .6 4 0. 66 0. 91 9. 81 0. 06 0. 53 9. 67 0. 0 0. 0 0. 0 0. 03 0. 03 0. 03 0. 0 0. 0 0. 0 co pa 69 .0 57 .0 70 .0 67 .0 64 .0 66 .0 78 .0 65 .0 69 .0 76 .0 70 .0 76 .0 75 .0 74 .0 76 .0 77 .0 69 .0 71 .0 77 .0 59 .0 71 .0 74 .0 47 .0 63 .0 63 .0 64 .0 72 .0 72 .0 he lla sw ag 71 .5 8 65 .7 6 71 .8 72 .1 67 .7 3 72 .0 2 73 .6 7 61 .0 68 .2 5 69 .9 2 69 .0 1 71 .5 7 72 .2 2 64 .8 70 .4 8 71 .6 4 77 .2 79 .0 78 .6 1 63 .3 1 71 .8 71 .4 2 54 .0 9 68 .6 9 69 .9 7 65 .2 9 69 .5 7 70 .3 pi qa 75 .7 3 73 .0 7 76 .1 7 75 .6 8 74 .6 5 76 .7 7 76 .7 1 74 .5 4 76 .1 2 75 .9 5 76 .4 4 76 .5 76 .3 3 74 .2 1 76 .7 7 76 .7 1 75 .9 5 77 .9 7 77 .2 72 .1 4 76 .5 75 .8 4 69 .9 7 75 .7 3 75 .6 8 74 .4 8 76 .7 1 75 .8 4 w in og ra nd e 60 .9 3 53 .5 9 58 .8 8 63 .3 58 .1 7 59 .6 7 65 .8 2 57 .2 2 61 .4 65 .9 59 .9 8 60 .7 7 67 .4 75 .4 5 77 .9 77 .0 3 62 .9 8 64 .4 8 66 .8 5 61 .8 62 .6 7 64 .5 6 52 .0 1 56 .2 7 59 .5 9 59 .6 7 61 .3 3 64 .4 w sc 60 .5 8 63 .4 6 63 .4 6 63 .4 6 63 .4 6 61 .5 4 63 .4 6 63 .4 6 63 .4 6 63 .4 6 57 .6 9 60 .5 8 62 .5 58 .6 5 53 .8 5 62 .5 60 .5 8 63 .4 6 63 .4 6 63 .4 6 60 .5 8 61 .5 4 64 .4 2 63 .4 6 63 .4 6 63 .4 6 63 .4 6 63 .4 6 w sc 27 3 75 .8 2 72 .5 3 75 .8 2 78 .7 5 64 .8 4 74 .7 3 79 .8 5 69 .2 3 76 .1 9 78 .3 9 72 .1 6 78 .0 2 76 .9 2 79 .8 5 84 .2 5 84 .9 8 76 .1 9 79 .4 9 81 .6 8 74 .7 3 78 .7 5 78 .7 5 58 .6 1 67 .0 3 72 .8 9 72 .8 9 76 .9 2 78 .7 5 m rp c 31 .8 6 31 .6 2 31 .6 2 31 .8 6 32 .1 1 31 .6 2 31 .6 2 31 .6 2 31 .8 6 32 .8 4 31 .6 2 31 .8 6 31 .6 2 34 .5 6 32 .3 5 32 .1 1 31 .6 2 31 .6 2 31 .6 2 74 .0 2 73 .0 4 70 .1 85 .7 8 73 .7 7 80 .8 8 31 .6 2 31 .6 2 31 .6 2 pa w s 55 .9 6 55 .7 6 55 .8 8 55 .8 55 .7 4 55 .6 4 56 .0 7 55 .4 3 55 .9 6 55 .7 3 55 .9 3 55 .6 9 57 .0 6 55 .7 56 .0 1 56 .9 1 55 .9 7 55 .9 7 55 .8 3 66 .7 7 60 .6 66 .2 6 44 .7 5 47 .4 4 52 .0 2 55 .8 55 .8 55 .8 6 qq p 69 .4 3 65 .2 9 66 .0 7 64 .2 8 61 .5 6 68 .0 7 66 .4 1 63 .1 3 63 .6 6 63 .8 7 65 .5 67 .1 8 68 .0 4 61 .0 2 69 .3 8 69 .5 63 .1 3 66 .1 2 63 .9 2 80 .6 1 76 .0 3 81 .3 6 49 .7 6 53 .9 4 68 .5 8 63 .1 9 63 .1 9 63 .1 9 rt e 70 .0 4 51 .2 6 74 .7 3 75 .0 9 48 .0 1 50 .5 4 68 .5 9 75 .4 5 79 .7 8 79 .7 8 54 .1 5 71 .4 8 68 .9 5 47 .6 5 53 .7 9 75 .8 1 47 .6 5 48 .0 1 77 .9 8 81 .9 5 76 .1 7 77 .6 2 58 .1 2 49 .1 57 .4 47 .2 9 47 .6 5 50 .5 4 sn li 34 .0 2 33 .4 4 50 .2 2 37 .3 4 37 .4 5 35 .9 4 45 .0 1 44 .5 1 33 .0 4 33 .3 4 41 .0 5 37 .4 4 62 .2 36 .9 9 40 .0 3 48 .6 7 41 .0 8 43 .9 9 49 .2 7 84 .9 5 87 .5 6 88 .8 2 35 .0 6 37 .1 7 43 .8 32 .9 5 32 .9 5 32 .9 5 m nl im 32 .1 4 34 .2 8 51 .6 41 .8 7 31 .2 9 31 .8 2 52 .3 3 39 .2 7 33 .3 8 37 .1 1 38 .4 35 .1 6 47 .5 34 .1 6 33 .2 1 50 .2 6 35 .5 1 35 .7 4 54 .2 9 87 .9 88 .8 8 89 .2 3 39 .9 6 40 .5 3 46 .5 2 32 .7 4 32 .7 4 32 .7 5 m nl im m 32 .0 1 35 .4 3 52 .1 5 43 .9 3 31 .7 6 32 .3 6 51 .9 9 38 .3 33 .5 9 37 .0 8 38 .0 6 34 .4 6 47 .4 1 33 .7 8 33 .2 6 50 .0 3 35 .4 6 35 .7 4 54 .1 2 87 .9 3 89 .5 2 89 .7 39 .3 40 .2 2 51 .0 6 32 .9 5 32 .9 5 33 .0 5 qn li 55 .1 9 52 .4 4 64 .2 1 57 .7 9 54 .7 57 .8 1 73 .9 50 .6 1 54 .1 8 51 .7 8 53 .9 1 61 .0 7 68 .7 9 54 .9 60 .3 1 71 .1 3 61 .4 3 63 .5 4 76 .2 6 79 .0 4 80 .1 8 82 .3 69 .4 9 56 .8 2 56 .7 6 50 .5 4 50 .6 3 50 .5 6 m ul tir c 44 .4 6 26 .0 2 56 .1 5 56 .7 1 43 .5 8 38 .2 7 44 .6 5 7. 77 27 .1 3 53 .1 6 53 .9 3 49 .3 1 59 .9 9 60 .0 9 61 .5 8 60 .3 8 26 .7 8 30 .6 3 38 .0 62 .7 7 74 .3 2 76 .4 2 51 .8 1 58 .2 3 47 .1 3 0. 0 36 .8 1 49 .5 2 op en bo ok qa 47 .0 44 .0 44 .8 46 .0 47 .6 49 .0 50 .0 45 .2 48 .0 50 .4 51 .4 51 .6 52 .6 49 .6 51 .8 52 .4 48 .0 49 .8 51 .0 45 .0 45 .4 45 .2 41 .0 43 .0 44 .2 44 .0 43 .4 45 .2 bo ol q 75 .2 6 59 .4 2 76 .6 7 76 .1 8 69 .1 1 72 .3 9 72 .0 8 67 .5 8 77 .7 4 77 .7 7 76 .0 6 77 .1 6 78 .5 79 .6 6 79 .5 1 78 .7 5 65 .7 8 65 .1 1 67 .0 9 60 .4 78 .1 3 78 .1 43 .5 5 55 .1 4 42 .6 6 37 .8 3 41 .9 6 53 .0 6 sq ua dv1 0. 06 0. 04 0. 06 1. 26 16 .4 1 28 .9 4 73 .1 5 86 .7 7 85 .6 7 86 .9 4 54 .6 1 58 .8 6 77 .4 28 .8 6 29 .2 2 72 .7 4 2. 88 10 .4 7 70 .0 6 0. 03 0. 16 7. 28 0. 07 0. 03 0. 22 0. 03 0. 79 16 .3 6 se nt im en t1 40 66 .8 5 56 .2 7 72 .7 81 .6 2 77 .9 9 84 .4 93 .5 9 83 .8 4 84 .9 6 91 .0 9 91 .3 6 89 .6 9 93 .5 9 63 .2 3 72 .7 90 .5 3 73 .8 2 84 .6 8 93 .5 9 48 .7 5 89 .9 7 92 .7 6 49 .5 8 54 .3 2 56 .8 2 92 .2 87 .4 7 92 .4 8 ss t2 63 .3 60 .2 1 86 .5 8 87 .0 4 62 .7 3 78 .4 4 93 .4 6 74 .4 3 65 .1 4 86 .9 3 66 .6 3 79 .3 6 92 .7 8 60 .5 5 76 .7 2 89 .4 5 54 .4 7 80 .9 6 93 .9 2 51 .9 5 71 .5 6 86 .9 3 48 .8 5 50 .9 2 50 .9 2 95 .5 3 96 .5 6 96 .5 6 ye lp 82 .9 51 .5 8 95 .9 6 96 .8 8 94 .9 5 93 .8 5 97 .7 5 96 .9 6 96 .2 4 96 .8 9 81 .8 7 83 .5 5 96 .8 8 84 .9 3 93 .2 96 .7 3 87 .8 9 90 .9 5 97 .8 4 52 .7 85 .2 2 96 .7 9 48 .0 52 .3 9 68 .6 3 96 .4 83 .2 9 92 .8 co m m on -g en 21 .1 4 0. 0 0. 0 0. 63 39 .2 41 .2 2 41 .4 5 20 .3 5 35 .8 2 36 .0 30 .7 4 33 .4 3 32 .9 28 .8 9 29 .8 3 36 .1 1 34 .6 2 31 .8 5 39 .2 8 0. 0 0. 0 0. 0 0. 0 0. 0 0. 0 0. 0 0. 0 15 .0 4 e2 enl g 34 .4 1 0. 0 0. 0 33 .5 6 27 .1 7 48 .7 7 48 .2 3 21 .7 3 44 .2 9 43 .3 4 16 .2 9 36 .6 8 39 .0 31 .2 6 44 .0 1 46 .1 4 38 .7 7 39 .9 8 43 .9 0. 01 0. 39 9. 64 0. 04 0. 04 0. 04 0. 0 24 .0 1 39 .3 4 da rt 32 .5 8 0. 1 0. 16 0. 22 29 .0 3 40 .8 9 35 .9 8 20 .7 3 24 .6 3 21 .9 1 16 .8 5 35 .8 5 33 .3 31 .0 6 37 .3 4 38 .8 6 28 .5 4 32 .8 8 41 .2 3 0. 02 0. 02 0. 05 0. 05 0. 05 0. 05 0. 0 0. 33 0. 15 ae sl c 3. 51 0. 02 0. 16 0. 42 2. 65 4. 01 1. 48 4. 58 4. 36 3. 78 4. 28 4. 52 1. 36 3. 41 4. 21 1. 89 2. 18 2. 79 1. 84 0. 0 0. 69 0. 78 0. 03 0. 01 0. 07 0. 0 1. 58 0. 79 ag -n ew s 40 .8 94 .4 2 94 .4 6 94 .4 1 39 .1 2 50 .3 3 88 .0 4 75 .1 1 52 .5 3 54 .5 3 70 .2 9 47 .3 2 87 .2 4 38 .1 8 39 .1 4 82 .7 1 49 .9 2 56 .3 4 87 .6 8 32 .1 4 53 .5 4 66 .6 7 25 .8 39 .6 1 64 .6 8 33 .7 31 .9 5 35 .3 3 gi ga w or d 3. 8 0. 0 0. 16 0. 09 6. 51 13 .2 8 1. 3 19 .3 2 14 .1 5 3. 16 4. 94 5. 13 5. 52 10 .4 5. 7 5. 64 7. 98 7. 91 7. 16 0. 0 3. 58 0. 52 0. 0 0. 0 0. 0 0. 0 0. 44 0. 14 A vg 46 .6 40 .4 4 49 .2 8 49 .8 6 46 .6 5 50 .5 6 58 .0 50 .7 2 52 .7 7 55 .1 9 51 .4 2 53 .7 4 59 .2 6 48 .2 7 51 .4 4 59 .1 4 47 .1 3 49 .9 5 58 .5 6 48 .8 4 54 .7 1 57 .4 1 38 .0 9 40 .7 44 .0 4 40 .8 3 43 .4 6 46 .7 5

heading: Future Work
text: We leave the exploration of integrating these methods in LM-Cocltail for future work. A related direction is utilizing model merging to do cross-task generalization. Most of these methods focus on merging parameter-efficient modules (e. g. , LoRA (Hu et al. , 2021), soft prompt(Lester et al. , 2021)). For the target task, some researchers (Ponti et al. , 2023; Wu et al. , 2023; Lv et al. , 2023) propose to aggregate the parameters
of lightweight task-specific experts that are learned from similar tasks to enhance the accuracy. Some works also have been proposed to merge prompt embeddings from lots of source tasks to the target domain (Vu et al. , 2021; Poth et al. , 2021; Sun et al. , 2023). The latest work is LoRAHub (Huang et al. , 2023). Given a few examples, it uses the black-box optimization tool Shiwa (Liu et al. , 2020) to combine multiple existing fine-tuned LoRA modules and generate a new LoRA module for the target task. In contrast to the above methods, we merge fine-tuned models and the base model with the aim of maintaining the general capabilities after finetuning. Meanwhile, to ensure performance on the target task, we employ losses from a small set of examples to filter out models that perform poorly on the target task. Besides, we merge the entire model instead of the parameter-efficient modules. The other direction relevant to our work is applying model merging in robust fine-tuning. Wortsman et al. (Wortsman et al. , 2022b) and Ilharco (Ilharco et al. , 2022b) both find that the fine-tuned clip (Radford et al. , 2021) model substantially improves accuracy on a given target distribution but reduces robustness to distribution shifts. To address this problem, they use a manually set coefficient to merge the fine-tuned model and the base model. Unlike these methods, we propose to utilize not only the base pre-trained model but also the specialist models from other tasks. In this way, our proposed method further improves the general capabilities and even can function in situations where fine-tuning is not feasible. Besides, we utilize a simple method to compute the merging weights for different models automatically