Pushing the Limits of Instruction Tuning via Alignment Tax Reduction (2024)

Abstract: Supervised fine-tuning (SFT) on instructionfollowing corpus is a crucial approach toward the alignment of large language models (LLMs). However, the performance of LLMs on standard knowledge and reasoning benchmarks tends to suffer from deterioration at the latter stage of the SFT process, echoing the phenomenon of alignment tax. Through our pilot study, we put a hypothesis that the data biases are probably one cause behind the phenomenon. To address the issue, we introduce a simple disperse-then-merge framework. To be concrete, we disperse the instructionfollowing data into portions and train multiple sub-models using different data portions. Then we merge multiple models into a single one via model merging techniques. Despite its simplicity, our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.1

heading: 1 Introduction
text: Trained on trillions of tokens from webpages (OpenAI, 2023; Bai et al., 2023; Google, 2023), large language models (LLMs) have demonstrated impressive capacity on obtaining general-purpose representations for various downstream NLP tasks. However, pre-trained language models may not follow human instructions (Ouyang et al., 2022) and produce toxic, hallucinated, or biased content (Sun et al., 2024; Huang et al., 2023; Zhang et al., 2023c). To address the issue, supervised fine-tuning (Ouyang et al., 2022) on instructionfollowing data has emerged as one of the de facto paradigms (Taori et al., 2023; Chiang et al., 2023) for aligning LLMs with human preferences.
†This work was done during internship at Tencent AI Lab. *Corresponding authors: Deng Cai (thisisjcykcd@gmail.com) and Rui Yan (ruiyan@ruc.edu.cn) 1The code is released at https://github.com/ TingchenFu/ACL24-ExpertFusion.
However, with the size of instruction-following data rising, it has been observed that the performance of LLM on standard knowledge and reasoning benchmarks does not always improve but exhibits degradation (Dou et al., 2023), i.e., the alignment tax (Bai et al., 2022), as is shown in Figure 1. In other words, simply scaling up the instruction-following data leads to a quick bump into the upper bound where the marginal return of increasing data size approaches zero or even minus. It is therefore non-trivial to unleash the full potential of large-scale instruction-following data.
Prior studies tend to attribute the alignment tax phenomenon to the low-quality samples within the instruction-following corpus (Chen et al., 2023; Cao et al., 2023), or the knowledge forgetting during the SFT process (Dou et al., 2023; Ren et al., 2024). However, our pilot study in Section 3 reveals that the quality issue and the catastrophic forgetting of pre-training knowledge are probably not the main cause of the alignment tax since the decline can be observed across corpora with varied
2967
sizes and quality.
By analyzing the trend of loss descent during the SFT process, we alternatively posit that the data biases fitted on the instruction data are probably one of the major causes behind it. Specifically, during the tuning process, LLMs fit on dataset biases while acquiring instruction-following ability. In the beginning, the acquisition of generalizable ability predominates so the performance on knowledge and reasoning benchmarks improves. However, during the tuning process, the learning of generalization quickly stagnates and the model tends to acquire more data biases instead, which harms the parametric knowledge of LLM and leads to a decline in related benchmarks.
We propose a frustratingly simple DTM (DisperseThen-Merge) framework composed of three steps: (1) we initially distribute the instruction-following data into several clusters and then (2) perform instruction tuning on each cluster of data to obtain a series of sub-models assimilating different data biases; (3) finally we merge the sub-models trained on each cluster into a single one in the weight space, such that the data bias of each sub-model is mitigated at fusion. Importantly, DTM ensures the reduction of alignment tax when instruction tuning with almost no extra cost at both training and inference.
To empirically verify the efficacy of the DTM framework, we conduct extensive experiments and evaluations across 9 benchmarks involving math reasoning, world knowledge, and code generation. The experiment results exhibit that DTM outperforms both (1) data selection methods that filter out low-quality samples (Dou et al., 2023); and (2) regularization and continue training methods that prevent the forgetting of knowledge learned from pre-training (Kirkpatrick et al., 2016; Rolnick et al., 2018). In particular, different from previous methods, DTM does not require any additional training and it incurs almost no extra cost at inference.
The contribution of this paper can be summarized as follows:
• We empirically verify and analyze the effect of alignment tax during the instruction tuning, thereby putting a hypothesis that the dataset biases are the reason behind the alignment tax. • We propose a frustratingly simple DTM framework in which the biases from instruction-following data are distributed and forgotten.

heading: 2 Related Work
text: Supervised Instruction Tuning. Supervised fine-tuning of LLMs on open-domain instructionfollowing data (Ouyang et al., 2022) is a promising approach for calibrating LLMs with human values, which is a critical prerequisite prior to their deployment in real-world scenarios (Xu et al., 2023c). Bypassing the complex and unstable proximal policy optimization algorithm Schulman et al. (2017) in the reinforcement learning from human feedback (RLHF) procedure (Ouyang et al., 2022), SFT only requires a high-quality instruction-following corpus collected from GPT-4 (OpenAI, 2023) or human annotator (Zhou et al., 2023; Conover et al., 2023) to tune on. In spite of its simpleness, a surge of recent models (Ding et al., 2023; Xu et al., 2023a; Geng et al., 2023; Xu et al., 2023b) prove the effectiveness of SFT with their impressive performance on both conventional knowledge and reasoning benchmarks (Hendrycks et al., 2021) and newly appeared instruction-following benchmarks (Li et al., 2023d; Zheng et al., 2023). However, Bai et al. (2022) point out that in particular cases, alignment of LLM is a double-edged sword, enhancing instruction-following ability at the sacrifice of capacity on the conventional knowledge and reasoning benchmark, or the alignment tax. Some follow-ups (Dou et al., 2023; Chen et al., 2023) conjecture that low-quality samples and interference of parametric knowledge are the reasons behind this. Different from previous works, in this study we propose a new perspective to understand the root cause of alignment tax.
Model Merging. Model merging is an effective technique to aggregate the capacity of multiple models. Distinct from model ensemble, merging techniques involve pruning (Yadav et al., 2023), rescaling (Yu et al., 2023), re-weighting (Matena and Raffel, 2022) or rotating (Singh and Jaggi, 2020) the parameters of multiple models before merging them into a single one in the weight space, therefore incurring no extra latency at inference. Different from previous works that apply model merging for multi-task learning (Yang et al., 2023b; Jin et al., 2023), machine unlearning (Hu et al., 2023; Daheim et al., 2023), domain transfer (Ilharco et al., 2023; Zhang et al., 2023a), multi-objective reinforcement learning (Rame et al., 2023; Jang et al., 2023), we utilize model merging for the alignment of LLM. Actually, model merging is closely related to the learned biases of neural networks.
Although small models with different prediction mechanisms can hardly be merged together without performance loss (Lubana et al., 2022; Juneja et al., 2023), it is different for large-scale fine-tuned models from pre-trained checkpoints, which could generally maintain their capacity when merged together (Qin et al., 2022; Gueta et al., 2023). Recently, Zaman et al. (2023) and Wan et al. (2024) have shown the possibility of fusing complementary knowledge or removing unintentional memory with the assistance of model merging.

heading: 3 Pilot Study
text: The existence of alignment tax indicates an upper bound of performance when directly increasing the data size at supervised fine-tuning. It is thus necessary to analyze the underlying cause for the alignment tax to unleash the full potential of instruction-following data. Specifically, we first examine the intuitions that data quality and knowledge forgetting are responsible for the decline in conventional knowledge and reasoning benchmarks (Section 3.1), and then posit our hypothesis that the biases during fitting the instruction-following data is probably one of the major causes (Section 3.2).

heading: 3.1 Are Data Quality and Knowledge Forgetting the Main Causes of the Alignment Tax?
text: The experiments are mainly conducted on Llama-27b (Touvron et al., 2023) with TÜLU-V2-mix. To examine the previously accepted data quality hypothesis (Chen et al., 2023), we employ the quality evaluator in Liu et al. (2024) to filter TÜLU-V2-mix samples, keeping only the samples with an above 2.5 quality score for tuning and the experiment re-
sults are shown in Figure 2. Besides, to verify the effect of pre-training knowledge forgetting, we mix the instruction-following corpus with an equivalent amount of pre-training data from Redpajama (Computer, 2023) for multi-tasking, and the experiment results are shown in Figrue 3.
From the experiment results, it is not challenging to discern the following points: • Data quality is probably NOT the main reason. Even if we filter out the low-quality samples within the instruction-following corpus with a quality evaluator (Liu et al., 2024), the alignment tax still exists as shown in Figure 2, suggesting that data quality is probably not the main cause behind the performance decline. • Knowledge forgetting is probably NOT the main reason. Although a significant amount of pre-training data is mixed into the pre-training corpus to alleviate the forgetting and intervention of parametric knowledge, from Figure 3 we can see the drop in performance of traditional knowledge and reasoning benchmarks can hardly be removed. Therefore, it is probably unreasonable to attribute alignment tax to knowledge forgetting.

heading: 3.2 Seek for Main Causes of the Alignment Tax
text: To understand the reason behind the alignment tax and in particular what is learned when alignment tax occurs, we propose to track the change of loss during the SFT process. In detail, we randomly split the dataset into 10 portions with equal sizes, training on 9 of them sequentially and leaving one for evaluation. Every after a portion is finished, we measure the loss reduction on the training set ∆Ltrain and the loss reduction on the validation
set ∆Lval. Intuitively, while ∆Lval reflects the enhancement in generalizable model capacity on instruction following, ∆Ltrain encompasses not only the generalizable instruction-following ability, but also the ungeneralizable data specific biases. To measure the composing proportion of the two components, we plot the ratio ∆Ltrain/∆Lval during the training process in Figure 4.
As is shown, the ratio is approximately 1.0 at the beginning, suggesting that generalizable instruction-following ability dominates at the initial of training. But as the SFT goes on, the ratio quickly inflates from 1.0 to nearly 20, indicating that the acquisition of data biases gradually outweighs other factors and becomes the major reason for loss reduction. Furthermore, to have a more intuitive understanding of data-specific biases, we exhibit the token-level biases by measuring the correlation between the per-token loss reduction on the training set and the validation set. Spearman’s ρ between the loss reduction on two sets is shown in Figure 5. From the figure, it becomes apparent that as the instruction tuning goes on, the fitting on training tokens gradually deviates from the generalizable ability. Meanwhile, some representative tokens with prominent loss reduction at the beginning and the end of training are shown in Figure 6. In a comparison between Figure 6a and Figure 6b, we can observe that the training loss reduction at the end can be mainly attributed to rare words and symbols, suggesting the existence of ungeneralizable data biases.
Therefore, we hypothesize that the datasetspecific biases and shortcuts (Wang et al., 2022; Du et al., 2021) are probably one of the primary contributors to the fitting of the training corpus. Once the assimilation of ungeneralizable dataset bi-
ases outweighs the growth of instruction-following capacity, the world knowledge and commonsense reasoning ability of LLM is damaged, thus causing the degradation in related benchmarks, or the alignment tax (Bai et al., 2022).

heading: 4 Methodology
text: As analyzed above, vanilla SFT on the full volume of instruction-following data suffers from the assimilation of dataset biases, leading to inefficiency in exploiting large-scale instruction-following corpus. Previously, Zaman et al. (2023) discovered that when two BERT-based classification models are merged together, the unshared knowledge within each model is mostly forgotten while the common knowledge is enhanced. Getting inspiration from this, to unleash the full potential of large-scale instruction-following data, we propose a DTM framework, as shown in Figure 7. In a nutshell, we disperse the instruction-following data into multiple portions to obtain a series of submodels with different data biases. Then through the
fusion of multiple sub-models, we can aggregate their instruction-following capacities and eliminate their dataset biases at the same time.

heading: 4.1 Instruction-following Data Distributing
text: As standard SFT, DTM assumes access to a base LLM M0 and an instruction-following corpus D = {(xi, yi)}Ni=1 with N samples where xi is the instruction prompt and yi is the response. The first step involves distributing the samples into nonoverlapped K clusters {D1,D2, . . . ,DK}. Numerous approaches can be employed to achieve the clustering of the training data. For instance, we can first obtain the embedding of an instruction sample exploiting an off-the-shelf sentence embedding model, or feed the sample into an LLM and use the pooling of the last hidden states as the sentence embedding alternatively. Once the embedding of instruction is obtained, K-means clustering based on cosine distance in embedding space is a good choice to divide the instruction-following corpus into K portions while other clustering schemes like random splitting are also acceptable.

heading: 4.2 Sub-model Training
text: After data distributing, the base LLM M0 is tuned on K portions of instruction-following data respectively with the standard next-token prediction objective, resulting in K instruction-tuned submodels M1,M2, . . . ,MK . It is worth mentioning that the sub-models are not impervious to biases; however, the biases they acquire vary from one another. In addition, according to the observed trend in Section 3, fitting on bias diminishes when the scale of instruction-following data narrows down, suggesting that the bias learned by M1,M2, . . . ,MK is less than the vanilla SFT counterpart.

heading: 4.3 Model Merging
text: The K tuned model M1,M2, . . . ,MK share common knowledge and capacity on instruction following but their data biases are distinct from each other. Consequently, the acquired capacity on instruction following is maintained, while their unique data biases are forgotten if we fuse the K sub-models together, according to Zaman et al. (2023). Various methods can be utilized to accomplish model fusion and simply taking the weighted average of K sub-models is the most straightforward strategy:
Mif = K∑
j=1
αjMij , (1)
where Mf is the fused model and the superscript denotes a single parameter in the model. αj is the merging weight of the j-th sub-model Mj and we have ∑K j=1 αj = 1, αj ≥ 0 (j = 1, 2, . . . ,K). If not specified otherwise, we use the weighted average of sub-models for its simplicity and ease of use.

heading: 5 Experiment

heading: 5.1 Experiment Setup
text: Data and Backbone In our experiment, we employ the TÜLU-V2-mix (Ivison et al., 2023) for SFT, a meticulously curated combination on the basis of TÜLU-V1-mix (Wang et al., 2023). It contains 326, 154 samples collected from 11 opensourced instruction-following corpora, which are either manually written by human annotators, converted from existing NLP benchmarks, or curated by GPT-4. As for the backbone, we employ the Llama-2-7b (Touvron et al., 2023) as our base LLM. The code is released to facilitate future relevant research.
Baseline Method We compare the proposed DTM framework with the following baselines: • Vanilla, or traditional SFT on the instructionfollowing data with language modeling objective. • L2-norm, where L2 regularization is incorporated in the training objective to circumvent the overfit on instruction-following data and interference with the parametric knowledge. • EWC (Elastic Weight Consolidation) (Kirkpatrick et al., 2016) is a typical regularization in the subfield of continue learning to alleviate the forgetting of previously learned knowledge. There, we apply EWC in SFT to mitigate the catastrophic forgetting of pre-training knowledge. • Replay (Rolnick et al., 2018) is another typical method for mitigating catastrophic forgetting in continue learning. In our implementation, we mix the pre-training data reconstructed by Redpajama (Computer, 2023) into the instructionfollowing corpus in a 1:1 ratio and perform multitask learning on plain language modeling and instruction-following to retain the pre-training knowledge. • Uniform Soup (Wortsman et al., 2022) is a similar recipe to ours in the sense that it fuses multiple trained models into a single one employing model merging techniques. However, in this case, multiple models are trained on the entire corpus with different hyper-parameter configurations. • MoE (Dou et al., 2023) is a recently proposed method to deal with the performance drop of LLM in knowledge-intensive benchmarks. Combining MoE with parameter-efficient fine-tuning, Dou et al. (2023) enables expert coordination for task utilization and full leverage of parametric knowledge. • Deita (Liu et al., 2023) is an automatic data selection strategy for alignment comprehensively considering the complexity, quality, and diversity of instruction-following data. In our implementation, we keep the samples with complexity scores exceeding 2.5 for training. Evaluation To have a comprehensive understanding on the efficacy of different training recipes, the evaluation encompasses the capacity of LLM in multiple aspects: math reasoning (GSM8K Cobbe et al., 2021), factual knowledge (MMLU Hendrycks et al., 2021), commonsense reasoning (BBH Suzgun et al., 2023, ARC-c Clark et al., 2018 and OpenBookQA Mihaylov et al., 2018), reading comprehension (RACE Lai et al., 2017), code generation (HumanEval Chen et al., 2021a and MBPP Austin et al., 2021) and truthfulness (TruthfulQA Lin et al., 2022), strictly following the evaluation protocol of Open LLM Leaderboard2. In addition, we assess their instruction following ability with MT-bench (Zheng et al., 2023) and Vicuna-bench (Chiang et al., 2023), two widely used instruction-following benchmarks.

heading: 5.2 Experiment Results
text: The main experiment results are shown in Table 1, in which we randomly distribute the instructionfollowing data into K = 4 clusters and utilize average weight merging (αj = 0.25, j = 1, 2, 3, 4) for fusion. From the table we can observe that our proposed approach outperforms its peers on most evaluation benchmarks, proving the effectiveness of our DTM framework. Meanwhile, the performance of Uniform Soup is notable, achieving the second-best results on three benchmarks. The difference between Uniform Soup and ours lies in that their sub-models for merging are trained on the full volume of data with different hyper2 https://huggingface.co/spaces/HuggingFaceH4/ open_llm_leaderboard
parameters. Consequently, the data biases of its sub-models are more likely to be overlapped and cannot be removed at merging. In addition, the performance of L2-norm and EWC also attains impressive performance on two benchmarks respectively, possibly due to the retention of pre-training knowledge through regularization techniques.
The effect of different clustering methods. To investigate the impact of different data clustering methods, we experiment with different sentence embedding models and different encoding schemes. In detail, we use MiniLM (Wang et al., 2020) and MPNet (Song et al., 2020) from the SentenceTransformers library (Reimers and Gurevych, 2019)3 to encode the instruction (I) or the response (R) or both of them (I+R) to obtain their dense representation for K-means clustering. The experiment results based on different clustering methods are shown in Table 2. From the table, it can be inferred that although the dense representation obtained via encoding response (R) is slightly better than other encoding schemes for clustering, none of those sophisticated clustering methods have an obvious advantage over simple random clustering.
3 https://www.sbert.net/
The effect of different merging methods. As for the model fusion, we experiment with several widely used merging techniques: (1) Fisher (Matena and Raffel, 2022) employs the approximated Fisher information matrix to approach the fused model with the highest joint probability. (2) Task Vector (Ilharco et al., 2023) subtracts the base LLM weight from the instruction-tuned model in the weight space to obtain the task vector and accomplish merging with vector arithmetic; (3) Tie Merge (Yadav et al., 2023) trims and prunes the models before merge and resolves the interference between multiple models. (4) DARE (Yu et al., 2023) refines task vector by dropout and re-scale before conducting vector arithmetic. The experiment results on different clustering methods are shown in Table 4. Similarly, it seems that no single merging method is apparently superior to others, and simple average weight merging is sufficient.
Performance on instruction following. The experiment results on instruction-following ability
are shown in Table 3. We can observe that our approach surpasses the vanilla SFT (5.19 v.s. 4.86 in MT-bench and 6.60 v.s. 6.26 in Vicuna-bench) and attains the best or the second-best performance, suggesting that our approach not only maintains the basic knowledge and reasoning ability of language model, but also improve the instruction-following ability. Notably, Uniform Soup exhibits strong instruction-following ability since their sub-models are trained on a full volume of data and therefore acquire stronger instruction-following capacity, although at the cost of more damage to world knowledge and commonsense reasoning ability.

heading: 6 Analysis
text: The SFT experiments on TÜLU-V2-mix have proven the efficacy of the proposed approach. To gain more in-depth insights, further exploration and analysis are detailed below.
Question1: How does DTM help? Answer1: Measuring the data biases or shortcuts on instruction following is challenging since we are agnostic to the specific form of the bias. Therefore, we choose to quantify the data biases of LLM through their error sets on MMLU (Hendrycks et al., 2021). We plot the Venn diagram for the error set of K = 4 sub-models in Figure 8. It can
be observed that their error sets share a large portion (48.7%) but every sub-model has its own error cases (accounting for 4.2%, 4.6%, 4.8% and 3.3% of the entire error cases respectively), attributing to their unique shortcut.
Next, we bucket the test case of MMLU into different bins according to their count of appearance n in the error sets. For example, n = 4 for the cases within the common intersection of four error sets while n = 1 for unique error cases of sub-models. Specifically, n = 0 denotes that the case does not belong to any error set or equivalently all sub-models can figure out the answer correctly. Then we plot the accuracy of the fused model on each bin in Figure 9. From the figure, the accuracy on the first bin (n = 0) is nearly approaching 100%, suggesting that the common knowledge is retained. On the other hand, the high accuracy on the second bin means that the unique error cases of four sub-models are likely to be correctly solved by the fused model, which is evidence for the forgetting of unique data biases in four sub-models.
Question2: Does DTM yield effective results across instruction-following data of varying sizes and domains?
Answer: To examine the robustness and generality of our approach, aside from TÜLU-V2-mix, we conduct experiments on other five widely used instruction-following corpora within or not within the TÜLU-V2-mix, namely GPT4-Alpaca (generic, 52,002 samples, Peng et al., 2023), Code-Alpaca (code, 20,022 samples, Chaudhary, 2023), Baize (Quora & StackOverflow & medicine, 158,183 samples, Xu et al., 2023b), Camel (STEM, 109,740 samples, Li et al., 2023a), Evol-Instruct (generic, 70,000 samples, Xu et al., 2023a) and LIMA (Stack Exchange and Reddit, 1,000 samples). The performance of our approach in comparison with vanilla SFT is shown in Table 5, from which we can conclude that DTM is not constrained by the domain of the instruction-following data, but its superiority is influenced by the data size.
Question3: How is model fusion in comparison to model ensemble? Answer: Different from model fusion which aggregates the parameter of multiple models in the weight space, model ensemble aggregates multiple models by manipulating their logits. To draw a comparison of their effects, we substitute the model merging procedure in Uniform Soup (Wortsman et al., 2022) and our approach with model ensemble, and the evaluation results on MMLU are shown in Table 7. From the table, model ensemble is almost on par with model fusion except that model fusion is marginally better than ensemble overall. However, the computation required by model ensemble is K (the number of sub-models) times larger than the model fusion, and thus its throughput is inferior to the model merge.
Question4: Does the proposed approach work on other base LLM? Answer: To answer the question, we conduct experiments with Mistral-7b (Jiang et al., 2023) and Baichuan-2-7b (Yang et al., 2023a), two renowned backbones with remarkable performance on Open LLM Leaderboard with similar parameter scale.
The experiment results are shown in Table 6, suggesting that DTM is agnostic to the base LLM and able to generalize to more capable LLMs.

heading: 7 Conclusion
text: In this study, we target the alignment tax during the SFT. Through a series of pilot studies, we hypothesize that data biases are the root cause for the decline in standard benchmarks after an LLM goes through the SFT process. To deal with the problem, we propose a simple three-step framework to disperse the biases apart and employ model merging techniques to mitigate the effect of data biases. Extensive experiments are conducted to empirically verify the efficacy of our approach and we hope our research will inspire more future work exploring the essence and mechanism of LLM alignment together with its effects on the capacity of LLM.

heading: Acknowledgement
text: This work was supported by the National Natural Science Foundation of China (NSFC Grant No. 62122089), Beijing Outstanding Young Scientist Program NO. BJJWZYJH012019100020098, and Intelligent Social Governance Platform, Major Innovation & Planning Interdisciplinary Platform for the "Double-First Class" Initiative, Renmin University of China, the Fundamental Research Funds for the Central Universities, and the Research Funds of Renmin University of China.
Limitations
The limitations of this study can be summarized as below: • In this work, we mainly focus on the alignment tax during the supervised fine-tuning process. Aside from SFT, there are multiple alternative approaches towards the alignment of LLM such as RRHF (Yuan et al., 2023), DPO (Rafailov et al., 2023), and their variants. However, we did not verify or discuss the alignment tax in other alignment methods and we would like to leave this for future work. • We generally utilize LoRA (Hu et al., 2022) as a parameter-efficient fine-tuning (PEFT) technique for SFT and do not perform experiments with other PEFT techniques such as adapter (Houlsby et al., 2019) or IA3 (Liu et al., 2022) or full-parameter fine-tuning.
Ethical Consideration
This paper has few ethical risks and will not pose a problem with ethics. First, the alignment of large language models is not a new task in natural language processing, and several papers about this task have been published at NLP conferences. Second, all the datasets and benchmarks used in this paper have been published in previous papers. Our work aims at better understanding and eliminating alignment tax towards the tax-free alignment and our approach should not be used for any malicious purpose.

heading: A More Details on Experiment Setup
text: A.1 More details on Instruction data In our experiment, we majorly employ the meticulously curated TÜLU-V2-mix4 (Li et al., 2023a) corpus for SFT. Composed of 11 subsets, TÜLUV2-mix includes 326, 154 samples, compared to 490, 445 in the V1 mixture. To reduce the computation cost required for fine-tuning, we only keep the first turn of dialogue in case there are multiple instructions and responses in a datum, and the data statistics for each subset consisting of the corpus are shown in Table 8. Besides, in Section 6, we perform SFT on other 5 corpora to investigate the generality of our approach. Among the five corpora, Alpaca-GPT4 (Peng et al., 2023) and CodeAlpaca (Chaudhary, 2023) are constituting components of TÜLU-V2-mix, while Baize (Xu et al., 2023b), Camel (Li et al., 2023a) and Evol-Instruct70k (Xu et al., 2023a) are external instruction corpora and their statistics are shown in Table 9.
A.2 More details on Evaluation Benchmarks In our experiment, to draw a comparison with our proposed DTM framework with other baseline methods, we evaluate on the following benchmarks:
• GSM8K (Cobbe et al., 2021) is a collection of 1,319 middle-school level math word problems with each question consisting of basic arithmetic operations (Azerbayev et al., 2023). Following (OpenAI, 2023), we experiment with 8-shot prompting and greedy decoding at inference.
• MMLU (Hendrycks et al., 2021) is a popular aggregated benchmark covering 57 tasks including elementary mathematics, US history, computer science, law, and more, which are categorized into 4 subsets: STEM, Humanities, Social Science and Others. Extensive world knowledge and problem-solving ability are required to attain a high score on this benchmark. We use 5-shot prompting at evaluation and report the overall accuracy.
• BBH (BIG-Bench Hard) (Suzgun et al., 2023) is a challenging subset of BIGBench (Srivastava et al., 2022) on which prior language models fall behind average humanraters. Composed of 23 particularly challenging tasks (27 sub-tasks), the benchmark
4 https://huggingface.co/datasets/allenai/
tulu-v2-sft-mixture
mainly focuses on LLM abilities in four aspects, namely multi-step arithmetic reasoning, natural language understanding, use of world knowledge, and multilingual knowledge and reasoning. Following Suzgun et al. (2023), we evaluate all models via greedy decoding and report the exact match between the generated output (after extracting the content behind the “the answer is” keyword) and the ground-truth label. 3-shot prompting and chain-of-thought prompting are employed as a common practice to improve performance.
• ARC (Clark et al., 2018) is a collection of genuine grad-school level science multiple-choice problems with two subsets, namely Easy Set (ARC-e) and Challenge Set (ARC-c). Our experiments are mainly conducted on ARC-c, which is composed of 1,172 problems that cannot be trivially solved by word co-occurrence algorithm or retrieval algorithm. We adopt zero-shot prompting and report the accuracy.
• OBQA (OpenBookQA) (Mihaylov et al., 2018) is a set of elementary level science multiple-choice problems. Modeled after the open book exams testing the understanding of a student on a specific subject, each question in the dataset is accompanied by a basic scientific fact and requires the possession of commonsense knowledge to combine the facts. Similarly, we adopt zero-shot prompting at inference and report the accuracy.
• RACE (Lai et al., 2017) is a large-scale reading comprehension benchmark, in which the problems are collected from the English exams for middle and high school Chinese students and cover a wide range of topics. Compared with other reading comprehension datasets, it requires more reasoning to work out the answer. Similar to the above two benchmarks, we adopt zero-shot prompting at inference and report the accuracy.
• HumanEval (Chen et al., 2021b) is a suit of 164 hand-written Python programming problems released by OpenAI, with each problem consisting of function signature, docstring, body, and several unit tests to validate the code produced by a language model. Following (Li et al., 2023c; Rozière et al., 2023), we use pass@k as our metric, which is the total frac-
tion of benchmark problems solved, where a problem is considered solved if any one of k code samples passes every test case. We adopt the simplest version of pass@k, namely pass@1, which is the likelihood that a problem is solved in a single attempt by the model. Greedy decoding is used for inference.
• MBPP (Austin et al., 2021) is another widely used test set for evaluating the code generation ability of language models, composed of 974 Python short functions and program textual descriptions. Similar to HumanEval, the performance of MBPP is evaluated by pass@1 and greedy decoding is adopted for inference.
• TruthfulQA (Lin et al., 2022) is a popular problem set for evaluating the truthfulness of LLM. Composed of 817 spanning 38 categories, it is widely used for benchmarking the hallucination of LLM (Zhang et al., 2023b; Chuang et al., 2024; Li et al., 2023b). We use the multiple-choice configuration of the benchmark and report the MC1 score, which is the fraction of benchmark problems where
models assign the highest scores to the best answer.
• Vicuna-bench (Chiang et al., 2023) is a recent benchmark with GPT-4 as a judge. Containing 80 questions spanning various categories such as roleplay, commonsense, and Fermi problems, it evaluates the instruction following proficiency of LLM.
• MT-bench (Zheng et al., 2023) is another rigorous benchmark for measuring both the conversation ability and instruction-following ability of language models. It contains 80 multi-turn questions across eight subjects: writing, roleplay, extraction, reasoning, mathematics, coding, knowledge I (STEM), and knowledge II (humanities/social science). We report the first-turn score since our instruction tuning only involves a single instructionresponse pair.
A.3 More Implementation details Our experiments are conducted on a cloud Linux server with Ubuntu 16.04 operating system. The
codes are written in Python 3.10 using the code from huggingface library5. The GPU type is the Nvidia Tesla V100 with 32GB GPU memory. The detailed hyper-parameter settings for training different base LLMs are shown in Table 10, which mostly follows Lee et al. (2023). We train each submodel for 3 epochs and use the following template for fine-tuning, which is borrowed from Taori et al. (2023):
Below is an instruction that describes a task. Write a response that appropriately completes the request.
### Instruction: {instruction} ### Response: {output} Note that the language modeling loss is only considered for the output part. We use the code from Abel6 (Chern et al., 2023), Open Instruct7, Language Model Evaluation Harness8 (Gao et al., 2023), Bigcode Evaluation Harness9 (Ben Allal et al., 2022) and Open Compass (Contributors, 2023) for evaluation.

heading: B More Experiment Results and Analysis
text: B.1 More Observations on Alignment Tax Our pilot study reveals the decline in MMLU and BBH when tuning Llama-2-7b and Llama-2-13b on the TÜLU-V2-mix. To have a more comprehensive understanding, we supplement the experiment on tinyllama-1.1b (Zhang et al., 2024), and the experimental results are shown in Table 10.
5 https://huggingface.co/ 6 https://github.com/GAIR-NLP/abel 7 https://github.com/allenai/open-instruct 8 https://github.com/EleutherAI/
lm-evaluation-harness 9 https://github.com/bigcode-project/ bigcode-evaluation-harness
B.2 The effect of the number of sub-models To investigate how different choices of K (the number of clusters and the number of sub-models) affect the effectiveness of the DTM framework, we vary the hyper-parameter K from 2 to 6 and the experiment results with different numbers of submodels are shown in Figure 11. From the figure we find that K = 4 attains the best performance among different choices of K. We gauge that there exists a trade-off between the acquisition of common knowledge and the forgetting of biases. If K is too small, the data-specific biases are not adequately dispersed. Thus the biases learned by each sub-model are too similar to be forgotten via merging. On the other hand, if K is too large, the average number of samples in each cluster narrows down and probably can not provide sufficient knowledge of instruction-following.
This is an appendix.

heading: Future Work
text: However, we did not verify or discuss the alignment tax in other alignment methods and we would like to leave this for future work. • We generally utilize LoRA (Hu et al. , 2022) as a parameter-efficient fine-tuning (PEFT) technique for SFT and do not perform experiments with other PEFT techniques such as adapter (Houlsby et al. , 2019) or IA3 (Liu et al. , 2022) or full-parameter fine-tuning