How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2024)

Abstract: Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, code generation, and instruction following. These abilities are further enhanced by supervised fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for enhancing individual capabilities, proprietary LLMs exhibit versatility across various skills. Therefore, understanding the facilitation of multiple abilities via SFT is paramount. In this study, we specificially focuses on the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during SFT. We propose four intriguing research questions to explore the association between model performance and various factors including data amount, composition ratio, model size and SFT strategies. Our experiments reveal that distinct capabilities scale differently and larger models generally show superior performance with same amount of data. Mathematical reasoning and code generation consistently improve with increasing data amount, whereas general abilities plateau after roughly a thousand samples. Moreover, we observe data composition appears to enhance various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful. Our findings also suggest the amount of composition data influences performance more than the composition ratio. In analysis of SFT strategies, we find that sequentially learning multiple skills risks catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT) strategy offers a promising solution to learn multiple abilities with different scaling patterns.

heading: 1 Introduction
text: Recent research has demonstrated the remarkable and versatile proficiency of large language models (LLMs) in dealing with a variety of real-world
* Work done during internships at Alibaba Group.
tasks expressed in natural languages (Ouyang et al., 2022a; Anil et al., 2023; OpenAI, 2023; Luo et al., 2023a), especially Information Extraction (IE) (Lu et al., 2022; Xu et al., 2023b; Zhao et al., 2023; Cheng et al., 2023d; Wang et al., 2023b; Zhang et al., 2024b; Li et al., 2024), Information Retrieval (IR) (Zhu et al., 2024; Liu et al., 2024c) and Spoken Language Understanding (SLU) (Hoscilowicz et al., 2024; Yin et al., 2024; Cheng et al., 2023a, 2024; Dong et al., 2023a). Among the tasks, LLMs especially emerge with three outstanding abilities in reasoning (Cobbe et al., 2021; Wei et al., 2022), coding (Chen et al., 2021), and aligning general human intentions (Ouyang et al., 2022a), which have drawn much attention from the LLM research community. In order to further incentivize such abilities, it necessitates supervised fine-tuning (SFT) stages on annotated task data.
However, existing research has mostly conducted separate SFT investigations on each of the three tasks, where reasoning and coding abilities require SFT on in-domain human-annotated or augmented data (Yuan et al., 2023b; Luo et al., 2023b; Yu et al., 2024) while diverse and complex human instructions are applauded for aligning human intentions (Wang et al., 2023d; Taori et al., 2023; Cheng et al., 2023c; Xu et al., 2023a; Zhou et al., 2023a; Wang et al., 2023a; Lu et al., 2023). As shown by the strong performance of proprietary LLMs such as GPT-4 (OpenAI, 2023) and Claude, LLMs have the potential to master all the tasks in one model. Therefore, it is of paramount importance to investigate the versatile performance of SFT with composite task data, and understanding and addressing the challenges posed by the data composition problem in the SFT stage is crucial for further enhancing the capabilities of LLMs in a comprehensive manner.
In essence, the tasks of reasoning, coding, and aligning human intentions are of different characteristics. Reasoning and coding tasks require ad-
177
hoc abilities of complex and detailed logic in decomposing task instructions and dealing with nonlinguistic and symbolic features (Chen et al., 2021; Huang and Chang, 2023), whereas aligning human intentions requires versatility and understanding obscure intentions expressed in human instructions (Lu et al., 2023). Given the fundamental difference among the tasks, multi-task learning with composite data fine-tuning for small-scaled pre-trained language models is prone to catastrophic forgetting (De Lange et al., 2022), hindering the fine-tuned performance of one model on separate tasks. Many efforts have been made to compensate for the phenomenon (Liang et al., 2021; Xu et al., 2021; Yuan et al., 2023a). There has also been research discovering that scaling up the pre-trained language model scale and the fine-tuning data scale are beneficial for zero-shot out-of-domain generalization on various linguistic tasks while leaving out the assessment of in-domain performance (Sanh et al., 2022; Chung et al., 2022a; Longpre et al., 2023). Given the increased capacity of LLMs, the multi-task performance by SFT on composite data of essentially different downstream tasks is less studied. Understanding the SFT performance with composite data and corresponding scaling patterns is of great utility in practice.
In this study, we focus on the data composition problem among mathematical reasoning, code generation, and general human-aligning abilities in SFT. We aim to comprehensively investigate the relationship between model performance and different factors including data amount, data composition ratio, model scales, and SFT training strategies. We also investigate how the relationship varies under different scales. Specifically, we focus on the following four research questions:
1. How do math reasoning, coding, and general abilities scale with SFT data amounts?
2. Are there performance conflicts when combining these three abilities in SFT?
3. What are the key factors that induce the performance conflicts?
4. What are the impacts of different SFT strategies for composite data?
To answer these questions, we conduct experiments on three benchmarks, which are GSM8K (Cobbe et al., 2021) for mathematical reasoning, HumanEval (Chen et al., 2021) for coding, and MT-Bench (Zheng et al., 2023) for general human alignment. We fine-tune LLMs on the related training data to activate these abilities. Furthermore, we
conduct extensive analysis regarding model parameter scales ranging from LLaMA 7B to 33B (Touvron et al., 2023) and explore four different SFT strategies shown in Figure 1: multi-task learning, sequential training, mixed sequential training, and dual-stage mixing fine-tuning (DMT), providing empirical guidance for learning a versatile LLM with composite SFT. The key findings of this paper can be summarized as follows:
• Different SFT abilities exhibit distinct scaling patterns, while larger models show better performances with the same data amount generally.
• Compared to single ability learning, multitask learning multiple abilities exhibits improvement in low-resource and decline in high-resource. Additionally, as the model size increases, there is a greater performance gain in low-resource settings for math and general abilities.
• Data amounts directly influence each ability, while the data ratio is insignificant.
• Multi-task learning lead to conflicts, while sequential training results in catastrophic forgetting. Our proposed DMT effectively alleviates both performance conflicts and catastrophic forgetting in the SFT phrase, achieving a balance between general and specialized abilities.

heading: 2 Related Works
text: Supervised Fine-Tuning of Large Language Models Large Language Models (LLMs) have shown notable zero-shot performance in various domains (Brown et al., 2020; Wu et al., 2021; Hou et al., 2024; Dong et al., 2023b; Zhou et al., 2024; Wu et al., 2023; Song et al., 2023), prompting further development to push the boundaries of these models. To delve deeper to their potential, LLMs are subjected to a Supervised Fine-Tuning (SFT) phase, enhancing their ability to solve tasks and align better with human instructions. Here, we extend the conventional definition of SFT to include various forms of sequence-to-sequence fine-tuning, such as fine-tuning for human alignment, instruction following, and domain-specific task optimization (Zhou et al., 2023b; Yuan et al., 2023c; Cheng et al., 2023b; Zhang et al., 2024a).
Recent research has delved into multi-task instruction fine-tuning of pre-trained LLMs to bolster their zero-shot performance across numerous downstream NLP tasks (Sanh et al., 2022). In an effort to encompass existing NLP tasks comprehensively, Chung et al.; Longpre et al. curated the expansive FLAN dataset specifically for instruction-based fine-tuning. LLMs, both open-source (Chung et al., 2022b) and proprietary (Singhal et al., 2022), finetuned with FLAN, have demonstrated enhanced zero-shot performance on a variety of unseen tasks.
While research has probed into the generalization capabilities of LLMs within out-of-distribution domains (Liu et al., 2024a; Yuan et al., 2024; Wang et al., 2024a), the effects of multi-task training on in-domain performance remain under-explored. With the ascent of proprietary models like ChatGPT, the focus on SFT for aligning LLMs with human intent has intensified (Ouyang et al., 2022b). Moving away from crowd-sourced SFT data, recent initiatives have generated SFT datasets from user logs within proprietary LLM platforms (Chiang et al., 2023; Wang et al., 2023a), employing the models themselves to assist in the data generation process (Wang et al., 2023d; Taori et al., 2023; Cheng et al., 2023d; Lei et al., 2023; Xu et al., 2023a; Xue et al., 2023). Additionally, methods to improve SFT data quality have been proposed, targeting more accurate alignment with human interactions (Zhou et al., 2023a; Wang et al., 2023c; Lu et al., 2023; Liu et al., 2024b).
Furthermore, SFT has proven beneficial for
LLMs in specialized areas such as mathematical reasoning (Cobbe et al., 2021; Hendrycks et al., 2021; Yuan et al., 2023b; Chen et al., 2024; Yue et al., 2023; Gou et al., 2024; Li et al., 2023; Yue et al., 2024) and code generation tasks (Chaudhary, 2023; Luo et al., 2023b; Wang et al., 2024b; Wei et al., 2023). Taking advantage of their advanced interactive capabilities, some researchers have leveraged supervised fine-tuned LLMs to compose commands that interface with external tools, thus enhancing the handling of assorted downstream applications (Shen et al., 2023; Yao et al., 2023b,a; Song et al., 2024; Fu et al., 2024). This paper examines the SFT performance using composite datasets, considering different model sizes and data amounts.
Scaling Laws in Large Language Models The exceptional performance of LLMs comes from scaling up model sizes, data amounts, and computational costs to massive scales. Therefore, it is crucial to explore the model performance across an exponential range of scales. Many endeavors have been made to discuss the scaling laws for pretraining (Anil et al., 2023; Hoffmann et al., 2022), transfer learning (Chronopoulou et al., 2019), preference modeling (Gao et al., 2022) and mathematical reasoning (Yuan et al., 2023b). In this paper, we also explore the SFT performance with composite data from the perspective of different scales of model sizes and data amounts.

heading: 3 Experiments
text: We have SFT datasets {D1, D2, ..., Dk} where each Di = {qi,j , ri,j}j contains queries and responses from one source. We consider each SFT dataset to correspond to one ability and we also have k in-domain metrics to measure them. We investigate the performances of in-domain metrics with different dataset compositions (D ⊂ ∪1≤i≤kDi) and training strategies on different sizes of LLMs.

heading: 3.1 Experiment Setup
text: We collect three SFT datasets {D1, D2, D3} including GSM8K RFT (Yuan et al., 2023b), Code Alpaca (Chaudhary, 2023), and ShareGPT (Chiang et al., 2023) to represent math reasoning, coding, and general human-aligning ability SFT dataset respectively. We will integrate a new SFT dataset D by these three datasets to investigate how data composition affects the model performances. We use GSM8K test set (Cobbe et al., 2021), HumanEval (Chen et al., 2021), and MT-Bench (Zheng et al., 2023) to measure abilities including math reasoning, coding, and general human-aligning. We use LLaMA (Touvron et al., 2023) series as our pretrained language models and use FastChat framework (Zheng et al., 2023) for fine-tuning. We finetune models with 3 epochs and a peak of 2e-5 learning rate. The batch size during SFT is 16. More details about SFT datasets, evaluation metrics, implementations and Training FLOPs can be found in Appendix A, B, C and D.

heading: 3.2 RQ1. Individual Ability Performance vs. Data Amount
text: The instruction following ability can be activated via SFT on datasets like ShareGPT which contain around 100 thousand samples. However, (Zhou et al., 2023a) demonstrates that strong base models can achieve human alignment with just 1000 samples. Specialized abilities such as math reasoning require a large amount of data (Cobbe et al., 2021; Yuan et al., 2023b), unlike general abilities. Therefore, it is crucial to investigate how each ability improves as the data amount increases.
Experimental Design: We conduct SFT on LLaMA of various sizes using {1, 1/4, 1/16, 1/64, 1/256} proportions of the training set obtained from GSM8K RFT, Code Alpaca, and ShareGPT seperately. This allowed us to evaluate each ability with various data sizes and model sizes.

heading: 1/256 1/64 1/16 1/4 1
text: Results and Analysis. Figure 2 shows the individual data scaling curves for different abilities after SFT. We find that: Different abilities exhibit different scaling curves. To be more specific, mathematical reasoning capability shows a positive correlation with the data amount across various model sizes which is consistent with (Yuan et al., 2023b). Similarly, general human-aligning ability demonstrates an almost monotonically increasing scaling curve. However, it is noteworthy that general ability emerges with only around 1k data samples (ranging from 1/256 to 1/64), and after reaching a certain threshold (1/64), their performances improve slowly. This further supports (Zhou et al., 2023a), indicating that a small amount of high-quality SFT data is possible for the emergence of general human-aligning ability in LLMs. On the other hand, code ability exhibits an irregular scaling curve when the model’s parameter count is small (7B & 13B). However, when the parameter count increases to 33B, its coding performance shows an approximately log-linear trend with the data amount. One possible explanation is that Code Alpaca and the samples in HumanEval have different distributions. Larger models can capture shared knowledge across code data distributions in the in-domain samples, which enables them to exhibit some level of generalization to out-of-distribution (OOD) samples. Another observation is larger models show better performances with the same data amount generally. The outlier is with very little data (1/256), smaller models may outperform larger models. If there is enough data, larger models have stable better performances.

heading: 3.3 RQ2. Performance Difference vs. Mixed Data Amount
text: We should deliver a versatile model that requires us to mix various SFT datasets and apply SFT. We want to ask how each ability varies due to SFT dataset mixtures. We investigate it with different amounts of mixed data and compare them with individual ability performance.
Experimental Design: For the individual source

heading: 1/256 1/64 1/16 1/4 1

heading: 1/256 1/64 1/16 1/4 1

heading: 1/256 1/64 1/16 1/4 1
text: setting, consistent with the setup in RQ1, we performed fine-tuning on LLaMA models of different sizes using {1, 1/4, 1/16, 1/64, 1/256} amounts of training data from GSM8K, Code Alpaca, and ShareGPT separately. For the mixed source setting, we sampled {1, 1/4, 1/16, 1/64, 1/256} amounts of training data from GSM8K, Code Alpaca, and ShareGPT, and directly mixed them according to the corresponding proportions. In this way, we constructed datasets with fixed proportions of different ability domains, while varying the total data amount. These datasets are then used for finetuning the LLaMA models 1.
Results and Analysis. Figure 3 presents results of LLaMA of different sizes on three benchmarks under the individual source and mixed source settings. The following observations are made: Abilities are improved with low-resource and are decreased with high-resource compared to individual source abilities. In the case of LLaMA-7B, compared to the data scaling curve of the individual source setting, the models fine-tuned with mixed source data consistently demonstrated performance conflicts among the three ability domains at high resources (100%). However, as the data volume decreased, a turning point in performance is observed between the two settings in the data range of 1/64 to 1/16. Notably, the models fine-tuned with mixed source data exhibited performance gains at low resources (1/256), indicating that SFT data from different sources benefit each other in a low-resource setting. However, when there is enough data, data from other sources could be viewed as noise for
1We also conduct "Equal Data Amount VS. Equal Data Proportion" experiments in Appendix H

heading: 1/256 1/64 1/16 1/4 1
text: Math Scaling with different data ratio
Code Scaling with different data ratio
General Scaling with different data ratio

heading: 1/256 1/64 1/16 1/4 1
text: in-domain generalization. As the model size increases, the performance gain in low-resource settings also increases for math and general abilities. In the case of the 13B and 33B models, it is obvious that the scaling curve for the mix source setting follows a similar trend observed in previous analyses, with the presence of performance intersection points as the data volume scales. However, a crucial distinction arises, whereby larger models exhibit more pronounced performance gains under low resources as the size of model parameters increases. The outlier is the LLaMA-7B (code only, 1/256). A possible reason is the introduction of a small amount of unseen code data easily disrupts the original code ability of the pretrained model, as supported by its low HumanEval score (less than 6). In conclusion, our finding implies that larger language models excel in acquiring general and specialized abilities from diverse data sources under low-resource conditions 2.

heading: 3.4 RQ3. Performance Difference vs. Data Composition Ratio
text: We observe ability conflicts in high-resource settings, and we want to investigate the reasons why the conflicts occur. Two possible factors are the data amount of other abilities is too high or the data ratio of other abilities is too high. Here we conduct experiments to investigate the data ratio factor.
Experimental Design: We consider coding and mathematics as a combined specialized data source, and the ShareGPT as the general data source. We designed three setups as follows which control the amount of one source of data and vary the ratio between general and specialized data.
1. Fixed general data, scaling specialized data: 2To validate the generalizability of our conclusions, we further conduct the more experiments on World Knowledge, Language Understanding and Translation in Appendix E.
We use a full training set of ShareGPT and sampled different proportions {1, 1/4, 1/16, 1/64, 1/256} of GSM8K RFT and Code Alpaca as a mixture.
2. Fixed specialized data, scaling general data: We use a full training set of GSM8K RFT and Code Alpaca and sample different proportions of ShareGPT as a mixture.
3. Fixed 1/64 general data, scaling specialized data: Motivated by LIMA’s setup (Zhou et al., 2023a), we used a 1/64 ShareGPT set (about 1500 examples) and sampled different proportions of GSM8K RFT and Code Alpaca as a mixture.
Results and Analysis. Q1: Does the performance of the model vary with different ratios of general and specialized data? As illustrated in the top three graphs of Figure 4, we conduct ablation studies of the data ratio (k) between specialized and general abilities. To be noticed ratio is normalized by data amount, for example, k = 1 means specialized use data amountgeneral use data amount = specialized all data amount general all data amount . We utilize a fixed specialized data setting (directly mixing 100% code & math data for training) and a fixed general data setting (100% general data for training) as the baseline and observe:
(1) With the increase in the ratio of general data from 1/256 to 1/1, Fixed specialized data, scaling general data setup exhibits similar performance to the setup that Fixed specialized abilities in terms of math reasoning. This suggests that variations in the data ratio k have minimal impact on math ability. We consider the reason that math and general abilities are non-conflict since they are too different in the semantic space. However, when considering HumanEval, the Fixed specialized data, scaling general data setup displays noticeable fluctuations compared to the baseline. We attribute this to the inclusion of a certain proportion of code data in ShareGPT. Due to the differences in data format and distribution, the presence of similar data features exacerbates the performance conflicts between abilities when the data ratio k increases. Further analysis of the distribution of different abilities is discussed in Section 4.1.
(2) With the increase in the ratio of specialized data from 1/256 to 1/1, the setup that Fixed general data, scaling specialized data displayed no significant performance changes compared to the baseline. This echoes our hypothesis that when there are significant differences in task formats and data distributions between different SFT
abilities, the impact of data ratio is minimal. However, when there is some degree of similarities, the data ratio can lead to noticeable performance fluctuations.
Q2: Under extremely limited general data resources, does the ratio of specialized data have an impact on the model’s performance? We further explore the impact of different ratios of specialized data when the model has just acquired a certain level of general human-aligning ability (k = 1/64). The bottom 3 graphs of Figure 4 present comparative experiments between two settings. We observe that regardless of whether the data amount for general capabilities is abundant (k = 1) or scarce (k = 1/64), the performance on MT-Bench shows no significant fluctuations with varying proportions of specialized data. Furthermore, in mathematical reasoning, 1/64 general data setup exhibited a scaling trend that is almost identical to the full general data setup. However, for coding ability, with the same amount of code data and different ratios, code abilities are different in the two settings. We still consider the reason is code data are partly related to ShareGPT data and cause the performance difference and provide an analysis in Discussion 4.2.

heading: 3.5 RQ4. Performance Difference vs. Training Strategies
text: We could feed these SFT datasets into models with different training strategies. In this section, We experiment with these settings and investigate how they influence each ability’s performance.
Experimental Design: Firstly, we introduce three kinds of naive training strategies as follows:
1. Multi-task learning: We directly mix different SFT data sources D = ∪1≤i≤kDi and applying SFT. If we view each data source as a different task, this can be viewed as multi-task learning.
2. Sequential Training: We sequentially apply SFT on each dataset. Specifically, we sequentially trained on coding, math reasoning, and the general ability dataset. Since the general ability is the most important one for human alignment, we put ShareGPT as our last dataset.
3. Mixed Sequential Training: We apply multitask learning on specialized datasets(code, math) first and apply SFT on the general ability dataset. These three approaches are presented in Figure 1.
Results and Analysis: Table 1 presents performances under different training strategies in terms
of mathematical reasoning, code generation, and general human-aligning ability. Multi-task learning preserves specialized abilities among these strategies while hurting the general ability most among them. Sequential training and mixed sequential training preserve general ability while losing too many specialized abilities. The observed outcome is in accordance with expectations, as during the final fine-tuning phase, the mixed sequential training strategy remains unaffected by specialized data, thereby effectively preserving its generalization capability. However, an inherent drawback of multistage training is the occurrence of catastrophic forgetting of prior knowledge, which motivates us to further explore methods that can alleviate catastrophic forgetting of specialized abilities while maximizing the preservation of general capability.
4. Dual-stage Mixed Fine-tuning (DMT): Based on our observation from RQ1 to RQ4, we propose a new training strategy that can reduce the ability conflict during multi-task learning and relieve the issue of catastrophic forgetting during sequential training. From RQ1, the model needs large data amounts to activate specialized abilities. From RQ2, multi-task learning with all amounts of specialized data and general data will hurt each ability. From RQ3, a small amount of specialized data will not affect the general ability performance. From RQ4, (mixed) sequential training forgets specialized abilities. So the model needs to learn large amounts of specialized data and should not forget them during learning general ability. A natural choice is to learn full amounts of specialized data first and add a small amount of specialized data to general data during the last stage of sequential training to prevent forgetting. As shown in Figure 1, we first apply SFT on the specialized dataset which is same as the first stage of the mixed sequential training strategy. For the second stage, we perform SFT with a mixed data source comprising a combination of the general data and varying proportions k (1, 1/2, 1/4, 1/8, 1/16, 1/32) of code and math data. Adding code and math data in the second stage helps models to recall the specialized ability. The results of DMT (k = 1/256) are presented in Table 1 and the detailed scaling analysis of proportion k can be found in the discussion.
Model Accuracy vs. DMT Strategies. In Table 1, LLaMA-7B with DMT (k = 1/256) strategy perform significant improvement in mathematical reasoning (32.6 to 41.92) and code generation (15.24 to 17.68) compared to the mixed sequential training
strategy, which indicates a significant alleviating effect of mixing specialized capability data in the last fine-tuning stage on catastrophic forgetting. Surprisingly, DMT (k = 1/256) even exhibits a slight improvement on MT-Bench, further highlighting its ability to alleviate catastrophic forgetting while effectively preserving general capability.
Regarding the 13B and 33B models, DMT (k = 1/256) demonstrates noticeable alleviation of catastrophic forgetting in mathematical reasoning (13B: 40.48 to 46.47 / 33B: 44.24 to 56.36) and code generation (13B: 18.3 to 19.5 / 33B: 24.4 to 25.5) compared to the mixed sequential training strategy. Additionally, it significantly retains its general capability (13B: 5.93 to 6.03 / 33B 6.43 to 6.69). Therefore, these results serve as additional validation of the efficacy of DMT in mitigating catastrophic forgetting while maintaining general capability 3.

heading: 4 Discussion

heading: 4.1 Visualization of Different SFT Abilities
text: In the aforementioned analysis of data composition, we observed a significant performance degradation when different data sources are directly mixed. In this section, our aim is to explore the potential mutual influence of semantic representation distributions among different data sources. Specifically, we randomly sampled 100 queries from CodeAlpaca, GSM8k RFT, and ShareGPT datasets and extracted the hidden layer representations located in the Middle layer (15th) of the model. Subsequently, we employed the t-SNE toolkit (Van der Maaten and Hinton, 2008) to visualize the representations of the three types of capabilities. The results in Figure 5 illustrate a notable collapse phenomenon in the semantic representations of both the original LLaMA-13b and LLaMA-13b with DMT (k=1/256). While both models exhibit a certain level of separation in the mathematical data representations, there remains a certain degree of overlap between the representations of code and general samples. In Appendix G, we further discuss the visualization of semantic spaces at different layers of LLaMA 7B & 13B.
3To verify the effectiveness of DMT strategy on relatively OOD benchmarks, we further evaluate it on MBPP and MATH in Appendix F.

heading: 4.2 Ablation of the Specialized Domains in ShareGPT
text: In RQ2, we observe using mixed data sources resulted in improved abilities under low-resource conditions but diminished abilities under high-resource conditions when compared to single data sources. However, the presence of coding and mathematical samples within the ShareGPT introduces uncertainty regarding whether the performance gain under low resources is solely attributed to these specific coding & mathematical data or other orthogonal samples in the general dataset (e.g., translation or extraction). Hence, the objective of this section is to investigate whether the conclusions drawn in Section 3.3 remain valid after removing the code and math samples within ShareGPT.
Experimental Design: We employed an openset tagger InsTag (Lu et al., 2023) to annotate samples in ShareGPT. To filter out data related to coding and mathematical abilities, we conduct regular expression matching to eliminate instances where the tags contain keywords “code” or “math”. Finally, we obtain a ShareGPT dataset devoid of any code or math-related information (reducing from 86K to 63K). In alignment with the settings in Section 3.3, we sampled different proportions of training data (1, 1/4, 1/16, 1/64, 1/256) from GSM8K,
Code Alpaca, and the modified ShareGPT dataset (without code math). These samples were directly mixed according to the corresponding proportions. Subsequently, the LLaMA models were fine-tuned by using this mixed dataset.
Results and Analysis. Figure 6 shows the results of our experiment. Removing the code and math from ShareGPT not only mitigates the performance conflicts among different abilities to some extent under high-resource conditions but also maintains stable gains in low-resource settings. We propose that the potential reason behind these findings lies in the differences in the distribution of code and math data between ShareGPT, CodeAlpaca, and GSM8K RFT datasets. This distribution gap introduces an extra noise during the SFT phrase, while its removal enables the model to better generalize coding and mathematical abilities. Furthermore, in low-resource scenarios, this phenomenon indicates that the code and math samples in ShareGPT are not the key factor contributing to performance improvements, but rather the diversity and variability of the data (Longpre et al., 2023). In summary, the presence of code math data within ShareGPT does not emerge as a key factor impacting the performance gains identified in Section 3.3, highlighting the generalization of our conclusions.

heading: 1/256 1/64 1/16 1/4 1

heading: 4.3 Specialized Data Amount in DMT
text: We investigate how different values of k influence model performance and results shown in Figure 5. When we adjust k from 0 to 1/256 (k = 0 is equal to mixed sequential training), the SFT models show significant improvements in both specialized ability and general human-aligning ability. On the contrary, as k increased from 1/4 to 1, the model exhibited a decline in general ability. We believe this is in line with the findings in RQ2, which concluded that high-resource settings lead to conflicts while low-resource settings lead to gains in mixed sources. Furthermore, as k increased from 1/256 to 1/4, we observe a linear inverse trend between general ability and specialized ability, especially an increase in general ability coincided with a decrease in specialized ability. This suggests k needs to be tuned based on specific requirements in order to achieve a balance between multiple abilities.

heading: 5 Conclusion
text: We explore the data composition in the SFT phase, focusing on mathematical reasoning, code generation, and general human-aligning abilities. We formulate four research questions to guide our investigation and analyze the scaling trends between different abilities and factors (e.g. data amount, data ratio, model parameters, and training strategies). Our findings reveal distinct scaling patterns among different abilities, with larger models demonstrating superior performance when trained with the same amount of data. Moreover, mixing data sources in the SFT phase improves performance in low-resource scenarios but diminishes in highresource scenarios. Interestingly, the phenomenon of low-resource gain becomes more prominent as the model parameter size increases. Furthermore, our observations indicate that data amount directly influences performance conflicts, whereas the impact of data ratio is insignificant within our experimental setup. Finally, regarding the SFT strategies, we demonstrate our proposed DMT strategy effectively alleviates performance conflicts, offering a
promising solution to activate multiple abilities.
Limitations
Due to our use of the large language model LLaMA-33B, the extensive computational resources and time required for both training and inference may limit its applicability. The datasets used in this article are all open source, so there are no ethical or moral issues; However, inappropriate prompts and noisy training corpora can potentially lead to privacy and bias issues with LLMs. Furthermore, the evaluation benchmark MT-Bench relies on GPT-4 for scoring, which may result in some variability in the results, and these may not always align perfectly with human judgment standards. In this paper, we primarily focus on three SFT capabilities that are of great interest in the LLMs community, including mathematical reasoning, code generation, and general human-aligned ability. To verify the generality of our conclusions, we further explore three additional SFT capabilities in the appendix. Nevertheless, there are still many other SFT capabilities (such as creative generation) within the LLMs community that have data composition issues waiting to be explored by researchers, which will also be the focus of our future research efforts.

heading: A SFT Datasets
text: We investigate the data composition issues of mathematical reasoning, coding, and general capabilities in the SFT stage from the following SFT datasets.
• Code Alpaca (Chaudhary, 2023) aims to build and share an instruction-following LLaMA model for code generation. which is fully based on Stanford Alpaca and contains 20K data used for fine-tuning the model. The Code Alpaca dataset has been open-sourced4.
• GSM8K RFT (Yuan et al., 2023b) is a mathematical dataset enhanced by integrating multiple reasoning paths based on the original GSM8K dataset (Cobbe et al., 2021) through the rejection sampling. It contains 7.5K questions and 110K responses in the training set. The GSM8k RFT dataset has been opensourced 5.
• ShareGPT refers to the multi-turn chatting histories used by Vicuna (Chiang et al., 2023). ShareGPT includes 86K human queries and responses from ChatGPT and other chatbots. The GSM8k RFT dataset has been opensourced 6.
The following table 2 presents the statistics of three datasets at different subset proportion (k).

heading: B Evaluation metrics
text: We use the following metrics to measure the aligned large language models.
4https://github.com/sahil280114/codealpaca 5https://github.com/OFA-Sys/gsm8k-ScRel 6Exact dataset of ShareGPT (https://sharegpt.com/) has not been released. We instead use a reproduced version from https://huggingface.co/datasets/ anon8231489123/ShareGPT_Vicuna_unfiltered cleaned raw dataset, and follow Vicuna preprocess.
• HumanEval (Chen et al., 2021) consists of 164 original programming problems, with an average of 9.6 test cases allocated to each problem. To ensure a thorough assessment of the functional correctness of LLMsynthesized code, HumanEval+ extends the number of test cases significantly, averaging at 774.8 test cases per problem. We use the same method as (Chen et al., 2021)to obtain unbiased estimates of Pass@k under greedy decoding. To facilitate the reproducibility of our results, we use the open-source github repository BigCode (Ben Allal et al., 2022) to evaluate all the HumanEval scores in this paper 7.
• GSM8K (Cobbe et al., 2021) is a math word problem dataset used to measure large language model math reasoning ability. We use the default test set to measure the model. We calculate the score based on greedy decoding accuracy (maj@1). In this paper, we use the open-source github repository gsm8k-ScRel 8
to evaluate all the GSM8k scores.
• MT-Bench (Zheng et al., 2023) is a significant benchmark that contribute to the evaluation and advancement of chatbot models and LLMs in different contexts. MT-Bench9
evaluates LLMs on multi-turn dialogues using comprehensive questions tailored to handling conversations. It provides a comprehensive set of questions specifically designed for assessing the capabilities of models in handling multi-turn dialogues.
We also supplement more benchmark evaluation results in the appendix F to verify the generalization of our conclusions:
• MATH (Hendrycks et al., 2021) is a dataset with challenging high-school math problems. Problems are classified into the following topics: Prealgebra, Algebra, Number Theory, Counting and Probability, Geometry, Intermediate Algebra, and Precalculus. Problems in MATH are harder and more diverse than in GSM8K. In this paper, we use the open-source github repository gsm8k-ScRel to evaluate all
7https://github.com/bigcode-project/ bigcode-evaluation-harness
8https://github.com/OFA-Sys/gsm8k-ScRel 9https://huggingface.co/spaces/lmsys/mt-bench
the MATH scores. We use 500 test problems from (Lightman et al., 2023) as out-of-domain math benchmark.
• MBPP (Austin et al., 2021) consists of around 1,000 crowd-sourced Python programming problems, designed to be solvable by entrylevel programmers, covering programming fundamentals, standard library functionality, and so on. Each problem consists of a task description, code solution and 3 automated test cases. To facilitate the reproducibility of our results, we use the open-source github repository BigCode (Ben Allal et al., 2022) to evaluate all the MBPP scores in this paper.
C Implementation Details
We fine-tune all the SFT datasets with 3 epochs and a batch size of 128 on NVIDIA A100 GPUs. We use 8 GPUs for 7B and 13B models, 16 GPUs for 33B models during fine-tuning. We use a peak learning rate of 2e-5 with a 3% learning rate warmup. We evaluate the results on the final epoch. We use greedy decode to calculate Pass@1 and maj@1. Since the scores of MT-bench will fluctuate, we conducted three experiments and took the average.
All experiments are conducted using the default template of the FastChat framework (Zheng et al., 2023), as shown in the figure below:
Prompt Template
A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions. USER: {Query} ASSISTANT:
To facilitate the replication of our results, all datasets and evaluation benchmarks used in our experiments have been open-sourced and their detailed sources are indicated. We will also opensource our code after the blind review process.

heading: D Estimating FLOPs of SFT
text: Training FLOPs. We mainly follow the notations of (Kaplan et al., 2020) here.
For each input sample of length nctx in SFT dataset (GSM8K, CodeAlpaca, ShareGPT), we can split it into two parts:
nctx = nQ + nR (1)
Ctrain ≈ 6NnctxNs (2)
where nQ, nR denotes the length of question and generated answers respectively. N ,Ns denotes the non-embedding parameters and the numbers of samples.
Therefore, We estimate the SFT FLOPs following (Kaplan et al., 2020) and GPU times in Table 3.
E Validation Experiments in More SFT Abilities
To validate the generalization of our conclusions, we selected representative datasets to evaluate the capabilities of large models across different dimensions. These dimensions include World Knowledge : WebQuestionsSP (Yih et al., 2016), Language Understanding: CoNLL 2003 (Tjong Kim Sang and De Meulder, 2003), and Translation: IWSLT14 (Cettolo et al., 2014)
Experimental Design: Align the settings of RQ1 and RQ2, we introduce two settings as follows:
1. Individual Domain: We conduct SFT on LLaMA of various sizes using {1, 1/2, 1/4, 1/8} proportions 10 of the training set obtained from WebQSP, CoNLL 2003, and IWSLT14 seperately. This allowed us to evaluate each ability with various data sizes and model sizes.
2. Mixed Domain: We sampled {1, 1/2, 1/4, 1/8} amounts of training data from WebQSP, CoNLL 2003, and IWSLT14, and directly mixed them according to the corresponding proportions. In this way, we constructed datasets with fixed proportions of different ability domains, while varying the total data amount. These datasets are then used for fine-tuning the LLaMA models.
Analysis. As shown in Figure 7 and Table 4, we have following observations.
For the individual domain, the performance (P, R, F1) of the model in the language understanding (NER) task shows a positive correlation with the scaling curve of data volume. These two abilities exhibit similar scaling curve trends as the mathematical ability performance in RQ1. In the case of world knowledge (WebQSP), a similar positive correlation trend is observed in terms of F1 and Hits@1. However, when the data ratio is reduced from 1/4 to 1/8, there is a significant performance fluctuation, particularly in the performance of translation ability, which shows a relatively irregular trend. These conclusions further support the core conclusion of RQ1 that different data exhibit different scaling curves.
For the mixed domains, the findings align with the conclusions in RQ2, where abilities are improved with low-resource and decreased with highresource compared to individual source abilities. This consistent conclusion holds for world knowledge, language understanding, and translation abilities.
10Because these three datasets have relatively small amounts of data (a few thousand), the scaling range is from 1/1 of the data volume to 1/8 of the data volume.

heading: 1/256 1/64 1/16 1/4 1

heading: F Results on OOD Benchmarks in Math and Code
text: To validate the generalization of our findings on other benchmarks, we utilized GSM8K and Code Alpaca as the training sets. We further evaluated the results on the individual domain, mixed domain, and different training strategies on other specialized ability benchmark, including MATH and MBPP, which is illustrated in Table 5 and Figure 8. We have the following findings:
(1) In the individual domain, LLaMA shows a positive correlation between performance in MATH and MBPP and the data volume (consistent with RQ1).
(2) Comparing the individual and mixed domains, LLaMA-7B exhibits a trade-off between high-resource performance conflict and lowresource performance gain in both MATH and MBPP (consistent with RQ2).
(3) Considering the general ability results shown in Table 1, we can observe that DMT maintains competitive results in MATH and MBPP while prioritizing general abilities. This further validates the effectiveness of DMT (consistent with RQ4).
G Visualization of Different Layers
In this section, we compared the visualization results of the baseline model of LLaMA-13B and DMT (k=1/256) in the starting layer (Layer1), mid-
dle layer (Layer15), and ending layer (Layer31) in Figure 9 and 10.
The visualization result of the starting layer are relatively chaotic, while the visualization results of the middle layer and the ending layer are clearer. And the results of the middle layer and the last layer are consistent in pointing out that both base model and model with DMT strategy exhibit a certain level of separation in the mathematical data representations, there remains a certain degree of overlap between the representations of code and general samples.

heading: H Equal Data Amount VS. Equal Data Proportion
text: In a realistic SFT phrase for training general LLM, the data amount for different abilities is likely to differ. Therefore, instead of controlling the same amount of data, we select to mix datasets with the same proportion of subsets to better simulate realworld scenarios in above experiments. In addition, We further supplement the experimental results using different abilities mixed with the equal data amount and compare them with the results using the equal subset proportion in Table 6.
Equal Data amount Setting: we utilize the data amount of GSM8k RFT as the baseline. We sampled data with proportions of 1/16, 1/64, 1/256, and mixed samples of the same data amount from Code alpaca and ShareGPT.
Equal Proportion Setting: we sampled data with proportions of 1/16, 1/64, 1/256 according to the subset proportions of each dataset and mixed them, which is aligned with the setup in RQ2.
It can be observed that there is not a significant difference in the results of the three benchmark tests between the two settings. Therefore, these findings do not significantly impact the main experimental conclusions presented in the paper.

heading: I Comparison Experiment of Different Training Sequences
text: To investigate the impact of training order on different SFT abilities, we have conducted additional experiments with six different training orders. The results and analysis of these experiments are provided in Table 7. Based on our findings, we conclude the following:
1. The SFT ability trained in the final stage tend to retain relatively good performance.
2. If general and code abilities are trained in the first two stages, there is a noticeable performance decrease in code capability, while math capability does not show significant impact. One possible reason is that the task format of code generation
and general ability exhibits similar data distributions (as discussed in RQ3 and Discussion1). This can result in a more severe catastrophic forgetting phenomenon during continuous fine-tuning.

heading: J Detailed Results of experiments
text: J.1 Results of Different Random Seeds
For each dataset, we employed random selection by utilizing a random function with three distinct seeds for sampling. Subsequently, we conducted a comparative analysis of the results obtained from different subsets on the three benchmark tests. The specific details are presented in Table 8. It can be observed that DMT maintains its superiority under three different random seed settings. The influence
of different subsets on experimental results is not a key factor and does not affect the overall trend.
J.2 Results of Single Source and Mixed Source
In Table 9 and Table 10, we report the detailed comparative results between mix domains and individual domains for LLaMA-7B, 3B and 33B, as the supplemental results in RQ2.
J.3 Results of Data Ratio (k) In Table 11, we report The detailed results of the data ratio (k) between specific abilities and general abilities on three benchmarks, as the supplemental results in RQ3.
J.4 Results of Specialized Data Amount of DMT
In Table 12, we report The detailed results of LLaMA-7B, 13B, 33B with different training strategies on three benchmarks, as the supplemental results in RQ4.
J.5 Results of MT-Bench In Figure 11, we report detailed results of LLaMA7B, 13B, 33B with different training strategies on MT-Bench, which include coding, extraction, humanities, math, reasoning, roleplay, stem and writing abilities.
J.6 Supplemental Results for Discussion In Figure 12, we report the t-SNE visualizations of LLaMA-7B and LLaMA-7B with DMT(k=1/256) strategy. What’s more, the bottom figure represents the scaling relationship of LLaMA-7B with DMT(k=1/256) under different values of K.
Moreover, in Table 13, we report The detailed results of LLaMA-7B, 13B, 33B with different training strategies on three benchmarks, as the supplemental results in RQ4.

heading: Future Work
text: Nevertheless, there are still many other SFT capabilities (such as creative generation) within the LLMs community that have data composition issues waiting to be explored by researchers, which will also be the focus of our future research efforts