Demystifying Instruction Mixing for Fine-tuning Large Language Models (2024)

Abstract: Instruction tuning significantly enhances the performance of large language models (LLMs) across various tasks. However, the procedure to optimizing the mixing of instruction datasets for LLM fine-tuning is still poorly understood. This study categorizes instructions into three primary types: NLP downstream tasks, coding, and general chat. We explore the effects of instruction tuning on different combinations of datasets on LLM performance, and find that certain instruction types are more advantageous for specific applications but can negatively impact other areas. This work provides insights into instruction mixtures, laying the foundations for future research.1

heading: 1 Introduction
text: Instruction tuning has been shown to have surprising efficacy for aligning large language models (LLMs) with human instructions (Chung et al., 2022; Li et al., 2023; Wu et al., 2023; Xu et al., 2023; Touvron et al., 2023; Muennighoff et al., 2023a; Gunasekar et al., 2023). Recent studies highlight the diverse ways in which instructions can enhance the different capabilities of LLMs. For instance, using general-purpose, chat-like instructions can improve the performance of LLMs as chat assistants (Chiang et al., 2023; Ouyang et al., 2022; Taori et al., 2023; Ding et al., 2023), while training LLMs on instructions based off NLP tasks improves their performance on NLP benchmarks (Sanh et al., 2022; Chung et al., 2022; Muennighoff et al., 2023b), and incorporating coding instructions enhances LLM code generation (Fu and Khot, 2022; Gunasekar et al., 2023). However, a key unresolved issue is determining how to combine various instruction datasets to optimize overall LLM performance.
1Code and data are available at: https://github.com/ Reason-Wang/InstructLLM.

heading: P3 Alpaca
text: In this paper, we aim to better understand the impact of instruction mixing across three critical areas: NLP downstream tasks, coding, and chat. The core of our investigation revolves around understanding the influence of instruction dataset distributions on model performance in these different areas. We first select representative instruction datasets: P3 (Sanh et al., 2022) for NLP downstream tasks, CodeAlpaca (Chaudhary, 2023) for code generation, and Alpaca (Taori et al., 2023) for general-purpose instructions. As shown in Figure 1, P3 is focused primarily on five tasks (including QA and classification), whereas Alpaca contains a vast array of instructions. Using a dependency parser, we identify over 1K unique root verbs from Alpaca’s instructions, with generate, create, and describe being the most frequent. CodeAlpaca, by contrast, is exclusively focused on coding tasks, and exhibits less variation compared to the others as exemplified in Table 3. We fine-tune models across all eight potential combinations of these instruction datasets, and carry out detailed evaluation of model performance in terms of NLP downstream tasks, coding proficiency, and chat capabilities.
Our main contribution in this work is to shed light on instruction mixing when fine-tuning LLMs through comprehensive experimentation. Our findings can be summarized as follows:
162
• Specific instruction datasets enhance LLM performance in their respective task areas. However, combining all instruction types does not uniformly improve performance across all tasks.
• Instructions reformulated from NLP downstream tasks (such as P3) can negatively impact the model’s conversational abilities. In contrast, instructions focused on coding not only improve coding proficiency but also enhance chat capabilities.
• Larger models, with their increased capacity, are able to make more effective use of a diverse range of instructions.

heading: 2 Related Work
text: Recent work has demonstrated that vanilla LLMs can follow general instructions if tuned with instructions and corresponding responses (Mishra et al., 2022; Sanh et al., 2022; Wang et al., 2022). For instance, Sanh et al. (2022) crafted an instructional dataset by reformulating supervised datasets with various prompts to create P3. However, despite their effectiveness in NLP tasks, these LLMs often diverge from human-like interactions in chatbot applications.
To facilitate general-purpose LLM fine-tuning, researchers has create general-purpose instructional data by human annotation (Conover et al., 2023) and automatic approaches (Wang et al., 2023b; Taori et al., 2023). Recent work has further expanded the dataset size (Wu et al., 2023), language coverage (Li et al., 2023), and task types (Chaudhary, 2023; Yue et al., 2023).
With increasing capabilities of LLMs and availability of instruction datasets, researchers have aimed to imbue a single model with diverse capabilities. Sengupta et al. (2023) attempted to blend different instruction datasets without considering the data volume and task types. Longpre et al. (2023) suggested that increasing the number of tasks and instruction diversity can enhance performance. In contrast, Anand et al. (2023) excluded P3 from their fine-tuning dataset, seemingly to enhance alignment. Nevertheless, none of these papers systematically studied the impact of the instruction mixture on the resulting LLM.
Concurrent to our work, Wang et al. (2023a) fine-tuned LLaMA models on 12 instruction-tuning datasets separately. By evaluating those model on 7
tasks, they found that different datasets can enhance model performance on individual tasks. They further identified the optimal dataset combination, and trained a single model to achieve the best overall performance. Novel to this work, we classify the instructions and model skills into three types, and conduct a deep analysis of the influence of data mixture on the models.

heading: 3 Experimental Setup
text: Datasets We select Alpaca (Taori et al., 2023) as the general instruction dataset to align models, in the form of 52K instruction–response pairs. We use P3 (Sanh et al., 2022) as our NLP task instruction dataset, which is reformatted for a wide range of NLP downstream tasks using diverse humanwritten templates. Since the number of samples in each task varies vastly, we randomly sample 1K instances from each subtask formatted with several corresponding prompts for diversity, resulting in 660K samples. For coding data, we choose CodeAlpaca (Chaudhary, 2023), which is an instruction dataset focusing on code generation. It contains 20K samples in different programming languages. To ensure a balanced comparison, we randomly sample a 20K subset from each dataset. Examples are provided in Table 3 in the Appendix.
Evaluation We divide the evaluation into three parts: NLP benchmark performance, code generation, and alignment evaluation (i.e., chat ability evaluation). For NLP benchmarks, we use ARC (Clark et al., 2018), Winogrande (Sakaguchi et al., 2021), PIQA (Bisk et al., 2020), MMLU (Hendrycks et al., 2020), RACE (Lai et al., 2017), and HellaSwag (Zellers et al., 2019). For coding, we use HumanEval (Chen et al., 2021), which tests the pass rate of the generate codes. For alignment evaluation, we use the FLASK (Ye et al., 2023) framework to score model alignment. We keep the eight most frequent alignment skills from the original evaluation set, resulting in 1,180 samples. Then we employ GPT-4 to assess model responses to each instruction sample based on human-written principles. See Appendix B for details of these skills.
Models We fine-tune LLaMA-2 7B and 13B (Touvron et al., 2023) models for two epochs in a generative way as in Radford et al. (2018), using a linear scheduler with a 3% warmup rate and a batch size of 64. The maximum learning rate is
5 × 10−5. The resources for training and evaluations are detailed in Appendix C.

heading: 4 Results
text: For the remainder of this paper, we denote the Alpaca, CodeAlpaca, and P3 datasets as A, C, P, respectively. For each model, we compare eight different data mixing strategies, denoted as None, A, C, P, AC, AP, CP, ACP, where None represents the vanilla model without fine-tuning, and each of the other settings represents the model fine-tuned with the corresponding dataset. For example, AC means the model is fine-tuned with both Alpaca and CodeAlpaca.

heading: 4.1 NLP Tasks and Code Benchmark Results
text: Table 1 shows the zero-shot results on the NLP and code generation benchmarks. Predictably each specialized instruction dataset improves the performance on the benchmarks they are designed for. In the no-mixture setting (comparing A, C, and P), models fine-tuned on P3 achieve the highest average score for NLP tasks, while models fine-tuned on CodeAlpaca excel in code generation benchmarks. Examining specific tasks reveals that a model’s performance on a specific task heavily relies on the similarity between the target task and the tasks it was fine-tuned on. For instance, Alpaca fine-tuned models excel in Race and HellaSwag, which involve the story completion task, similar to the Alpaca instruction format. On the other hand, P3 fine-tuned models perform well on ARC and Winogrande, which involve multiple-choice QA and cloze tests, which are well represented in P3.
In the mixture setting, it’s evident that including specialized data consistently boosts model performance in corresponding benchmarks compared to models without such data. For example, P, PA, PC, and PCA perform better than None, A, C, and CA on NLP downstream tasks. Focusing on the code benchmarks, incorporating general instructions consistently improves coding performance. For the 7B model, AC improves performance by +1.28 and +0.61 compared to C, while the improvements are −0.80 (outlier) and +3.05 for the 13B models. Another interesting finding is that the 13B models perform best with the ACP mixture, while the 7B models perform best with AC. This
suggests that larger models can better learn from varied instructions more effectively than smaller models.
These findings highlight the importance of considering model size and target usage when designing the instruction mixture.
Mixing with Different Ratios While it is clear that mixing specialized instructions is vital for benchmark performance, how the mixing ratio correlates with the performance is also important. As Figure 2 shows, with the number of general instructions fixed to 20K, scores in both NLP task and code benchmarks first decrease and then increase as the ratio of specialized instructions increases. They both peak when the ratio is 1.5, and drop back slightly when the ratio is increased further to 2.0. We hypothesize that this is because the model overfits to the specialized instructions when there are too many such instructions.
Number of instances Figure 2 also shows the performance change with respect to the number of fine-tuning data instances. We mix each type of instruction with the same number. We find that the performance over both benchmarks plateaus when the number of instances is larger than 10K.

heading: 4.2 Alignment Skill Results
text: Table 2 shows the alignment skills results. We adopt the same setup as FLASK, using GPT-4-0613 to access the alignment skills and scaling the scores to the range [0, 100].
From Table 2 we make the following observations: (1) All three types of instructions improve
model alignment compared to the vanilla LLM. Among these instructions, Alpaca stands out as the most effective. It contains general-purpose instructions and human-like responses, making it a better fit for aligning models with humans. (2) While CodeAlpaca alone doesn’t notably enhance alignment abilities, combining it with general instructions results in a substantial improvement of +0.6 (7B) and +1.6 (13B) points; these improvements are mainly due to better compression, commonsense understanding, completeness, and conciseness. (3) Mixing P3 data causes a drop of −2.8 (7B) and −3.6 (13B) in alignment skills, suggesting that P3 has a negative impact on fine-tuning chatbot LLMs.

heading: 5 Conclusion
text: In this paper, we investigated different data mixing strategies in instruction fine-tuning. We measured models against diverse benchmarks and alignment skills. We find that general instructions provide better alignment as well as performance on NLP benchmarks, code instructions improve coding and alignment skills, while NLP task instructions hinder alignment skills when combined with other instruction types.
Limitations
Our work is subject to several limitations that should be addressed in future research. (1) We only use LLaMA-2 7B and 13B models in our experiments. Other models of varying sizes should be used to further verify our findings. We acknowl-
edge that the model’s behavior may vary with different sizes, and that usually, larger models have stronger capabilities, and hence may be able to handle more instructions without performance degradation. (2) In this paper, we limit our instruction dataset to 20K and mainly compare the 1:1 ratio of all instruction types. We leave the exploration of the impact of more instructions and mixing ratios to future work.
We acknowledge these limitations and propose that future work should focus on addressing them to help the community better understand the impact of instruction mixture on LLMs.

heading: A Examples of Instruction Types
text: Table 3 list examples from Alpaca, CodeAlpaca, and P3 dataset.

heading: B Alignment Skills Demonstration
text: The FLASK framework annotates each instruction with three skills that is needed to respond to the instruction. We select 8 most frequent skills and filter out instructions annotated with other skills, resulting 1,180 instructions in the evaluation set. The following are demonstrations of each alignment skill from the annotation prompt.
Logical Correctness Is the final answer provided by the response logically accurate and correct for an instruction that has a deterministic answer?
Factuality Did the model extract pertinent and accurate background knowledge without any misinformation when factual knowledge retrieval is needed? Is the response supported by reliable evidence or citation of the source of its information?
Commonse Understanding Is the model accurately interpreting world concepts for instructions that require a simulation of the expected result or necessitate commonsense or spatial reasoning?
Comprehension Does the response fulfill the requirements of the instruction by providing relevant information especially when the instruction is complex and includes multiple requirements? This includes responding in accordance with the explicit and implicit purpose of given instruction.
Completeness Does the response provide a sufficient explanation? Comprehensiveness and thoroughness of the response should be considered, which depends on the breadth of topics covered and the level of detail provided within each topic.
Insightfulness Is the response creative, original or novel, including new perspectives or interpretations of existing information?
Readability Is the response structured to promote readability and coherence? Does the response exhibit excellent organization?
Conciseness Is the response presented in a concise manner for the reader without any unnecessary information?
The prompt for alignment skill assessment are provided in Figure 3. For how a response corresponds to a specific level of an alignment skill and other details, please refer to their repository.2

heading: C Resources
text: We use 4×A100 to train LLaMA-2-7B and 8×A100 to train LLaMA-2-13B. Each 20k data takes about 2 hours. For all experiments, training takes about 288 A100 GPU hours.
For evaluation, we use GPT-4, where each input has an average of 950 tokens and each output has an average of 293 tokens. All evaluations cost about $760.
2https://github.com/kaistAI/FLASK

heading: Future Work
text: Limitations
Our work is subject to several limitations that should be addressed in future research. (1) We only use LLaMA-2 7B and 13B models in our experiments. Other models of varying sizes should be used to further verify our findings. We acknowl-
edge that the model’s behavior may vary with different sizes, and that usually, larger models have stronger capabilities, and hence may be able to handle more instructions without performance degradation. (2) In this paper, we limit our instruction dataset to 20K and mainly compare the 1:1 ratio of all instruction types. We leave the exploration of the impact of more instructions and mixing ratios to future work. We acknowledge these limitations and propose that future work should focus on addressing them to help the community better understand the impact of instruction mixture on LLMs