Instruct-SkillMix: A Powerful Pipeline for LLM Instruction Tuning

Authors: Simran Kaur, Simon Park, Anirudh Goyal, Sanjeev Arora
Primary area: foundation or frontier models, including LLMs
Keywords: instruction tuning, high quality synthetic data, diverse synthetic data

Abstract:
We introduce INSTRUCT-SKILLMIX, an automated approach for creating diverse, high quality SFT data for instruction-following. The pipeline involves two stages, each leveraging an existing powerful LLM: (1) Skill extraction: uses the LLM to extract core “skills” for instruction-following by directly prompting the model. This is inspired by “LLM metacognition” of (Didolkar et al., 2024); (2) Data generation: uses the powerful LLM to generate (instruction, response) data that
exhibit a randomly chosen pair of these skills. Here, the use of random skill combinations promotes diversity and difficulty. The estimated cost of creating the dataset is under $600. 

Vanilla SFT (i.e., no PPO, DPO, or RL methods) on data generated from INSTRUCT-SKILLMIX leads to strong gains on instruction following benchmarks such as AlpacaEval 2.0, MT-Bench, and WildBench. With just 4K examples, LLaMA-3-8B-Base achieves 42.76% length-controlled win rate on AlpacaEval 2.0, a level similar to frontier models like Claude 3 Opus and LLaMA-3.1-405B-Instruct. Ablation studies also suggest plausible reasons for why creating open instruction-tuning datasets via naive crowd-sourcing has proved difficult. In our dataset,adding 20% low quality answers (“shirkers”) causes a noticeable degradation in performance.

The INSTRUCT-SKILLMIX pipeline seems flexible and adaptable to other settings.

Decision: accept (poster)
Reviewer scores: 3, 5, 6

Reviews and discussion:
<thread>
creation date:
1729871052146

thread_type:
official_review

<thread_object>
creator: reviewer_jh2b

<content>
summary:
This paper focuses on constructing high-quality data for enhancing base LLMs’ instruction following capability. To achieve this goal, the authors propose a novel pipeline, called Instruct-SkillMix, which naturally combines diversity and difficulty in the data generation process. SFT with the resulting SFT dataset leads to very impressive performance gains on several established benchmarks. In particular, the data generation is fully automatic and the size of dataset can scale up easily.

strengths:
- It finds that directly prompting a strong LLM to identify crucial skills achieves better performance than extracting skills from existing IFT datasets.
- The performance of using merely thousands of Instruct-SkillMix data is impressive.
- The data generation pipeline is fully automated and has nearly no human intervention.
- It conducts detailed ablation studies and shows the contributions of different components.
- It reveals that even a small amount of low-quality data greatly harms the instruction following performance.

weaknesses:
- The type of queries and topics could be relevant to coverage of data. I think it might be worth to do ablation study on the query and topic types.

questions:
Kindly refer to the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730644718555

thread_type:
official_review

<thread_object>
creator: reviewer_ohap

<content>
summary:
The authors propose a new instruction tuning data generation pipeline, INSTRUCT-SkillsMIX. They prompt a strong LLM to identify some key instruction following skills. They then use these skills to produce useful instruction following data. They show strong results on instruction following benchmarks where an LLM is used as a judge.

strengths:
- This paper shows very strong performance on benchmarks where LLMs are used as a judge.
- The InstructSkillMix framework is novel and interesting. Moreover, it does not require any seed data, which is beneficial.

weaknesses:
- The baseline methods are not fair: the main comparison is to Alpaca 52K, which is really old and known to be a low quality dataset. I think the authors should try comparing their dataset to stronger datasets such as ShareGPT with the responses regenerated by GPT4-Turbo.
- In my opinion, section 1.1 is somewhat misleading. The authors (in line 70-75) say it is a mystery why public instruction tuning does not match the performance of proprietary instruct models. However, these proprietary models are trained in a variety of stages, and with distillation and RL techniques. It is not expected that instruction tuning alone can match the performance of proprietary models.
- Table 4 suggests that the main reason for the good performance of InstructSkillMix is that a stronger model is used for distillation compared to previous IFT datasets. With the same judge, Alpaca-1K longest performs similarly to InstructSkill Mix (although Alpaca 1K longest is a weak dataset in my opinion: instructions are created using text-davinci-003). Alpaca-1K longest does perform worse on the length-controlled benchmark, but this is not a fair comparison since Alpaca-1K longest is specifically biased to encourage longer responses. 
- The authors claim that InstructSkillMix is a more performant data generation method than UltraChat and Vicuna (line 511). Although InstructSkillMix seems to be a strong method, my guess is that the primary reason that InstructSkillMix outperforms UltraChat and Vicuna is due to a stronger teacher model. If the authors want to make this claim, I think they should regenerate responses from UltraChat and ShareGPT using the same teacher model InstructSkillMix uses.
- In my opinion relying only on AlpacaEval 2.0 and MTBench is a bit limited. It would be beneficial for the authors to evaluate their models on other tasks, including mathematical reasoning (MATH, GSM8K), instruction following (IFEval), and knowledge benchmarks (MMLU).

questions:
- Can you compare performance to ShareGPT with the responses regenerate with GPT4-Turbo?
- Can the authors discuss weakness 3?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682597051

thread_type:
official_review

<thread_object>
creator: reviewer_mpdq

<content>
summary:
The paper introduces INSTRUCT-SKILLMIX, a pipeline for creating instruction-tuning datasets using large language models. The method involves two stages: (1) extracting instruction-following skills using an LLM's metacognitive abilities, and (2) generating synthetic (instruction, response) pairs using random combinations of these skills. Using just 4K examples, the authors demonstrate that base models fine-tuned on this data achieve competitive performance on instruction-following benchmarks compared to much larger models.

strengths:
The paper presents a novel approach to synthetic data generation that achieves strong results with only 4K examples, suggesting an efficient path forward for instruction tuning. The empirical validation is well-designed, testing across multiple benchmarks and models while including careful ablation studies that isolate the effects of different components. 

The method is cost-effective, requiring only about $600 compared to traditional human annotation approaches. 

The authors provide some analysis of how low-quality data affects model performance, offering practical insights for dataset creation. 

The paper also test both their preferred method and a seed-dataset dependent variant, providing comparative insights.

weaknesses:
The paper's most significant limitation is the performance plateau at 4K examples, with no clear explanation or analysis of learning curves as dataset size increases. This is compounded by limited investigation of whether different architectures or model sizes might hit different ceilings. 

The evaluation methodology relies heavily on AlpacaEval 2.0 and lacks assessment of long-form generation and multi-turn conversations. The use of both teacher and grader models from the same model family (GPT-4) raises concerns about potential systematic biases. 

Also, the methodology lacks a principled approach for determining the optimal number of skills or combinations, and provides no systematic quality metrics for the generated data. 

The paper provides limited investigation of how different teacher models might affect results. Lack of this raises questions about the method's generalizability. 

The relationship between skills and model performance remains inadequately explored, with no clear metrics for assessing skill quality or coverage.

questions:
Have you investigated whether different model architectures or sizes hit the 4K example ceiling at different points?

Could you explain the choice of k=2 for skill combinations? Have you explored other values?

How would the method perform with different teacher models (e.g., Claude, PaLM)?

Would it be possible that combining synthetic data with human annotations potentially break through the 4K example ceiling?

Could you elaborate on potential approaches for quality control in the data generation process?

Could you provide analysis of model performance on longer-form tasks and multi-turn conversations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732389125445

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to All Reviewers

comment:
We thank the reviewers for their thoughtful reviews on our work, and for recognizing the novelty of our pipeline, as well as the strong results achieved via SFT on the resulting dataset.

We first address issues raised by more than one reviewer, and mention updates to the results since submission.

## **Primary Contribution of Our Work**

We would like to emphasize the following points as the main contribution of our work
1. We surpass all prior open SFT-only instruction-tuning efforts by a *wide* margin. 
- We enable small open models to achieve competitive performance to large proprietary models only via SFT. No RL. 
2. Our method is extremely cost effective (estimated data cost $600) 
- We avoid human intervention / annotation. The automated pipeline seems usable for other purposes.
- We use orders of magnitude smaller datasets (our 1K dataset beats older methods that used 50k to 1M examples)


## **Gains from pipeline vs. strong teacher model, Part 1: Rewriting responses to ShareGPT and UltraChat**

To disentangle the advantage of Instruct-SkillMix from the potential benefit of using a stronger teacher model, we initially compared the results of SFT on two datasets (generated by the same model): Alpaca-1K Longest and Instruct-SkillMix-D-1K (Table 4). 

One reviewer found this insufficient and desired a similar comparison with UltraChat and ShareGPT. We regenerate responses to random 1K examples from UltraChat and ShareGPT using the same teacher model as used in Instruct-SkillMix. We present results in the table below, showing SFT on Instruct-SkillMix is superior to them, especially in the most important metric of LC win rate. We think the LC win rate is due to the high density of information (per word) elicited by our SkillMix idea for generating diverse and interesting queries. 

| Generation Model  | Dataset                                                | WR(%) | LC WR(%) | MT-Bench |
|-------------------|------------------------------------------------------------------|---------|---------|--------|
| GPT-4-Turbo | Alpaca 1K Longest (response regenerated)          | 35.23 | 19.62 | 6.99 |
| GPT-4-Turbo | Alpaca 1K Random (response regenerated)         | 20.85 | 23.48  | 6.93 |
| GPT-4-Turbo | Ultrachat 1K (response regenerated)                    | 37.10 | 25.64 | **7.39** |
| GPT-4-Turbo | ShareGPT 1K (response regenerated)                  | 30.06 | 26.01 | 7.19 |
| GPT-4-Turbo | Instruct-SkillMix-D-1K                                            | 33.87 | 27.48 | 6.92  |
| GPT-4-Turbo | Instruct-SkillMix-1K                                                | **41.97** | **38.48** | 7.33  |


## **Gains from pipeline vs. strong teacher model, Part 2: Generalizability of pipeline to other teacher models**

Our pipeline indeed primarily utilizes models from the GPT-4 family. Below we report experiments with the same pipeline but using Claude-Sonnet-3.5 in all stages–- curating skills and query types, followed by Q&A generation. We report performance on AlpacaEval 2.0 (with GPT-4 as the grader) in the table below:


| Generation Model  | Dataset                                                | WR(%) | LC WR(%) | MT-Bench |
|-------------------|------------------------------------------------------------------|---------|---------|--------|
| Claude 3.5 Sonnet | Alpaca 1K Longest (response regenerated) | 22.10 | 19.12 | **7.13**  |
| Claude 3.5 Sonnet | ShareGPT 1K (response regenerated)         | 21.00 | 19.77 | 7.06  |
| Claude 3.5 Sonnet | Instruct-SkillMix-1K                                       | **25.74** | **25.54** | 6.88  |

Observe that for a fixed generator model, SFT on Instruct-SkillMix outperforms Alpaca-1K Longest and ShareGPT-1K. This affirms that the benefits observed via SFT on Instruct-SkillMix arise from our pipeline.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a method for creating synthetic instruction fine-tuning data using a teacher model. The method firstly instructs a strong teacher LLM to generate a list of skills useful for instruction following, and then ask the LLM to generate instructions/queries which require 1 or 2 randomly selected skills, after which the teacher LLM is again used to generate the response. The method achieved impressive instruction-following performance using merely 1k-4k data points. Various ablation studies are performed to verify that the proposed method for dataset construction (while controlling other factors such as the teacher model and the grader model) can indeed lead to significant performance improvement.

The reviewers generally agreed that the paper has a number of strengths: the proposed method is novel, the experiments and ablation studies are well designed, the method is cost-effective, the paper additionally offers insights as to how the addition of low-quality data degrades the performance, the pipeline is fully automated not requiring any human intervention.

In the initial review, the reviewers pointed out some concerns, mostly regarding the fairness of comparisons and whether the reported performance advantage indeed stems from the novel methodology.  After reading the paper and the discussions, I think the authors' rebuttal sufficiently addressed these concerns. Below I list some major concerns:
- The performance plateau at 4K examples (Reviewer MpdQ): The authors responded that all methods plateau eventually since its performance is upper-bounded by the teacher model, and different models can plateau at different points. Also, this plateau at 4K examples is already quite high, and the performance achieved by the proposed method is already impressive in various experiments (surpassing the baselines and the instruction-tuned models).
- Potential systematic bias since the teacher and grader are from the same family (Reviewer MpdQ): The authors pointed to experiments in which they used a different grader model and also added a new experiment during rebuttal. The proposed method still consistently performs better than the baseline methods in these experiments. I think this is sufficient to justify that the performance advantage of the proposed method is not significantly affected by this systematic bias.
- How the teacher model affects the performance (Reviewer MpdQ): The added results during rebuttal sufficiently demonstrated that the proposed method performs better than the baselines across different teacher models.
- Should compare with other synthetic data method (ShareGPT and Ultrachat) with regenerated responses using GPT-4 (Reviewer oHAp): The authors added the corresponding results requested by the reviewer, and the performance advantage of the proposed method is consistent in these new experiments.
- The performance improvement of the proposed pipeline is due to the use of stronger teacher model (Reviewer oHAp): The authors' response to this point also makes sense to me. The newly added experiments showed that even if the same strong teacher model is used to regenerate the responses for other synthetic data method, the proposed method still performs better.
- Lack of evaluation for long-form generation and multi-turn conversations (Reviewer MpdQ): I think since the proposed method for instruction dataset construction is novel, evaluating it in standard single-turn conversation benchmarks seems good enough to me (as long as fair comparison is ensured).

I also agree with the follow-up comment from Reviewer oHAp that the reported numbers in this paper are partially due to the improved capability of the GPT-4 teacher model. However, the authors have used the new experiments to demonstrate that even if the same teacher model is used, the proposed method still works better. I suggest the authors revise and soften some of the potential overclaims pointed out by Reviewer oHAp (such as "by a wide margin").

Overall, I think the paper makes a useful and important contribution to instruction fine-tuning based on teacher models, and the method could inspire future works in this area as well. Therefore, acceptance is recommended.