Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Authors: Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin
Primary area: datasets and benchmarks
Keywords: Dataset, LLM, alignment, synthetic, supervised fine-tuning

Abstract:
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie.  Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the pre-query templates up to the position reserved for user messages, thanks to their auto-regressive nature.  We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We further introduce extensions of Magpie for filtering, generating multi-turn, preference optimization, domain-specific and multilingual datasets. We perform a comprehensive analysis of the Magpie-generated data. To compare Magpie-generated data with other public instruction datasets (e.g., ShareGPT, WildChat, Evol-Instruct, UltraChat, OpenHermes, Tulu-V2-Mix, GenQA), we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that using Magpie for supervised fine-tuning (SFT) solely can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. We also show that in some tasks, models supervised fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through SFT and subsequent preference optimization. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

Decision: accept (poster)
Reviewer scores: 3, 8, 6

Reviews and discussion:
<thread>
creation date:
1730590914278

thread_type:
official_review

<thread_object>
creator: reviewer_dhck

<content>
summary:
MAGPIE is a scalable method for synthesizing high-quality instruction data from aligned large language models (LLMs) without requiring prompt engineering or seed questions. By leveraging the auto-regressive nature of models like Llama-3-Instruct, MAGPIE efficiently generates diverse instruction-response pairs through an automated two-step process, extendable to multi-turn, domain-specific, and multilingual datasets. Extensive analysis shows that models fine-tuned with MAGPIE-generated data outperform those using public datasets and can rival proprietary models fine-tuned on much larger datasets. MAGPIE’s versatility is further demonstrated through extensions for preference optimization and task-specific data generation, making it a significant contribution to scalable LLM alignment and democratized AI research.

strengths:
- Good contribution to open source community as the dataset is quite large and diverse
- Interesting finding that aligned LLMs can be used to generate synthetic user instructions with just a system prompt
- Detailed insights on the generated SFT and Alignment pairs were presented
- The framework would be very useful for the open source community
- Authors also present a solution to overcome some limitations of the magpie dataset by developing a booster dataset

weaknesses:
- LLMs are typically trained (loss is computed) only on the assistant turns (during SFT, DPO, RLHF etc) and it is not clear how aligned models are able to produce user instructions. The paper does not provide enough insights to understand this. While this may not be directly in scope of the paper, it would help in increasing confidence in the presented approach.

questions:
An important aspect of the work is the fact that aligned LLMs can generate user prompts with just a system prompt. 

For instance, the below system prompt would help generate user prompts for code related tasks.

"You are an AI assistant designed to provide helpful, step-by-step guidance on coding problems. The user will ask you a wide range of coding questions. Your purpose is to assist users in understanding coding concepts, working through code, and arriving at the correct solutions"

If an LLM can indeed produce a lot of diverse user prompts when sampled correctly, the novelty would seem limited. The work does go into depth on filtering etc. However fact remains that the LLM is able to generate most of the prompts. If this point can be addressed, the work would definitely be very convincing.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730688329875

thread_type:
official_review

<thread_object>
creator: reviewer_d1jb

<content>
summary:
The paper introduces a new dataset for instruction fine-tuning and preference optimization called Magpie. The dataset is curated by prompting either LLama-3-70B-Instruct or LLama-3-8B-Instruct with the pre-query template or user tags and relying on the next token prediction ability of the model to elicit instructions. The instructions collected are then used to gather responses from LLMs. To generate instruction & responses that are multiturn or relevant to a certain task, the authors add relevant system prompts to steer the model ouptuts. Experiments across multiple baseline datasets, base models and comparison on multiple benchmark shows that effectiveness of Magpie dataset. The paper also has qualitative analysis highlighting various attributes of the dataset.

strengths:
1. The paper introduces a simple & cost effective method of generating alignment data at scale, without relying on closed models or expensive human annotations.
2. Results on Alpaca Eval & Arena hard show significant improvements compared to other baseline datasets.
3. The paper presents an exhaustive quantative & qualitative experiments on question difficulty, safety, task categories, etc to highlight the attributes in Magpie.
4. The authors also experiments with various filtering configuration to improve the quality of final set of instructions.

weaknesses:
1. The Magpie dataset is not conversational in nature. The MT variant of the dataset has at maximum 2 turns. In contrast, datasets like Ultrachat, Open Assistant and WildChat are multi-turn with maximum upto 7-8 turns. Also, the paper has no experiments or analysis on conversational benchmark and multi-turn nature of Magpie in comparison to existing datasets.
2. The paper has very limited improvements on OpenLLM leaderboard in Table 3 when compared to improvements in Table 2. Why is this the case, more dicussion around this is needed.
3. The paper does not have any analysis on contamination or information leakage in the dataset for the evaluation benchmarks used. Although the dataset is purely sythetic, and does not use any seed data the chances of leakage are less, but as there is no control over what the LLM generates, the chances that LLM results in generic instructions that end up being similar to evaluation datasets are high.
4. The method for generating questions is not controlled at all. The method relies only on the next token prediction, which might lead to simple instructions. The only way to steer the generation process is through system prompts, but it is hard to judge how effective is that, as the paper has no ablations on system prompts.
5. As the generation method is fully un-controlled, how does its ensures diverse instructions? T-SNE plot in figure 7 or the similarity analysis in Figure 3 using all-mpnet-base-v2 model is not enough to gauge the diversity.
6. The paper mentions that the method can be used to generate domain specific and multilingual datasets in the abstract. However, there are no evaluations on multilingual or domain specific data.

questions:
Based on the Weaknesses, the questions are as follows:
1. How does the performance of Magpie look on multiturn benchamark? 
2. Does Magpie generation process even scale for longer turns?
3. Why there are limited improvements in Table 3 in comparison to massive improvements in Table 2?
4. As the data generation process is un-contolled, contamination analysis in Magpie datasets would be helpful.
5. How effective are system prompts for providing some control over the generation process? Is there any way to measure that? Are there any domain specific results?
6. How the generation process ensures diversity of instructions?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730749450598

thread_type:
official_review

<thread_object>
creator: reviewer_aneq

<content>
summary:
This work proposes Magpie, which is a way to prompt aligned LLMs to generate alignment datasets. Unlike most previous studies in this area (LLM-generated alignment data), Magpie removes the need of a seed set of instructions and heavy prompt engineering efforts. Instead of prompting aligned LLMs with an meaningful instruction, Magpie directly puts the "pre-instruction" token, i.e. one of the formatting delimiter tokens, to LLMs and let the LLMs generate single-turn instructions "freely". With limited extra prompt engineering efforts and an extra reward model, the method can be extended to multi-turn, multi-lingual, multi-domain data, and DPO cases (preference data). The experimental results indicate that datasets generated through Magpie are of high quality, yielding superior performances on AlpacaEval2, ArenaHard, and WildBench, as well as comparable performances on general benchmarks like MMLU, GSM8K, etc.

strengths:
- The proposed method is simple yet effective, according to the experimental results. In particular, the generated DPO data show a significant performance boost.

- The ablation study is comprehensive, considering different settings/datasets/baselines as well as different aspects like domain-specific capabilities and safety, though the result analysis lacks depth.

weaknesses:
- The presentation can be improved by putting more experimental results in the main context instead of the appendix. The proposed method is simple and easy to understand, similar to most prompt engineering related studies. The experimental results along with analysis and hypothesis test are the most important. For example, Section 3 is less important than Appendix F in terms of demonstrating the method's pros and cons.

- The experiments are well designed but lack deep analysis, which leaves many questions unanswered and makes it hard for other researchers to understand the method's best use cases and limitations. Please refer to Questions for details.

- For those most understanding results, e.g. surpassing GPT-4-Turbo(1106) on AlpacaEval 2, qualitative results are needed, i.e. random samples of actual generations for those benchmarks from all models in comparison. It is known that LLM-based judge could be biased and misleading.

- It would be better to add more alignment benchmarks like IFEval.

questions:
1. It is not clear how the model used to generate data affects the final performance. The experimental results show that MAGPIE-Pro-300K-Raw has limited improvements over MAGPIE-Air-300K-Raw. This is counterintuitive and needs more insights.

2. Following up above, one possibility is that the base model is the same (llama-8b). From this point of view, the relationship between the model to be aligned and the model to generate data is crucial. More analysis is required.

3. We can also see that MAGPIE-Pro-300K-Filtered show much better performances when used on Llama than QWen. Of course, this could mean Llama base model is better, but could also be limitations of the proposed method. The proposed method essentially relies on the model itself to control the diversity of generated instructions, which is controlled by the pre-training data. If the model to be aligned was trained with the same pre-training data, we can intuitively expect better performances and less hallucination. If that is the case, it becomes kind of a chicken-egg thing, limiting the usefulness to the proposed method.

4. Why is a separate two-step process necessary? The aligned model should be able to generate instruction and response in one pass?

5. The DPO results are surprisingly good. How does the reward model's quality affect the final performances?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733301484929

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of the Rebuttal

comment:
Dear Reviewers and Area Chairs,

Thank you very much for your time reviewing our paper and for the insightful feedback and comments. Below, we summarize our responses to the reviewers' concerns and highlight additional experiments.

### Summary of Strengths Mentioned by Reviewers

**Innovative and Simple Approach**: Magpie is simple yet effective, particularly in generating high-quality DPO data (`aNEq`), while requiring no closed models or expensive human annotations (`D1Jb`).

**Strong Empirical Results**: Magpie demonstrates superior performance on key benchmarks like Alpaca Eval 2 and ArenaHard (`D1Jb`), supported by comprehensive ablation studies across different settings, datasets, and baselines (`aNEq`).

**Comprehensive Analysis**: Our work includes detailed quantitative and qualitative experiments, examining question difficulty, safety, and task categories (`D1Jb`).

**Open Source Impact**: Our contribution to the open-source community is significant, offering a large and diverse dataset (`DhCK`). Notably, we discovered that aligned LLMs can generate synthetic instructions using simple pre-query templates (`DhCK`).

---

### Highlights of Our Clarification and New Experiments

**Extended Evaluation Results:** We provided comprehensive new results on multiple benchmarks highlighted by reviewers, including IFEval (`aNEq`), MT-Bench (`D1Jb`), and contamination analysis (`D1Jb`). These results demonstrate Magpie's superior performance compared to baselines. Additionally, we performed detailed ablation analyses across different base models and response generators, showing universal effectiveness (`aNEq`). We also examined the impact of reward model quality on final model performance in DPO (`aNEq`).

**Additional Technical Validations:** We validated the effectiveness of system prompts in controlling task categories (e.g., 3% to 95% for code tasks) and demonstrated Magpie's instruction diversity through Topic Diversity metrics (`D1Jb`). New benchmarks confirmed Magpie's strong performance in generating domain-specific (code) and multilingual (Chinese) datasets (`D1Jb`).

**Design Justification:** We explained how aligned LLMs can generate user instructions via shared token embeddings and backpropagation (`DhCK`) and clarified the advantages of our two-step process for flexibility and resource efficiency (`aNEq`).

---

### Conclusion

We sincerely thank all reviewers for their thoughtful feedback. Our responses and new experiments have demonstrated Magpie's effectiveness, versatility, and significant contribution to open-source LLM alignment research. We are committed to incorporating these insights to strengthen our paper. Unfortunately, without further engagement from Reviewer `aNEq` during the rebuttal, we are unable to address any remaining concerns, which could significantly impact the overall assessment of our work. We hope that the reviewers and Area Chairs will consider our comprehensive responses and new experimental results when making their final decision.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission introduces Magpie, a method for automatically generating large-scale high-quality alignment data for LLMs. Its idea is to leverage aligned LLMs such as Llama-3-Instruct to generate instructions by prompting them with pre-defined templates, producing millions of instructions and responses. The proposed method includes extensions for filtering, multi-turn generation, preference optimization, and multilingual data. The authors conducted experiments by comparing the generated data to other public instruction datasets, and results show that models fine-tuned with Magpie outperform those trained on other public datasets. Interestingly, the authors demonstrate that Magpie's performance is competitive with Llama-3-8B-Instruct (10 million data points through SFT and subsequent preference optimization). 

The reviewers identified the strengths of this work as:
- the paper introduced method is  simple and efficient. it helps generate alignment data at scale, which helps reduce expensive human annotations and reliance on closed models.
- the authors demonstrate significant improvements on Alpaca Eval and Arena. Extensive quantitative and qualitative experiments --- covering cover different aspects such as question difficulty, safety, and task categories ---  are performed. 

The concerns are raised by the reviewers on: 
- 1) Limited Conversational Data and Multi-Turn Analysis: the Magpie dataset lacks conversational depth (with a maximum of 2 turns). Given the importance of multi-turn dialog, comparisons with related datasets or analysis on conversational benchmarks would further the significance of this work.
- 2) there are concerns about the lack of diversity in instructions, and no ablation studies are provided for system prompts to guide generation.
- 3) though the method claims to generate domain-specific and multilingual datasets, no evaluations or experiments are provided for these aspects, limiting the generalizability of the claims.

First of all, it is unfortunate that only three out of five assigned reviewers submitted reviews. During the rebuttal, the authors managed to convince reviewer D1Jb to increase their score from 6 to 8. Reviewer DhCK decided to keep their rating to 6. Reviewer aNEq did not engage with the author rebuttal until the last minute and eventually decided to keep their rating as 3. 

The AC went through reviewer aNEq's review, the authors' rebuttal to it, and the reviewer's engagement. It is difficult for the AC to follow the rational to given a rating of 3 based on the three weaknesses (a. move more resutls from appendix to main; b. general comments on more deep analysis and c. qualitative results) given in the review. The authors' rebuttal covers detailed (and very long) responses to these questions, and more importantly, extensive experimental results are also included. It would be much appreciated if the reviewer could engage more with the authors before the ddl (extended one). 

Overall, this work's strength does not lie in its technical novelty. Rather, it shows strong empirical results with comprehensive analysis, and additional extensive experiments provided during rebuttal. In other words, this work offers a simple idea to build alignment data with strong empirical demonstration. All factors considered, this work receives an acceptance recommendation. The authors are suggested to include the new results (during the rebuttal) in the camera ready as "implicitly promised" if accepted.