Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing
Authors: Zhangchen Xu
Fengqing Jiang
Luyao Niu
Yuntian Deng
Radha Poovendran
Yejin Choi
Bill Yuchen Lin
Venue: ICLR 2025 Poster (2025)
Keywords: Dataset
LLM
alignment
synthetic
supervised fine-tuning
Primary area: datasets and benchmarks
TL;DR: This paper presents a scalable method to synthesize a 4M publicly available dataset for LLM alignment.

Abstract: High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruct, have open weights, their alignment data remain private, which hinders the democratization of AI. High human labor costs and a limited, predefined scope for prompting prevent existing open-source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present a self-synthesis method for generating large-scale alignment data named Magpie.  Our key observation is that aligned LLMs like Llama-3-Instruct can generate a user query when we input only the pre-query templates up to the position reserved for user messages, thanks to their auto-regressive nature.  We use this method to prompt Llama-3-Instruct and generate 4 million instructions along with their corresponding responses. We further introduce extensions of Magpie for filtering, generating multi-turn, preference optimization, domain-specific and multilingual datasets. We perform a comprehensive analysis of the Magpie-generated data. To compare Magpie-generated data with other public instruction datasets (e.g., ShareGPT, WildChat, Evol-Instruct, UltraChat, OpenHermes, Tulu-V2-Mix, GenQA), we fine-tune Llama-3-8B-Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that using Magpie for supervised fine-tuning (SFT) solely can surpass the performance of previous public datasets utilized for both SFT and preference optimization, such as direct preference optimization with UltraFeedback. We also show that in some tasks, models supervised fine-tuned with Magpie perform comparably to the official Llama-3-8B-Instruct, despite the latter being enhanced with 10 million data points through SFT and subsequent preference optimization. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.

Published as a conference paper at ICLR 2025
MAGPIE: ALIGNMENT DATA SYNTHESIS FROM SCRATCH BY PROMPTING ALIGNED LLMs WITH NOTHING
Zhangchen Xu
Fengqing Jiang
Luyao Niu
Yuntian Deng
Radha Poovendran
Yejin Choi
Bill Yuchen Lin
University of Washington
Allen Institute for Al
https::/maspiee-lign.. wtt.:ii-aaiigg....uu........ github.
http:://a..commmm...eee align
ABSTRACT
High-quality instruction data is critical for aligning large language models (LLMs). Although some models, such as Llama-3-Instruet, have open weights, their align- ment data remain private, which hinders the democratization of AI. High human labor costs and limited, predefined scope for prompting prevent existing open- source data creation methods from scaling effectively, potentially limiting the diversity and quality of public alignment datasets. Is it possible to synthesize high-quality instruction data at scale by extracting it directly from an aligned LLM? We present self-syutheeiis method for generating surge-scale alignment data named M AGPIE. Our key observation is that aligned I.LMs like -llma-3-Innsren Instruct can generate
query when input only the pre-query templates up the position reserved for user messages, thanks to their auto-regressive nature. We use this method to prompt -lastae--nstruct and generate 4 million instructions along with their corresponding responses. We further introduce extensions of MAG- PIE for filtering. generating multi-turn, preference optimization, domain- and multilingual datasets. We perform comprehensive analysis of the MAGPIE- generated data. To compare MAGPIE-generated data with other public instruction datasets (e.g., ShareGPT, WildChat, Evol-Instruct, UltraChat, OpenHermes, Tulu- V2-Mix, GenQA), we fine-tune Llama-3-8B Base with each dataset and evaluate the performance of the fine-tuned models. Our results indicate that using MAGPIE for supervised fine-tuning (SFT) solely can surpass the performance previous public datasets utilized for both SFT and preference optimization, such direct preference optimization with UltraFeedback. We also show that in some tasks, models supervised fine-tuned with MAGPIE perform comparably the official Jlama-3-8B-Instruce despite the latter being enhanced with 10 million data points through SFT and subsequent preference optimization. This advantage is evident on alignment benchmarks such as AlpacaEval, ArenaHard, and WildBench.
INTRODUCTION
Large language models (LLMss) such as GPT-4 (Achiam et al. 2023) and Llama-3 (Meta 2024) have become integral Al applications due to their exceptional performance on wide array of tasks by following instructions. The success of LLMs is heavily reliant on the data used for instruction fine-tuning, which equips them handle a diverse range of tasks, including those not encountered during training. The effectiveness of instruction tuning depends crucially on access to high-quality instruction datasets. However, the alignment datasets used for fine-tuning models like I Hama--3-Instrue are typically private, even when the model weights are open, which impedes the democratization of Al and limits scientific research for understanding and enhancing L.LM alignment.
To address the challenges in constructing high-quality instruction datasets, researchers have developed two main approaches. The first type of method involves human effort to generate and curate instruction
data (Databricks) 2023 Köpf et al.) 2023 Zhao et al. 2024 Zheng et al. 2024 2023). which is both time tine-consuming and labor-intensive (Liu et al. (222a)). In contrast, the second type of methods
uses LLMs to produce synthetic instructions (Ding et al. 2023] Yin al. 2023 Li et al.) 2024a]]
Sun et al. 2023, Taori et al., 2023) Wang et al. [023]
[2223]20242 2024c| Xu et al. 2023a Li et al. 2023a).
Although these methods reduce human effort, its success heavily depends on prompt engineering

---

Published as a conference paper at ICLR 2025
Demain-Specfiic System Prompts (Optional)
</> fx
Instruction
Resperce
Step 1
<Istart, header__id r_idd_ buser end header >
materials Uuse to build a nest?
Brsperse
Reward Model
Nuaoed
Step 2
<lstart_ header buser end header >d]>
Response
Paspros
Building nest That's wonderful project!
What materials should use build nest?
T
Inpat Dhelts les foe diferonce
Filters
Dupput
< [joot_id] sildd/pe start head sassasstt header_ _id>>
LLM
Intrution
Bast valtea
SFT
Ban
cilieral
MAGITE MI
Figure 1: This figure illustrates MAGPIE, the process of self-symthesizing alignment data from aligned LLMs (e.g.. 1aama-3-8B-Instruue
create a high-quality instruction dataset. In Step we input only the pre-query template into the aligned LLM and generate an instruction along with its response using auto-regressive generation. In Step 2. we use a combination of post-query template and another pre-query template to wrap the instruction generated from Step 1, prompting the LLM to generate the response. This completes the construction of the instruction dataset. MAGPIE efficiently generates diverse and high-quality instruction data, which can be further extended to multi-turn (MAGPIE-MT), preference optimization (PO),, domain-specifit and multilingual datasets.
and the careful selection of initial seed questions. as the dataset size grows. Despite ongoing efforts, the scalable creation of high-quality and diverse instruction datasets continues be challenging problem.
The diversity of synthetic data tends to decrease
it possible to synthesize high-quality instructions at scale by directly extracting data from advanced aligned I.MMs? A typical input template, the query, and the post-query template. For instance, an input to Llama-2-chat could be "I "
 Hi! I/INST]", ", where (INST) is the pre-query template and |/ [NINS]] the post-query template. These templates are predefined by the creators of the aligned LLMs to ensure the correct prompting of the models. We observe that when we only input the pre-query template aligned LLMs such as Llama-3-Instruct they self-synthesti
aligned L.M contains three key components: the pre-query
a user query due to their auto-regressive nature. Our experiments indicate that these random user queries are of high quality and great diversity, suggesting that the abilities learned during the alignment process are effectively utilized.
Based on these findings, we developed a self- eeff-synthesis method to construct high-quality instruction datasets at scale, named MAGPIE (as illustrated in Figure D). Unlike existing methods, our approach does not rely on prompt engineering or seed questions. Instead, it directly constructs instruction data by prompting aligned LLMs with a pre-query template for sampling instructions. We also demonstrated the extensibility of MAGPIE in generating multi-tuma ttrnttuum preference optimization, domain- specific, and multilingual datasets. We applied MAGPIE to the Baatmaa---B-Instrree and 3-ama-3- 70B-Instruct models, creating two instruction datasets: MAGPIE-Air and MAGPIE-Pro, respectively.
Our MAGPEE-Air and MAGPIE-Pro datasets were created using 206 and 614 GPU hours, respectively, without any human intervention or API access to production L.LMs like GPT-4. The statistics and advantages of MAGPIE datasets compared to existing ones are summarized in Table in Appendix A We perform comprehensive analysis of these two datasets in Section 3 allowing practitioners to filter and select data instances for fine-tuning models according to their particular needs.
To compare MAGPIE data with other public instruction datasets (e.g., ShareGPT (Chiang et al. 2023). WildChat (Zhao et al. (2024), Evol Instruct (Xu et al. 2023a). UltraChat (Ding et al. (023). OpenHermes (Teknium (203aa b). GenQA (Chen et al. 2024). Tulu V2 Mix (Ivison et al. (223)), we conducted supervised fine-tuning (SFT) of the L.lama--881 3-8B-Base model with each dataset and assess the performance of the fine-tuned models on alignment benchmarks such as AlpacaEval 2 (Li et al.) 2023b), Arena-Hard (Li et al. 2024b), and WildBench (Lin et al. 2024). Our results show that models supervised fine-tuned with MAGPIE achieve superior performance, even surpassing models that utilize both SFT and direct preference optimization (DPO) (Rafailov et al. 2023) with UltraFeedback (Cui et al. (202).. Notably, MAGPIE-aligned models outperform the official Llama-3-88 Instructtt model on AlpacaEval 2, despite the latter being fine-tuned with over 10 million data points for SET and subsequent preference optimization. Not only does M MAGPIE excel in SFT alone compared to prior
2

---

Published as a conference paper at ICLR 2025
public datasets, but also delivers the best results when combined with preference optimization methods such as DPO. By leveraging MAGPIE extensions to generate high-quality preference optimization datasets, MAGPIE-aligned Llama-3 models can even outperform GPT-4-Turbod 106) on AlpacaEval 2. These findings show the exceptional quality of instruction data generated by MAGPIE, enabling it to outperform even the official, extensively optimized, and proprietary LLMs..
MAGPIE: A SCALABLE METHOD TO SYNTHESIZE ALIGNMENT DATA
Chat Templates of Aligned LLMs. For an aligned LLM (e.g., Llama-3-8B --8B--nsttuct), the input sequence can be represented as x = T pre-query @ @ [post-querg Here, is the user query (e.g.. "What material should use to build nest?"), while T pre-query and pust-query are pre-query and post-query templates. The pre-query template shows up before the user query. and the post-query template is defined as the conversation template between the user query and the LLM response. These templates are defined by the model provider to ensure the correct prompting. For example, for Llama- 3-8B-Insturet model, Tpre- = < aattt_ttrraeettte header id >user< eeddeen__ddddl header_id >, and T postt-quen query pot_id|>< start headereeaerr___ id wassistanssss
surd__hader____
2.1 MAGPIE PIPELINE
Overview of MAGPIE. In what follows, describe our lightweight and effective method, MAGPIE, to synthesize alignment data from aligned LLMs. An instance of instruction data consists of at least one multiple instruction-respone pairs. Each pair specifies the roles of instruction provider (e.g.. user) and follower (e.g., assistant), along with their instruction and response. As shown in Figure MAGPIE consists of two steps: (1) instruction generation, and (2) response generation. The MAGPIE pipeline can be fully automated without any human intervention, and can be readily adapted for the generation of multi-turn, preference, and dopaif- datasets, as detailed in Section 2.2 We describe each step in the following.
Step 1: Instruction Generation. The goal of this step is to generate an instruction for each instance of instruction data. Given an open-weight aligned LLM (e.g., 1imma---70B-Insttreet MAGPIE crafts a pre-query template in the format of the predefined instruction template of the LLM. Note that the auto-regressive LLM has been fine-tuned using instruction data in the format of the pre-query template. Thus, the LLM autonomously generates an instruction when the pre-query template crafted by MAGPIE is given as an input. MAGPIE stops generating the instruction once the LLM produces an end-of-sequence token. Sending the crafted query to the I.L. multiple times leads to set of instructions. We note that compared with existing synthetic approaches (Ding et al. 2023 Li et al.) 2024al Taori et al., 2023 Wang et al., 2003]] 2024c| [u et al., 2023a b). MAGPIE does not require specific prompt engineering techniques since the crafted query follows the format of the predefined instruction template. In addition, MAGPIE autonomously generates instructions without using any seed question, ensuring the diversity of generated instructions.
Step 2: Response Generation. The goal of this step is to generate responses to the instructions obtained from Step 1. MAGPIE sends these instructions to the LLM to generate the corresponding responses. Combining the roles of instruction provider and follower. the instructions from Step and the responses generated in Step 2 yields the instruction dataset. We note that separating instruction and response generation offers several key advantages, including more flexible generation configurations between instructions and responses, where instruction generation benefits from higher temperature settings to maximize diversity, while response generation requires lower temperature for accuracy and reliability. addition, this approach provides modular flexibility, allowing users to generate instructions independently and later pair them with responses from various sources.
Applicability of MAGPIE on Different LLMss. MAGPIE can readily deployed to state-of-the-art open-weight language models including but not limited to Llama-3 (Meta) (024)), Llama-3. 1/33. (Dubey et al.) 2024), Qwen2 (Yang et al. 2024a). Qwen2.5 (Yang et al.) 2024b). Gemma-2 (Team) et al. 2024), and Phi-3 (Abdin et al.] 2024). Please refer to Appendix A for detailed support and corresponding datasets.
Remark. MAGPIE generates high-quality instructions even when the instruction loss is masked during alignment. We hypothesize that LLMs retain an implicit memorization of instruction distributions. We leave it potential future research problem.
3

---

Published as a conference paper at ICLR 2025
2.2 MAGPIE EXTENSIONS
Dataset Filtering. MAGPIE allows practitioners to select instruction data from the raw dataset generated from the above two steps based on their needs. In Appendix [ we explores potential filter configurations with eight available metrics for users to customize their own MAGPIR datasets. We also provide 6 off-the-shelf filter configurations and discuss their performance in Appendix [:55
Generating Multi-Turn Instruction Datasets. MAGPIE can be readily extended to generate multi- turn instruction datasets. To construct a multi-turn dataset (denoted as MAGPIE-MT), we initially follow Steps 1 and 2 to generate the first turn of instruction and response. For subsequent turns, we append the pre-query template to the end of the full prompt from the previous round of communication. We observe that the model may occasionally forget its role as the user, especially for the 8B model. To mitigate this, we employ a system prompt designed to control the behavior of the LLM and reinforce its awareness of the multi-round conversation context. The full prompt for building the instructions of MGGPEE-MT can be found in Figure [4] in Appendix G We follow the procedure in Step 2 of Section 2.1 to generate responses form the multi-turn instruction dataset.
Generating Preference Optimization Datasets. Leveraging the diverse and high-quality instructions produced by MAGPIE, we present simple and effective method for generating preference optimiza- tion data, inspired by Meng et al. (2024) and Tran et al. (023). We first select a small proportion of high-quality instructions from the raw dataset generated by the MAGPIE pipeline, ensuring diverse task categories. For each selected instruction, we sample responses from the aligned LLM k times, using a temperature of T < 1. We then employ a reward model (RM) to annotate scores for these responses. The response with the highest RM score is labeled as the chosen response, while the one with the lowest RM score is designated as the rejected response.
Generating Domain-Specifer and Multilingual Datasets. In certain scenarios, users may wish to fine-tune LLMs using domain-specific or multilingual instruction data to enhance performance within specific domains or languages. To address this need, we introduce lightweight method to control both the task category and the language of generated instructions. Our approach involves guiding LLMs through tailored system prompt, specifying that the model is chathot designed for a particular domain and outlining the types of user queries it might encounter. Examples of system prompts designed control the generation of math, code, translation, and multilingual instructions are illustrated in Figure 15
in Appendix G
Furthermore, note that domain-specific and multilingual instruction data can also be generated using models that are tailored to particular fields. MAGPIE demonstrates broad applicability beyond diverse chat models, extending to specialized code models (c.g.. CeepSeek-Coders V2 (Zhu al.) 2024)) and math models (c.g.. Breen--Math-BB-mnae unique strengths and specializations of different models, MAGPIE can create a rich and diverse corpus of instructions. Examples of MAGPIE-generate instructions from different domain-spedn models and multilingual models are provided in Appendix !
(Yang et al. (024a)). By leveraging the
3 DATASET ANALYSIS
To demonstrate the effectiveness of MAGPIE compared with baseline methods for generating diverse high-quality aligmment datasets, we apply MAGPIE to the Llama-3-8B-Ing Instrrctt and Llama-3-70B- Instruct models (Meta) 2024) to construct two instruction datasets: Llama-34 referred to as MAGPIE- Air) and -aama-3-MMGPI---- (hereafter referred to as MAGPIE-PrO), respec- tively. Examples of instances in both datasets can be found in Appendix [ In this section, present a comprehensive analysis of the MAGPIE-Air and MAGPIE-Prer Pro datasets, including topic coverage, difficulty, quality, similarity of instructions, and the quality of the responses.
(hereafter
3.1 DATASET COVERAGE
We follow Zhao et al. (024)) and analyze the coverage of MAGPIE-Pro in the embedding space. Specifically, we use the a ll mpnet baae-baaevv22 embedding mode calculate the input embed-
dings, and employ I-SNE (an der Maaten & Hinton 2008) to project these embeddings into a
4

---

Published as a conference paper at ICLR 2025
two-dimensional space. We adopt three synthetic datasets as baselines, including Alpaca Taori et al. ).23), Evol Instruct (Xu et al. 2023a). and UltraChat (Ding et al. 2023), to demonstrate the coverage of MAGPIE-Pro. The detailed analysis can be found in Appendix D.I We observe that the t-SNE plot MAGPTE-Pro encompasses the area covered by the plots of A Alpaca, Evol Instruct, and UltraChat. This suggests that MAGPIE-Pro provides broader more diverse range of topics. We also follow Wang et al. al. (2023) and present the most common verbs and their top direct noun objects instructions in Appendix indicating the diverse topic coverage of MAGPIE dataset.
3.2 DATASET ATTRIBUTES
Attribute: Task Categories of Instructions. We Llama-3-8B-Instreee to categorize the in- stances in MAGPIE-Pro (see Figure in Appendix [.I] for detail). The prompts used to query Mlama---B-Instreee can be found in Appendix G Our observations indicate that over half of the tasks in MAGPIE-Pro pertain to information seeking, making it the predominant category. This is followed by tasks involving creative writing, advice seeking, planning, and math. This distribution over the task categories aligns with the practical requests from human users
(Li et al. 2023b).
Attribute: Quality of Instructions. Similar to methods in (Chen et al., 2023), we prompt the - model to assess the quality of each instruction in MAGPIE- Air and MAGPIE-Pro, categorizing them as "very poor', 'poor', 'average', 'good', and 'excellent'. We present the histograms of qualities for both datasets in Figure (a). We have the following two observations. First, both datasets are of high quality, with the majority of instances rated "average" quality of MAGPIE-Pro surpasses that of AGPIE-AÍrr. We hypothesize that this is due the enhanced capabilities of Llama-3-70B compared with Llama-3-8B.
40%
Pro
Porcentage
30%
1.5%
(6) Statistics on Inpur Quality
higher. In addition, the overall
70%
Air
60%-
Percentage
Attribute: Difficulty of Instructions. We the Llama- 3-8B-Instruct model to rate the difficulty of each instruc- tion in MAGPIE-AN Air and MAGPIE-Pro. Each instruction can be labeled as 'very easy', 'easy', 'medium', 'hard', or "very hard'. Figure (b) presents the histograms of the levels of difficulty for MAGPIE-Air and MAGPTE-Pro. We observe that the distributions across difficulty levels are similar for MAGPIE-AIr and MAGPIE-Pro. Some instructions in MAGPIE-Pro are more challeng- ing than those MAGPIE-Air because MAGPIE-Pro is generated by a more capable model (i.e., Llama-3-70BB 7nntsssuuu
20%
10%
0%
very casy
medium hardevery hard
Statistics on Input Difficulty
The statistics of instruction difficulty and quality.
Figure 2:
Attribute: Instruction Similarity. We quantify the simi- larity among instructions generated by MAGPIE remove repetitive instructions. We measure the similarity using minimum neighbor distance in the embedding space, Specifically, we first represent all instructions in the em- bedding space using the all all-mpnet- base v2 embed- ding model. For any given instruction, we then calculate the minimum distance from the instruction its nearest neighbors in the embedding space using Facebook AI Sim- ilarity Search (FAISS) (Douze al. (224).. The minimum neighbor distances of instructions in MAGPIE-Air after removing repetitions are summarized Figure (a).
52903-
12003
0.75
0
044
565
1.55
3,00
1.25
1.45
Embedding Distance
153923
renuuct Reward
Base Roward
15
13
30
Quality of Responses. We assess the quality of responses using rewards assigned by a reward model, also
Attribute:
(b) Reward Difference between Base Model and Instruct Modal
Figure 3: This figure summarizes the minimum neighbor distances and reward differences.
denoted as r" For each instance in our dataset, calculate reward difference as r" Thases where Tbase is the reward assigned by the same reward model to the
5

---

Published as a conference paper at ICLR 2025
response generated by the Llama-3 base model for the same instruction. We use URIAL Lin et al. ) to elicit responses from the base model. A positive reward difference indicates that the response from dataset is of higher quality, and could potentially benefit instruction tuning. In our (Xiong et al. 2024) as the reward model. Our results the reward difference presented in Figure (b.
experiments, we follow Lambert et al. (2024) and choose F's Fsfai perr r
3.3
SAFETY ANALYSIS
We use -lama-Gurrd22 TTea 2024) to analyze the safety of MAGPIE-Air and MAGPIE-Pro. Our results indicate that both datasets are predominantly safe, with less than 1% of the data potentially containing harmful instructions or responses. Please see Appendix D.2 for detailed safety analysis.
3.4
COST ANALYSIS
We perform experiments on a server with four NVIDIA A100-SXM4-80GB GPUs, an AMD EPYC 7763 64-Core Processor, and 512 GB of RAM, using the VLLM inference framework (wwon et al. 2023). The models are loaded in the bfloat16 format.
When creating the 3M MAGPIE-AIr dataset, our MAGPIE spent 1.55 and 50 hours generate the instructions (Step 1) and responses (Step 2), respectively. For the IM MAGPIE-Pro dataset, MAGPIE used 3.5 and 150 hours generate the instructions (Step 1) and responses (Step 2), respectively. Compared to existing approaches to create instruction datasets, the pipeline of MAGPIE is fully automated without any human intervention or API access to advanced commercial models such as GPT-4 (Achiam et al. 2023). Consequently, MAGPIE is cost-effective and scalable. On average, implementing MAGPIE on a cloud serve would weuldd incur costs $0.12 and $1.1 per 1,000 data instances for MAGPIE-Air and MAGPIE-PrO, respectively.
3.5 ADDITIONAL ANALYSIS
Additional dataset analysis, including the impact of generation configurations on the quality and difficulty of the generated instructions, is in Appendix D.3 Ablation analysis on annotating models for assessing quality and difficulty is in Appendix D.4 Contamination analysis is in Appendix D.5
4 PERFORMANCE ANALYSIS
In this section, we evaluate the quality of MAGPIE-generated datasets by utilizing them to align base models including Llama-3 (etta 2024). Qwen1.5 (Bai al. 2023). and Qwen2 (Yang et al. (004a).
4.1 EXPERIMENTAL SETUPS
Baselines for Supervised Fine-Tuning and Preference Optimization. We compare the family of instruction datasets generated by MAGPIE with eight SOTA open-source instruction datasets:
ShareGPT (Chiang et al. 2023). WildChat (Zhao al. 2024). Evol Instruct (Xu et al. 2023a),
UltraChat (Ding et al.] 2023). GenQA (Chen et al.) 2024). OpenHermes (Teknium 2023a).
OpenHermes 2.5 (Teknium (023b), and Tulu V2 Mix (Ivison al. 2023). ShareGPT and WildChat
are representative human- written datasets containing 112K and 652K high-quality multi-round conversations between humans and GPT, respectively. Evol Instruct, UltraChat, and GenQA are representative open-source synthetic datasets. Following Meng et al.] (222)), we use the 208K sanitized version of Ultrachat provided by HuggingFacef] OpenHermes
OpenHermes 2.5, and Tulu V2 Mix are crowd-sourced datasets consisting of a mix of diverse open-source instruction datasets, with 243K, 1M, and 326K conversations, respectively. We also create an instruction dataset with 100K conversations using the Self-Instruct (Wang al. 2023) and Moma-3-8B-Instreeee model, denoted as Sesf-Instruct (Llama-3).
We compare the models aligned using MAGPIE with preference optimization baselines using direct preference optimization (DPO) (Rafailov et al. 2023). Specifically, we follow Meng et al. (2044)
hatp:///aaamaaaaa......oo com http:/lass..oommsoeiieeeeeeeeeee htttas//aaaaaaabs.ccoooeeeeeeeep

---

Published as a conference paper at ICLR 2025
and use the models fine-tuned with the UltraChat dataset (for instruction tuning) and Ultrafeedback dataset (for preference optimization) (Cui et al. 2023).
MAGPIE Setups. To demonstrate the quality of MAGPIE-generran instruction datasets for SFT, select the first 300K MAGPIE-Air and MAGPIE-PrO raw datasets generated by 3lama-3-8B-Instrree models, respectively. Apart from these datasets, also applied the filters detailed in Appendix Cand created two filtered datasets: MAGPIE-Air-Filtered and MAGPIE- Pro-Filtered, each contains 300K conversations. For preference optimization, we generate twO additional datasets: MAGPIE-Air-DPO (generated by Llama-3-8B-Inin ----t--utttttt and MAGPIE-PrO-DPO (generated by 3-0-ma----7OBIntttttt with and T 0.8, each contains 100K conversations. We use RLEE 1ow http//ArooRMMMaaaa
and
0 (Wang et al. 2024a) as the reward model.
Model Alignment Details. For supervised fine-tuning, we follow Touvron al. (2023) and use cosine learning rate schedule with an initial learning rate of 2 x 10 when fine-tuning Llama-3,
Qwen1.5 and Qwen2 base models. The maximum sequence length is 8192. For DPO, we use cosine learning rate of 5 x 10 The detailed parameters can be found in Appendix B.2 We follow the official instruction templates of each model.
Evaluation Benchmarks. We evaluate the performance of the aligned models using two widely adopted cnstruction-following benchmarks: AlpacaEval (Li et al. 2023b) and Arena-Hard (Li et al.) 2024b). AlpacaEval 2 consists of 805 representative instructions chosen from real user interactions. Arena-Hard is an enhanced version of MT-Bench (Zheng et al. 2023), containing 500 challenging user queries. Both benchmarks employ GPT evaluator assess responses generated by the model of interest and a baseline model. Specifically, we use GPT-4-Turbo (1 106) and 1aama-3-8B-Inttruue as baselines for AlpacaEval 2. By default, Arena-Hard uses GPT-4 (0314) as its baseline model.
Metrics. We adopt two metrics to measure the capabilities of instruction-followinng of fine-tuned models. The first metric is the win rate (WR), which calculates the fraction of responses that favored by the GPT evaluator. This metric is applied in both benchmarks including AlpacaEval 2 and Arena-Hard. The second metric the length-controlled win rate (LC) (Dubois al. 2024). a debiased version of WR. The GPT evaluator considers the lengths of responses generated by the baseline model and model under evaluation when computing LC. By accounting for response length, LC reduces its impact on the win rate. This metric is specifically applied to the AlpacaEval 2 benchmark (Li et al. 2023b).
More Experimental Setups. We provide more detailed descriptions of our experimental setups. including more model alignment details and benchmark decoding hyper-parameters in Appendix
4.2 EXPERIMENTAL RESULTS
Creative
Tasks
MAGPIE datasets outperform baselines with SFT only. In Table [ we compare the performance of Llama-3 models fine-tuned with instruction datasets generated by MAGPIE against those supervised fine- tuned with baseline datasets. Using the AlpacaEval 2 benchmark, we observe that both the LC and WR of our supervised fine-tuned models surpass all those models fine-tuned with baseline SFT datasets. This indicates that the datasets generated by MAGPIE are of higher quality, leading to significantly enhanced instruction- following capabilities. A similar observation is made when using the Arena-Hard evaluation benchmark. We highlight that the Llama-3 base models supervised fine- tuned with instruction datasets generated by MAGPIE outperform even those models that have undergone preference optimization (i.e., STF followed by DPO), which further emphasizes the high quality of data gen- erated by MAGPIE.
Coding
Dabugging
Reasoning
Planning
Info Seeking
Magale dChat
Shane
UhraChat
Evel Instuct
OperHermas
To V2 Mishune
Figure 4:
This figure shows the perfor- mance breakdown by category of MAGPIE- Pro and baselines on WildBench.
httss::/huugiingffce.cooo
/Mata/////aaaaa ama-a-a---mmm 88- issruct
7

---

Published as a conference paper at ICLR 2025
Table 1: This table compares the performance of models instruction-tunee on the Llama-3-8B base models using MAGPIE--ennraile datasets and baseline datasets. We observe that models aligned with our datasets significantly outperform those aligned with baseline datasets of the same order of magnitude in terms of data size. In addition, our fine-tuned models achieve comparable performance to the official aligned model, despite only undergoing SFT with a much smaller dataset. Numbers in bold indicate that MAGPIE outperforms the official Llama-3-8B3 B-B-a-----I model.
AlpacaEval
Arena-Hard
Aligument Sctup
#Convs
GPT-4-Turbo (1106)
Llama-3-8B-Innae
(Base LLM Llama-3-8B)
LC (%)
WR (%)
SD
1.C (5)
WR
SD
WR(S)
SFT +Self-Instruct (1 (Jlama3)) otne al. 2023
100K
7.21
5.18
0.7
17.86
12.73
1.05
4.0
*ShrreGPT (Chiang
112K
9.73
7.2
0.81
27.26
18.32
1.18
6.5
+Evol InstrucT ex 20455
143K
8.52
6.25
0.76
20.16
14,98
1.1
5.1
+OpenHernes Tekmum 20354
243K
9.94
6.27
0.73
29.19
17.92
1.16
4,4
+Tulu V2
A075
326K
9.91
7.94
0.8G
24.28
18.64
1.18
5.4
+WildChat (Zhho ill
652K
14.62
10.58
0.92
34.85
26.57
1.32
8.7
+Openl lermey
Tekmium
20233
1M
12.89
1,30
+GenQAA (Chen et al.
2024
6.47M
9.05
7.11
0.82
21.90
16.09
1.12
3.0
+UltaChar Ding al A123
208K
8.29
5.44
0.71
23.95
15.12
1.11
3.6
DPO
+UlraFerdbscck eireen
2023
64K
18.36
17.33
1.14
44.42
42,36
1.46
14.8
SIT
MAGPIR K00K Raw
300K
21.99
21.65
1,21
48.63
48.06
1.42
15.8
MAGPIR-Air 3ir.30KK........ Filterel D
300K
22.66
23.99
1.24
49.27
50.8
1.44
14.9
-MAGPIE-Air
100K
50.43
79.64
SFT
MAGPIE-PrO- Prtt--Pro-00O-Rm
Raw
300K
21.65
22.19
1.2
49.65
50.84
1.42
15.9
MAGPIR Pro. 300K Fillerel
300K
25.08
29.47
1.35
52.12
53.43
1.44
18.9
DPO
+MAGPIE Pro-DPO
100K
50,10
53.53
1.45
78.52
80.82
1.17
35.7
Liama-3-B--lstrrrr
10M
22.92
22.57
20.6
To investigate the advantages of MAGPIE across different task categories, we compare the performance of models fine-tuned with MAGPIE-Pro with baseline datasets using WildBench benchmark (Lin et al. 2024). This benchmark consists of 1024 tasks carefully selected from real-world human-L.LM conversation logs. The results are demonstrated in Figure 4 outperforms baseline datasets across categories.
We observe that MAGPIE consistently
Models aligned with data generated by MAGPIE achieve comparable or even higher perfor- mance to the official aligned model, but with fewer data. In Table I we also compare the performance of models aligned with data generated by MAGPIE against the official aligned model (Llama-3-8B-Instruue We observe that the Llama-3-8B base model supervised fine-tuned with data from MAGPIE outperforms Llama-3-88 8B-isstruct using the AlpacaEval benchmark. For example, when Llama-3-8B- Baa-maa---B-Inntruuee -ns-rutttt is chosen as the baseline model of AlpacaEval 2, we observe that LCC of Llama-3-8B base models fine-tuned with instruction data from MAGPIE exceeds 50%, indicating a preference for our SFT models over the official aligned model. In addition, when DPO is applied, our aligned model demonstrates remarkable performance gains. Specifically, it outperforms the official Llama-3-88 3-BBB- In-strtttt model both the AlpacaEval 2 and Arena- Hard benchmarks. Most notably, model even surpasses GPT-4-Turbo( 106) on AlpacaEval 2. Finally, we highlight that our alignment process
more than 400K data, whereas the official aligned models aligned with more than 10M data samples. This demonstrates the high quality of the data generated by MAGPIE.
MAGPIE can enhance the performance of other backbone models. Table 2 illustrates the efficacy of MAGPIE when applied to generate instruction dataset and fine-tune other base models, i.c., Qwen2- 1.5B, Qwenl.5-4B, and Swenl.5-7B. The results demonstrate that our fine-tuned models achieve better performance than the official aligned models, which have undergone both supervised fine- tuning and preference tuning. These results underscore the effectiveness of MAGPIE and the quality of its generated instructions. In addition, we apply MAGPIE--gneratee datasets to align I.lama-.. Minitron-4B- -Biitthhh-----hh----hh-- Width-WWaaasss -ast (Sreenivas et al. (0244) and -----aa...-8--Inaee using SFT followed by DPO. The resulting aligned model, which term MagpieLM, achieves remarkable performance and ranks first among popular open-source instruction models with fewer than 10 billion parameters. The details of MagpieLM are deferred Appendix B
(Dubey et al. 2024)
Performance of MAGPIE on More Benchmarks. We report the performance of models supervised fine-tuned using MAGPIE-Air and MAGPIE-PrO, evaluated across range of tasks featured on the Huggingface Open LLM Leaderboard (Beeching et al. 2023) in Table 3 The tasks includes MMLU (Hendrycks et al.
2020 ARC Challenge (Clark et al. 2018), HellaSwag (Zellers et al. 2019)
8

---

Published as a conference paper at ICLR 2025
Table 2: This table compares the performance of models instruction-ture on the Qwen base models using the MAGPIE-Pro-3004 -ii--Prro000--iteerre dataset and the official instruction-tun tundd models. The Qwen base model enhanced with MAGPIE outperforms the official instruction -unondd model.
AlpacaEval
Aligument Setup
GPT--Turbo (1106)
Official Aligned Model Ref.
LC (%)
WR (%)
SD
LC (%)
WR (%)
SD
Qwen2-1. (wven2-1.5B-Insstin Base Model + MAGPIE
3,91
3.00
0.54
50
50
Qwen2-1.5B
3.48
5.32
0.67
56.66
66.27
1.50
Qwenl.5-44
Qwen 1ooni.5-400 5-4BBChat Base Model + MAGPIE
5,89
4.74
0.67
50
9.1
10.96
0.93
68.09
72.42
1.42
Qwen S-7B-hhat
14.75
11.77
0.97
50
50
Qwenl.5-7B
Base Model + MAGPIE
15.10
18.51
1.14
46.28
58.53
1.44
Table 3: This table compares the performance of models supervised-fne--tneed MAGPIE-Air, MAGPIE-Pro, and MAGPIE-PrO-Mix against baselines and official instruct model across various downstream benchmarks. All models are supervised- -ineeedu-i--teeeed on the Llama-8B base models.
Alignment Setup
MMLU
ARC
HelluSwag
TruthfuIQA
WinoGrunde
GSMSK
MMLU-Redux
Average
ShareGarT
66.13
58.45
48,67
Instract
65.62
42.91
GezQA
Openllemees
OpeaHHemes V2 Mic
47.99
46.07
WildChat
65.95
59.22
$1.39
53.18
15.30
48.75
$2.59
62:34
UlraChar
65.23
62.12
81.68
52.76
75.53
50.57
50.75
62.66
MAGrin-A--- S00---A----------- Filteeeltteeedd
64.45
61.01
79:00
53.48
72.38
$2.24
52.54
62:25
MAGPIE: Pro- 300K Fillered
64.25
60.41
80.52
52.46
73.33
47.92
52.16
61.58
MAGPIE-PP P'o Mix Filtered
65.05
39.64
80.72
50.51
73.24
63.08
56.34
64.21
61.52
75.67
72.14
71.72
TruthfulQA (Lin al. 2021),
WinoGrande (Levesque et al. 2012), and GSM8K (Cobbe al. 2021
We also perform experiments on ALMLU-Redul (Gema et al. (0024) with zero-shot prompting. Our experimental results demonstrate that models fine-tuned with MAGPIE-Air and MAGPIE-PAR achieve comparable performance to the official instruct model and other baselines.
We note that the performance of MAGPIE may degrade on reasoning tasks, which is attributed the small proportion of reasoning instructions in MAGPIE-Air and MAGPIE-PrO datasets. In response, we provide a supplementary "booster" dataset containing 150K math, code, and reasoning instructions
using the MAGPIE extension mentioned in Section 2.2 We combine this booster dataset with AAGPIE-Pro-300K-T
and create MAGPIE- Pro- Pro-Mix-Fiiie Filteree Experimental results presented in Table 3 demonstrate that the model supervised fine-tuned using the mixed dataset effectively addresses the initial weakness in reasoning tasks. Notably, this new model ranks among the top-3 of all model checkpoints, performing only slightly weaker than OpenHermes 2.5 (1M conversations) and Llama-3-8BB-Instann (>M0M conversations). This significant improvement showcases the flexibility and adaptability of the MAGPIE framework in generating task-specific instruction data.
Additional Experimental Results. We defer additional experimental results and analysis of multi- turn datasets, i.e., MAGPIE-Air-MT and MAGPIE-PrO-MT, Appendix F. We conduct detailed comparison between MAGPIE and Self-Instruct in Appendix F.2 We demonstrate the performance of domain-specific and multi-lingua MAGPIE datasets Appendix F.3 In addition, ablations on data quantity, quality, filter designs, and response generator are deferred in Appendices F.4 F.5 and F.6 MAGPIE model's performance on trustworthiness and istructioo-folllwinng benchmarks reported in Appendices F.7 aanddF F.8 The ablation analysis on the impact of reward models on DPO data performance is presented in Appendix F99
RELATED WORK
LLM Alignment. Instruction tuning (Wei et al.) 2022 and preference tuning (Bai et al. 2022) widely used to align the responses of LLMs with human values. Instruction tuning utilizes an instruction dataset to fine-tune LLMs, where each instruction data consists of one turn or multiple turns of instructions and desired responses. The performance of instruction tuning heavily relies

---

Published as a conference paper at ICLR 2025
the quality of instruction data (Taori et al. 2023 Wang al. 2023 Zhou et al. 2023a Preference
tuning further improves responses of LLMs using reinforcement learning human feedback (RLHF)
(Bai et al. 2022) preference optimization (Azar et al., 2024 Ethayarajh et al. 2024 Hong et al.)
2024 Rafailov et al. 2023) based on preference dataset.
Alignment Dataset Construction. We classify the existing methods of creating datasets for model aligmment into two main categories: human interactions with LLMS and synthetic instruction genera- tion. To create datasets for alignment, previous studies have collected human interactions with LLMs @Databricks 2023; Zhao et al.
2024; Zheng et al. 20224] 2023 Köpf et al. 2023). However, manually crafting instructions is not only consuming and labor-intensive, but may also incorporate toxic content (Zhao et al.) (2024). Another category of approaches (Wang et al., 2023 Taori et al., 2023
Xu et al. 2023a bj Wang et al. 2024c Sun et al. 2023 Wu et al. (024) focus on prompting L.LMs to generate synthetic instruction datasets, beginning with small set of human-annotater seed in- structions and expanding these through few-shot prompting. However, these methods face a diversity challenge, as few-shot prompting often results in new instructions that are too similar to the original seed questions (Li et al. 2024a). To enhance coverage, some research (Ding et al.) 20233 Li et al. 2024a) summarizes world knowledge and employs it to generate synthetic datasets. We note that
MAGPIE dataset also belongs to the synthetic dataset. However, we leverage the prompt template
without any requirement for seed questions or prompt engineering.
Compared to the above two main categories, alignment data can also be generated by transforming existing data (Wang al. [022) [ann et al.,, [022] Gandhi et al., (024).. However, the constrained variety of NLP tasks in these datasets may impede the ability of tuned LLMs to generalize in real-
world scenarios (Li al. 2024a). There are also mixture datasets (e.g.. [Ivison al. 2023 Teknium
2023a: Liu et al. 2024b; Zhou et al. (023aa) that combine or select high-quality instruction data from
various existing open-source source instruction datasets to enhance coverage (Ivison et al. 2023 Teknium
2023a) and/or improve overall performance (Liu et al.) [0224b] Zhou et 2023ap. There are also data synthesis methods focusing on improving reasoning and math abilities Yuue et al.
2023 2024),
which can be further merged with MAGPIE for creating a better mixture of data for instruction tuning.
Training Data Extraction. Language models have the capability to memorize examples from their training datasets, potentially enabling malicious users to extract private information (Brown et al.
2022. Biderman et al. 2023 Carlini et al. 202T). Pioneering work (Krishna et al. 2020 Carlini
al. 2021 Nasr al. 2023) has demonstrated that it is possible to extract private pre-training data
from BERT Devlin et al. 2018)) GPT-2 (Radford et al.] 2018). and ChatGPT (Achiam et al. 2023),
respectively. Yu et al. (2023) propose several techniques including adjusting sampling strategies to better extract training datasets from language models. Recently, Kassem et al. (2024) propose a black- box prompt optimization method that uses attacker L.LM to extract high levels of memorization in a victim LLM. Wang et al.] (2024b) leverage membership inference attack (MIA) to extract fine-tuning datasets from fine-tuned language models. |ai al. (2024) extract the training dataset of production language models via special characters (e.g., structural symbols of JSON files, and # in emails and online posts). Different from the prior work, we aim to create publicly available alignment datasets with minimal human effort by leveraging the remarkable generation capabilities of LLMs, rather than extracting private training data from LLMs.
CONCLUSION
In this paper, we developed scalable method, MAGPIE, to synthesize instruction data for fine-tuning large language models. MAGPIE leveraged the predefined instruction templates of open-weight LLMs and crafted prompt specifying only the role of instruction provider. Given the crafted prompt, the LLM then generated detailed instructions due to their auto-regressive nature. MAGPIE then sent the generated instructions to the L.MM to generate corresponding responses. These pairs of instructions and responses constituted the instruction dataset. We used Llama-3-8B-instruu to label the instruction dataset and developed a filtering technique to select effective data instances for instruction tuning. We fine-tuned the Llama-3-8B base model using the selected data, and demonstrated that the fine- tuned model outperformed those fine-tuned using all baselines. Moreover, our fine-tuned models outperformed the official aligned model, Llama-3-8B- Instruutt which has been instruction-tune and preference-optimizee using more than 10M data instances. This highlighted the quality of the instruction data synthesized by MAGPIE.
10

---

Published as a conference paper at ICLR 2025
LIMITATIONS, DISCUSSIONS, AND ETHICAL CONSIDERATIONS
Limitations and Discussions. MAGPIE-aligned LLMs demonstrate strong performance on instruc- tion following benchmarks compared to the official Llama-3-8B-Instruce However, we observe performance degradation on math and reasoning benchmarks. Although we leverage the techniques described Section t. generate specialized booster reasoning datasets, there is still a performance gap between MAGPIE-aligge aligned LLMs and the official models. Enhancing the reasoning ability of MAGPIE-aligned models presents a promising direction for future research.
Societal Impact and Potential Harmful Consequences. The primary objective of this paper is to develop a scalable method synthesize instruction data to enhance the caaruction-followiin capabilities of LLMs, and thus align them with human values. However, the data generated by MAGPIE may contain harmful instructions and/or responses, which may lead to unsafe behaviors used raw in instruction tuning. Our empirical evaluations indicate that such harmful data instances constitute less than 1 % of the dataset. Our data filtering technique in Appendix can identify and remove these instances, thus mitigating the risk.
ACKNOWLEDGMENT
The research of Z. Xu, F. Jiang, L. Niu, and R. Poovendran is partially supported by the Air Force Office of Scientific Research (AFOSR) under grant FA9550-23-11 1-0208,, the National Science Foundation (NSF) AI Institute for Agent-based Cyber Threat Intelligence and Operation (ACTION) under grant IIS 2229876, and the Office of Naval Research (ONR) under grant N0014-23-1-2386. The research of Choi is partially supported by NSF under grant DMS-2134012, IARPA HIATUS under grant 2022-2207220000, and ONR under grant 400001-24-1-2207. Results presented in this
paper were partially obtained using the Chameleon testbed Keahey al. (020) supported by the National Science Foundation.
This work is supported part by funds provided by the National Science Foundation, Department of Homeland Security, and IBM. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF or its federal agency and industry partners.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally your phone. arXiv preprint arXiv:2404. 14219, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint Xr0: 087744. 2023.
Mohammad Gheshlaghi Azar, Zhaohan Daniel Guo, Bilal Piot, Remi Munos, Mark Rowland, Michal Valko, and Daniele Calandriello. A general theoretical paradigm understand learning from human preferences. 4447 4455. PMLR, 2024.
International Conference on Artificial Intelligence and Statistics, pp.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang, Jian Yang. Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Yang Bai, Ge Pei, Jindong Gu, Yong Yang, and Xingjun Ma. Special characters attack: Toward scalable training data extraction from large language models. arXiv preprint arXiv:2405.05990. 2024.
11

---

Published as a conference paper at ICLR 2025
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stamislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph. Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodss, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, and Jared Kaplan. Training a helpful and harmless assistant with reinforcement learning from human feedback, 2022.
Edward Beeching, Clémentine Fourrier. Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. Open Ilm leader- board. httpss// Chuggingfaces co/ wwtpspppce/pppppp.c..ee 1m- www.lleedrrrd...oooooooooooooood 11m leaderboard, 2023.
Stella Biderman, Usvsn Prashanth, Lintang Sutawika, Hailey Schoelkopf, Quentin Anthony, Shivan- shu Purohit, and Edward Raff. Emergent and predictable memorization in large language models. Advances in Neural Information Processing Systems, 36, 2023.
Hannah Brown, Katherine Lee, Fatemehsadat Mireshghallah, Reza Shokri, and Florian Tramèr. What does it mean for language model to preserve privacy? In Proceedings of the 2022 ACM Conference on Fairness, Accountability and Transparency, pp. 2280-2292, 2022.
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert- Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, al. Extracting training data from large language models. In 30th USENIX Security Symposium (USENIX Security 21), pp. 2633-26531 2021.
Jiuhai Chen, Rifaa Qadri, Yuxin Wen, Neel Jain, John Kirchenbauer, Tianyi Zhou, and Tom Gold- stein. Genqa: Generating millions of instructions from handful of prompts. arXiv preprint arXiv:2406. 10323, 2024.
Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srinivasan, Tianyi Zhou, Heng Huang, et al. Alpagasus: Training better alpaca with fewer data. arXiv preprint arXiv:2307.08700 2023.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.0333 2021.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* * chatgpt quality, March 2023. URL httpps
//imsys. hrwpsyss.rrg/blg/222233333 30000000-0--vaa///---
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 18Xiv:1803.088 1803005477 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, al. Training verifiers solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Wei Zhu, Yuan Ni, Guotong Xie, Zhiyuan Liu, and Maosong Sun. Ultrafeedback: Boosting language models with high-quality feedback. arXiv preprint arXiv:2310.01377, 2023.
Databricks. Databricks dolly- 2023. URL
databricks betaabrr.ckk/ddttbbriikkk.
Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference-free evaluations of generated text. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 19960-10977, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguis- tics. doi: 10. 30653//1/2022.mmllpp 753. URL https: h..tps://aclantholooyy.. ozgg///2222olcyyro22222221
lemn p-lmp-mmm-- ma 753
12

---

Published as a conference paper at ICLR 2025
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.044 0430004405, 2018.
Ming-Wei
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou. Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint arXiv:2305.14233, 2023.
Matthijs Douze, Alexandr Guzhva, Chengqi Deng. Jeff Johnson, Gergely Szilvasy, Mazaré, Maria Lomeli, Lucas Hosseini, and Hervé Jégou. The faiss library. 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama herd of models. arXiv preprint arXiv:2407.21783, 2024.
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controlled alpacaeval: A simple way debias automatic evaluators, arXiv preprint FrxXxi:24440044755 04475, 2024.
Kawin Ethayarajh, Winnie Xu, Niklas Muennighoff, Dan Jurafsky, and Douwe Kiela. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.01300 2024.
Saumya Gandhi, Ritu Gala, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig. Better synthetic data by retrieving and transforming existing datasets. arXiv preprint 2YXiv:2404.1331,, 2024.
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang. Laria Reynolds. Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. A framework for few-shot language model evaluation, 07 2024. URL https:// http:///eeooo.coom wttp:///o//eoooe/eee0000000
Aryo Pradipta Gema, Joshua Ong Jun Leang, Giwon Hong, Alessio Devoto, Alberto Carlo Maria Mancino, Rohit Saxena, Xuanli He, Yu Zhao, Xiaotang Du, Mohammad Reza Ghasemi Madani, et al. Are we done with mmlu? arXiv preprint 2rXiv:2406.00422 2024.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprins arXiv:2009.03300 030000 2020.
Jiwoo Hong, Noah Lee, and James Thome Reference-free monolithic preference optimization with odds ratio. arXiv preprint arXiv:2403.0769) 2024.
Yue Huang, Lichao Sun, Haoran Wang. Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang. Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing, Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitiik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jicyu Zhao, Jiliang Tang, Jindong Wang. Joaquin Vanschoren, John Mitchell, Kai Shu. Kaidi Xu, Kai-Wei Chang, Lifang He. Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin-Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Yang Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang, Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. TrustIlm: Trustworthiness in large language models. In Forty-first International Conference on Machine Learning, 2024. URL htttpp:://opnneeveete http://ewwwee..ttteeeeeeeeeeeeee forum? wtwp:ummmddbbwuoOwwwwwww
Hamish Ivison, Yizhong Wang. Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi. Joel Jang, David Wadden, Noah A Smith, Iz Beltagy, et al. Camels in a changing climate: Enhancing Im adaptation with tulu 2. arXiv preprint arXiv:2311. 2023.
Aly M Kassem, Omar Mahmoud, Niloofar Mireshghallah, Hyunwoo Kim, Yulia Tsvetkov, Yejin Choi, Sherif Saad, and Santu Rana. Alpaca against vicuna: Using Ilms to uncover memorization of Imss. arXiv preprint 20l::204004800 2024.
13

---

Published as a conference paper at ICLR 2025
Kate Keahey, Jason Anderson, Zhuo Zhen, Pierre Riteau, Paul Ruth, Dan Stanzione, Mert Cevik, Jacob Colleran, Haryadi Gunawi, Cody Hammock, Joe Mambretti, Alexander Barnes, François Halbach, Alex Rocha, and Joe Stubbs. Lessons learned from the chameleon testbed. In Proceedings of the 2020 USENIX Annual Technical Conference (USENIX ATC 20).. USENIX Association, July 2020.
Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi Rui Tam, Keith Stevens, Abdullah Barhoum, Duc Nguyen, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick. Openassistant conversations democratizing large language model alignment. In Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 47669-4768 Curran Associates, Inc., 2023. oc paperr files//e//// httpaapprrrrlleeeeeee/////0///
URL http:::/ppooceeddngs..eeerrrrrre
99 Datasets and Benchmarks.
pdf
Kalpesh Krishna, Gaurav Singh Tomar, Ankur P. Parikh, Nicolas Papernot, and Mohit lyyer. Thieves sesame street! model extraction of bert-based apis. In International Conference on Learning Representations, 2020. Utt.tttss: //o//httpsss//ppeeeeeiise
5
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, et al. Rewardbench: Evaluating reward models for language modeling. arXiv preprint arXiv:2403. 13787, 2024.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thirteenth international conference on the principles of knowledge representation and reasoning. 2012.
Guohao Li, Hasan Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. Camel: Com- municative agents for" mind" exploration of large language model society. Advances in Neural Information Processing Systems, 36:51991-52000 2023a.
Haoran Li, Qingxiu Dong, Zhengyang Tang, Chaojun Wang, Xingxing Zhang, Haoyang Huang. Shao- han Huang. Xiaolong Huang, Zeqiang Huang, Dongdong Zhang, et al. Synthetic data (almost) from scratch: Generalized instruction tuning for language models. arXiv preprint arXiv:2402. 13064, 2024a.
Tianle Li, Wei-Lin Chiang. Evan Frick, Lisa Dunlap, Banghua Zhu, Joseph E. Gonzalez, and Ion Stoica. From live data to high-quality benchmarks: The arena-hard pipeline, April 2024b. URL
https:// htttps://lmsys... http:///mmmsss.orrol////////////
04 ott.:///ooo222204400000---eerrrr
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-follovinn models. https:: / / thub wttp:/tto/otooeeeeeee lacisssullbbbbbe latp:laaca_eeaaaaaaaaaaaaaaaaaaa 2023b.
Bill Yuchen Lin, Abhilasha Ravichander, Ximing Lu, Nouha Dziri, Melanie Sclar, Khyathi Chandu, Chandra Bhagavatula, and Yejin Choi. The unlocking spell on base llms: Rethinking alignment via in-context learning, arXiv preprint arXiv:2312.01552 2023.
Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. Wildbench: Benchmarking language models with challenging tasks from real users in the wild, 2024. URL htttp:://huggingfaee.. co/ h
spaces/ wttpees//lleaaaiianaa
Stephanic Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Mcasuring how models mimic human falsehoods. arXiv preprini arXiv:2109.07955 2021.
14

---

Published as a conference paper at ICLR 2025
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang, Jinmeng Rao, Steven Zheng, Daiyi Peng. Diyi Yang, Denny Zhou, et al. Best practices and lessons learned on synthetic data for language models. arXiv preprint arXivv2404.07503, 2024a.
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. The Twelfth International Conference Learning Representations, 2024b. URL / net/forum 3tt.-ooouuiiiid-3311 TTdee
Yu Meng, Mengzhou Xia, and Danqi Chen. Simpo: Simple preference optimization with a reference- free reward. arXiv preprint arXiv:2405.14734 2024.
Meta. Llama 3. https: //atps://ai.. h cot.e/a..co//oo////oo//////o het//meeaaaeee 11 -amamm/// 2024.
Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher Choquette-Choo, Eric Wallace, Florian Tramèr, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:3311.10035
2023.
OpenAL. Tiktoken.
2024.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea
Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference Neural Information Processing Systems, 2023. URL https
/pppneeiieww..... net/
www///eetoo/roee fmr 2id Het.---HHSSSSSS IIPASIX
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak. De- bajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj. Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022. URLp:ttss:/ http://w.e.eoo...ooooeeeeeeee
Sharath Turuvekere Sreenivas, Saurav Muralidharan, Raviraj Joshi, Marcin Chochowski, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro, Jan Kautz, and Pavlo Molchanov. Llm pruning and distillation in practice: The minitron approach. arXiv preprint arXiv:2408. 11796 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36, 2023.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsumori B. Hashimoto. Stanford alpaca: An instruction-folloviii llama model.
https: tht.tt........./ggg www.tt//aaaeeoo coom/tatss-- 1ab/ stanford_dd aaapoorddaaaaaaaaaaaaaaappppaaaa 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, et al. Gemma 2: Improving open language models practical size. arXiv preprint arXiv:2408.00/18 2024.
Llama Team. Meta llama guard 2. httpa////////ggg.....o////////// http://hw..ccoommmaaaalaaaaaaaap /ooard22MMDELL. BATT___ARD_mmm md. 2024.
blob/main/
Teknium. Openhermes dataset, 2023a. URL http:://hogginnffee.ccc/atttt
Beknium/openherro
Teknium. Openhermee 2.5: An open dataset of synthetic data for generalist Ilm assistants, 2023b.
URL Utl.htttss:/hhhgging httpaaac.w..eeaaaaeeaeat/a//aa/a
15

---

Published as a conference paper at ICLR 2025
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXhv:2307.09288 2023.
Hoang Tran, Chris Glaze, and Braden Hancock. Iterative DPO alignment. Technical report, Snorkel AI, 2023.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008.
Haoxiang Wang, Wei Xiong, Tengyang Xie, Han Zhao, and Tong Zhang. Interpretable preferences via multi-objective reward modeling and mixture-of-experts. In EMNLP, 2024a.
Jeffrey G Wang, Jason Wang, Marvin Li, and Seth Neel. Pandora's 's white-box: Increased training data leakage in open llms. arXiv preprint arXiv:2402.17012 2024b.
Yizhong Wang. Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap. Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro. Tanay Dixit, and Xudong Shen. aaperr-Nauurallnstuuttiosss Generalization via declarative instructions on 1600+ NLP tasks. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods Natural Language Processing, pp- 5085-5109, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics, doi: 10.18653/v1/2000 2022. www.emmeipmmm.....mmmmmmmmmmmmmm
URL https: ac http///caaaantoooog.ooooo /rg/
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannanch Hajishirzi. Self-instruct: Aligning language models with self self-generated instructions. In Proceedings of the sl Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13484-13508, Toronto, Canada, 2023. Association for Computational Linguistics.
Zifeng Wang, Chun-Liang Li, Vincent Perot, Long T Le, Jin Miao, Zizhao Zhang, Chen- Yu Lee, and Tomas Pfister. Codeclm: Aligning language models with tailored synthetic data. arXiv preprint 2578552 2024c.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022. URL http:htps:://// /ttttps://openrevieee wwtp:/r.eeeffuurmmeeee
1d LdZggrzoozzdq
Siyuan Wu, Yue Huang, Chujie Gao. Dongping Chen, Qihui Zhang, Yao Wan, Tianyi Zhou, Xi- angliang Zhang, Jianfeng Gao, Chaowei Xiao, and Lichao Sun. Unigen: A unified framework for textual dataset generation using large language models. ArXiv, abs/2406.1 18966, 2024. URL https:: www.nteetttt..m inmmentticchhh weschocchcac ar... wtwp/oorpssaaooooooooooo ID 227074441
Wei Xiong, Hanze Dong, Chenlu Ye, Ziqi Wang, Han Zhong, Heng Ji, Nan Jiang, and Tong Zhang. Iterative preference learning from human feedback: Bridging theory and practice for rihf under kl-constraint, 2024.
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. WizardIm: Empowering large language models to follow complex instructions. arXiv preprint ar044 12244, 2023a.
Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. Baize: with parameter-efficient tuning self-chat data. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6268-6278, Singapore, December 2023b. Association for Computational Linguis- URL hitps: htttps://acaattaaett http:///aaaatttty....ooo org/2023
An open-source chat model
tics. doi: 10. 18.18/631lll http0/2eemmlpmmmmiiiia lemn 1 p- i 385
16

---

Published as a conference paper at ICLR 2025
An Yang. Baosong Yang. Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li. Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024a.
An Yang. Baosong Yang, Beichen Zhang. Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingieng Xue, Pei Zhang, Qin Zhu, Rui Men. Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren. Yang Fan. Yang Su, Yichang Zhang. Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. arXiv preprint 2024b.
Da Yin, Xiao Liu, Fan Yin, Ming Zhong, Hritik Bansal, Jiawei Han, and Kai-Wei Chang. Dynosaur: A dynamic growth paradigm for instruction-tumne tunnnng data curation. arXiv preprint ar arXiv:2305.14327, 2023.
Weichen Yu, Tianyu Pang, Qian Liu, Chao Du, Bingyi Kang. Yan Huang, Min Lin, and Shuicheng Yan. Bag of tricks for training data extraction from language models. In International Conference on Machine Learning, pp. 40306-40320. PMIR, 2023.
Zhaojian Yu, Xin Zhang, Ning Shang, Yangyu Huang, Can Xu, Yishujie Zhao, Wenxiang Hu, and Qiufeng Yin. Wavecoder: Widespread and versatile enhancement for code large language models by instruction tuning. In Proceedings of the 62nd Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). pp. 5140-5153, 2024.
Xiang Yue, Xingwei Qu. Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning. ArXiv, abs/2309.05653, 2023. URL https:ts:://att 261696597
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web, 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 19Xiv:1005007830 2019.
Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi. and Yuntian Deng, Wildchat: Im chatGPT interaction logs in the wild. In The Twelfth International Conference on Learning Representations, 2024.
net. feee:/forum?id=BB Bal2d=i11112221b
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging -uuM-es-a-judge with MT-bench and chatbot arena. In Thirty-sevnnth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https http://owooomiiiccooooooo.oo....
/openreview
Lianmin Zheng, We Chiang, Ying Sheng. Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric Xing, Joseph F. Gonzalez, Ion Stoica, and Hao Zhang. LMSYS- chat-1m: large-scale real-world LLM conversation dataset. In The Twelfth International Conference on Learning Representations, 2024. URL httpt:tt////mm httl:tttt///opeeeeeeee /ttpn//v////eeeeeeeeemmee wttpi/ewwwweeeeeeeeeeeeeeeeeeeee net forum?
ld=BofDKxfwtt
Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao. Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36, 2023a.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-followin evaluation for large language models. arXiv preprint arXiv:2311.07911 2023b.
Qihao Zhu, Daya Guo, Zhihong Shao, Dejian Yang, Peiyi Wang, Runxin Xu, Y Wu, Yukun Li, Huazuo Gao, Shirong Ma. et al. Deepseck-coder-. Breaking the barrier of clored-source models in code intelligence. arXiv preprint arXiv:2406.1 1931, 2024.
17

---

Published as a conference paper at ICLR 2025
A STATISTICS OF INSTRUCTION DATASETS GENERATED BY MAGPIE COMPARED TO OTHER INSTRUCTION DATASETS.
MAGPIE can be readily deployed to state-of-the-ar open-weigg model families, including but not limited to Llama-3 (Meta 2024), Llama-3.14 1/3.3 (Dubey et al. 2024). Qwen2 (Yang et 2024a).
Qwen2.5 Yang et al. 2024b), Gemma-2 (Team et al. 2024). and Phi-3 (Abdin et al. 2024 models.
In what follows, we compare datasets generated by MAGPIE with the above model families compared to other state-of-the-ar instruction datasets. The MAGPIE dataset family encompasses over 13.4 million diverse and high-quality instructions and corresponding responses generated from state-of- the-art open-source models. This corpus represents the largest alignment dataset for LLMs that does not rely on human-wrinn written questions or employ complex multi-stage pipelines.
Table 4: Statisties of MAGPIE family compared other instruction datasets. Tokens are counted using the tiktoken library (OpenAl 2024). Links to the MAGPIE datasets provided in the text.
Instruction
Daraset Name
#Convs
#Tums
Human
Response Generator
#Tokens / Tum
TTotal Tokens
Alpuca (Tuor et al 2023
52K
1
Low
text-davinci 003
67.38
3.5M
Synthetic
Evol Instng
ChaGPT GhaGGT
473.33
UltrChaat DDing 21 4024
208K
3.16
Low
376.58
238M
Dally Databricks 2025
15K
1
High
ChaGGT
94,61
1.42M
Human
Share (Zhang 24 al 2025
WildChat 2.h0 et 20.5
652K
112K
2.52
4.79
High
High
GPT:3.5 & GPT-4
ChaGPT
727.09 -818.884
465.38
_306
852M
201M
LMSYS-
ZZenng
High
Mix
260.37::540.0
Delta Liu et al. 2024b
9.5K
22.02
Mix
372.78 1158
74M
Mixture
Openl
20230
243K
Mix
297,86
72M
Tulu V2 Mixiure [Vison et 2025
326K
2.31
Mix
411.9
285M
MAGPIE-Llama-3ae
3M
No
--am--3-8--lsstuee
426.39
1,28B
MAGP 1-3 Damis -T
300K
No
LJama-3-8BB 1-8B-lnstruet
610.80=00.ss
366M
MRGFIE-LIamBS
Llama- 1-nna-3-0B--
[211 de
MAGP (-- --m
300K
No
Llama-3-70B 3-770-Instttd
554,53-1380
333M
MAGPIE
M L-P L- 1-MM0--M
MAGPIE Lumma
300K
IM
No
Llama-3 1-ntt--ntttttct
Llama-3 3.0 Instrrutt
552.53 40
482.35
482M
331M
MAGPIE L Laama P.3-
IM
No
Llama 3. 3-B Instruct
$68.59 _291
569M
3M
No
Qwen2. 78- Instruct
577.87
1,73B
MAPPIE MAdPIE-Qwen.- Qwen. Pro
IM
No
Been2-72B-Insttee
424.87
424M
MATIPI Q-eeee 3-Plo
IM
No
Owen2.5-7BBInattee
693.31
693M
Gemmal.PPP Pro
Gemma-
483.90 80
M AAAIPII-PPiiS- Prii--P Pro
IM
No
mn--3-Medium-latrre
391.38_24:432
391M
B MAGPIELM
Benchmark Scores Small/Medium Language Models (<10B)
Greedy Decoding
Alpaca Eval 2 LC
Arena Hard
WidBench Score
Maggiet BB-Chat
58.18
45.4
44.72
Gemmna
48.54
467
4270
MagoleLN 4B-Chat
40.92
24,8
$2.37
Phk-3 5-ilnnknnttucc
22:31
316
Model
inatruct
23.0
Phi-s Misi-tsisstt
13:23
21.8
MiniCPNS-48
11.08
688
13.00
Figure 5: This figure shows the performance of MAGPIELM-4B- and MAGPIELM-SB-CA compared with baselines. MAGPIELM significantly outperforms baselines of similar sizes.
18

---

Published as a conference paper at ICLR 2025
In this section, discuss the details of our MAGPIELM. To construct the SFT and DPO datasets for aligning MAGPIELM, we select 550K and 200K high-quality instructions, respectively, from the MAGPIE family. These instructions cover diverse categories, ensuring comprehensive training set. We then generate corresponding responses using the Gemma-2-9b-it model (TTeam et al. 2024).
The benchmark performance of both models is demonstrated in Figure 5 Notably, MAGPIELM significantly outperforms other baselines similar model sizes across multiple benchmarks, including Alpaca Eval (Li et al. 2023b), Arena Hard (Li al. 2024b), and Wildbench (Lin al. 2024).
C
FILTER SETUPS
In this section, we explore potential filter configurations for selecting high-quality instructional data for fine-tuning purposes. We provide the following metrics enable users to customize their filtered MAGPIE dataset:
1. Input Length:
The total number of characters in the instructions.
2. Output Length: The total number of characters in the responses.
3. Task Category: The specific category of the instructions. See Appendix D.] for details.
4. Input Quality:
The clarity, specificity, and coherence of the instructions, rated as 'very poor', 'poor', 'average', 'good', and 'excellent'.
5. Input Difficulty: The level of knowledge required to address the task described in the instruction, rated as 'very easy", 'easy', 'medium', 'hard', "very hard'.
6. Minimum Neighbor Distance: The embedding distance to the nearest neighbor. Can be used for filtering out repetitive or similar instances.
7. Reward: Denoted as r* See Section 3 for details. This metric can be used to filter out low-quality responses, such as repetitions or refusals.
8. Reward Difference: Denoted as r
See Section 3 for details.
We provide several off-the-seee aff-the-shelf configurations, as demonstrated in Table 5 We defer the detailed performance analysis of each filter configuration for MAGPIE-Pro to Appendix F.5
Table Different filter configurations we provide. We note that the Output Length filter is applied last. Specifically, this filter selects the & instances of the longest responses. In our experiments, we empirically set 7i 12, and 72
Source Dataset
Pilter Name
#Coevs
I Legtth
Input
Output Lnggth
Tisk Calegory
Input Quality
Input Difficuity
Min Neighbor Distance
Reward
Rewand Difference
MAGGBB-Arr
Filter
300K
Longest
good
medium
300K
Longest
average
n >-
Filir2
300K
Longest
2 good
2 ewsy
MAGPLE-Pro
Filter3
300K
Longest
>
Filter4
300K
Longest
good
easy
>.0
casy
Filter6
200K
Longesi
50% easy + 50% easy
>.0
>n
D MORE DATASET ANALYSIS
This section provides additional dataset analysis, complementing the discussions in Section Statistics including lengths of instructions and responses are illustrated in Figure
D.1 ADDITIONAL ANALYSIS ON DATASET COVERAGE AND ATTRIBUTES.
Dataset Coverage Measured by T-SNE and UMAP. Figure presents the t-SNE and UMAP plots of MAGPIE, Alpaca, Evol Instruct, and UltraChat. Each t-SNE and UMAP plot is generated by randomly sampling 10,000 instructions from the associated dataset. We observe that the t-SNE and UMAP plot of MAGPIE encompasses the area covered the plots of Alpaca, Evol Instruct, and UltraChat. This suggests that MAGPIE datasets provides a broader or more diverse range of topics, highlighting its extensive coverage across varied themes and subjects.
19

---

Published as a conference paper at ICLR 2025
$0.00
in Longth bokions)
16469
1o 199 140
tollens)
-20
400
-00
2011 2015 45 do to
-60
-40
-20
20
40
60
Alpaca
Evol Instruct
UltraChat
Magpie
2930-
2000
Figure 7: This figure compares the t-SNE plot of MAGPIE-Pro with those of Alpaca, Evol Instruct, and UltraChat, each of
ico wo oo wa 180 imt 100
Length
which sampled with 10,000 instructions. The t-SNE plot of MAGPIE-PrO encompasses the area covered by the other plots, demonstrating the comprehensive coverage of M MAGPE-PPro.
Figure 6: Lengths of instructions and responses in MAGPIE-Air/Po.
UMAP Visualization of Input Embeddings
Alpaca
Evol Instruct
Magpie-Air
10
10
12
14
Figure 8: This figure compares the UMAP plot of MAGPIE-Air with those of Alpaca, Evol Instruct, and UltraChat, each of which is sampled with 10,000 instructions. The UMAP plot of MAGPIE-Ai encompasses the area covered by the other plots, demonstrating the comprehensive coverage of MAGPIE-Air.
20

---

Published as a conference paper at ICLR 2025
Task Categories of MAGPIE-Pro and MAGPIE-Air. Figure 9 tributions for MAGPIE-PE Pro and MAGPIE-AIr, as labeled by L Instrrucc. We observe that the task category distributions of these two datasets are largely similar, however, MAGPIE- Pro exhibits higher percentage creative writing tasks.
illustrates the task category dis-
tading besaging
Editing
Other
Brainstoman
Solin Enisoning
Reasoning
Cresove Writgg
Math
Math
Information Seeking
Information Seeking
Planning
Planning
Advice Seeking
Advice
Creative
Seeking
Writing
(a) Task categories of MAGPIE-Pro.
(b) Task categories of MAGPIE-Air.
Figure This figure visualizes the task category of MAGPIE-Pro and MAGPIE-Air by topic tags.
Topic Diversity of MAGPIE-Pro and MAGPIE-Air. To validate the diversity of generated instruc- tions, conducted additional analysis using Topic Diversity metric from UltraChat (Ding al.) 2023). Our results are summarized in Table The results demonstrate that our generated instructions indeed more diverse in topic compared with other baselines.
Table Comparison of Topic Diversity across Different Synthetic Datasets.
Dataset
Alpaca
Evol Instruct
UltraChat
Magpie-Air
Magpie-
Topic Diversity (L)
0.13
0.09
0.10
0.05
0.06
Visualization of Root Verbs and Their Direct Noun Objects. Figure 10 visualizes the top common root verbs and their direct noun objects of MAGPIE-Air dataset. This indicates the diverse topic coverage MAGPIE-Air.
D.2 ADDITIONAL SAFETY ANALYSIS
Table 7 illustrates the percentage different unsafe categories MAGPIE-AR and MAGPIE-Pres as labeled by
[Team] 2024). We have two key observations. First, the proportion of data containing potentially harmful queries is minimal, with less than 1% for both datasets. Second, majority of unsafe responses fall into the category of specialized advice, which includes responses that may offer specialized financial, medical, legal advice, or suggest that dangerous activities or objects are safe.
Table 7: This table shows the percentage of different unsafe categories of MAGPIE-Air and MAGPIE-
Pro tagged by Llama-Guard-2 Team (2024) model.
Vicker
Nor-Vielen
Sen-Rcairal
Scaui Expletaaiin
Spacislsual
Privazy
Propent
Indieciiiintt: Wespors
Scani Contest
MaGRi--Ar 198%
0/001 %
IMP19
6.00%
0.009
0.00%
0.00%
00189
01005 0.00%%
80099
01889 0.036%
MAGFIE Pro 99.347%
0.001%
0.049%
0.00%%
0.000%
0.446
0,015%
0,074%
0.014%
0L004R
0.019%
21

---

Published as a conference paper at ICLR 2025
friend
dataset
trip
trouble
party
plan
have
write
story
need
nuvel
lerie
help
issue
concepy
Figure 10: This figure demonstrates the top 20 most common root verbs (shown in the inner circle) and their top direct noun objects (shown in the outer circle) within the MAGPIE-Air dataset. This indicates that MAGPIE encompasses broad range of topies.
D.3 ABLATION ANALYSIS ON GENERATION CONFIGURATIONS
44 X357
32M
3295
Aenpp Ooddit Zore
Temteppore
126
2.80
2.20
C00
1p-3
Tops
Top
(a) Average Quality Scores of MAGPIE-Air
(b) Average Difficulty Scores of MAGPIE-Air
Average Minimum Neighbor Distances of MAGPIE-Air
Figure 11: This figure illustrates the impact of varying decoding parameters the quality, difficulty, and diversity of generated instructions. We observe that while higher temperature and top-p values may decrease the overall quality, they tend increase both the difficulty and diversity of the instructions.
Without System
55%
With System
Ablation Analysis Decoding Parameters. We conduct an ablation analysis on the decoding parameters used in generating instruction with MAGPIE. Specifically, temperatures (i.e., 1, 1.1, and 1.2) and top-p values (i.e., 1, 0.995, and 0.99) during Step 1 of MAGPIE. We three metrics, Average Quality Score, Average Difficulty Score and Average Minimum Neighbor Distance to characterize the quality, difficulty, and diversity of instructions using dif- ferent decoding parameters. The Average Quality Score is calculated by averaging the ratings of all data within a specific temperature-top-p pair, on a scale from 1 ('very poor') to ('excellent'). Similarly, the Average Difficulty Score is rated on a scale from ('very easy') ) to ('very hard'). The Average Minimum Neighbor Distance is calculated by averaging the minimum neighbor distances, as defined in Section [3] for all data generated using the same decoding parameters.
41%
three different
25%
15%
eery 200r
posr
sewnion 9955 Pote let
(e)
Compariion Ingal Quality Prumpta
System
43%
30%
20%
1096
very 4a5y
Companioo
Dilliculy
Syaeam Phampts
Figure 12: This figure compares the input quality and difficulty with and without system prompts.
22

---

Published as a conference paper at ICLR 2025
The findings are summarized in Figure 11 slightly decrease the overall quality of instructions, while simultancously increasing the difficulty and remarkably enhancing the diversity of the instructions generated. The selection of these hyper- parameters should be tailored to the user's specific requirements, balancing the trade-offs between quality, difficulty, and diversity.
We observe that higher temperature and top-p values may
Ablation A nnlysis on the System Prompt. Figure 12 compares the use of system prompt compared with not using it in Step of MAGPIE. Since the Llama-3 model does have an official system prompt, we use the default system prompt from Vicuna (Chiang et al. 2023): A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's ss questions. We observe that using a system prompt generally results in a decrease in the overall quality of instructions, and the instructions are easier. Consequently, we recommend not appending system prompts in default settings.
Lama-3
D.4 IMPACT OF ANNOTATING MODELS
Qwen2
200-
We note that LLMs may occasionally favor its own response (Deutsch al. 2022). In what follows, we conduct experiments to evaluate the impact of annotating models when labeling quality and difficulty of the MAGPIE-Air dataset. We used the (rren-2-7B-nnstrveee model (outside the Llama-3 family) to annotate the quality and difficulty of our MAGPIE-Air dataset.
20%
10%
on
1879 poo
peer
evarage good & bocsee
(4)
Staristic Quality Using Offferent Annotators
[ama-]
Qwan2
The statistics are summarized in Figure 13
20%
Our findings show that even when evaluated by Qwen-2-7B- Instruct, the MAGPIE-Air dataset maintains high quality and difficulty, which is even higher than those originally annotated by Llama-3- 8B- Instruct. This suggests that our dataset's 's quality is robust across different annotators.
10%
1ary may
maciim
harpesery had
1b1 Input Mifcuityy Using Different Annotators
Figure 13: the impact of different annotators on evaluating the instruction qual- ity and difficulty.
This figure compares
D.5 CONTAMINATION ANALYSIS
We conduct a contamination analysis of MAGPIE- MAGriE--enerated instructions using the Alpaca Eval and Arena Hard benchmarks. Following (Yu et al. 2024), we use an embedding model to convert both MAGPIE instructions and benchmark questions into embeddings, calculating cosine similarity scores to assess potential overlap.
Our analysis reveals that Arena Hard shows no evidence of data contamination, while some similarities were found between Alpaca Eval 2 and MAGPIE datasets generated by Llama-3, with few entries showing cosine similarity greater than 0.9. Notably, the affected questions account for at most 10 out of 805 benchmark questions. This is expected, Alpaca Eval data is constructed from mixture of existing instruction datasets, many of which contain common questions such as "What's the capital of Australia?", "Can you code?", and "What's your name?" We note that even with this small degree of overlap, MAGPIE consistently outperformed the baselines, demonstrating its robustness and showing no significant impact on benchmark evaluations.
E DETAILED EXPERIMENTAL. SETUPS
E.I EXPERIMENTAL SETUPS FOR GENERATING MAGPIE-AIR AND MAGPIE-PRO
As detailed in Appendix D33 varying decoding parameters Step can significantly influence the quality, difficulty, and diversity of the generated instructions. To optimize the trade-off's among these attributes, we employ diverse decoding parameters for the generation of MAGPIE-Air and MAGPIE-Pro. Table presents the configurations of MAGPIE-Air and MAGPIE-Pro, showcasing how diverse decoding parameters shape each dataset.
We employ greedy decoding to generate responses in Step 2 for MAGPIE-Air and MAGPIE-Pro The intuition that the word with the highest probability is more likely to originate from the model's training dataset.
23

---

Published as a conference paper at ICLR 2025
Table This table demonstrates the configurations of generating instructions of MAGPIE-Air and MAGPIE-Pro datasets with varying decoding parameters.
Decoding Parameters
Dataset
Total #Convs
Temperature
Top-p
#Conys
1.0
1.00
300K
1.0
0.995
300K
1.0
0.990
300K
1.1
1.00
300K
1.1
0.995
300K
1.1
0,990
300K
MAGPIE-Air
3M
1.2
1.00
300K
1.2
0.995
300K
1.2
0.990
300K
1,25
1.00
100K
1,25
0,995
100K
1,25
0.990
100K
1.0
1.00
300K
1.1
0.995
300K
MAGPIE-PrO
1.2
0.995
300K
1.25
0.990
100K
E.2 EXPERIMENTAL SETUPS FOR INSTRUCTION TUNING AND PREFERENCE TUNING
Supervised Fine- Tunnng Hyper-parametert Table 9 demonstrates the detailed supervised fine- tuning hyper-parameters. These experiments were conducted using Axolot
Table This table shows the seper-parameters for supervised fine-tuning.
Hyper-parameter
Value
Learning Rate
2
10 10-5
Number Epochs
2
Number of Devices
4
Per-device Batch Size
1
Gradient Accumulation Steps
8
Effective Batch Size
32
Optimizer
Adamw with [3s = (0.9,).999) and e = 10
Learning Rate Scheduler
cosine
Warmup Steps
100
Max Sequence Length
8192
Preference Tuning Hyper-parameters. Table [ the detailed DPO hyper-parament for aligning Llama-3-8B using MAGPIE-Air-DPO and MAGPIE-Pto-DPO These experiments were conducted using Alignment Handbook
Decoding parameters for evaluation benchmarks. For Arena-Hard (Li et al. 2024b) and Wild- Bench (Lin et al. 2024), we follow its default setting and use greedy decoding for all settings. For AlpacaEval 2 (Li et al. 2023b) which allows the model provider to specify decoding parameters, we also employ greedy decoding in all experiments with a slightly increased repetition penalty (RP= 1.2) to mitigate the potential repetitive outputs during the generation.
24

---

Published as a conference paper at ICLR 2025
Table 10: This table shows the pyperr-parmmetes for direct preference optimization.
Hyper-parameter
Value
Learning Rate
5
10
Number Epochs
1
Number Devices
4
Per-device Batch Size
2
Gradient Accumulation Steps
16
Effective Batch Size
128
Optimizer
Adamw with A3 = (0.9, and 10
Learning Rate Scheduler
cosine
Warmup Ratio
10%
F ADDITIONAL EXPERIMENTAL RESULTS
F.1 PERFORMANCE OF MAGPIE-MT
Table II compares the performance of MAGPIE-Air-MT and MAGPIE-PrO-ME with their respec- tive single-tur turn counterparts. We observe that the multi-ture datasets have enhanced performance, particularly in the Arena-Hard benchmark.
Table 11: This table compares the performance the multi-turn versions, MAGPIE-Air-MY and LAGPIE-ProMME with their single-turn counterparts. All models are instruction-tumee tuned on the Llama- 8B base models.
AlpacaEval 2
Arena-Hard
GPr-4-Tubo 106)
Dataset
LC (%)
WR (%)
SD
LCC (5)
WR (%)
SD
WR (%)
Single-Turn
22.66
23.99
1.24
49.27
50.80
1.44
14.9
MAGPIE-Air
MT
22.98
24.02
1.27
49.63
51.42
1.40
15.5
Single-Turn
25,15
26.50
1.30
50.52
$2.98
1.43
18.9
MAGPIE-Pro
MT
24.21
25.19
1.28
52.92
54.80
1.41
20.4
F.2 COMPARE MAGPIE AND SELF-INSTRUCT USING LLAMA-3-8B-INSS
To compare the performance of MAGPIE and other synthetic dataset generation methods using the same model, we follow the official Self-Instruct /Wang ct al., 2023) setup and generate 100K supervised fine-tuning dataset using Llama-3-8B-Instruct For fair comparison, select the first 100K data samples from the MAGPIE-Air dataset generated by of models fine-tuned with these two datasets is shown in the table 12
The performance
Table 12: This table compares the performance of models fine-tuned using 100K instructionnfollooinng datasets generated by Self-Instruct and MAGPIE. All models are sepervised-fine-tuee the Llama-8B base models. We observe that MAGPIE significantly outperforms Self-Instruct across all benchmarks.
AlpacaEval Llama-3-8B-Instrue
Arena-Hard
GPT-4-Turbo (1106)
Dataset
#Convs
LC (%)
WR (%)
SD
LC (56
WR (%)
SD
WR (%)
MAGPIF-Air-100K
100K
20.17
21.33
1.21
46.82
48.76
1.44
15.7
Self Instruct (Llama-3)
100K
7.21
5.18
17.86
12.73
1.05
4.0
We observe significant performance gap between models fine-tuned with datasets generated by Self- Instruct and our MAGPIE. Our analysis revealed that the instruction format in taaf--nsstuttgeeeratte datasets is predominantly constrained by the patterns defined in the seed instructions, resulting in
25

---

Published as a conference paper at ICLR 2025
lack of diversity. This comparison indicates the novelty of our MAGPIE in generating diverse qugh-quality instructions without any seed questions.
F.3 PERFORMANCE OF DOMAIN-SPECCE SPECIFIC AND MULTILINGUAL MAGPIE DATASETS
Domain Specific Data Evaluation. We choose code data as representative domain-specific data. We generate domain-speciife data using the code instruction system prompt detailed in Appendix Using 1ann2.55-22BInstrrur as data generator, we create 100K synthetic code instructions via MAGPIE. We then fine-tune both Llama-3-8B base and 1-ama---B--Ima dataset. The models are evaluated on HumanEval (Chen et al. 2021). The results shown in Table [3 demonstrate that our MAGPIE- -aGeeeegenerated code dataset effectively enhances Llama-3's performance on code-related tasks, validating MAGPIE'S applicability to domain-specific instruction tuning.
G
models using this
Table 13: Performance Comparison on HumanEval.
Aligmment Setup
Pass @
Pass @ 10
Pass@ 100
Llama- -lama-3-8B--
0.5574
0.7174
0.8049
Llama-3-8B-base + MAGPIE-Code--00K
0.5327
0.7134
0.8293
Llama-3-8B-Instruu + MAGPIE-Code-100K
0.5768
0.7334
0.8232
Multilingual Data Evaluation. We evaluate MAGPIE'S multilingual capabilities using Chinese as our representative language case. Following the method described in Section 2.2, we use Qwen2- 72B-Instruce to generate 200K Chinese synthetic instructions. We then fine-tunes the Llama-3-8B base model with this dataset and evaluated its performance using multilingual MT-Bench. The results are presented Table The results demonstrate that models fine-tuned with our Chinese MAGPIE dataset outperform the official Llama-3-8B --ae-------InntuuC multilingual MT-Bench (zh-cn). This suggests MAGPIE'S s applicability to generate high-quality multilingual datasets.
Table 14: Performance Comparison on Chinese MT-Bench.
Alignment Setup
Zh MT-Bench
Meta-LLamaa-3-8- --nttrru
7.75
Llama-3-8B-bas + MAGPIE-Chinese-200
7.80
Llama-3-8B-base MAGPIE-CChinese-200 + MAGPIE-PrO-MT
7.96
F.4 ABLATION ANALYSIS ON DATA QUANTITY AND QUALITY
In what follows, we compare within the family of datasets generated by MAGPIE in Table 15 These datasets differ in size, deployment of filtering, and models used to generate data. We observe that as the dataset's size increases, the fine-tuned model's performance improves, indicating that data quantity plays critical role in enhancing cnstructiio-followweee capabilities. Furthermore, the model fine-tuned with MAGPIE-PrO- Prro-oOO-FFileeee even higher amounts of raw data. This demonstrates the effectiveness of our filtering technique, and underscores the importance of data quality. Finally, we observe that the models fine-tuned with MAGPIE-PAS consistently outperform those fine-tuned with MAGPIE-Air. The reason is that MAGPIE-PrO is generated by the more capable model, i.e., Llama-3-70B-Ima Instruct.
outperforms those fine-tuned with the same or
F.5 ABLATION ANALYSIS ON FILTER DESIGNS
conduct an ablation analysis on various filter designs within MAGPIE-Pro to assess their impact on the performance of supervised fine-tuned models. The results are presented in Table [6 We observe that different filtering strategies yield optimal performance on different benchmarks, and no single filter consistently achieves the best performance across all benchmarks. Therefore, determining how to select instructional data to enhance the performance in supervised fine-tuning is an interesting topic for future research.
26

---

Published as a conference paper at ICLR 2025
Table 15: This table compares MAGPIE datasets within its family that differ in size, deployment of filtering, and models used to generate data. All models are supervised-fine fine-tuned on the Llama-8B base models.
AlpacaEval 2
Arena-Hard
GPT-4- (1106)
IJama-3-a
LC
WR (%)
SD
LC (%)
WR (9)
SD
WR(S)
300K-Rac -Ro
300K
21.99
21.65
1.21
48.63
48.06
1.42
15.8
MAGPIE-AN
3M Raw
3M
22.
21.09
1.20
50.57
48.40
1.42
16.1
Fo0t-riteered
300K
22.66
23.99
1.24
49.27
50.8
1.44
14.9
300K Raw
300K
21.65
22.19
1.2
49.65
50.84
1.42
15.9
IM-Raw
1M
24.16
23.93
1.25
49.97
50.34
143
16.7
MAGPIE-Pro
100K-Filtered
100K
20.47
24.52
1.25
47.92
52.75
1.43
17.2
200K Fitterd
200K
22.11
26.02
1.26
51.17
56.76
1.41
15.9
300K-Fittred
300K
29.47
1.35
52.12
53.43
1.44
MAGPIE-Air+ MAGPIE-Pro
4M-Raw
24.08
1.26
51.96
$2.08
1.42
15.5
Table 16: This table compares the performance of different filter designs within MAGPIE-Pro. All models are supervised seerviseddfine-ttue fine-tuned on the Llama-88 8B base models.
AlpacaEval Llama-3-8B-Inster
Arena-Hard
GPT:4-Turbo (1106)
Dataset and Filter
L.C (%)
WR (%)
S1>
I.C. (%)
WR (%))
SD
WR (5)
Filter
25.08
29.47
1.35
52.12
53.43
1.44
18.9
Filter 25.15
26.50
1.30
50,52
52.98
1.43
18.9
Filter
23.90
25.21
1.25
51.45
53.64
1.41
18.3
MAGPIE-Pro
Filter
24.20
25.33
52.43
54.34
1.43
Filter 24.85
25.12
1.26
52.12
53.43
1.44
18.4
Filter 6 23.20
28.43
1.26
51.34
$7.29
1.41
17.9
F.6 ABLATION ANALYSIS ON RESPONSE GENERATOR
To investigate the impact of the response generator on the supervised fine-tuning performance using MAGPIE, we conduct an ablation study by replacing the response generator with Qwwnn227B-IInstrutt We note that the performance of Qwen-2-7B- Instruct is comparable to, or slightly weaker than, Llama-3-8B 3-8-----Btut--ttet The results are summarized in Table 17
Yang et al. 2024a) within MAGPIE-Air-300K-TTT
We observe that although there is slight performance degradation, the model fine-tuned using Owen-2-7B-Innstrnn as the response generator still outperforms all baselines, including those using GPT-4 as the response generator. These findings indicate two key points: (1) The success of MAGPIE depends little on the specific response generator used, and (2) the instructions generated by MAGPIE are high quality and diversity.
Table 17: This table compares the impact of different response generators on the model performance.
All models supervised-fine-tume the Llama-8B base models.
AlpacaEval
Arena-Hard
Response Generator
GPT-4-Turbo (1106)
Llama- 3-B-a---uetnntt
LC (%)
WR (%)
SD
LC (%)
WR (%)
SD
WR (%)
Llama-3-8P 3-8--Intruutt
22.66
1.24
49.27
50.80
14.9
Dwwen2-BB-Instruce
15.01
15.60
1.05
41.09
42.07
1.47
13.7
F.7 TRUSTWORTHINESS OF MAGPIE-ALIGNER MODELS
In what follows, conduct experiments to compare MAGPIE model and (lama-3-8B-Insrruue on the TrustLLM benchmark (Huang et all 2024). The results for safety, faimness, ethics, privacy, and robustness summarized in Table
27

---

Published as a conference paper at ICLR 2025
We observe that our supervised seservvsed--ine-tuue fine-tuneddd model slightly underperforms Llama-3-8P 3--8B-Insttuct in terms safety and fairness. However, it outperforms the official instruct model on ethics, privacy, and robustness. Considering that fine-tuned model uses much fewer data samples (300K compared to over 10M), these results again highlight the high quality of data generated by MAGPIE.
Table 18: This table compares the performance of model supervised- seiissedd-iie-tueee ttnn- using MAGPIE Pro- 300K-Filtered and the official I-s-trrsutt on the TrustLLM benchmark (Huang et al. 2024).
TrustLLM
Evaluation/Dataset
-lama-3-BB 3-8B-Instructt
MAGPIE Pro-300K- Filtered
Jailbreak (RtAf)
0.93
0.80
Safety
Misuse (RtAf)
0.85
0.80
Exaggerated Safety (RIA.)
0.54
0.52
Stereotype Recognition (Acet)
0.49
0.40
Stereotype Query Test (RtAf)
1.00
0.99
Fairness
Disparagement Sex (p-valuef)
0.99
0.99
Disparagement Race (p-value")
0.55
0.47
Social Chemistry 101 (Acct)
0.94
0.63
ETHICS (Acet)
0.65
0.69
Ethics
MoralChoice (Acef)
0.97
0.95
MoralChoice (RIAT)
0.97
0.98
Privacy Awareness-Normal (RtA1)
0.33
0.71
Privacy
Privacy Awareness Awareness-Augmeeneed (RtAf)
1.00
0,98
Privacy Leakage (RtAf)
0.66
0.87
AdvGlue (RobustScoret)
0.42
0.58
Robustness
OOD Detection (RIAf)
0.37
0.26
OOD Generalization (FI-Scoref)
0.83
0.84
F.8
IFEVAL EVALUATIONS OF MAGPIE-ALIGNED MODELS AND BASELINES
We compare the models fine-tuned with MAGPIE against baselines on IFEval Zhou et (2023b)
using the IM-Evaluation-HHreeee framework Gao et al. (2024). The results are presented in Table
Our results demonstrate that MAGPIE- geeeeregeneetted datasets achieve comparable prompt-level and instruction-level strict accuracy scores to existing baseline datasets. Moreover, MAGPIE exhibits significantly higher performance in both prompt-level and instruction-level loose accuracy metrics. These findings indicate the high quality of MAGPIF--enerated datasets.
Table 19: This table compares the performance of models fine-tuned using MAGPIE and other baseline datasets on the IFEval benchmark (Zhou et al. 20236).
Alignment Data
prompt_levell level_strict inst_ Jevel__triice
prompt_leve levellt__eeee__ee Joose
inst_level_Jone
Self-Instruct (Llama-3)
0,333
0.465
0.372
0.501
ShareGPT
0.331
0.454
0.372
0.492
Evol Instruct
0.344
0.463
0.377
0.494
OpenHermes
0.340
0.453
0.377
0.488
Tulu V2 Mix
0.338
0.458
0.370
0.499
WildChat
0.372
0.489
0.423
0.538
OpenHermes 2.5
0.381
0.493
0.436
0.536
GenQA
0.307
0.458
0.331
0.484
Ultrachat
0.298
0.421
0.346
0.466
MAGPIE-Air-BOn 1irr-000--BBB -Ra
0,366
0.489
0.477
0,590
MAGPIE-Air-300 Xir-300K-Filtene
0.355
0.484
0.475
0.597
MAGPIE-Air- -MT
0.368
0.496
0.495
0.614
MAGPIE-Pros MAGPIE-Pro-300K
0.338
0.472
0.455
0,582
0.298
0.432
0.401
0,529
Pro-33KK
0.336
0.452
0.455
0.568
28

---

Published as a conference paper at ICLR 2025
F.9 ABLATION ANALYSIS ON THE IMPACT OF REWARD MODELS ON DPO PERFORMANCE
To investigate how the choice of reward model influences DPO performance, we performed an ablation study using reward models: and SatrriXXC/ssfaix-LLaMMA3-3
(Wang et al. (024a) (Xiong et al. 2024). Notably, ArmoRM exhibited stronger performance on RewardBench (Lambert al. 2024). Following the procedure described in Section 2.2. we constructed DPO datasets based on scores from both reward models and evaluated their performance using AlpacaEval2 and Arena Hard benchmarks.
As shown in Table 20 our results demonstrate that employing higher-performinn performing reward model (ArmoRM) yields better-pertee peatte-performing LLM during DPO. This underscores the critical role of reward model quality in enhancing alignment and overall model performance.
Table 20: Comparison of Llama-3-8B DPO-trained models using different reward models. Metrics are AE2 LC, AE2 WR, and AH Score.
Instruction Model
AE2 LC
AE2 WR
All Score
Llama-3-88 --rr-----BB---r---- B----ir-DO-AAAee MI-DPO-AAmoRM
45.48
50.43
35.9
Blama-3-8B-AM Aii---r---------- DD--P--sfaiXC
37.87
45.08
33.1
Llama-3-8B-Pra Pel--Po---O-Ammeete
50.10
53.53
35.7
43.49
50.75
34.2
G PROMPT TEMPLATES
G.1 PROMPT TEMPLATES FOR MAGPIE EXTENSION
This section presents the prompt template used to generate MAGPIE-MT and control instruction tasks, detailed in Figure 14 and Figure 15 respectively.
Prompt for generating MAGPIE-MT
Beegin__o__txx(<<<// htw.hher_illsssseemeeeeeeeeeeeee
You are helpful Al assistant. The user will engage a multi-round conversation with you, asking initial questions and following up with additional related questions. Your goal is to provide thorough, relevant and insightful responses to help the user with their (ireesss<((oodd>>((((sss (ddd<(starthhhader__ddl heeder_id/>sserr/ppppppppeeeeeep Sirdd__hade_ii_ id]>
(espoonse)<eoo__dd><((tttt hhaaartt_haaer__iia hedeeeeddddddddsseeeeeeddd header id|>
Figure 14: Prompt for generating MAGPIE-MT. We take Llama-3 3-8B-Instruct as an example. The placeholder (inttrucioon) and (response) ; are from the first turn.
G.2 PROMPT TEMPLATES FOR EVALUATION
Here, we present the prompt template employed generate task categories, quality, and difficulty, as detailed Figure 16 Figure 17 and Figure [8] respectively. The placeholder input represents the instructions be evaluated.
H EXAMPLE OUTPUTS OF MAGPIE MODELS
In the following Figure 19 and Figure 20
present example outputs generated by the MAGPIE
models and compare them with GPT-4- Turbo (1 (1106), using Alpaca Eval 2 for evaluation.
29

---

Published as a conference paper at ICLR 2025
System Prompt Template
<lbegin of textt)/><//strttteeee header idj=system<jend header id/>
[System Tromppt)(eeo_idd/x(((ttr http://ooo__oooeaaaaaaaaaaerdddr
_dl
System prompt for controlling math instruction tasks
You are AI assistant designed provide helpful, step-by-step guidance solving math problems, The user will ask you wide range of complex mathematical questions. Your purpose is to assist users in understanding mathematical concepts, working through equations, and arriving at the correct solutions.
System prompt for controlling code instruction tasks
You are an Al assistant designed to provide helpful, step-by-step guidance on coding problems. The user will ask you wide range of coding questions. Your purpose is to assist users in understanding coding concepts, working through code, and arriving at the correct solutions.
System prompt for controlling translation tasks
You are an Al assistant designed to provide accurate and contextually appropriate translations. Users will ask you to translate text between various languages. Your purpose is to assist users understanding and conveying meaning across languages, maintaining the original context and nuances.
System prompt for controlling multilingual instruction generation (Japanese Math)
t A17 1 # 1 5 / 2 2
<
< 2 7 ステップ
Figure 15: Prompts for controlling instruction generation tasks. These examples illustrate how to in generating instructions for specific domains: mathematics, coding, translation, and multilingual tasks. To adapt this approach for different instruction tasks, replace the System Pr Prompt placeholder in the System Prompt Template with appropriate domain-specific prompt.
guide
30

---

Published as a conference paper at ICLR 2025
Prompt for generating task categories
# Instruction
Please label the task tags for the user query.
## User Query "(input)""
## Tagging the user input
Please label the task tags for the user query. You will need to analyze the user query and select the most relevant task tag from the list below.
all task tags -
"Information seeking", # Users ask for specific information or facts about various topics. "Reasoning"; # Queries require logical thinking, problem-solving processing of complex ideas.
"Planning", # Users need assistance in creating plans or strategies for activities and projects.
"Editing", # Involves editing, rephrasing, proofreading, or other tasks related to the composition of general written content.
"Coding Debugging", Users seek help with writing, reviewing, fixing code programming.
"Math", Queries related to mathematical concepts, problems, and calculations.
"Role playing", # Users engage in scenarios requiring ChatGPT to adopt character persona.
"Data analysis", Requests involve interpreting data, statistics, or performing analytical tasks.
"Creative writing", # Users seek assistance with crafting stories, poems, or other creative texts.
"Advice seeking", Users ask for recommendations or guidance on various personal or professional issues.
"Brainstormingg, # Involves generating ideas, creative thinking, or exploring possibilities. "Others" # Any queries that do not fit into the above categories are of miscellaneous nature.
### Output Format:
Note that you only select a single primary tag. Other applicable tags can be added to the list of other tags. Now, please output your tags below in json format by filling the placeholders <...t
HI
"primary_tag" "<primary tag>", "other_tags" ["<tag 1>", "<tag 2>",
B
Figure 16: Prompt for generating task categories
31

---

Published as a conference paper at ICLR 2025
Prompt for generating quality of instructions
# Instruction
You need to rate the quality of the user query based on its clarity, specificity, and coherence. The rating scale as follows:
very poor: The query is unclear, vague, or incoherent. It lacks essential information and context.
poor:
The query is somewhat unclear lacks important details. It requires significant clarification.
average:
The query is moderately clear and specific. may require some additional information for complete understanding.
-good: The query is clear, specific, and mostly well-formed. provides sufficient context for understanding the user's intent.
excellent: The query is very clear, specific, and well-articulated. It contains all the necessary information and context for providing a comprehensive response.
## User Query "(input)""
## Output Format
Given the user query, you first need to give assessment, highlighting the strengths and/or weaknesses of the user query. Then, you need output rating from very poor excellent by filling in the placeholders in [...]:
H[
"explanation": "[...]". qualityuality":: "[very por//poo/avenggggo///ecccllll"""
B
Figure 17: Prompt for generating quality of instructions
Prompt for generating difficulty of instructions
# Instruction
You first need to identify the given user intent and then label the difficulty level of the user query based on the content of the user query.
## User Query "(input)""
## Output Format
Given the user query, in your output, you first need to identify the user intent and the knowledge needed to solve the task in the user query. Then, rate the difficulty level of the user query 'very easy', 'easy', 'medium', "hard', 'very hard'.
Now, please output the user intent and difficulty level below in a json format by filling the placeholders in [...]:
[[
"intent": "The user wants to [....].
"knowledge": "To solve this problem, the models need to know [....]", "difficulty": "[very easyllasylmeedum/hrrdeeyyy hard]"
B
Figure Prompt for generating difficulty of instructions
32

---

Published as a conference paper at ICLR 2025
Question
Hi, I'm trying to solve crossword puzzle, but I've never done one of these before. Can you help me out?
GPT-4-Turbo(1 106)
Absolutely! I'd be delighted help you with your erossword puzzle. Just tell the clues and the number of letters needed for each answer (and any letters you may have already filled in), and I'll do my best to help you find the solutions. If you have any specific questions about how to approach solving crossword puzzles in general, feel free to ask those as well!
MAGPIE- ProDDP
=*Crossword Crusader's Guide
Welcome the world of crosswords! We're thrilled you navigating your first puzzle. This primer will the basics, lips, and get you
## Understandiin the Grid--
crossward puzzle consists grid black squares (abstacies) and squares (answer spacos).
The grid is divided into rows and columns, usually labeled with numbers. Each answer spans one more squares horizontally (across) or vertically (down).
Kay Concepts
1. -=Cluss:: Those are the instrcctinns to fill the answers. Clues are usually writton above (across) or to the left (down) of the answer
..Across clues- are read
right, sentence.
--Down cluns. are read from top to bottom, like column.
2. wAnswer Langthw:: The number of squares an answer accupies is indicated by digitts) next to the clue. For example, "5-letter answer"
"Down: 7*
.:Wardplay-:: Crosswards
employ wordalay, -nnngramm.::: Hearrange letters to form the answar.
moans might uso:
.Homophoness.: Words with simlla sounds (6.g "bank"" and "bank").
--Double Defintions-:: Two saparate definitions
..Prefixes/Sutlleee Clues might ask
with specitic prefix suffix.
## ..Tips for Solving..
your way up. 3. .Use the grid.: Pay attention to the grid's structure. Look petterns, doesn't contict with other Don't be atraid guess..: you're unsure, try few possibilties,
1. --Start easy clues.:: Bogin
answers (3-5 lotters)
2. .iFil abvicus answerg..: Look for words that are easily recognizable or have only one possible anawer.
and blocked areas help guide answers.
4. ..Chark for intorsections-: When
intersections.
can always erase try again.
6. .Use the "X" mathod=:: When unsure, write an 'X' the square to indicate possible answar. This halps you keep track of possibilties.
7. Take broaks..:: Crosswords can be mentally taxing. Take broak to rofresh your mind.
Conssutt
you're stuck,
special
## ..Additooo onal Resources
Crossword XWordinto, Cruciverb.
Watch video tutorials or You channels dedicated crossworde, like Croesword Comner or The Croesword Guy,
## ..GGod Lucki..
Now, go ahead and dive into your crossword puzzle. Romember, the more you practice, the better you'll become. Have fun, and happy solving!
Figure Comparison between outputs of MAGPIE-Pro-DPO and GPT-4-Turbo (1 106) (Example 1).
33

---

Published as a conference paper at ICLR 2025
Question
Do you know why turkeys became the official food of thanksgiving?
GPT-4-Turbo( 1106)
The tradition of eating turkey for Thankagiving rooted the history of the Untee States. There are low reasoons vhy turkey becamo the conterpicce the Tharksgiving moal:
1. wHistorical Accountsre:
Although there no definitve rocord that turkey was caten at the very first Thanksgiving in 1621, when the Pligrims colebratod large widdow likely a converient choice such gathorings.
the
Native Americans, It large group poople
America,
2. Practicality..:
Turkoys abundant in North America for their egge, and cows, which were more useful for their milk, turkeys were otten kept for their meat and could be slaughtered without signficant loss to the homestead.
practical choice for celebratory moal. Unikk chickens, which
valuablo
--Iffuence of Writings-::
Josepha prominent and editor the 19th century, campaigned establishment Thankagiving as national holiday, She published recipes and descrigtions Thankssgvving moals that included turkey, which may have infucnced the tradition Americans sought to replicate the ideal least she depicted.
4. Presidential Tradition.::
In 1863, during the American CMil War. President Abraham Lincoin prociaimed Thanksglving a national hollday to be celebrated on the last Thursday in November. As the holiday became more formalzed, so did the menu, with turkey the center.
5. -Symbolic and Practical Choloa..:
By the 19th and early 20th conturies, turkay had become the symbalic and practical choice for Thanksgiving was large enough to feed crowd, reletively inexpensive, and not something that Americans would eat day, which made it special the
As result of these factors, turkey became entrenched as the traditional main dish of Thankagiving celebrations the United States.
MAGPIE-PrO-DPO
delightful queetion historical background. While exclusively single turkey's Tharkegiving fascinating blend of cultural, economic, and symbolic factors, Here's the story:
NP --arly American Tradition 17000--
colonial England, harvest festivais were common, celebrating bounty or "thankagiving" featured roasted meats, bread, and vegetables. Widdoww like ducks, geese, and even swans, were popular choices. Turkeys, although present, weren't the primary dish.
gatherings. often called "harvest
HP The Pigrrms Influence (1621)..
The famous Pilgrim Ieggri--nndian harvest least Plymouth Coleny 1621, often mislabeled the "first Tharkegiving, played significant role. along with com,
It's likely wildtowl, including turkeys, Pllgrims, being English, were familiar with turkey consumption, which was already staple their homeland.
fully
local foods.
#P -Economic and Ecological Factors (1700s-18) 1600s).
the American colonies grew. so the turkey population. The bird thrived in the New World's forests, where it was abundant and essy to hunt. This led to:
gatherings. 2. Dietary shift-:; As the American diet transltione from primarlyy fish and game to more domesticated animals, turkeys became a convenient, familir, and nutritiuss alternative beef and park.
::vaallbbllty..: Turkeys
relatively inexpensive, making
practical
3. EEooooiccal edvantage--: Turkeys well-suited the New World's climate could be raleed small farms, reducing dependence imported meats.
XW --SSyboliam and Cultural Evolution (1800s-1900s)--
turkey's a popularity
lamily getherings
grow.
associated
1. .-Abuddance and Plenty--: Turkeys represented the harvest's bounty and the nation's prosperity.
2. Family Tradition-:: The turkey became symbol togethernese and shared meals, which essential American culture. --National ldentity..:
United States grew, the tur key's association with Thanksgi ving helped and shared heritage.
national dentily
## ThThe Official Food Thanksgiving (1930s)--
In the 1980s, the turkey industry. which had been growing steadily, began promote the bird as the quintessentil Thanksghving food. This effort was led by the National Turkey Federation (now the National Turkey Federation of America) and supported by the poultry industry. larmers, and food manufacturers.
In 1947, the National Turkey Federation officially declared the turkey the 'traditional food Thanksgiving " This dectaration was not government decree, but rather cultural recognition of the bird's widespread acceptance and symmbolic significance.
The turkey became the official food Thankagiving due to combination of historical, aconomic, ecological, and cultural factors. Over time, the bird's availabiit,, familianity symbolism the perfect choice for national harvest celebration. Today, the turkey integral part American Thanksgiving traditons. Happy Thanksgiving!
Figure 20: Comparison between outputs of MAGPIE-Pro-DPO and GPT-4-Turbo (1 (110) (Example 2).
34

---

Published as a conference paper at ICLR 2025
1 MAGPIE EXAMPLES
In this section, we demonstrate several representative instructions generated by MAGPIE with different task categories. The following instructions are generated by the Llama-3-70B--nsttmel model.
Information Seeking
A few days ago, I was at a restaurant and got a cup of coffee. However, when went to take sip, I realized it waited, noticed that coffee is actually two different colors. The part that was closest to the surface of the coffee is a lighter color, and the part that deeper is darker color. Have you ever observed this phenomenon before?
little too hot. So, decided to let it cool down for few minutes. As I
Reasoning
You have 3 switches but they are not labeled. Each switch corresponds to one of three light bulbs in room. Each light bulb is either many times as you want, but you can only enter the room one time to observe the bulbs. How can you figure which switch corresponds to which light bulb?
the switches and off off. You
Planning
You are the Founder of Financial Planning Company, As professional financial advisor, you are scheduled to meet a new client tomorrow. Specifically, what are you planning to do to prepare for this meeting?
Editing
What is the best way re-write the sentence: "We call this the 'core' product the "core' offering" using proper quotation marks and avoiding the word "this"?
Coding & Debugging
Write a Python program that calculates the total cost customer's order. The program should ask for the customer's 's name, the number of items they want to purchase, and the price each item. It should then calculate the total cost by multiplying the number of items by the price of each item and adding 8% sales tax. The program should display the customer's name, the number of items, the price of each item, and the total cost, including sales tax.
Math
In the following problem, please use integers to solve it. A water tank has 1000 I. of water. On the first day, 1/5 of the water was drained. On the second day, 3/10 the remaining water was drained. On the third day, 2/5 of the remaining water was drained. On the fourth day, 3/4 the remaining water was drained. How many liters of water are left after the fourth day?
Role Playing
this game, you will be the host, and I will be the contestant. You will ask me a series of questions, and I will try to answer them correctly. The questions will be multiple choice, and will have 25% chance of getting the correct answer if I just randomly guess. However, I am clever contestant, and will try to use logic and reasoning to increase my chances of getting the correct answer.
35

---

Published as a conference paper at ICLR 2025
Data Analysis
The personnel manager company tasked with finding the average salary of new hires. She has collected data on the salaries of 13 new hires. She wants to know if there is a statistical difference between the average salary of new hires and the national average salary. The national average salary is $45,000. The sample of new hires has mean salary of $42,800 and a standard deviation of $4,200.
Creative Writing
Write a paragraph about mythical creature that you created. The creature is small, no larger than a house cat. It has shimmering scales that reflect light, and can emit soft, pulsing glow from its body. has large, round eyes that seem to see right through you, but with a gentle kindness. It has a soft, melodious voice, and can communicate with humans through a form of telepathy.
Advice Seeking
How do you handle stress and overwhelm?
Brainstorming
Can you give me some ideas for a spontaneous, fun and memorable birthday celebration for my partner?
Others
What does "sdrawkcaB" mean?
MAGPIE can also generate domain-specific instructions using models that are tailored to particular fields, as mentioned in Section 2.2 The following instructions are generated by DeepSeek-Coder-V. (Yang et al. 2024a), respectively.
(Zuu al. 2024) and
DeepSeek-Coder- V2 (Code Instruction)
You are given list of emails. You need to write Python function that returns the domain, excluding the @ symbol, for each email.
1ean2--Math-7B-Insttte (Math Instruction)
A rectangle with length 12 units and width 8 units is scaled by factor of 2 to form new rectangle, Determine the dimensions of the new rectangle and calculate its area. Compare the area of the new rectangle the area of the original rectangle.
We note that MAGPIE's capabilities extend beyond generating English datasets to producing diverse multilingual datasets. The following instructions are generated by the Qwen2-72B Instrrctt model.
Chinese
M
W SEE FF (M0+, E.
German
Das Debate-Systen 'Oxford-Oberhann wird bei ersten Auseinanderrsetzungen verwendet.
Bitte erklären sie, wie dieses System funktioniert.
36

---

Published as a conference paper at ICLR 2025
Spanish
Según la encuesta anual de satisfacción al cliente que acabamos de realizar, parece que la satisfacción general de los clientes con nuestro rendimiento ha disminuido. y ?Podrías preparar una presentación detallada para la reunión del lunes que analice los resultados, identifique las áreas problemáticas y proporeione posibles soluciones basadas en los datos recogidos?
Portuguese
Ho comprato una nuova inchiostriera sulla quale è presente la scritta "Non manipolare". Cosa signidica?
Italian
Crie um exemplo de uma conversa entre dois personagens, um MC de hip hop e um pianista clássico, discutindo sobre seus estilos favoritos de música.
37


Reviews:
[{"review_id": "ddPXaCI68K", "reviewer": "Reviewer_aNEq", "summary": "This work proposes Magpie, which is a way to prompt aligned LLMs to generate alignment datasets. Unlike most previous studies in this area (LLM-generated alignment data), Magpie removes the need of a seed set of instructions and heavy prompt engineering efforts. Instead of prompting aligned LLMs with an meaningful instruction, Magpie directly puts the \"pre-instruction\" token, i.e. one of the formatting delimiter tokens, to LLMs and let the LLMs generate single-turn instructions \"freely\". With limited extra prompt engineering efforts and an extra reward model, the method can be extended to multi-turn, multi-lingual, multi-domain data, and DPO cases (preference data). The experimental results indicate that datasets generated through Magpie are of high quality, yielding superior performances on AlpacaEval2, ArenaHard, and WildBench, as well as comparable performances on general benchmarks like MMLU, GSM8K, etc.", "questions": "1. It is not clear how the model used to generate data affects the final performance. The experimental results show that MAGPIE-Pro-300K-Raw has limited improvements over MAGPIE-Air-300K-Raw. This is counterintuitive and needs more insights.\n\n2. Following up above, one possibility is that the base model is the same (llama-8b). From this point of view, the relationship between the model to be aligned and the model to generate data is crucial. More analysis is required.\n\n3. We can also see that MAGPIE-Pro-300K-Filtered show much better performances when used on Llama than QWen. Of course, this could mean Llama base model is better, but could also be limitations of the proposed method. The proposed method essentially relies on the model itself to control the diversity of generated instructions, which is controlled by the pre-training data. If the model to be aligned was trained with the same pre-training data, we can intuitively expect better performances and less hallucination. If that is the case, it becomes kind of a chicken-egg thing, limiting the usefulness to the proposed method.\n\n4. Why is a separate two-step process necessary? The aligned model should be able to generate instruction and response in one pass?\n\n5. The DPO results are surprisingly good. How does the reward model's quality affect the final performances?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method is simple yet effective, according to the experimental results. In particular, the generated DPO data show a significant performance boost.\n\n- The ablation study is comprehensive, considering different settings/datasets/baselines as well as different aspects like domain-specific capabilities and safety, though the result analysis lacks depth.", "weaknesses": "- The presentation can be improved by putting more experimental results in the main context instead of the appendix. The proposed method is simple and easy to understand, similar to most prompt engineering related studies. The experimental results along with analysis and hypothesis test are the most important. For example, Section 3 is less important than Appendix F in terms of demonstrating the method's pros and cons.\n\n- The experiments are well designed but lack deep analysis, which leaves many questions unanswered and makes it hard for other researchers to understand the method's best use cases and limitations. Please refer to Questions for details.\n\n- For those most understanding results, e.g. surpassing GPT-4-Turbo(1106) on AlpacaEval 2, qualitative results are needed, i.e. random samples of actual generations for those benchmarks from all models in comparison. It is known that LLM-based judge could be biased and misleading.\n\n- It would be better to add more alignment benchmarks like IFEval.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BbWLyPcuyE", "reviewer": "Reviewer_D1Jb", "summary": "The paper introduces a new dataset for instruction fine-tuning and preference optimization called Magpie. The dataset is curated by prompting either LLama-3-70B-Instruct or LLama-3-8B-Instruct with the pre-query template or user tags and relying on the next token prediction ability of the model to elicit instructions. The instructions collected are then used to gather responses from LLMs. To generate instruction & responses that are multiturn or relevant to a certain task, the authors add relevant system prompts to steer the model ouptuts. Experiments across multiple baseline datasets, base models and comparison on multiple benchmark shows that effectiveness of Magpie dataset. The paper also has qualitative analysis highlighting various attributes of the dataset.", "questions": "Based on the Weaknesses, the questions are as follows:\n1. How does the performance of Magpie look on multiturn benchamark? \n2. Does Magpie generation process even scale for longer turns?\n3. Why there are limited improvements in Table 3 in comparison to massive improvements in Table 2?\n4. As the data generation process is un-contolled, contamination analysis in Magpie datasets would be helpful.\n5. How effective are system prompts for providing some control over the generation process? Is there any way to measure that? Are there any domain specific results?\n6. How the generation process ensures diversity of instructions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper introduces a simple & cost effective method of generating alignment data at scale, without relying on closed models or expensive human annotations.\n2. Results on Alpaca Eval & Arena hard show significant improvements compared to other baseline datasets.\n3. The paper presents an exhaustive quantative & qualitative experiments on question difficulty, safety, task categories, etc to highlight the attributes in Magpie.\n4. The authors also experiments with various filtering configuration to improve the quality of final set of instructions.", "weaknesses": "1. The Magpie dataset is not conversational in nature. The MT variant of the dataset has at maximum 2 turns. In contrast, datasets like Ultrachat, Open Assistant and WildChat are multi-turn with maximum upto 7-8 turns. Also, the paper has no experiments or analysis on conversational benchmark and multi-turn nature of Magpie in comparison to existing datasets.\n2. The paper has very limited improvements on OpenLLM leaderboard in Table 3 when compared to improvements in Table 2. Why is this the case, more dicussion around this is needed.\n3. The paper does not have any analysis on contamination or information leakage in the dataset for the evaluation benchmarks used. Although the dataset is purely sythetic, and does not use any seed data the chances of leakage are less, but as there is no control over what the LLM generates, the chances that LLM results in generic instructions that end up being similar to evaluation datasets are high.\n4. The method for generating questions is not controlled at all. The method relies only on the next token prediction, which might lead to simple instructions. The only way to steer the generation process is through system prompts, but it is hard to judge how effective is that, as the paper has no ablations on system prompts.\n5. As the generation method is fully un-controlled, how does its ensures diverse instructions? T-SNE plot in figure 7 or the similarity analysis in Figure 3 using all-mpnet-base-v2 model is not enough to gauge the diversity.\n6. The paper mentions that the method can be used to generate domain specific and multilingual datasets in the abstract. However, there are no evaluations on multilingual or domain specific data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Idc0ojE9yp", "reviewer": "Reviewer_DhCK", "summary": "MAGPIE is a scalable method for synthesizing high-quality instruction data from aligned large language models (LLMs) without requiring prompt engineering or seed questions. By leveraging the auto-regressive nature of models like Llama-3-Instruct, MAGPIE efficiently generates diverse instruction-response pairs through an automated two-step process, extendable to multi-turn, domain-specific, and multilingual datasets. Extensive analysis shows that models fine-tuned with MAGPIE-generated data outperform those using public datasets and can rival proprietary models fine-tuned on much larger datasets. MAGPIE’s versatility is further demonstrated through extensions for preference optimization and task-specific data generation, making it a significant contribution to scalable LLM alignment and democratized AI research.", "questions": "An important aspect of the work is the fact that aligned LLMs can generate user prompts with just a system prompt. \n\nFor instance, the below system prompt would help generate user prompts for code related tasks.\n\n\"You are an AI assistant designed to provide helpful, step-by-step guidance on coding problems. The user will ask you a wide range of coding questions. Your purpose is to assist users in understanding coding concepts, working through code, and arriving at the correct solutions\"\n\nIf an LLM can indeed produce a lot of diverse user prompts when sampled correctly, the novelty would seem limited. The work does go into depth on filtering etc. However fact remains that the LLM is able to generate most of the prompts. If this point can be addressed, the work would definitely be very convincing.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Good contribution to open source community as the dataset is quite large and diverse\n- Interesting finding that aligned LLMs can be used to generate synthetic user instructions with just a system prompt\n- Detailed insights on the generated SFT and Alignment pairs were presented\n- The framework would be very useful for the open source community\n- Authors also present a solution to overcome some limitations of the magpie dataset by developing a booster dataset", "weaknesses": "- LLMs are typically trained (loss is computed) only on the assistant turns (during SFT, DPO, RLHF etc) and it is not clear how aligned models are able to produce user instructions. The paper does not provide enough insights to understand this. While this may not be directly in scope of the paper, it would help in increasing confidence in the presented approach.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)