Montessori-Instruct: Generate Influential Training Data Tailored for Student Learning
Authors: Xiaochuan Li
Zichun Yu
Chenyan Xiong
Venue: ICLR 2025 Poster (2025)
Keywords: synthetic data
data influence
instruction tuning
Primary area: foundation or frontier models, including LLMs

Abstract: Synthetic data has been widely used to train large language models, but their generative nature inevitably introduces noisy, non-informative, and misleading learning signals. In this paper, we propose Montessori-Instruct, a novel data synthesis framework that tailors the data synthesis ability of the teacher language model toward the student language model's learning process. Specifically, we utilize local data influence of synthetic training data points on students to characterize students' learning preferences. Then, we train the teacher model with Direct Preference Optimization (DPO) to generate synthetic data tailored toward student learning preferences. Experiments with Llama3-8B-Instruct (teacher) and Llama3-8B (student) on Alpaca Eval and MT-Bench demonstrate that Montessori-Instruct significantly outperforms standard synthesis methods by 18.35\% and 46.24\% relatively. Our method also beats data synthesized by a stronger teacher model, GPT-4o. Further analysis confirms the benefits of teacher's learning to generate more influential training data in the student's improved learning, the advantages of local data influence in accurately measuring student preferences, and the robustness of Montessori-Instruct across different student models. Our code and data are open-sourced at https://github.com/cxcscmu/Montessori-Instruct.

Published as a conference paper at ICLR 2025
MONTESSORI-INSTRUCET GENERATE INFLUENTIAL TRAINING DATA TAILORED FOR STUDENT LEARNING
Xiaochuan Li Zichun Yu", Chenyan Xiong*
School of Software, Tsinghua University "Language Technologies Institute, Carnegie Mellon University li- 1i-xc200 0 @mmail 1s tsinguu.. edu. cn z aichnyye andrew. cmu.edu ex0cs cmu edu
ABSTRACT
Synthetic data has been widely used to train large language models, but their gener- ative nature inevitably introduces noisy, non-informative. and misleading learning signals. In this paper, we propose CONTESSORI-INSTREE framework that tailors the data synthesis ability of the teacher language model toward the student language model's learning process. Specifically, we utilize local data influence of synthetic training data points on students to characterize students' learning preferences. Then, we train the teacher model with Direct Preference Optimization (DPO) to generate synthetic data tailored toward student learning pref- erences. Experiments with Llama3-8B-Insttee
a novel data synthesis
(teacher) and Llama3-8B (student) on Alpaca Eval and MT-Bench demonstrate that Montessori-Instruce significantly outperforms standard synthesis methods by 18.35% and 46.24% relatively. Our method also beats data synthesized by a stronger teacher model, GPT-4o. Further analysis confirms the benefits of teacher's learning to generate more influential train- ing data in the student's 's improved learning, the advantages of local data influence in accurately measuring student preferences, and the robustness of Montessori- Instruct across different student models. Our code and data are open-sourced at https://gi com cxcs cmc httpssccc/ooootttssseeee Instructt
INTRODUCTION
Synthetic training data is highly effective in various applications of large language models (LLMs) (Lu
al., 2023), spanning from general pretraining (Allal et al., 2024 Zhou al.] 2024), instruction-
tuning (Tong al. 2024) to domain-specifie scenarios such mathematics (Yu et al. 2023) and
coding (Jiang et al. 2024). The advantages synthetic data include its low cost, convenience, and
flexibility, making them an appealing choice for scaling up training data (Yue et al. 2024). mitigating the shortage of human labels (Chang et al. 2024), and improving data diversity (Sun et al. 2023).
Typical data synthesis methods (Wang et al. 2023) employ instruction-tuned teacher model and prompt it with seed data to generate synthetic training data for a student model. It is widely observed that the teacher-generated data can be noisy and oo-iiffrratiie (Bauer et al. 2024). their simple and uniform format may lead to pattern overfitting (Chen et al. 2024). and their biased and ungrounded content can introduce ambiguity in AI alignment (Liu et al. 2024). These are fundamental challenges of synthetic data as they can mislead students and sometimes even result in model collapse (Shumailoy al. 2023a Seddik et al. 2024).
a novel data synthesis framework designed to generate more tailored and informative data by directly optimizing the synthesis ability of the teacher toward the student's is learning preferences. We first leverage influence functions (Koh & Liang 2017. Yu al. 2024b) to precisely measure the utility of synthetic data its ability to effectively train the students. Then, we optimize the parameters of the teacher model according to the student's preferences through Direct Preference Optimization (DPO) (Rafailov et al., 2024). The preference- optimized teacher then synthesizes influential training data for the students. As shown in Figure [
In this paper, we propose MONTESSORI-INSTRUUS
Part of this work is done while visiting CMU.

---

Published as a conference paper at ICLR 2025
Conermat
dasiett
Proegt
stake
Flour
DPQ
Obtah
Teacher
Student
Teacher
Student
Teacher
Student
Teacher
Student
Syntheabe
Systhoin
Inrractions
Hhspecsss
Samande
Prefeseraa
(a) Soff-Instruct
(b) Relf-Reward
(c) LLM2LLM
(d) Montessori Instructt
Figure I: Data synthesis methods with standard teacher (data synthesizer) and student (target) setups.
rather than employing LLM-as-a-judge (Zheng al. 2024) to evaluate and filter data by quality (Yuan
et al., 2024) or prompting teachers to generate harder examples (Lee et al. 2024) Montessori-Instrt directly optimizes the teacher according students' learning preferences, leading more customized, flexible, and effective synthetic training data for the students.
Our experiments use MontessoriiInstruus to synthesize 10K instruuction-responee pairs with Llama3- 8B-Instruct (Meta 2024) as teacher and train Bamma--8B/Tinyllmmaa I. (Zhang et al., 2024) as students. The results show that paattsssor-Innstrre achieves relative improvements of 18.35% and 46.24% over Instuuctt on in-domain Alpaca Eval (Dubois et al. 2024) and out-of-domaad MT- Bench (Zheng et al., (024). respectively. The benefits of Montessor Instruct are more pronounced compared to state-of-the-ar data synthesis methods such as RewadR and LLM2LLM, as well as data synthesized by the cutting-edge L.IMM, GPT-4o (OpenAl 2024). The results on a wide range of general NLP tasks (c.g., MMLU (Hendrycks et al. (020) and GSM8K (Cobbe al. 202I further demonstrate the generalization capabilities of Contessori-Instrucee
Further analyses reveal strong correlation between the teacher's optimization process and the student's 's performance, demonstrating that Montessori-Instrus enables the teacher to generate data aligned with students' preferences to enhance its learning. Ablation studies highlight the advantages of using data influence to reflect students' preferences, the effectiveness of optimizing the teacher parameters over solely bootstrapping the data, and the robustness of Montessori--nstrrun across different seed data, multiple iterations, and a variety of student models.
Our main contributions summarized as follows:
1. We propose Montessori-Instruuc a novel data synthesis framework that tailors the data synthesis ability of the teacher toward the student's learning.
2. We incorporate influence functions to accurately capture the student's data preferences and effectively guide the teacher's 's optimization directions.
3. Our empirical results demonstrate the effectiveness and robustness of Montessori- Instruct in improving students' learning outcomes by tailoring synthetic data generation to align with student learning preferences.
RELATED WORK
Synthetic data has been shown highly effective in various applications of large language models (Lu
et al. 2023), including pretraining (Allal et al.) 2024 Zhou et al.) 2024). iiatrrcciiontuunnng (Tong al.
2024. Yue al. (024). mathematics Yu et al.) 2023) and coding (Jiang et al.) 2024). Typical approaches like Instruct (Wang et al. 2023) leverages
instruction -unded teacher to generate instruction-respoonee pairs given a small amount seed data. Following the similar pipeline, Self- Guide (Zhao et al.) 2024) and Self-Alignment (Sun et al. 2023 Guo et al. 2024) further enhance data quality for specific tasks, such as safety, truthfulness, and instruction-follovin by carefully curating task-relevant seeds. In parallel, Instruction Backtranslation (Li et al. 2023) and Bonito (Nayak et al.
2024) collect massive texts from the internet as responses, prompt LLMS to synthesize instructions reversely, and select high-quality candidates.
Despite its promising potential, synthetic data primarily rely on the teacher's free-form generations, thus is inevitably often biased, non-informative, and misleading (Bauer et al. 2024 Liu et al. 2024).
2

---

Published as a conference paper at ICLR 2025
The discrepancy between synthetic data and real-world sources often results in a misalignment with human values and preferences (Liu et al. (022). raising the risk of training student models that are
biased (Feng al. 2023 Liu et al. (2021) ,, ungrounded (Liu al.) 2022; Patel & Pavlick| (022)) or misrepresentative of real-world scenarios (Ji et al. 2023] Hu et al. 2024b). It is also observed that task-specific synthetic data often lacks diversity YYu et al. (024a). whereas general synthetic data suffers from pattern overfitting (Chen et al., 2024) and the memorization of the synthesis model's training data (Van Breugel et al.] 2023). Another challenge of synthetic data training the
phenomenon of model collapse (Shu et al. [2023), where the massive noise in unregulated synthetic data leads to the disappearance of the tails of the original content distribution and ineffective student
models (Seddik et al. 2024).
To address these limitations, researchers have explored various approaches to improve the utility of synthetic data (Shu et al., 2023) Wang et al. 2024). One line of work focuses on filtering out noisy synthetic data, using techniques like ranking synthetic data with an additional reward model (Shu
et al. 2023). verifying the truthfulness of responses via programs (Dong et al. 2024), prompting LLMs to judge the data quality (Zheng et al. 2024), and ensemble of multiple teacher (Lee et al. 2023). One can also directly adjust the teacher's synthesis strategies to generate more useful data for students (Lee et al., 2024; Yuan et al.] 2024). For instance, LLM2LLM (Lee et al., (022)) collects data points that the student answers incorrectly and prompts the teacher to bootstrap similar data, thereby generating targeted data to strengthen the student's weaknesses. Another potential path, such
as Self-Reward (Yuan al., 2024). is to employ LLM-as-a-judge (Zheng al.) (224) assign each
response a discrete reward score and optimize the student to generate highly rewarding responses.
The last body of related work is data influence functions (Hampel 1974), a commonly used technique for measuring the utility of data on a model's performance. Intluence function (Hampell 1974]
Koh & Liang 2017 Bae et al. 2022) quantifies the change in reference loss when data point is upweighted in the traminn set (Koh & Liang 2017). It often serves as a theoretical tool to analyze data utility (Choe et al. 2024) and attribute model behavior (Park et al. 2023). Recent work has applied influence functions to facilitate model-aware data selection in pretraining or Instruction-tunie
using first-order approximation (Xia et al.) 2024), linear datamodels (Engstrom et al. 2024), and data influence models (Yu et al.) 2024b). These methods have been shown to be more effective than traditional rule-based techniques in data selection, mostly notably in the pretraining stage (Engstrom
et al. 2024 Yu et al. 2024b).
3 MONTESSORI-INSTRUC
This section first introduces the overall framework of MONTESSORI-INSTRUCT ($ 3.1) and then elaborates its two main components: local data influence collection ($ 3.2) and student-preference- guided teacher optimization 3.3
3.1 OVERALL FRAMEWORK
Standard data synthesis methods (Wang et al. 2023 Yuan et al. 2024 Lee et al. 2024) begin with a teacher model M and a seed prompt p formed using a few-shot sample of example data. The teacher model processes the seed p to generate a set of N new instructions, { < < N}. that follow similar format to the seed but with a variety of contents. Each generated instruction x; is then used to prompt the teacher to synthesize the corresponding response y4- This yields set of Instruction-respon pairs (<., y4) < < N} that are then used train the student model m.
Montessori- -ansess upgrades this standard data synthesis pipeline with the optimization of the teacher www.e-eeneeeeiuieeeeee teacher optimization
model toward the student's learning preferences. The starts with prompting the teacher to generate probing dataset 'D probing using Self- Self-Instruct and then collecting these data points' local data influence I' on the student model ($ 3.2). The collected data preferences form the preference dataset D Preference. and Montessori-Instree the teacher model via Direct Preference Optimization (DPO) (Rafailov et al. 2024) ($ 3.3 The optimized teacher then generates the actual training dataset to train the student model can be iterated multiple rounds to continually refine the teacher according to the student's updated preferences. This process is illustrated in Figure and discussed in detail in the next two sections.
uses it to update
The process
3

---

Published as a conference paper at ICLR 2025
Generate Selection Dataset
Collect Local Data Influence
Construct Prefereence
seed
Instruction
Collect
Infivence
instruction
Student
response
Prompi
Generate
seed
Teacher
Preference dataset
seed
instruction
Student
instruction
resporis
influence
Update
Figure 2: Ceeeent-Prefeenee--Gee Guided teacher optimization in Montessori Centessori-Instruct..
3.2 LOCAL DATA INFLUENCE COLLECTION
A key component of our framework is to precisely measure the utility of synthetic data, i.e., how good they are improving the student's s learning outcomes. This question is often approached using influence functions (Weisberg & Cook 1982, Koh & Liang 2017), which was designed to quantify changes in reference loss when a data point (zy, ys) is upweighted in the training sets Park et al. (2023), thus reflecting the utility of this data point to the student's learning.
In order to efficiently calculate the data influence, we follow Yu et al. (2024b) and approximate influence functions locally, using the change of the model's reference loss before and after training on a single data point (I,, y:):
Im( Dret) R LD A(yi m;;m)) + L(Drer m);
where L(Dret m) B(a.y)>Dar 6(y x; m))
where Dier denotes the reference data that measure the student's capability, and E(yla; m) is the loss of student m on an input-output pair (z, y). A(ys (r;m) refers to the optimization operation of student m on data (z,, 9i). e.g., one-step training with Adam (Kingma & Ba 2015) on (Ii, B4).
The local data influence, (m;; Dret), represents how the instruction-response pair (æ,, yi) impacts the student's learning outcome as measured on the reference data. A positive Imm indicates that the data benefits the student's reference performance, while negative Im, shows the opposite. A complete theoretical derivation local data influence provided in Appendix B
3.3
PREFERENE GEAIIEEEEEEGGGUIEDD TEACHER OPTIMIZATION
After calculating local data influence for each instruction in the probing dataset 2 probing pair every two instructions with positive and negative influence, along with their corresponding seed prompt p. to construct the preference dataset:
D Prreference
{(p x
Im
Pret) < 0 < In ;
We then apply DPO to optimize the teacher model M toward the student's learning preferences:
(x p) M(x (p) log
M² p)
(MP Ep.p. 1 >>>Dgnttnne loga log
M(x p)) )]
(4)
where 3 is a parameter that controls the deviation from the initial teacher M and a is the logistic function. The updated teacher, M after multiple iterations, is then used synthesize the training data for the student model m.
4 EXPERIMENTAL METHODOLOGIES
This section details our main experimental setups, including a thorough configuration of the data synthesis process, the chosen baselines, and the evaluation methods.
4

---

Published as a conference paper at ICLR 2025
Data Synthesis Process. We choose Llama3-8B 1-----B-nnnnse Instructct (Meta 2024 as the teacher, and train Llama3-8B (Meta 2024) and Tinyllama-L... L. (Zhang et al. 2024) as students. We merge the text in instruction and input fields of Alpaca GPT-4 dataset (Taori et al. 2023), consisting of 52K entries, to create our seed pool. We follow the 8-shot seed proposed in Self-Instruct (Wang et al. 2023 to prompt the teacher to generate instructions, with 6 out of the 8 randomly sampled from the seed pool and 2 sampled from the synthetic instructions in the teacher's previous iterations. Detailed prompts are provided in Figure 13
Following Yuan et al. (024), we initially use the unoptimized teacher model to synthesize 1K data to warm up the student. Then, we generate 4 instructions for each seed and response for each instruction and filter out similar instructions whose Rough-L score exceeds 0.7, resulting in a probing dataset of 10K poomp-innsrruciio--ssspnn triplets. For each instruction-respons pair the probing dataset, we collect local data influence using the loss difference of the student model on the reference data (Alpaca GPT-4) before and after one-step training. Then, we construct a preference dataset comprising 6,792 entries, where each entry represents seed-instruction pair with positive and negative influences. This preference dataset is used to train the teacher with Direct Preference Optimization (DPO) (Rafailov al., 2024). Finally, data to train the student from scratch. In the subsequent iterations, we optimize the teacher using similar steps, but with the updated student from last iteration to collect data influence. For both the teacher and student training. we utilize Adam optimizer (Loshchilov & Hutter] 2019) along with WSD scheduler (Hu al. 2024a). Both models are trained for one epoch. For teacher's 's generation, we use v vLLM (Kwon et al. (202)) as our decoding engine and provide specific decoding parameters
use the optimized teacher to synthesize 10K
in Table More details can be found in Appendix
Baselines. We compare our method against several mainstream data synthesis baselines. The simplest baseline is Self-Instruct (Wang et al. 2023), where we the unoptimized teacher to synthesize data. Additionally, we select GPT-4o (OpenAl 2024) as a stronger teacher to synthesize an equivalent amount of data for comparison. Another baseline is Self-Reward (Yuan et al. 2024). which employs an aLM-as-a-judge (Zheng et al.] (024)) to assign ratings from I to 5 points to its self-synthesized responses. Since we find in our preliminary experiments that Llama3-8B lacks the ability to effectively score its own responses, we instead employ GPT-4o as an external judge to score the student's responses. The results of the original $ (D.3 The final baseline is LLM2LLM (Lee et al. 2024), which evaluates the student's accuracy on its seed set and filters out those that result in incorrect answers. our case, we define data points with the highest 50% training loss as incorrect examples. The teacher is then prompted to bootstrap data similar to the incorrectly answered seeds. To align with our setting, we uniformly conduct two rounds of iterations for Self-Reward and LLM2ILM. For all methods, we synthesize 10K instruction-respone pairs to train the student models.
are reported in the Appendix
Evaluation Methods. We use Alpaca Eval 2.0 (Dubois et al. 2024) as the in-domain evaluation to assess the model's instruction-following ability. We utilize gpt-4-ttue 4--uuboo--ooo---- courbo-202- 2024 11224-444 000-0-4-000 04-09 as the evaluator and uniformly compare all methods against the student model trained with Self-Instruce The evaluation metrics are standard Winng Rate (WR) and Length Control Winning Rate (LC-WR). For head-to-heall winning rate, we employ the evaluation prompt in both pairwise orders, and if the results disagree, we count as a tie. Additionally, we evaluate the model's generalization performance across six oft-of-domain tasks, including MT-Bench (Zheng et al.) 2024), ARC- AhCl-naleenge (25-shot) (Clark let al. 2018). GSMSK (8-shot) (Cobbe et al. (2021). HellaSwag wagg (8-shot) (Zellers et al.) 2019). GPQA (0-shot) (Rein et al. 2023), and MMLU (0-shot) (Hendrycks et al. 2020). These tasks span areas such as multi-turn dialogue, knowledge based question answering, mathematics, and natural language reasoming, offering a thorough assessment of our approach's 's effectiveness. For MT-Bench, we report the score out of 10 judged by gpt tet-uuuroo20200 -04--44-0 04-09. For other tasks, we report normalized accuracy if is included in the evaluation results, otherwise, standard accuracy.
5 EVALUATION RESULTS
This section evaluates the effectiveness of Montessori-Insttrne ($ 5.1.. illustrates the correlation
between the teacher's learning and the student's performance ($ 5.2). conducts comprehensive ablation studies on the effectiveness of local data influence, the optimization of the teacher, the seed
5

---

Published as a conference paper at ICLR 2025
Table Evaluation training 8B/1.IB students with different data synthesis methods. Adoption of stronger teacher model (GPT-4o) is indicated by All else use Mama3-8B-Instrun as the teacher model. The best and second- best performances are marked in bold and underscore, respectively.
In- Domain
Out Ou--OF Of-Domain
Alpaca Eval 2.0
MT-Bench
MMLU
GPQA
ARC-C
GSMSK
HellaSwag
Methods
LC-WR
Score
Accuracy
8B Setting:
Student-Llama3-88t
No fine-tuning
2.09%
3.39%
5.597
62.15
24.33
57.85
51.25
81.96
Self-Instruct 50%
50%
6.490
62.42
31.92
59.98
58.76
80.93
Self-Instruct
54.95%
56.39%
5.918
63.41
30.13
60.58
50.42
81.42
Self-Reward" 51.87%
Iteration
55.38%
6.713
62,46
28.19
59.84
53.60
81 .04
Iteration
53.49%
57.32%
6.798
62.02
29.08
60.64
56.37
81,13
LLM2LLM
Iteration
51.49%
53.12%
6.531
62.18
29.12
57.49
55.28
80.49
Iteration
52.63%
55.02%
6.519
62.46
30.04
59.65
$7.75
80.57
Montessori Instrrcct
Iteration
54.92% 56.37 %
58.59%
6.903
62.93
29.91
62.97
58.76
81.22
Iteration
60.15%
7.163
63.47
31.36
60.17
60.02
81.98
1.1B Setting: Student=Tinyllann
No fine-tuning 17.89%
17.56%
1.020
26.16
23.88
37.12
1.97
62.61
Self-Instruct 50%
2.154
26.21
24.78
37.97
1.82
62.47
Self-Instruct
54.02%
55.02%
1,928
26.64
24.33
38.82
2.20
63.17
Self-Reward"
Iteration
47.62%
48.34%
1.804
26.34
23.92
37.64
1.76
62.27
Iteration
46.48%
46.95%
1.717
26.09
24.62
38.03
1.76
62.79
LLM2LLM
Iteration
52.03%
52.75%
2.243
25.87
24.51
36.86
2.24
62.15
Iteration
51.64%
53.52%
2.192
25.62
24.84
36.74
2.31
62.08
Montessori- Instruct
Iteration
53.25%
51.77% 54.68%
2.485 2.526
26.23
23.92 24.88
37.97
2.35 2.82
62,59
Iteration 54.37%
26.47
38.05
63.54
data and multiple iterations ($ 5.3
and then demonstrates the generalization of the synthetic data
from Montessori-Instren ($ 5.4).
5.1 OVERALL PERFORMANCE
Table presents the overall performance of cuntessorriInnstrune compared with the state-of-the-arr data synthesis methods. In the 8B setting. Montessori--nstruue significantly outperforms Self- Self-Instruct by 6.37% LC-WR and 10.15% WR on Alpaca Eval. Notably, our method still surpasses Sesf-Instruct with GPT-4o as the teacher, suggesting that stronger LLM does not necessarily produce more beneficial data than weaker LLM that is tailored the student's needs. Compared to Self-Reward and LLM2LLM, Montessori-Instrue This underscores the advantage of directly optimizing the teacher model's parameters toward the student's 's preferences derived from data influence.
consistently shows better performance across both iterations.
In addition in-domain evaluation, Montessori- Instrucc also outperforms all the baselines out-of- domain tasks, achieving maximum improvements of 0.673 and 0.372 on the MT-Bench in the 8B and I.IB settings, respectively. This indicates that the teacher optimized by our method does not overfit the reference tasks and maintains strong robustness and generalization capabilities, whereas other baselines suffer from performance degradation on out-of-domain tasks.
5.2 CORRELATION BETWEEN TEACHER'S LEARNING AND STUDENT'S PERFORMANCE
This set of experiments examines how the teacher is progressively optimized to align with student preferences, thereby enhancing the student's 's performance. We first zoom in on the teacher's learning process investigate its progressive impact student models. Figures 3a and 3b compare the

---

Published as a conference paper at ICLR 2025
Learning Margin
DPQ Magh Stacla)) EaiLCWAK
Learning Margin
Dio degln Cloeeed Mf (rnnaan
M MT-Beach Score
dago-2200 dign
Posttive Infiuence Data 89
1:30
Alpaca Evall
Count
20
Toucherw splate
Probing Dataset Size
Probing Dataset Size
Local Data Inflownce
Studemt Training
(a) Alpaca Eval
(b) MT-Bench
(c) Data influence
(d) Positive influence
Figure 3: Figures (a) and (b) illustrate the correlation between the teacher's learning process and the performance of the student trained on data synthesized by the intermediate teachers in Alpaca Eval and MT-Bench. Figure (c) depicts how the distribution of the local data influence of the teacher's synthetic data shifts as the teacher is progressively updated. Figure (d) presents the proportion of training data with positive local data influence during the student's training.
performance of students trained using synthetic data generated from the teacher's intermediate checkpoints. The learning margin reflects the teacher's learning process, representing the average difference between selected rewards and corresponding rejected rewards in DPO. A larger margin indicates that the teacher is more likely to generate the selected synthetic data. The results indicate a positive correlation between the student's performance and the teacher's optimization progress.
We then select several teacher checkpoints to examine the properties of their synthetic data, aiming to identify changes occurring as the teacher learns. Specifically, we focus on the distribution of local data influence in the synthetic data, defined as the change in the model's 's reference loss before and after training on a single data point, which indicates the utility of that data for the model. The baseline reference loss is the loss on the reference set prior to one-step training, i.e., Equation [ As shown in Figure 3c we observe that as the teacher optimized, the distribution of its synthetic data shifts towards the positive side, indicating an increased proportion of data with positive local influence in its synthetic outputs. From the student's perspective (Figure 3d), which shows the changes in the proportion of data with positive local influence in the next training batch, this proportion decreases over time during training. However, the data generated by the updated teacher consistently maintains higher proportion of positive influence compared to regular teacher.
In summary. we attribute the improved performance achieved by Montessori-Instrus to the teacher's continuously enhanced ability to synthesize data with higher local influence, by using DPO to distinguish data with varying influence values. The positive correlation between student performance and the increased proportion of training data with positive local influence leads to more effective learning, thereby improving the student's 's overall performance.
5.3 ABLATION STUDIES
This subsection demonstrates the effectiveness of the methodological design in Contessori-Instternn through four ablation studies, summarized in Table 2 The yellow lines show ablations on data point utility evaluation methods. The red lines represent optimization for responses based on instructions and optimization for teacher models. The blue lines cover various seed data types: OOD) (Out-Or-Domaing ID (In-Domain), and Test (direct use of the test set).
Effectiveness of Local Data Influence. To evaluate the impact of different methods for obtaining the influence of data point, compare our local data influence against two additional baselines: (1) L--dde-a-Judge (Zheng et al. 2024). which leverages GPT-4o to directly assign a 1-5 score to each instruction-responss pair, inspired by Relf-Reward, and (2) Training loss, which directly uses the training loss of each data point as its influence score, inspired by L.LM2LLM. As shown in the yellow lines in table our local data influence consistently outperforms both baselines by significant margin. This indicates that local data influence is a more effective metric for capturing students' fine- -raained data preferences compared to the other methods.
Effectiveness of Teacher Optimization. To analyze the effectiveness of the optimization strategy on the teacher, compare method with additional ablation baselines: (1) Bootstrap:
7

---

Published as a conference paper at ICLR 2025
Table 2: Ablation studies on the effectiveness of the methodological design in Montessori-Instruuct All experiments were conducted on the Llama3-8B students.
Alpaca Eval 2.0
MT--Becch
MMLU
GPQA
ARC-C GSM8K
HellaSwag
Methodological design
L.C-W
WR
Score
Accuracy
Effectiveness of Local Data Influence
LLM-as-a-Judge
53.42%
54.93%
6.731
62.93
29.75
62.09
58.82
81.05
Training loss
52.34%
54.99%
6.656
62.54
29.89
61.48
58.76
80.93
Local data influence (Ours)
54.92%
58.59%
62.93
29.91
62.97
58.76
81.22
Effectiveness of Teacher Optimization
Bootstrup
50.59%
48.14%
6.618
60.67
25.19 19
57.95
58.13
80.46
Response optimization
51.59%
54.22%
6.556
62.43
27.45
60.42
56.38
81.04
Instruction optimization (Ours)
54.92%
58.59%
6.903
62.93
29.91
62.97
58.76
81.22
Effectiveness Seed Data
Open Assistant (OOD)
52.28%
54.76%
6.706
62.86
29.74
62.29
58.42
81.24
Alpaca GPT4 (ID) (Ours)
54.92%
58.59%
6,903
62.93
29.91
62.97
58.76
81.22
Alpaca Eval (Test)
$7.64 %
61.36% %
7.147
62.93
30.44
63.06
60.80
81.09
Muntesaar--tsirnct Wine
Seliinatruct Wins
1ef Wins (in 1eft va. Right
Twe
Right Wins
Moctessort- instruct
Mostesseci-nstroocc
51.9
24.3
23.0
36,8
32.9
30.3
Self festruct
Montessoci Instruct
Montessors Instruct
46.7
25.8
27.5
40.2
30.7
29.1
SelfInstmeet
Monlesoori- -nattuut A4
(a) rates of iterations compared to Self-Instruct
(b) Win rates compared between different iterations
Head-to head win rates for evaluating 8B models among the Self- Instruct baseline and three successive iterations updated using Montessori Instrucct.
Figure 4:
bootstrap the top 50% influential data by utilizing it as the seed, and (2) Response optimization: we optimize the teacher by the student's local data influence of different responses given an instruction. As shown in red lines in table optimizing the teacher is generally better than merely bootstrapping influential data, highlighting the necessity of adapting the teacher the student's needs. Furthermore, instruction optimization (Montessori-Insttee
outperforms response optimization across all tasks. We attribute this to the smaller search space of response optimization, which limits the headroom for teacher improvement compared to instruction optimization.
Effectiveness of Seed Data. This study examines the impact of the seed data by varying its relevance to the evaluation tasks. In addition to the Alpaca GPT-4 (in-domain seed data) used the main experiments, we also utilize Open Assistant and Alpaca Eval alternative seed data. Open Assistant represents an out-of-domain seed, whereas Alpaca Eval is directly sampled from the evaluation task. Blue lines in table 2 demonstrates that using Alpaca Eval leads the best performance on itself while using Open Assistant is less effective compared to in-domain seed data. For more general NLP benchmarks, changing the seed data results in only slight differences performance. This indicates that our method is robust enough to enhance the synthesis ability of teachers, even when using different seeds.
Effectiveness of Multiple Iterations. We examine the performance differences when applying Montessori-Instruct over multiple iterations. each iteration, we begin by constructing a probing dataset of 2K samples to collect local data influence on the student model from the previous iteration, followed by updating the previous teacher. As shown in Figure 4a] Montessori-Instruct continues to outperform Self-Instruct across three iterations, achieving peak head-to-head win rates of 51.90 51.9%. The results in Figure illustrate the comparison between different iterations, demonstrating that previous iterations. We attribute these gains to the
Montessori- Instruct yield improvements Montessori Instruct's 's ability to capture the data preferences of students at different iterations and to tailor influential data according their evolving needs.
8

---

Published as a conference paper at ICLR 2025
90
90-
90-
90
72
Peclanttust
Score
72
pestralzed tacllanduut
72
MT Bench Score
72
postabad
Faod antr
10
MI-Bench Score
Alpaca Eval
*SUUStesoe LESTIUTE
Alpuca Eval
traxdassen antruct
Alpoca Evaal
Alpuca Eval
Franstesioninneie insinut
194
061
20
M
18
1.11
L.C WW..M
WI
LICA/LN Menchh
L.C.
(a) Llama3-8B
(b) Qwenl.5-7B
(c) Mistral-7B
(d) Gemma2-9B
Figure 5: Evaluation results of training four different student models using synthetic data generated by teacher optimized for the data preferences of the 1.1B student.
5.4 GENERALIZATION ABILITY OF THE SYNTHESIZED DATA
In this experiment, we study the generalization ability of our teacher optimized toward a small student (L.1B)'s preferences. Specifically, utilize the data synthesized by this teacher to train four different student models-Llamas
(Meta 2024), Mistral-7B (Jiang et al. 2023), Qwen 1,5-7B (Bai et al. 2023), and Gemma2-9B (Team et 2024). As shown in Figure 5 the data synthesized by one teacher leads to consistent performance gains across all the students compared to Self-Instruct. This finding implies we can directly deploy an optimized teacher generate data for variety of student models, enhancing their performance with low expense.
5.5
CASE STUDY
In this section, we present several cases visualize the differences be- tween the instructions synthesized by Self-Instruct and by Montessori- Instruct, and showcase the chosen and rejected data pairs that reflect what the teacher learns during our optimiza- tion. Figure(oJsowss the word analysis of root verbs and their corresponding nouns. We identify the top most common root verbs (inner circle) and their top 4 direct noun objects (outer circle) in the generated instructions. The results indicate that, compared to Self-Instruct, Montessori-Instruet guides the teacher to synthesize more on writing instructions and providing specific, informative examples, while reducing the frequency of simple commands like summarizing and translating.
aanggr write
concept
grudd
recipe
concept
paara graph
explain
points
write
waste
waste
sontencs eoesine
area
lagt
(a)
(b) Montessori-Lnstte
Figure 6: The most common root verbs (inner circle) and their top direct noun objects (outer circle) in generated in- structions
Table 3 compares the chosen and rejected data pairs given the same prompt. Our method discards low-utility data, such as explanations of simple concepts and sentence translations. and increases the likelihood of generating complex and informative instructions. This further demonstrates the effectiveness of using local data influence to differentiate data utility.
DISCUSSION AND LIMITATIONS
Synthetic Data Scale. We synthesize 10K data points to verify the effectiveness of our innovative data synthesis framework. While this 10K dataset outperforms other baselines and demonstrates strong generalization, its effectiveness when scaled to the volume required for production-level fine-tuning (around 100K) remains unclear. Expanding the synthetic data volume may introduce redundancy, a phenomenon commonly observed in data synthesis (Bauer et al. 2024 Liu al. 2024). would be meaningful to study how to balance the quantity and the diversity of the synthetic data, while this is orthogonal to our main contribution.

---

Published as a conference paper at ICLR 2025
Table 3: Examples of instruction-respons pairs with positive and negative influences
Positive
Negative
What the most efficient way to transport given package across a city, considering traffic congestion, distance, and cost?
[nstruction]:
[Instruction]: Explain the concept of artificial intelligence in 100 words.
[Response]: Artificial intelligence (AI) refers to the development of computer systems that can perform tasks that typically require hu- man intelligence, such as learning, problem- solving, and secision-mmkknng Al systems use algorithms and data to analyze and interpret in- formation, allowing them to make predictions, classify objects, and generate insights. They can be trained on large datasets to improve their performance over time, and can even adapt to new situations and learn from experience. Al has many applications, including robotics, ural language processing, image recognition. and expert systems, and is increasingly being used in industries such as healthcare, finance, and transportation.
[Response]:
The most efficient way to transport pack- age across a city depends on several factors, including the package's s size, weight, fragility, and urgency, as well as the city's 's traffic patterns, infrastructure, and regulations. Here are some general considerations and options to minimize costs, reduce congestion, and ensure timely delivery: Disttacce and Route Optimization**:
* Use online route planners or transportation manage- ment software to determine the shortest and most efficient route.
* Consider alternative routes modes of transportation if the primary route congested or closed.
2. Traftic Congestion Mitigation**:
Avoid peak hours (7-9 am and 4-6 pm) when possible, traffic congestion typically highest during these times...
[Istrruction]: Can you write script for a public service announcement about the importance of recycling?
[Instruction]: Can you translate the following text from english to spanish: "the hotel room was comfortable and clean." "
[Response]: Here is a script for public service announce- ment about the importance of recycling:
[Response]: The translation of the text from English to Spanish is:
[Opening shot of beautiful park or nature reserve, with narrator speaking]
"El cuarto hotel era cómodo y limpio."
Narrator: "Imagine a world where our planet is thriving,
Here's 's a breakdown of the translation:
where wildlife roams free, and where our communities are healthy and strong. But get there, need to take action." "
"the hotel room" = ='el cuarto de hotel"
"was" "era" (verb conjugation)
[Cut to shot of a person holding a plastic bottle and throw- ing it away]
"comfortable" "cómodo"
"and"= = "y" (conjunction)
Narrator: "One of the simplest and most effective ways to
"clean" = "limpio"
make a difference is by recycling."...
So, the complete sentence in Spanish is: "El
cuarto de hotel era cómodo y limpio."
Overhead. Montessori- Instruct introduces additional computational cost. Compared Wang let al.] (2023), training an 8B model using our method increases the average processing time per data by 5.8 seconds (see the Appendix [ for details). At the instruction finetuning stage, compute is less an issue compared to pretraining. The scale is smaller, and generating data is faster and cheaper than human annotations. Additionally, the most time-intensive step our beetoo-"ccoleetting local data -cnn -cnn be independently parallelized on heterogeneous compute systems, allowing for easy acceleration. As demonstrated $5.44, Montessori-Instrus exhibits strong generalization capabilities. In practice, one can use smaller model to collect data influence for updating the teacher and then apply the updated teacher to synthesize data for larger models.
influence"
CONCLUSION
In this paper, we propose Montessori-Instruutt, a novel data synthesis framework that tailors the teacher for student learning. Montessori-Insiiee Instrruct leverages local data influence to reflect the student's learning preferences and to optimize the teacher to produce more influential synthetic training data. Experimental results demonstrate that Montessori-Instrucc significantly outperforms state-of-the-art data synthesis methods in both in-domain and out-of-domain evaluations, exceeding the performance of data generated by stronger teacher models like GPT-4o. Further analyses confirm the benefits of optimizing the teacher toward the student's preferences in improving student performances. Ablation studies validate the benefits of using local data influence to reflect data utility and highlight the benefits of optimizing the teacher over bootstrapping. Our work successfully demonstrates the potential of incorporating the student's 's learning preferences into teacher optimization, and we hope it inspires further exploration of more effective synthetic data generation frameworks.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We sincerely thank Shi Yu and Zhenghao Liu for discussing ideas and providing helpful feedback on this work. We also extend our gratitude Alex Xu for testing our Github repository.
REFERENCES
Loubna Ben Allal, Anton dia: How to create large-scale synthetic data for pre-training large language
Lozhkov, and Daniel van Strien. Cosmope-
models, 2024.
URL h htw.aaawww........aaa..eeaaaaaaa -oreraeeeeeeeeete-------------ee
cosmopedia-hove sasaa-howwto-oo www.tereeeeeeeeeeeeeee lareattelarge-saaae sca Accessed: 2024-09-09.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If influence functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 077799937767, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li. Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang. Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang. Hao Yang. Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. Qwen technical report. arXiv preprint arXiv:2309. 200,:999,.66,, 2023.
André Bauer, Simon Trapp. Michael Stenger, Robert Leppich, Samuel Kounev, Mark Leznik, Kyle Chard, and Ian Foster. Comprehensive exploration of synthetic data generation: A survey, arXiv preprint arXiv:2401.0223 025544, 2024.
calflops. calftops: flops and params calculate tool for neural networks. https://github.crm
Mr bttxJ/calculate-Fiic.. pytorch 2024.
Hsin-Yo Yu Chang, Pei-Yu Yu Chen, Tun-Hsiang Chou, Chang-Sheng Kao, Hsuan-Yun Yun Yu, Yen-Ting Lin, and Yun-Nung Chen. A survey of data synthesis approaches, 2024. URL http:::tt/// //arxiv
org/abs/ 2407. 03672
Jie Chen, Yupeng Zhang, Bingning Wang, Wayne Xin Zhao, Ji-Rong Wen, and Weipeng Chen. Unveiling the flaws: Exploring imperfections in synthetic data and mitigation strategies for large language models. ArXir, abs/2406. 12397, 2024. URL WtRLhttps::///t //ap::://api. semant
org//Crrps ID: 270562705555 788
Sang Keun Choe, Hwijeen Ahn, Juhan Bae, Kewen Zhao, Minsoo Kang, Youngseog Chung, Adithya Pratapa, Willie Neiswanger, Emma Strubell, Teruko Mitamura, al. What is your data worth to gpt? llm-scale data valuation with influence functions. arXiv preprint arXiv:2405. 20..24055....44 13954, 2024.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge. arXiv preprint arXiv:1803.054:3 2018.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Guanting Dong, Keming Lu, Chengpeng Li, Tingyu Xia, Bowen Yu, Chang Zhou, and Jingren Zhou. Self-play with execution feedback: Improving instruction-folloiing capabilities of large language models. ArXiv, abs/2406. 13522, 2024. URL htttt://t:///tt semant ccschol 1 ar
orc/CorpusID:2706200
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto. Length-controoled alpacaeval: A simple way debias automatic evaluators. arXiv preprint 244445204.04445, 2024.
11

---

Published as a conference paper at ICLR 2025
Logan Engstrom, Axel Feldmann, and Aleksander Madry. Dsdm: Model-aware dataset selection with datamodels, 2024. URL https: http:///axxvvvvvv.o attp:///bb/2420...1.
Shangbin Feng, Chan Young Park, Yuhan Liu, and Yulia Tsvetkov. From pretraining data to language models to downstream tasks: Tracking the trails of political biases leading to unfair nlp models, 2023. URL BtLLatts://aaaatt
org/ abs/ / 305. 08283
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Man- grulkar, Mare Sun, and Benjamin Bossan, Accelerate: Training and inference at scale made simple, efficient and adaptable.
htw.:ttwww...m.uuiiii...i.ametee http///ugginggggggccec wwc.ee/e.a..eeeacceeteaeeeeeeeee 2022.
com
Hongyi Guo, Yuanshun Yao, Wei Shen, Jiaheng Wei, Xiaoying Zhang, Zhaoran Wang, and Yang Liu. Human-instruction-fiee I1m self--aiggmment with limited samples. ArXiv, abs/2401.06685, 2024.
URL https: / /api. wwwant www.iccccchhaacc org/ Ttrg/CorpusIDi 66999538
Frank R Hampel. The influence curve and its role in robust estimation. Journal of the american statistical association, 69(346):383-333, 1974.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint wwXiv:2009.03300, 2020.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Minicpm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06:3 06395, 2024a.
Xuming Hu, Junzhe Chen, Xiaochuan Li, Yufei Guo, Lijie Wen, Philip S. Yu, and Zhijiang Guo. Towards understanding factual knowledge of large language models. In The Twelfth International Conference on Learning Representations, 2024b. URLpttps: /tattps://openrevieee net.eeewweeeeeteeeeeeeeeeeeeeeee / 1d 90evMUdods
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang. Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.0682. 2023.
Juyong Jiang, Fan Wang, Jiasi Shen, Sungju Kim, and Sunghun Kim. A survey on large language models for code generation. arXiv preprint arXiv:2406.005// 2024.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, San Diega, CA, USA, 2015.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMIR, 2017.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nicholas Lee, Thanakul Wattanawong, Sehoon Kim, Karttikeya Mangalam, Sheng Shen, Gopala Anumanchipali, Michael W Mahoney, Kurt Keutzer, and Amir Gholami. Llm2llm: Boosting llms with novel iterative data enhancement. arXiv preprint 2403::2403./5044 2024.
Young-Sur Lee, Md Arafat Sultan, Yousef Eu-Kur Tahira Naseem Asim Munawar, Radu Florian, Salim Roukos, and Ramón Fernández Astudillo. Ensemble-instruet: Generating instruction- tuning data with a heterogeneous mixture of Ims. ArXiv, 10.119611, 2023. htthpp www.atticcccoooooo http://owa....aaa....oooooooooaa
12

---

Published as a conference paper at ICLR 2025
Xian Li. Ping Yu, Chunting Zhou, Timo Schick, Omer Levy, Luke Zettlemoyer, Jason Weston, and Mike Lewis. Self-alignment with instruction backtranslation, arXiv preprint arXiv:2308.06259, 2023.
Ruibo Liu, Chenyan Jia, Jason Wei, Guangxuan Xu, Lili Wang, and Soroush Vosoughi. Mitigating political bias in language models through reinforced calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 14857-14866, 2021.
Ruibo Liu, Jason Wei, Shixiang Shane Gu, Te-Yen Wu, Soroush Vosoughi, Claire Cui, Denny Zhou, and Andrew M Dai. Mind's eye: Grounded language model reasoning through simulation. arXiv preprint arXiv:2210.05352 2022.
Ruibo Liu, Jerry Wei, Fangyu Liu, Chenglei Si, Yanzhe Zhang. Jinmeng Rao, Steven Zheng, Daiyi Peng. Diyi Yang, Denny Zhou, al. Best practices and lessons learned synthetic data for language models. arXiv preprint arXiv:2404.07503, 2024.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019. URL https
/awxiv.crg/abs/
Yingzhou Lu, Minjie Shen. Huazheng Wang, Xiao Wang. Capucine van Rechem, and Wenqi Wei. Machine learning for synthetic data generation: a review. arXiv preprint arXiv:2302.000002 2023.
Meta. The llaaa herd of models, 2024. URL http:pp///aaxx...rr////////////
Nihal V. Nayak, Yiyang Nan, Avi Trost, and Stephen H. Bach. Learning to generate instruction
tuning datasets for zero-shot task adaptation. ArXiv, htds/2402.183344 2024. URL https
wwt. 7/api. www.sseeaaaniiioaee.e.e.e http:ooaa...oooooorr.sssa.2222
OpenAl. Gpt4 url, 2024. URL htLLtttts////hhtpttttt
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior at scale. arXiv preprint 2.t.::2333..11186. 2023.
Roma Patel and Ellie Pavlick. Mapping language models to grounded conceptual spaces. In International conference on learning representations, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: A graduate-level google-proof q&a benchmark. arXiv preprint arXiv:2311. 12022, 2023.
Nikhil Sardana, Jacob Portes, Sasha Doubov, and Jonathan Frankle. Beyond chinchilla-optimal: Accounting for inference language model scaling laws. arXiv preprint arXiv:2401 .004480 2023.
Mohamed Amine Seddik, Suci-War Chen, Soufiane Hayou, Pierre Youssef, and Mérouane Debbah. How bad is training synthetic data? a statistical analysis of language model collapse. ArXiv, abs/24004050090) 2024. URL htt.s::/// /api. www.enicccccolaaee.....a. ott.aaaccccccoo.....oooooooooapa
org/Corpus
269005923
Lei Shu, Liangchen Luo, Jayakumar Hoskere, Yun Zhu, Canoee Liu, Simon Tong, Jindong Chen, and Lei Meng. Rewritelm: An instruction-tumet large language model for text rewriting. In AAAI Conference on Artificial Intelligence, 2023. URL https: sww.apiisossttttcccccc...a..aa. org/
CooppusD::2588778
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Anderson. The curse recursion: Training on generated data makes models forget. ArXiv, abs/2305. 17493, 2023a. httLhttp::///
Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot, and Ross Ander- son. The curse of recursion: Training on generated data makes models forget. arXiv preprint arXiv:2305. /7493, 2023b.
13

---

Published as a conference paper at ICLR 2025
Charlie Snell, Jachoon Lee, Kelvin Xu, and Aviral Kumar. Scaling Ilm test-time compute optimally can be more effective than scaling model parameters. arXiv preprint arXiv:2408.033144 2024.
Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David D. Cox, Yiming Yang, and Chuang Gan. Principle-driven self-alignment of language models from scratch with minimal human supervision. ArXiv, 0ass23055.03044 2023. URL https:
7/api. //api
sema nt cacho a wwtpaa..o../o.oooooe//.... ND:
Rohan Taori, Ishaan Guirajani, Tianyi Zhang. Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Stanford alpaca: An instruction-follovin Ilama model. https: / www.ootttt-oooooo..oooooooooooee labp http//tffoodddapcaaaaaa 2023.
Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, al. Gemma 2: Improving open language models at practical size, arXiv preprint arXiv:2408.00116 2024.
Yuxuan Tong, Xiwen Zhang, Rui Wang, Ruidong Wu, and Junxian He. Dart-math: Difficulty-aware rejection tuning for mathematical problem-solving. 2024. URL htttp://arxiv. org/ /bs/
2407 10007136699 13690
Boris Van Breugel, Zhaozhi Qian, and Mihaela Van Der Schaar. Synthetic data, real errors: how (not) to publish and use synthetic data. In International Conference on Machine Learning, pp. 17793-34808. PMLR, 2023.
Leandro von Werra, Younes Belkada, Lewis Tunstall, Edward Beeching, Tristan Thrush, Nathan Lambert, and Shengyi Huang. Trl: Transformer reinforcement learning. htttp:://qthuu:
2020.
Tianlu Wang, Ili Kulikov, Olga Golovneva, Ping Yu, Weizhe Yuan, Jane Dwivedi- Yu,
Richard Yuanzhe Pang, Maryam Fazel-Zarandi, Jason Weston, and Xian Li. Self-taught evaluators. arXiv preprint arXiv:2408. 24000.000000006666 2024.
Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. In The 6lst Annual Meeting Of The Association For Computational Linguistics, 2023.
Sanford Weisberg and R Dennis Cook. Residuals and influence in regression. 1982.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. In ICLR 2024 Workshop on Navigating and Addressing Data Problems for Foundation Models, 2024.
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your mathematical questions for large language models. arXiv preprint arXiv:2309.12284, 2023.
Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. Large language model as attributed training data generator: A tale of diversity and bias. Advances in Neural Information Processing Systems, 36, 2024a.
Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. arXiv preprint 24060.2406.06046, 2024b.
Weizhe Yuan, Richard Yuanzhe Pang. Kyunghyun Cho, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston. Self-rewarding language models. arXiv preprint arXiv:2401.100222 2024.
Xiang Yue, Tuney Zheng, Ge Zhang, and Wenhu Chen. Mammoth2: Scaling instructions from the web. arXiv preprint All::200500544 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv: 190.00780 2019.
14

---

Published as a conference paper at ICLR 2025
Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu. Tinyllama: An open-source small language model, 2024.
Chenyang Zhao, Xueying Jia, Vijay Viswanathan, Tongshuang Wu, and Graham Neubig, Self-guide: Better task-specific instruction following via self-synthetic finetuning. ArXiv, abs/2407. 2024. htttp::///app
Yanli Zhao, Andrew Gu, Rohan Varma, Liang Luo, Chien-Chin Huang, Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, Alban Desmaison, Can Balioglu, Pritam Damania, Bernard Nguyen, Geeta Chauhan, Yuchen Hao, Ajit Mathews, and Shen Li. Pytorch fidp: Experiences on scaling fully sharded data parallel, 2023. URL/ttts:://axxii www.iii..rrraeee.oe abs///b///// att.abb//.2330..
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging Ilm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36, 2024.
Kun Zhou, Beichen Zhang, Jiapeng Wang, Zhipeng Chen, Wayne Xin Zhao, Jing Sha, Zhichao Sheng, Shijin Wang, and Ji-Rong Wen. Jiuzhang3. O: Efficiently improving mathematical reasoning by training small data synthesis models. arXiv preprint arXiv:2405. /4365, 2024.
15

---

Published as a conference paper at ICLR 2025
A TRAINING DETAILS
The hyperparameters used during training teachers and students are as follows. We employ the A optimizer (Loshchilov & Hutter 2019) with WSD scheduler (Hu et al. 2024a). For SFT, the 8B model utilizes a maximum learning rate of 5e while the 1B model uses Ie- The WSD scheduler is configured with warmup ratio of 0.1, stable ratio of 0.5, and decay ratio of 0.4, with the learning rate decaying to one-thousandth of the maximum. The epoch is set to 1, batch size is set to 32 and the dropout is We mask non-targe non-target tokens, calculating the loss only on target tokens. If the student model does not have chat template itself, we apply the Llama3-8B formatted chat template, as shown in 7 with bos token, eos token and pad token set to < start header id >> < leen__he _eaderr d and < send_hedeer_ic >, respectively. For DPO, we use a learning rate of 1e-0, set 0.1, and use batch size of 2, while other parameters remain the same as in SFT.
Figure 7: Chat Template
Chat Template
{8 if messages ['ree"""' == system' $} {8 set offset 8)
1% else %}
{8 set offset $}
{5 endif 8]
{{ bos.token }}
{8 for message in messages 'user") !=
{8 if (message r'rel' (Loop.index0 2 offset) $
If raise exception Conversation roles must al alternate userass stantuserass sttteeerrrssssstt........ stant....
}}
(s endif
|[ <|start_header.id message (' ('role'] +
< <lendhheader.id]> message content') | trim +
eos.token }} {s endfor $)
{s if aed..eensraaiion..oonn $ {{ <<ssstrtt header 'assistant'
<<) <<lend header id/>
n' {$ endif 8}
We Hugging Face TRL codebase (von Werra et al. 2020) perform both full parameters fine- tuning and direct preference optimization. For the 8B model, we employ the Hugging Face Accelerate codebase (Gugger al., (022) to facilitate FSDP training (Zhao al. 2023). All the parameters introduced in this section are summarized in Table 4
Table 4: Training Parameters
Method
Learning Rate
Weight Decay
Warmup Ratio
Stable Ratio
Decay Ratio
SFT
5.0c
0.0
0.1
0.5
0.4
DPO
1.0c
0.0
0.1
0.5
0.4
Method
Minium Learning Rate
Epoch
Per Device Train Batch Size
Gradient Accunulation
Train Butch Size
SFT
5.0e
2
32
DPO
2
Method
Max Length
Dropout
BF16
Flash Attention 2
Beta
SFT
1024
0.0
True
True Truc
DPO
1024
0.0
True
0.1
16

---

Published as a conference paper at ICLR 2025
B THEORETICAL GUARANTEE OF LOCAL DATA INFLUENCE
This section provides detailed explanation of the derivation for computing local data influence and the rationale behind its effectiveness. We referred to the derivation method in Yu et al. (0024b). We use Pier to represent the reference set and m represent the student model that we calculate local data influence on. The derivation begins with the standard influence functions Koh & Liang (2017); Weisherg Weishere & Cook (1982) which quantify the change in reference loss when a data point J; upweighted by small We denote the optimal model state after the upweighting me,a; arg mi + j=1 (xx m) + eC(x;; m) and simplify the optimal model under 0 case (i.e., no upweighting) as m. The influence of upweighting is then given by:
der dL(Dret me.zi)
Im (x:: Dret)
l==0
(5)
de
dm le=0
= - Mn m)
(6)
de
= -V m)) H V [(a m))
where Hn, = - from Eq. to 7 is given by building quadratic approximation to the empirical risk around and iperforming a single Newton step as shown in Koh Liang (2017). Now let's consider the scenario
Ca, | m) is the Hessian matrix, which is positive definite. The derivation
in which z; is incorporated into the training data. In this case, L. and the parameter difference due the inclusion of Tú mi further represented as:
V (xx m) and the influence in Eq. 7 can be
Im (r.; Dret) R mn m) (m] a.. m) = n(e(Dref
(8)
m m= L(D m))
(9)
x LD(ree m) + L(Dref mm...)).
(10)
So far, we have successfully derived the method (Eq. TOp of caleulating local data influence used in $ 3.2 Using the supervised fine-tuning algorithm A, we denote the model state as A y Fe; m). which is updated on the synthetic data point (æ,, y1) for one step. Replacing the variables in Eq. I0 with the notation of our method, we can obtain:
Im (2:; Dret) -D(rre A(ya ong m)) L(Dret m)
(11)
C STATISTICS ON SYNTHESIS DATA
We plot the top 20 most common root verbs (inner circle) and their top 4 direct noun objects (outer circle) in the generated instructions by Self-Instruct (Figure B). the first iteration of Montessori- Instruct (Figure D. and the second iteration of Pantessori-Insstrue (Figure I0]. respectively.
We observe an increasing trend in instructions such 'write,' "provide," and 'make.' well consistent trend for instructions like 'explain' and 'describe.' These commands typically require more general detailed information and lead longer, more complex responses. Meanwhile, commands like 'translate' and 'calculate" show decline, as they usually require straightforward answers and simpler formats. This outcome demonstrates that Montessori Instruct helps the teacher model generate more detailed and informative instructions, thereby improving student performance.
We also plot the distribution of tokenized instructions and responses generated by Self-Instruct and Montessori-IInttruc for comparison. As shown in Figures [nd 12 there is an increasing trend in the length of instructions, while the length of responses remains relatively unchanged. This aligns with our design. which focuses optimizing instructions based on prompts rather than optimizing responses based on instructions. The increased length of instructions also reflects the teacher's data synthesis strategy shifting toward more complex and informative instructions.
17

---

Published as a conference paper at ICLR 2025
Figure 8: The top 20 most common root verbs (inner circle) and their top direct noun objects (outer circle) in the generated instructions by Self-Instrus
aarrgraae
story
difference
differences
points
article
write
waste
summariza
concept
consumption
explain
pollution
reduce
footprint
translate
sentence
lackot
describe
calcolate
text
phrase
differences
use
plan
Bonm
tha
Dnssg
steps
process
manngger
sources
example
d
exomples
eaia al
Figure 9: The top 20 most common root verbs (inner circle) and their top direct noun objects (outer circle) in the generated instructions by Montessori-Instte (iteration 1)
diffe fffreecces
guide
Bilorrmtcs
concept
story
waste
stress
consumption
recipe
pollution
explain
area
reduce
write
paragraph
calculate
example
provide
examples
describe
organioe
ingeeve
conatdor
skill
process
ifferences
use
summarien
translato
pla nanning
discovers
give
make
include
factors
Bookshe H
steps
manager
Dack
ingredients
auradients
transporiation
Intredii
points
butter
senfence
suitcase
cale
phrase
talent
base
radiu
who
trip
18

---

Published as a conference paper at ICLR 2025
Figure 10: The top 20 most common root verbs (inner circle) and their top direct noun objects
(outer circle) in the generated instructions by MontessoriiInstuue (iteration 2)
email
tydleeeered
concept
differences
recipe
steps
ditferen difference
ingredients
explain
guide
write
include
steps
describe
process
differences
make
provide
create
shitb
dish
serves
dish
conaddr
butter
plan
use
Hap
pizza
that
example
examples es
ingredients
aurds
that
people
factors
trip
Figure 11: Distribution of tokenized instructions generated by Self-Instruct and Montessori ciattessori-Insrrett
2000
# Instructions
* Instructions
1000
1500
750
1000
500
500
250
20
40
60
60
100
0
20
40
60
80
100
Tokenized Instruction Length
Tokenized Instruction Length
(a) Self-Instruch
(b) centessori-Instruue
Figure 12: Distribution of tokenized responses generated by Self-Instrue and Montessori-Innse Instruct
Responses
600
Responses
400
200
#
600
1200
Tokenized Response Length
Tokenized Response Length
(a) Self-Instrus
(b) Montessori Instruct
19

---

Published as a conference paper at ICLR 2025
D ADDITIONAL EXPERIMENTAL DETAILS
D.1 PROMPTS USED FOR INSTRUCTION GENERATION.
In this section, present the prompts used in centessori-Instruue Figure 13 illustrates how prompt the teacher model to generate new instructions. We begin by outlining some requirements for the teacher, followed by inserting 8-shot seed examples sampled from both the seed pool and the data pool generated in the previous iteration. We then extract the instruction from the teacher's output using regex matching and filter out those with incorrect formats.
Figure 14 displays the prompt used in our ablation studies on the effectiveness of Local Data Influence. In this study, we evaluated different methods for assessing the utility of synthetic data, one of which involved using ALM-as-aaJudes- (Zheng et al. (2024). We adapted the prompt from Self-Reward (Yuan al., 2024) and added an additional point to evaluate the quality of the instruction, resulting maximum score of 6 points.
Figure 13: Prompt for Generating Instructions
Prompt
Generate an instruction. This instruction should be a question that humans would be ask. It can in imperative or interrog- ative form. We will use the instructions you generate to train models, so you must ensure that the instructions generated are of high quality and correct and also keep the instruction clear and concise.
You should:
1. Briefly explain why you generate this instruction.
2. Think about whether you need to add some input to this instruction so that can be answered directly. (For example, for tasks that involve summarizing, you need to provide the paragraph to be summarized).
3. Return you output strictly following the format:
Your generated instruction should strictly follow the following formar:
<instruet <instruetion><<OUR YOUR
INSTRUCTION HERE><YOUR
INPUT HERE></inst. cennstruution.
there is no need add imputs answer instruction. you can skip the <YOUR INPUT HERE> part. If you need to add inputs, just replace the <YOUR INPUT HERE> with the input. Now here are some examples of reference instructions, and please generate only one instruction.
D.2 DECODING STRATEGIES
We list all the parameters used for decoding outputs from language models in Table 5 Separate parameters are used for generating instructions and responses. A higher temperature used for instruction generation to encourage diversity, enabling us to leverage local data influence to identify more informative instructions. For responses, we use a temperature of 0.6 to reduce uncertainty. Additionally, two penalty techniques are employed to mitigate duplication issues during synthesis.
D.3 REWARD RESULTS WITHOUT THE EXTERNAL JUDGE
In this section, we report the results of the original Self-Reward (Yuan et al. 2024) method. Self- Reward requires the student model to generate responses given instructions, and then assess their own responses by generating judgments and scores ranging from to 5 using LLM-as-a-Juee a--ddee (Zheng et al. 2024). It then employs Direct Preference Optimization (DPO) to encourage the student to synthesize higher-scoring responses. However, this approach demands a high level instruction- following ability from the student model. The authors of Rewadd employ Llama2-70B as the
20

---

Published as a conference paper at ICLR 2025
Figure 14:
LLM-as-a-Judee Prompt for evaluating instructions and corresponding responses in our ablation studies on the effectiveness of Local Data Influence
Prompt
Review the user's instruction and the corresponding response using the additive 6-point scoring system described below, Points accumulated based the satisfaction of each crite- rion:
Add I point the response relevant and provides some in- formation related the user's inquiry, even if is incomplete or contains some irrelevant content.
Add another point the response addresses a substantial
portion of the user's question, does not completely resolve query or provide a direct answer.
Award third point if the response answers the basic
ments of the user's question a useful way, regardless whether it seems to have been written by an Al Assistant or if it has elements typically found in blogs or search results.
Grant fourth point if the response is clearly written from an Al Assistant's 's perspective, addressing the user's question directly and comprehensively. and is well-organized and help- ful, even if there is slight room for improvement in clarity, conciseness or focus.
Bestow fifth point for response that is impeccably tailored to the user's question by Al Assistant, without extraneous information, reflecting expert knowledge, and demonstrating high-quality, engaging. and insightful answer.
Award an additional point you consider this instruction be of moderate difficulty, requiring thought and analysis rather than being straightforward task. User:
<INSTRUCT INNOUCTTIOONEE SON_OON_EERRRR HERR>>>>>
(response>>RRSSPOEEE BEEEEOONNSSSSSRSSSESEEEE //E__EERRR///EE. r/reponse/e
After examining the user's instruction and the response:
Briefly justify your total score, up to 100 words.
Conclude with the score using the format: \Score:
<total points>"
Remember to assess from the Al Assistant perspective, uti-
lizing web search knowledge necessary. To evaluate the response in alignment with this additive scoring model, we'll systematically attribute points based on the outlined criteria.
Table Decoding Parameters using vLLM
Generate Instruction
Generate Responses
temperature
0.6
top-p
0.9
0.9
frenuency-penalty
0
0
presence_penalty
1
1
repetition penalty
1.5
max_token
1024
1024
student model for this reason. our experimental setup with Llama3- 8B and TinyLlama-1.1. 1.11, both models lack sufficient instruction-follet following capabilities and fail to produce detailed judgments and valid scores. For example, Llama3-8B's s scores are skewed, clustering around 4 and 5, making difficult differentiate between responses. The 1.1B model's scores even do not follow the rules in
21

---

Published as a conference paper at ICLR 2025
Table 6: Evaluation of training 8B/I.IB students using the original Self-Reward settings compared to without relying on external judges.
In Domain
Out-Of-Domain
Alpaca Eval 2.0
MT-Bench
MMLU
GPQA
ARC-C
GSMSK
wellaSwag
Methods
LC-WR
WR
Score
Accuracy
8B Setting:
Student-Llama3-81
No fine-tuning
2.09%
3.39%
5.597
62.15
24.33
57.85
51,25
81.96
Self-Instruct
50%
50%
6.490
62.42
31.92
59.98
58.76
80.93
Self-Reward
Iteration
2.45%
4.06%
5,442
61.79
24.30
57.81
49.92
80.75
Iteration 2
2.69%
4.71%
5.428
61.79
23.58
57.64
49.53
80.17
1.1B Setting:
Student=Tinyllama- 1.1B
No fine-tuning
17.89%
17.56%
1.020
26.16
23.88
37.12
1.97
62.61
Self Instruct
50%
50%
2.154
26.21
24.78
37.97
1.82
62.47
Self Reward
lteration 7.79%
8.13%
1.000
23.58
22.30
36.55
0.94
61.92
Iteration 6.34%
7.57%
1.000
23.44
22.06
36.49
0.98
61.24
the prompt and fall outside the specified to 5 range. Therefore, in our main experiment, we use GPT-4o an external judge to score the student responses. Nonetheless, we also report results here based on the original Rewf-Reward settings, where the model judges own responses without relying on more powerful external model.
E COST ANALYSIS
E. TIME OVERLOAD
Compared to Self-Instruct (Wang al. 2023). our method introduces additional overhead in: (1) collecting local data influence to construct the preference dataset ($ 3.2 (2) and performing DPO optimization the teacher model ($ 3.3 The majority of the computational overhead arises from collecting local data influence. This process begins by generating instructions and responses to create a probing dataset, distinct from the training set used for fine-tuning the student, and used solely for calculating local data influence. Then, traverse the entire probing dataset, fine-tuning the student model on each individual data point to collect its corresponding local influence. For each data point, loading the student's warmed-up checkpoint from disk, training for one step, and evaluating on the reference dataset are the primary time-consuming steps. We provide a detailed breakdown of the time required for these steps table and calculate the average time needed run the entire Montessori -ontessori-Instruct process and resulte in the final student model. The calculations are based on probing dataset and training dataset, each consisting of 10K entries.
However, there are two simple ways to reduce the time demand for Montessori Instrrcc... First, the process of collecting local data influence can be parallelized independently on a heterogeneous compute system speed up execution, with need for communication between systems- common bottleneck in distributed training. In our experiments, we utilize H100 GPUs to accelerate this process. Second, as demonstrated in our experiments ($ 5.4). MontessoriiInnstuu shows strong generalization capabilities. In practice, a smaller model can be used to collect data influence for updating the teacher, which can then synthesize data for larger models. This approach significantly reduces the computational overhead compared to using larger models directly for collecting local data influence.
22

---

Published as a conference paper at ICLR 2025
Table 7: Time Overload Statistics
Task
Sub task
8B
1B
generate instructions
0.372s
generate responses
0.031s
load warmuped ckpt from disk
2.69s
1.08s
collect local data influence / per data
fine-tune for one step eval reference set
4.12s
0.79s
4.19s
1.26s
total
13.403s
3.533s
Task
8B
Time for DPO Training per data
0.362s
Task
Method
8B
Self-Instruet Montessorii
0.486s
0.422s
Time for obtaining the final student model / per data
5.842s
1.834s
E.2 COST-PERFORMANT RELATIONSHIP
We provide further clarification on the cost-performance relationship of our method compared all baselines. We analyzed the Performance-FLLOPs curve of four methods, with a particular focus on the changes in Self-Instruct's Alpaca Eval and MT-Bench Score their FLOPs increase to levels comparable to those of Montessori-Instruct We scale the FLOPs of Self-Instruct by synthesizing additional data. We also marked the Performance-FLOW FLOPs relationship of the two baselines, LLM2LLM and Self-Reward, in the following figures.
62
LC LC Win Rate(%)
62
MT-Bench Score
7.3
Win Rate(%)
58
58
7.0
54
54
6.7
50
50
6.4
Self belf-lnstruct
Self Insruct Monkesori-nnemcc LLM2LLM
Self Instrut
46
46
Montesori--nmnne
L.MM2LMM
LLM2LLM
Self-Reward
Self Rowsed
Self-Reward
3
9
12
18 21
3
12
15 18 21
5.50
18 21 FLOPs (x (x120)
9 12
FLOPs (x1020)
FLOPs (x1029
(a) Alpaca Eval WR
(b) Alpaca Eval LC-WR
(c) MT-Bench
Figure 15: The Performance-FLOP curve for all four methods.
can be seen that Self-Instrue quickly reached the upper bound during the scaling-up process, and even with more FLOPs, no better performance improvement can be achieved. The reason may be that the data generated by Self-Instruct is severely homogenized, In contrast, the upper bound of our method is significantly better and continuously grows when we invest more FLOPs into it.
Then we give a computational result of the FLOPs estimated for four methods, as well as the pretraining and cest-time-scaling The detailed derivation is provided in E.3 The main FLOPSs for Montessori-Instrnee come from processing probing data. In the Table T we used 10K probing data to utilize the most resources to achieve the best performance, but as the Figure 3a and Figure 3b suggests, using around 1K probing data can already achieve better performance than other baselines. To make fair comparison, calculate the FLOPs under 1K probing data. We estimate the FLOPs as follows (Llama3-8B. (aaman--B-Instiee IB-Insttuct as the teacher, Llama3-8P as the student):
Self-Instruct: 1.34 x 1020 FLOPs
Self-Reward 2.11 x 10 FLOPs
LLM2LLM: 2.3 x a0 FLOPs
Montessori- 6.43 x 1020 FLOPS
23

---

Published as a conference paper at ICLR 2025
Pretrain Llama3-8B. 1.87 x 10 FLOPs
Inference-Timee Time Scaling: 1.60 x 1023 FLOPs
We can see that Montessori-Instruuc's FLOPs are 7 times less than Self-Reward. Furthermore, if we use the proxy model (Yu al. 2024b), such as a smaller-sized model (e.g.. 1B parameters for assisting an 8B model) to process probing data, Montessori's s FLOPs can further reduce to 1.92 1030 FLOPs. This makes comparable to Insfruintt while still outperforming it. Using proxy model has promising potential for enhancing both efficiency and performance, which leave for future work. Regarding the pretraining, since the computational cost during the SFT phase significantly lower than that during the pretraining phase 110 times smaller), even if increase resource investment in SFT, its overall consumption remains minimal. Recent work has focused on scaling inference time achieve better performance (Snell al. 2024). However, the inference-time scaling FLOPs are also significantly larger than those of SFT, being approximately 10 times greater, according to Sardana al. (2023). Nevertheless, our teacher training represents one-time cost, As demonstrated in Section 5.4, the optimized teacher can assist multiple students in improving their performance without the need for retraining from scratch.
E.3 DERIVATION OF FLOPS
When generating synthetic data, the input window includes both prompt and seed data, we set the input length to 2048.
For instruction-basee input/output, the input/output length is 128.
For response-based input/output, the input/output length is 1024.
For judgment-basee input/output using LLM, the input/output length is 1024.
We define the computational cost of generating one token for an input of length 128 as one unit F. During instruction fine-tuning, the input and output lengths are 128 and 1024, respectively. The backward FLOPs are approximately twice the forward FLOPs. For one data sample, the training FLOPs can be estimated as:
1024F 3072.F
FLOPs calculations based on ccalfloss (2024). where F= 1.927 FLOPs.
24

---

Published as a conference paper at ICLR 2025
Method
FLOPs (F)
Self-Reward
Synthesize 10K instructions from seed Synthesize 4 responses per instruction Generate 3 judgments per response
16.F x 128 10K 20480K
40K x 1024F = 40960KF
40K 8F 1024
983040KF
Train with 10K pairs using DPO
DPO1uK
Synthesizes 2K instruction-response- judge sets
(16F x 128 + Fx 1024 + 8F x 1024)
x2K 22528KJ
Perform SFT on student
2K
Total
2 1100MF + DPO
LLM2LLM
Synthesize 10K instructions from seed
16F x 128 x 10K 20480KF
Generate 1 response per instruction
10240KF
Student responds to each instruction Resynthesize 10K instructions
F 1024 x 10K = 10240KF
20480KF
Generate 1 response instruction
10240KF
Perform SFT on student
SFT
Total
R 120M
Montessori
Synthesize 1K instructions from seed
2048KFF
Generate 1 response per instruction
1024KF
Train student with each instruction Evaluate trained student on validation set Perform DPO updates teacher with 6K samples Resynthesize 10K instructions
SFT 1OK
1KF
1024
256 262144KK
DPO 20480KFF
Generate 1 response per instruction
10240KF
Perform SFT on student
SFT
Total
340MJF + DPO
Use a 1B model for probing data
100MF + DPO
Self-Instruct
Synthesize 1K instructions from seed
2048KF
Generate 1 response per instruction
1024KF
Perform SFT on student
SFT10K
Total
2 70MJ
Table 8: FLOPs Computation Table for Different Methods
F EXPERIMENTS UNDER THE SELF-EVOLVE SETTING
In our primary experiment, leveraged teacher model generate tailored synthetic data aimed enhancing the capabilities of a different student model. Here, we shift our focus to explore whether LLMs can harness synthetic data generated by themselves achieve self-improvement-t paradigm we term the "Self-Evolve" setting. To investigate this, we adapt our MontessooriInstructt framework by aligning the student model with the teacher model. Starting from an identical check- point, the model generates synthetic data for itself, employing influence scores to identify the most beneficial and tailored samples, and subsequently performs Direct Preference Optimization on itself. Notably, the fine-tuning process begins anew from the initial checkpoint, rather than building upon a pPOt-DPO state. We evaluate this paradigm using both 1a-ma3-8B-Instruun an instruction-tur model, and Llama3-8B, its pretrained version, to assess the potential of self-imprrvemmnt, The results are presented Table
Our findings reveal that Llama3-8B -BB-sssttttt achieves superior performance across all benchmarks under the self-evolve setting, exhibiting a consistent upward trend in capability. Remarkably, even the non- con-instruction-tuee -uuned Llama3-8B demonstrates self self-improvement at the 8B parameter scale. However, while Llama3-8B exhibits gains with each iteration, the rate of improvement diminishes over time. This suggests that the pretrained model struggles surpass its instruction-tunee version
25

---

Published as a conference paper at ICLR 2025
Table iepl-impoveeett performance of Llama3 models across different iterations. The Winning Rate (WR) and Length-Control Winning Rate (LC-WR) are compared to the Llama3-8h -er---B--nnnteee model. The best performances are marked in bold.
Alpaca Eval 2.0
MT-Bench
Methods
WR
LC-WR
Score
Maama3-B-IInstruce
50.00%
50.00%
7.472
Llama3-8B (No fine-tuning)
3.39%
2.09%
5.597
Teacher-Student=LLama-3
Iteration 1
26.76%
26.53%
6.224
Iteration 2
35.42%
34.76%
6.308
Iteration 3
39.84%
38.12%
6.386
Iteration
53.74%
52.51%
7.563
Iteration 2 Iteration 3
56.78%
54.84%
7.595
58.62%
56.12%
7.611
through self-evolution alone at this stage. We attribute this limitation to the suboptimal quality and restricted diversity of the synthetic data produced by the models themselves. [Shumaloov et al. (2023b) reveals that the perplexity synthetic training data tends converge toward low-value range after multiple iterations, offering diminishing returns in terms of novel and beneficial information for model enhancement. We hope that future research will devise innovative strategies to bridge the gap between synthetic and organic data, unlocking the full potential of self-evolving LLMs.
26


Reviews:
[{"review_id": "oAafNHmG2z", "reviewer": "Reviewer_hB25", "summary": "This paper introduces Montessori-Instruct, a new framework for generating synthetic training data to enhance the learning of student language models. The authors propose a method that aligns data synthesis by a teacher model (e.g., Llama3-8B-Instruct) with the learning process of the student model (e.g., Llama3-8B). The approach involves assessing the local data influence of synthetic data points on the student model to understand its learning preferences, followed by training the teacher model using Direct Preference Optimization (DPO) to generate data better suited for the student. Experiments show that Montessori-Instruct significantly improves performance on benchmarks like Alpaca Eval and MT-Bench.", "questions": "See weaknesses above", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The concept of aligning the teacher model to better match the student model’s learning preferences is well-motivated.\n\n* The method of assessing the impact of individual data points on student learning (i.e., local data influence) is a interesting approach backed by theoretical guarantees.", "weaknesses": "* Including experiments with multiple runs and reporting the mean and standard deviation would strengthen the reliability of the results.\n\n* The performance gains, particularly on out-of-domain benchmarks, appear minimal when weighed against the additional computational cost involved in training the teacher model.\n\n* The idea of local data influence, which is the crucial algorithm for the proposed solution, is not new but taken from a previous paper, as the authors have mentioned. This reliance on existing techniques may reduce the perceived originality of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "q66vQQ0Of5", "reviewer": "Reviewer_5Czb", "summary": "This paper proposes a customized synthetic data generation pipeline that seeks to improve the expert model towards generating more useful and debiased data for the student model by aligning the expert model towards data distribution with higher influences on the student model via DPO. Specifically, this paper (1) first adopts an established influence function in active learning to measure the utility of a single data point in the probing set, (2) then they construct preference data using the positive and negative data influences sampled from a same prompt, and use this data to align the expert model. Finally, they regenerate data from the updated expert model and finetune the student model with this data. While the method proposed in this paper generally make sense, it suffers from a lack of ablation study and analysis on the additional cost (e.g. manual curation cost of the reference set, per sample influence inference cost, training cost of DPO on expert model) incurred by this data selection process.", "questions": "See the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a synthetic data generation pipeline to customizedly align the expert model towards higher data utility over the student model. Specifically, \n+ the method proposed is sound and can intuitively address the OOD issue with the synthetic data to enhance the generalizability of the student model.\n+ this paper has good presentations and conveys the idea clearly.", "weaknesses": "The paper lacks more informative ablation study and in-depth explanation and analysis, especially,\n+ How do the authors pick the reference dataset and determine its size? It seems this dataset is critical to judge the influence of the data samples on the student model and highly determine the performance of the fine-tuned student model later. \n+ Can the author provide some ablation study on the size of the reference dataset and its selection method?\n+ How many samples do the authors obtain for each prompt in order to get a pair of positive and negative instruction? What are the additional costs behind these?\n+ Calculating the inference for each data point in the probing set seems a bit costly as it involves an LLM optimization (even if just one step). Can the authors provide an ablation study on the sample size of the probing set and the final performance?\n+ It also introduces some other additional costs as the proposed method involves DPO on the expert model, while most of the baselines do not. As the expert model is large in size, this training cost seems also non-negligible.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wbhmD6cBs7", "reviewer": "Reviewer_PpzE", "summary": "This paper introduces Montessori-Instruct, a novel framework that optimizes the teacher model's data generation capabilities by aligning them with student learning preferences. The framework uses local data influence measurements to estimate how synthetic data impacts student learning, then employs DPO to tune the teacher model accordingly. Experiments with Llama-3 and TinyLlama models showed significant improvements over baseline methods.", "questions": "- In lines 226-228, how were the 6,792 preference pairs collected from the 10K probing dataset?\n- Will the general capabilities of the optimized teacher model deteriorate?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- A novel data influence-driven approach where the teacher model actively learns to generate training data by optimizing against student's learning preferences, in contrast to previous works that use a static teacher.\n- The experimental comparisons and benchmarks are comprehensive, the proposed method shows significant improvements, and the ablation and analysis experiments are thorough.", "weaknesses": "The main drawback of this work is the excessive computational overhead beyond the objective of training the student model. Although the authors discussed this additional computation in detail in Section 6 and Appendix E, I believe that the impact on the paper's practicality cannot be dismissed through discussion alone. The majority of the additional computational cost comes from local data influence for the student, where one step of training is required for each sample, along with evaluation on the reference dataset. To accelerate this process, the authors used 8xH100, which is a very expensive overhead. This makes me question whether we could actually achieve similar gains by investing these resources in using stronger teacher models and building more sophisticated reward model pipelines[1]. I encourage the authors to (1) discuss potential directions that could directly reduce the unit cost of local data influence, rather than encouraging more computational resources, thereby enhancing the promise of local data influence-based approaches. (2) present the additional monetary costs for each method (i.e., APIs used for baseline methods and computational resources used for acceleration) to improve the fairness and transparency of the comparison.\n\n[1] Snell, Charlie, et al. \"Scaling llm test-time compute optimally can be more effective than scaling model parameters.\" _arXiv preprint arXiv:2408.03314_ (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "izLUAwtFyF", "reviewer": "Reviewer_D2rt", "summary": "This paper introduces MONTESSORI-INSTRUCT, a framework to generate synthetic data for training student language models - tailored for the student model's learing process / ability. The method first uses local data influence to measure the utility of synthetic data points for student learning. Then, it optimizes a teacher model with DPO to generate more effective synthetic training data by aligning with the student model's learning preferences. \n\nThe authors evaluate MONTESSORI-INSTRUCT using `Llama3-8B-Instruct` as the teacher and `Llama3-8B/Tinyllama-1`.1B as students. Evaluation on Alpaca Eval and MT-Bench shows that Montessori-Instruct outperforms existing methods like Self-Instruct. The authors also show that Montessori-Instruct can beat GPT-4o on in-domain and out-of-domain evaluation. Ablation studies highlight the effectiveness of using data influence to capture student preferences, the benefits of optimizing teacher parameters, and the robustness across different configurations.", "questions": "1. Is there an easy way to see how the effectiveness of the framework scales with the data? You are welcome to scale down / scale up, to a point where the experiment is reasonable.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Writing**. \n- Introduction is well-written. The motivation is clear, the problem is well-defined, key contributions are listed and aligned with the structure of the paper.\n- Related work is well-written, extensive, and up-to-date. Related works is also well-organized in logic, and set up a good foundation between existing line of works and the proposed method.\n\n**Evaluation**.\n- Evaluation has good coverage of different baselines, and the selection of baselines are realistic. Generally, the evaluation is comprehensive and thorough.\n- Ablation study is comprehensive and thorough. The ablation studies the effectiveness of teacher optimization, seed data, (multiple) iterations, and very clearly demonstrates the generalizability of the method.\n\n**Originality**. The idea of the paper is novel, and well-motivated. \n\n**Significance**. The proposed method is a good contribution to the field of synthetic data generation for language model training. The proposed method is generalizable to other domains, though with additional overhead as stated in the limitation section.", "weaknesses": "**Scale of experiment.** The limitation section points out that in the experiment, the scale is chosen as a fixed 10k data points. Ablation on the scale of experiment will be helpful to show the generalizability of the method on the scale of data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)