Pedagogically-Inspired Data Synthesis for Language Model Knowledge Distillation
Authors: Bowei He
Yankai Chen
Xiaokun Zhang
Linghe Kong
Philip S. Yu
Xue Liu
Chen Ma
Venue: ICLR 2026 Poster (2026)
Keywords: Data Synthesis
Large Language Model
Knowledge Distillation
Primary area: foundation or frontier models, including LLMs
TL;DR: This paper introduces a pedagogically-inspired data synthesis framework that distills knowledge from teacher to student language models through deficiency diagnosis, curriculum structuring, and stage-wise adaptation.

Abstract: Knowledge distillation from Large Language Models (LLMs) to smaller models has emerged as a critical technique for deploying efficient AI systems. However, current methods for distillation via synthetic data lack pedagogical awareness, treating knowledge transfer as a one-off data synthesis and training task rather than a systematic learning process. In this paper, we propose a novel pedagogically-inspired framework for LLM knowledge distillation that draws from fundamental educational principles. Our approach introduces a three-stage pipeline—**Knowledge Identifier**, **Organizer**, and **Adapter** (**IOA**)—that systematically identifies knowledge deficiencies in student models, organizes knowledge delivery through progressive curricula, and adapts representations to match the cognitive capacity of student models. We integrate Bloom's Mastery Learning Principles and Vygotsky's Zone of Proximal Development to create a dynamic distillation process where student models approach teacher model's performance on prerequisite knowledge before advancing, and new knowledge is introduced with controlled, gradual difficulty increments. Extensive experiments using LLaMA-3.1/3.2 and Qwen2.5 as student models demonstrate that IOA achieves significant improvements over baseline distillation methods, with student models retaining 94.7\% of teacher performance on DollyEval while using less than 1/10th of the parameters. Our framework particularly excels in complex reasoning tasks, showing 19.2\% improvement on MATH and 22.3\% on HumanEval compared with state-of-the-art baselines.

Published as a conference paper at ICLR 2026
PEDAGOGICALLL-INSPPREE DATA SYNTHESIS FOR LANGUAGE MODEL KNOWLEDGE DISTILLATION
Bowei Hel:?, Yankai Chen 1.2,, Xiaokun Zhang,, Linghe Kong", Philip S. Yu?,
Li Xue Liu 1.2 Chen Ma 1 MBZUAL, McGill, CityUH, 4 SITU, 5 UIC
ABSTRACT
Knowledge distillation from Large Language Models (LLMs) to smaller models has emerged as a critical technique for deploying efficient Al systems. However, cur- rent methods for distillation via synthetic data lack pedagogical awareness, treating knowledge transfer as a one-off data synthesis and training task rather than a sys- tematic learning process. In this paper, we propose novel pedagogically -insiired framework for LLM knowledge distillation that draws from fundamental educa- tional principles. Our approach introduces three- stage pipeline- Pipeline-Knoweee Knowledge Identifier, Organizer, and Adapter (IOA) that systematically identifies knowl- edge deficiencies in student models, organizes knowledge delivery through progres- sive curricula, and adapts representations to match the cognitive capacity of student models. We integrate Bloom's 's Mastery Learning Principles and Vygotsky's 's Zone of Proximal Development to create dynamic distillation process where student models approach teacher model's performance on prerequísite knowledge before advancing, and new knowledge is introduced with controlled, gradual difficulty increments. Extensive experiments using I.L.M-331//3 1/3.2 and Qwen2.5 as student models demonstrate that IOA achieves significant improvements over baseline distillation methods, with student models retaining 94.7% of teacher performance DollyEval while using less than 1/10th of the parameters. Our framework particularly excels in complex reasoning tasks, showing 19.2% improvement on MATH and 22.3% on HumanEval compared with state-of-the-art baselines.
INTRODUCTION
Knowledge distillation (KD) has been a widely adopted approach to compress a large teacher model into smaller student model by transferring knowledge (Hinton et al. 2015). Early-stage language model distillation methods mostly rely on minimizing the discrepancy of logit (output
distribution) (Gu et al., 2024, Zhang et al., 2023) or intermediate layer representations (Jiao et al.. 2020: Sun et al. 2020) between the teacher model and student model. However, due to the tokenizer mismatching and access limitation of proprietary large language model (LLM) parameters like OpenAI ol (Jacch et al. 2024) and GPT-4 (Achiam et al. 2023), many recent distillation methods instead train the smaller LLM (SLM) using synthetic data generated by the teacher model. In fact, there have been many successful applications of LLM distillation with synthetic data, like Stanford's Alpaca model (Taorí et al., 2023). which fine-tuned the open-sourced LLaMA-7B model (Touvron al. 2023) with synthetic data from OpenAl's -ext-davinci-0033 (i.e. the original ChatGPT model), improving its dnstruction-follovii
capability to comparable level. For the most recent reasoning models, DeepSeek (Guo et al. 2025) distilled DeepSeek- R1's outputs with simple fine-tuning, enabling the efficient DeepSeck-RI RI-Distill -waiill--wen-BBB trained from Qwen2.5-7B (Qwen) 2024) to outperform specially designed reasoning model QwQ- DWQQ-2B-PPP 3-----eeewwwww (Qwen) 2025). The distilled model DeepSeek-R -Distill-Qwen-a even achieved higher performance than OpenAl-ol-mini on complex reasoning tasks like math, code, and science question answering (QA).
Some works have proposed different data synthesis strategies to distill LLM's knowledge and reasoning abilities into SLMs. One line of work (Yu et al. 2024) focuses on bootstrapping seed
"Corresponding authors: yankai chenea org, chenma@ typ. edu. .hk

---

Published as a conference paper at ICLR 2026
questions by rewriting them from multiple perspectives, thereby enhancing distillation generalizability. Another line (Hsieh et al. 2023] Chen et al. (0244b Dai et al. (024) emphasizes eliciting and distilling the teacher LLM's reasoning ability. Adversarial distillation frameworks such as Lion (Jiang et al. 2023) leverage feedback by identifying hard instructions and generating new ones to iteratively strengthen the student model's reasoning ability. Beyond these, counterfactual data synthesis (Feng et al. 2024a) has been proposed encourage more robust reasoning by exposing the student alternative problem- solving perspectives. To further safeguard the quality of the synthesized data, heuristic filtering strategies and rejection sampling mechanisms (Zhao et al. 2025) have also been adopted.
(CoT) reasoning traces to improve the student's 's step-by-steg
Although these methods achieve acceptable perfor- mance, many of them primarily focus on improving the quality or diversity of synthetic data either in a
Knowledge Teaching
single-shot or iterative manner. Considering the anal- ogy of LLMs as teachers and SLMs as students (as
Teacher
Student
shown in Figure D. these works did not explicitly model KD systematic knowledge teaching pro-
What to teach?
When to teach?
How to teach?
cess, in which teaching contents and strategies should be dynamically and selectively adapted according the student model's 's prior knowledge and learning progress. In fact, there are several fundamental prin- ciples in pedagogy (Bloom] 1968; Vygotsky & Cole 1978) which can inspire better distillation design: (I) What to teach: identifying and targeting critical knowledge deficiencies. In the absence of specific knowledge units, even strong reasoning abilities may not be sufficient to obtain correct answers. For in- stance, student with strong logical reasoning skills but no prior exposure to logarithms would be unable to solve equation such as (xr) 3. (2) When to teach:
Knowledge
Knowledge Organization
Knowledge Adaptation
Identification
Knowledge Distillation
LLM
SLM
Figure I:
Analogy between real education and language model knowledge distillation.
organizing knowledge delivery through progressively complex curricula. Designing a curriculum in which the knowledge components are sequentially connected and gradually increase in difficulty, while ensuring that students have solid grasp of prior knowledge before progressing to the next stage-often leads to more effective learning outcomes. (3) How to teach: adapting knowledge representation to the learner's cognitive level. For example, in mathematics education, complex concepts like derivatives are often first introduced using intuitive, visual representations such as slope diagrams or motion- based analogies (e.g., the speed of moving car), before transitioning to formal symbolic notation.
Based on such inspirations, we analyze the drawbacks of existing methods as follows: Identification: Current synthetic data generation lacks knowledge- identify the specific knowledge that SLMs require to develop certain capabilities. The deficiency of such targeted knowledge remains prominent in SLMs. 2) Knowledge Organization: Existing synthetic data is generated without pedagogical organization, missing critical opportunities to optimize knowledge learning trajectories for SLMs. Ignoring the sequence of knowledge and failing maintain an appropriate teaching pace make it difficult for SLMs to develop deep understanding and mastery of concepts. As result, SLMS often rely mechanical memorization, which undermines their ability to generalize in real-world scenarios. 3) Knowledge Adaptation: Most synthesis methods overlook SLM-adaptive knowledge representation, resulting in suboptimal knowledge absorption even with high-quality data. In fact, the cognitive level often cannot accommodate the knowledge involving abstract concepts or complex derivation process; such knowledge should be adapted using figurative metaphors and broken down into sub-steps to facilitate the knowledge distillation.
1) Knowledge targeting and fails to
To effectively tackle these challenges, we propose a knowledge-awar aware three-stage data synthesis framework for distillation, ddentifier-Organize--be
(IOA), that explicitly answers what to teach, when to teach, and how to teach through (i) deficiency diagnosis and dependency-awar sequens targeting, (ii) curriculum sequencing with mastery gating, and (iii) cognitively aligned representation adaptation. We operationalize Bloom's 's Mastery Learning(Bloom 1968) and Vygotsky's 's Zone of Proximal Development (ZPD) (Vygotsky & Cole 1978) into concrete criteria: topological curricula, bounded difficulty increments, and stage-wise advancement rules, together with prompting templates for data synthesis. Empirically, across instruction following and reasoning benchmarks, our IOA
2

---

Published as a conference paper at ICLR 2026
framework achieves obvious gains over distillation baselines and consumes relately moderate time when taking 1LaMA-3.1/3.2 and Qwen2.5 as student models. This underscores the effectiveness and efficiency of pedagogically inspired data synthesis for language model knowledge distillation.
2 RELATED WORKS
Synthetic Data for LLMs Synthetic data has been widely employed in the LLM scenario, because such -Lne-geretted text can significantly reduce the human annotation costs, overcome the data scarcity in specific low-resources scenarios (Nadas al.] 2025), and enhance the diversity of exiting web corpus (Chen et al. 2024a), thus further facilitating the extension of scaling laws. Previous works have explored how to augment the corpus formats and styles with synthesis data in pretraining phase (Hao et al. 2025). Besides, some researchers investigated the "model collapse" phenomenon when utilizing the synthetic data to recursively pretrain new language models (Feng et al.) 2024b] Seddik et al. 2024 Zhu et al. 2024 Gerstgrasser et al. 2024). In the post-training phase, synthetic data has also been utilized to distill LLM knowledge and capabilities into smaller models (Tooi] et al. 2023 Guo et al. 2025). Recently, synthetic interaction trajectories have even been employed to strengthen LLM's agentic capabilities (Prabhakar et al. 2025). In addition to performance benefits, another advantage of symthetic data lies on protecting the privacy in case of the sensitive information leakage from the original data (Flemings & Annavaram (024). In this paper, our focused scenario is that SLMs have exhibited base knowledge obtained from pretraining, but needs effective distillation via synthetic data to gain more difficult knowledge and more complex task-completion capabilities.
Language Model Distillation Knowledge distillation has been successfully applied to language models to enrich model internal knowledge and enhance various capabilities like instruction following and reasoning. Previous representative distillation approaches include logit/output distribution
distillation (Gu et al.) 2024 ZZhan et al. 2023 Ko et al. 2024 Wang et al. 2025b Ko et al. 2025)
and intermediate layer representation distillation (Jiao et 2020; Liang et al. 2023 Sy al.)
2024). Especially, on-policy distillation (Agarwal et al.) 2024) is an effective approach to tackle the
distribution mismatch issue between output sequences seen during training and those generated by the student during inference. Some previous works have also explored integrating curriculum learning into model distillation by ordering "easy-to-hard" training samples according to the teacher-student output distribution divergence (Zhu et al. 2021 I) or learning from successive intermediate checkpoints of the teacher progressively (Panigrahi et al. 2025 Gupta & Karmalkar 2025). However, these types of white-box schemes are often unpractical when distilling knowledge from most advanced LLMs, because many of them are closed- commercial LLMs, like OpenAl ol (Jacch et al. 2024) and Gemini 2.5 (Comanici et al. 2025). In contrast, the black-box distillation with L.MM- generated synthetic data has become a more common and realistic way. Some recent works (Zhou al. 2024 Wang al. (2025a) has noticed the significance of such data synthesis and proposed corresponding approaches, like augmenting the original seed mathematical questions and answers with L.Mss Yu et al. 2024). eliciting the teacher LLM's Chain-of Thouuhh reasoning process (Hsieh et al. 2023: Chen et al. 2024b; Dai et al. 2024 Yang et al. 2025), , and introducing counterfactual data generation (Feng et al.) 2004a). Notably, Oera series works (Mukherjee et al., 2023) Mitra et al.,
2023) highlight that adapting knowledge representations in synthetic data to the student's 's cognitive
level can produce substantial gains. Different from previous works, our this work mainly explores how to teach a student model with a large black-box teacher via pedagogically guided data synthesis.
3 METHODOLOGY
3.1 PROBLEM FORMULATION
Given teacher LLM T with parameters oy and student SLM S with parameters 0g (where [0sl < [0r)). ). the goal is to generate a synthetic dataset Dayn that maximizes the knowledge transfer from T to S. By trained on distillation data Deym. the parameters of student model S are updated to 0g. whose performance is further evaluated on the specific targeted downstream tasks. Conventional approaches generate Day by prompting T with seed data Deced {d1, d2, ...; d
Dayn = U(dr d.g = T(d;; = 1.....,JJ
(1)
1=1
3

---

Published as a conference paper at ICLR 2026
Targeted Knowledge Selaction
Curricutum Sequence Construction
Conventional Data Synthesis
Organizer
Diffieulty Increment Centrol
Dapendonoy Graph Constructio
Seed Data
Knowl
Progressive Learning
Knowladge Defficiency Diagnosis
Knowladge Representation
Synthesis Prompt
Adaptation
Generated Data
Identifier
Student Model
Adapater
Teacher Model
Romedial Data Generation
Figure 2: Pedagogiicallyinspire data synthesis framework for language model knowledge distillation.
where J; represents the number of synthetic samples generated from the i-th seed data d,. Here, assume access to a compact seed dataset that provides minimal yet sufficient coverage of the target domain, and is partitioned into training and validation splits. The training portion serves as the initial contexts for synthesis, while the validation split later enables diagnostic evaluation.
3.2 IDENTIFIER: KNOWLEDGE DEFICIENCY DIAGNOSIS AND TARGETING
The knowledge Identifier module first diagnoses knowledge deficiencies by systematically evalu- ating performance disparities between teacher and student models across fine-grained knowledge components, and then prioritizes most critical knowledge gaps by analyzing knowledge dependency.
Knowledge Deficiency Diagnosis Rather than treating capabilities monolithic entities, ap- proach first decomposes complex domains (c.g.. mathematical reasoning) into constituent knowl- edge modules, enabling identification of specific deficiencies that impede student model perfor- mance. For a given target capability domain DD (e.g., mathematical problem-solving), we employ hierarchical decomposition strategy that organizes knowledge across multiple granular levels. Taking mathematics as an example, we query the teacher LLM to structure knowledge units as: D (K, K2,-.. ...g Km where K, [K,,,, ki,,, ki,,. Here, K₃ represents major knowledge categories (e.g., algebra, geometry, calculus) and k., denotes specific knowledge modules within each category (e.g., linear equations, quadratic functions, trigonometric identities). This structure fa- cilitates both comprehensive coverage and targeted intervention. Note this knowledge hierarchy keeps fixed once generated for each domain. Based on this, given teacher model T and a student model S, carefully constructed probe tasks Ps for each knowledge
we evaluate their respective capabilities module The data of Px comes from the validation part of seed data D (see Appendix for details). The performance gap is quantified
A(k) (k) Ps(k) EPEP. - - = vill (k)
(2)
EpEp. (pT(p) 11
where (k-(k) and (k) represent the score of teacher and student models on knowledge module (accuracy ROUGE-L depending on the probe task), respectively. Knowledge modules are classified as deficient when (kk) > Tgaps typically set to 0.3 to ensure meaningful performance disparities warrant targeted intervention.
Targeted Knowledge Selection Understanding prerequisite relationships is crucial for establishing pedagogically sound learning trajectories. We construct directed acyelic graph = E) where vertices V represent knowledge modules and edges E encode prerequisite dependencies. The dependency strength between modules k; and k; is empirically determined through conditional performance analysis:
Pr()s) Ps( Ps(k)) Z Thigh) Ps(kj
(a(ks) Pr(k:)
Tlow)
Dependency(k; k.))
(k
Pr(k.)
Pa(k)
Thigh)
4

---

Published as a conference paper at ICLR 2026
where Thigh and Tow two thresholds to indicate if student model has mastered the knowledge module k enough during the learning process. which are set as 0.9 and 0.7, respectively. This metric captures how mastery of prerequisite knowledge k; influences performance
dependent knowledge ky. Dependencies with strength above threshold Tdep = 0.3 are incorporated into the graph structure. Here, compute Ps (ky) under different conditions in Eq. 3 we simply record
Ps(ks) first exceeds Thigh or falls below Now- the student's performance on k, at the moments when Pr(k.
Beside, to handle rare cyclic knowledge dependencies in this graph, we detect cycles and break them by removing the weakest edge (lowest dependency score), ensuring that the resulting structure remains valid DAG for curriculum construction. To prioritize distillation efforts, we establish severity ranking mechanism that considers both absolute performance gaps and relative importance within the knowledge hierarchy. The deficiency severity score is computed as:
Severity(k) (k) (1-
Σ
Dependency(k: + k')
[K(KK)
KEN(()
where the first term captures the absolute performance gap between teacher and student models. The second term measures the knowledge module's connectivity within the dependency graph G, where ((k)) represents neighboring modules and Dependency(k: + k') denotes dependence strength. The weighting parameters is empirically set 0.7, prioritizing performance gaps while accounting for structural importance. The identifier outputs prioritized list of deficient knowledge modules Kusrget fk,, ..,,.. k ranked by severity scores, along with their associated dependency structures. This targeted selection ensures that subsequent synthetic data generation focuses the most critical knowledge gaps, maximizing distillation efficiency. Specifically, modules are selected such that:
Karrgtt Top-m Top-mm(ueecciont (Ceeiiientt Severity(-))
(5)
where Kedeticien = {kk (k) > }gup}-
(6)
The threshold m is dynamically adjusted based on available computational resources and desired distillation scope, typically set to cover approximately 20-30% of difficient modules Ketefcient to balance comprehensiveness with tractability.
3.3 ORGANIZER: PROGRESSIVE CURRICULUM DESIGN WITH MASTERY LEARNING
The knowledge Organizer module transforms the deficient knowledge modules prioritized by the Identifier into a pedagogically structured learning sequence. Drawing from Bloom's 's Mastery Learning Principles (Bloom) 1968) and Vygotsky's s Zone of Proximal Development (Vygotsky & Cole 1978) (see Appendix foro details), this module ensures that knowledge acquisition follows optimal learning trajectories while maintaining an appropriate progression of difficulty.
Curriculum Sequence Construction Given the targeted knowledge modules Kinrger and their depen- dency graph G from the Identifier, we construct a learning sequence S (&1, 82, 8,,) where each stage s; contains a subset of semantically similar knowledge modules to be learned simultaneously (e.g., linear equations and systems of linear equations). The sequence construction follows topological ordering of the dependency graph while respecting pedagogical constraints:
[k Karggt Vk' (rerequisstes<()),
U s3).
J<b
Here, the Prerequisites(() function returns the set of prerequisite knowledge modules identified from the dependency graph in the Identifier stage, formally defined as Prerequisites(A) = { Dependency(k* + k) > Tetp }. This ensures that all prerequisite knowledge is mastered before introducing dependent knowledge. Notably, to create gentle learning curves and reduce cognitive over- load, we apply Vygotsky's i 7.P principles by controlling difficulty increments between consecutive learning stages. The difficulty difference between stages is constrained as:
Σ Ps(k)
Ps(k) < TZPD
PPs(H)
(8)
88+1
88
RE
where
Σ (k) represents the average difficulty level of knowledge modules in stage 8;, and
|&a|
ke#:
TZPD 0.15 ensures that difficulty increases remain within the student model's learning capacity.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 Pedagogiically-Insiiie Data Synthesis for Language Model Knowledge Distillation
Require:
Teacher model T' Student model S, Target domain D, Seed data Sooo
Ensure: Fine-tuned student model S"
// Identifier: Deficiency Diagmosis and Targeting
2: moddles +
CrapreeeeDDeeeeeeeeerrrrrrrh A t EvaluateGaps(T, (Kmouuull
Kearget f- PrioritieeCriiicaCiasssA. G) 4: // Organizer: Curriculum Sequence Construction
3: G
5: S +- OrgamizeL_arminggqqueneee
G, S)
6: // Organizer: Mlastery-Based Progressive Learning
7: for = 1 to [S] do
8: // Adapter: Cognitive Level Alignment
9: Dyn -
Dsood)
S S FineeeTune(S Dsyn) Ps EvaluateStage(S, 8,); Pr EvaluateStage(T, s;)
11:
while minkes. Pr(
Ps(k)
Tmastery do
12: S 5- FineTTune(S, GenerateRemeddillaatt... S;, B,,Dseedi Dseea))
13: end while
14:
end for
15:
return S"
Mastery-Based Progressive Learning Following Bloom's criterion-reffrrncee evaluation, we implement strict mastery requirements between learning stages. For each knowledge module k in stage s;, the SLM must achieve relative performance before progressing to learning next stage 81417
True
if min
Ps(k) (r(
mastery
Progress(s; - S1+1)
False
otherwise.
where Tiusstery = 0.9 requires the student model to achieve 90% of the teacher model's performance before advancement. Otherwise, the remedial data will be synthesized to continue learning knowledge in this stage untill mastery. This prevents knowledge gaps from accumulating and ensures that the student model has sufficiently mastered each prerequisite concept relative to the teacher's ss capability.
3.4 ADAPTER: KNOWLEDGE REPRESENTATION ADAPTATION FOR COGNITIVE ALIGNMENT
The knowledge Adapter module transforms the structured curriculum from the Organizer into cognitively appropriate representations that align with the student model's learning capacity. Instead of querying the teacher model T with Desed and then directly utilizing the obtained synthesized data Dsyn. the Adapter systematically modifies content delivery match the cognitive constraints of student models, ensuring genuine understanding rather than mechanical memorization. The adaptation is performed on the subset of Dreed corresponding to the knowledge units scheduled for current stage In detail, the knowledge adaptation is conducted from following five perspectives:
Abstract Concept Concretization The Adapter transforms abstract mathematical concepts into concrete, intuitive representations using analogical reasoning. For instance, derivatives are initially introduced "measuring how fast something changes, like car's speedometer" before presenting the formal limit definition. Complex logarithmic relationships begin with concrete scenarios: "If bacteria double every hour, how many doublings reach 256 from 1?" This grounds abstract concepts in familiar experiences, thus facilitating the understanding and absorbing knowledge in SLM.
Complex Reasoning Decomposition Multi-step reasoning processes are systematically disag- gregated into atomic cognitive operations. For example, for math problems, the process is broken into sequential sub-procedures: information extraction + relationship identification + equation formation solution execution + verification. Each component mastered independently before integration, while building comprehensive reasoning capabilities. Note that if obtained results cannot pass the verification, the corresponding reasoning process will be filtered out.
Mi Cognitive Load Management Information density and complexity are carefully regulated to match SLMs' cognitive constraints. Systems of equations begin with 2x2 integer coefficient cases before progressing to larger or fractional systems. Complex problems are divided into manageable segments with explicit intermediate verification, ensuring sustainable learning progression.

---

Published as a conference paper at ICLR 2026
2 Representation Format Optimization Knowledge encoding is restructured using templated frameworks and structured pedagogical formats. Standardized solution templates provide consistent scaffolding, for example: "Step 1: Identify coefficients. Step 2: Calculate discriminant. Step 3: Apply formula." Multiple worked examples following the same templates, while varying surface features, enable pattern extraction and facilitate robust knowledge generalization.
Linguistic Complexity Reduction Content undergoes systematic simplification across vocabulary, syntax, and discourse structure. Mathematical terminology is replaced with simpler equivalents when possible ("reciprocal" for "multiplicative inverse"). Complex sentences with subordinate clauses are decomposed into direct statements. Besides, clear signaling words ("first," "therefore," "next") are integrated into texts to provide explicit logical structure guidance through reasoning processes.
3.5
OVERALL KNOWLEDGE DISTILLATION FRAMEWORK WITH DATA SYNTHESIS
Our pedagogically-insne inspired framework integrates the Identifier, Organizer, and Adapter modules into coherent distillation pipeline that systematically transfers knowledge from teacher student models through structured synthetic data generation. As shown in Algorithm T the framework operates in iterative cycles, continuously adapting the training data at each stage based student model progress. The distillation process begins with a deficiency diagnosis and targeting phase where the Identifier evaluates performance gaps across knowledge modules and prioritizes the most critical gaps. The Organizer then constructs progressive learning curriculum respecting prerequisite relationships, the zone of proximal development, and mastery requirements. In each curriculum stage, the Adapter generates cognitively appropriate data, which is used to train student models until achieving mastery.
4 EXPERIMENTS
4.1
EXPERIMENT SETUP
Teacher and Student Models: We utilize two most representative large reasoning LLMs as teacher models: closed-sourced OpenAl ol (Jaech et al. 2024 and open-sourced DeepSeek-RI (Guo et al..
2025 It is conservatively estimated that both models at least have over 100B parameters (Abacha ct al. 2024). As for student models, we select recent small language models from Qwen family
(Qwen2.5-3BB, Qwen2.5-7B, Qwen2.5-144 (Qwen 2024h) and LLaMA family (LLaMA3.1188 (Meta) AI. 2024a), LLaMA3.2-3B (Meta AI, 2024b)). Due to page limitation, we only provide results on two 3B models in the main text, leaving others in Appendix F
Evaluation Benchmarks We employ two main types of benchmarks to comprehensively evaluate the
effectiveness of different methods: Instruction Following: Dolly Evaluation (Conover al., 2023) and
Vicuna Evaluation (Chiang et al.
2023); Reasoning: math problem solving (GSM8K (Cobbe et al.
202I). MATH (Lewkowyez al. 2022). AIME2024 (Mathematical Association of Americal 2024 ).
code generation EvumanEval (Chen et al. 202T), MBPP (Austin et 2021)) LiveCodeBench (Jain
al. 2025)). ) academic knowledge reasoning (GPQA-Diamonl abbreviated as GPQA-D (Rein
et al. 2024)). As for the evaluation metrics, we adopt ROUGE-L and Pass @ k(k 1) for instruction
following and reasoning benchmarks respectively, referring previous literature (He et al. 2025 Guo
al. 2025).
Baselines We primarily compare our framework with following several types of data synthesis-
based language model distillation approaches: Vanilla Synthesis: Self-Instruct (Wang et al. 2023).
LaMini (Wu et al. 2024); Adversarial Synthesis: Lion (Jiang et al. 2023); CoT Synthesis: Distilling
Step-by Step (DSS) (Hsieh et al. 2023). CasCoD (Dai al. 2024); Counterfactual Synthesis: Coun-
terDistill (Feng et al. 2024a): : Mulfi-Agent Synthesis: Star-Agents (Zhou et al.
2024). MADA (Wang
al. 2025a).
Besides, we have provided the detailed seed data construction in Appendix B detailed introductions
baselines and evaluation settings in Appendix Brvendi(CanddD implementation details and hyperparameter settings in Appendix E Supplementary experiments on agentic tasks can been in Appendix G
4.2 MAIN RESULTS AND ANALYSIS
Overall Performance Comparison Tables and repport results with OpenAl ol and DeepSeck-RI teachers. IOA consistently achieves the best performance across both Qwen2.5-3B and L.M.23.2-
7

---

Published as a conference paper at ICLR 2026
Table I: Comparison among different data synthesis-based distillation approaches various evalua- tion benchmarks for LLaMA and Qwen models. OpenAI ol is taken as the teacher LLM. Bold and underline indicate the best and second-best results, respectively. indicates statistically significant.
Model
Method
DailyEval
VicumnEoal
GSMBK
MATH
AIME2024
HumsnEasl
MRPPP
LiuhhnBBechh
GPQA.D
Undietlled
25.37
23.82
37.24
5.79
2.13
22.46
31.58
14.72
7:05
Self instrot
32.18
29.36
43.69
7.12
2.97
25,83
36.27
16.84
9.28
LaMini
33.94
31.15
45.28
7.96
3
26.83
57.49
17.42
9.65
Lion
34.73
32.62
68.36
9.43
3,83
28,95
39.14
19.74
10.46
Owen2.5:8
DSS
35.25
33.45
89.57
10.64
4.08
30.57
40.03
20.85
10.94
CasCaD
35.65
50.46
11.71
11.19
CounteDDisill
34.25
33.08
89.84
11.19
4.77
31.15
40.35
21.29
11.05
Stu--Ageuts
36,08
51.25
12.47
5.23
11.67
MADA
56.42
35.09
$2.04
13.15
5.54
33.90
42.18
23.06
11:03
IOA
38.16
65.7y
15.53
13.74
Undistlled
23.68
21.97
34.13
5.12
1.80
20.33
29.63
13.24
7.14
Self EnstrEnstret
30,42
27.84
40.57
6.31
2.45
23,91
33.82
15.25
6.54
LaMini
31.77
29.63
42:25
7,03
2.73
25.15
34.94
13.92
8.91
Lion
32,91
30.86
45.63
8.59
3.26
27.14
36.53
18.16
9.72
LLMMA32-30
DSS
33.63
31.57
46.75
9.68
3.74
28.52
57.35
19.26
10.08
CasCaD
34,05
10:43
CounterDisill
33.19
31.12
47.06
10.43
4.35
29.05
37.71
19.63
10.28
Stu-Ageuts
48.14
30.26
10.64
MADA
35.82
38.21
48.82
11.26
4.58
31.18
39.13
21.02
10.87
IOA
36,86
53,07'
14.12*
5.47
44.25
24.05"
12.15*
Table 2: Comparison among different data synthesis-based distillation approaches on various evalua- tion benchmarks for LLaMA and Qwen models. DeepSeek-R1 is taken as the teacher LLM. Bold and underline indicate the best and second- results, respectively. indicates statistically significant.
Model
Method
DollyDval
Vicunalival
GSMSK
MATII
AIME2024
IuumnnEval
MBPP
LiveCodeBench
GPQA D
Undistiled
25.39
23.84
37.25
5.82
2.12
22.45
31.59
14.73
7.94
Self-lastrus
32.14
20.35
43.74
7.13
2.95
9.36
LaMini
33.92
31.13
45.27
7.94
3,02
26.86
37.53
17.43
9.63
35.01
32.93
40.12
9.79
3,82
39.63
10:62
Qwe226.38
DSS
35.58
33.81
50.32
11.14
4.30
31,63
40.71
21
11.18
CasCal
36.13
34.25
51.18
12.29
4,91
32.63
41.52
22.25
11.54
CounterDiiill
34.72
33.37
50.74
11.82
5.08
37.17
41.03
21
11.29
Sar Agents
36.62
35.07
52.15
13.21
5.58
33.89
42.37
23.14
11.92
MADA
37.01
52.81
13.48
12:27
IOA (Oun)
37,77
36,29
$6,88
16,02
6.57
42.08
39.32*
27.739
14,49
Undistiled
23.69
22.01
34.12
5.09
1,82
20.34
29.61
13.21
7.12
Self-lsttrs
30.41
27.53
40.53
6.29
2.53
23.92
8.52
LaMini
$1.82
29.58
42.24
7.02
2.71
25.13
34.92 92
15.91
8.94
Lion
33.12
31.02
66.29
8.87
3.37
27.81
36.94
18.54
9.83
DSS
33.91
31.91
47.63
10.27
3.88
29.31
37.87
19.64
10:37
CasCal
34.65
32.28
48:47
11,46
4.32
30.25
38.72
20.27
10.72
CounterDisill
33,63
48.02
11,00
4.48
29.94
38:25
10.51
Sar Agents
34.97
33.01
89.09
11.91
4.74
31.00
20.92
10.94
MADA
35.55
89.72
12.28
4.95
39.82
11.13
IOA (Ouss)
36.75
34.72
53.31
14.01
5.07
39.37
44.97*
24.88
$2,47
reasoning, and coding tasks. On instruction-followin tasks (DollyEval, VicunaEvvl),, TOA outperforms strong baselines like MADA by approximately 1.5-2.0 points. On mathematical reasoning. IOA reaches 15.53/14.02 (Qwen/ILLMM with ol), and further 16.02/14.81 with DeepSeek-RI, showing the added benefit of reasoning-orient teacher. For code benchmarks, the gains are most pronounced: HumanEval improves from 33-34 (MADA) to over 40 with IOA, a relative gain exceeding 20%, with MBPP and LiveCodeBench showing similar trends. On knowledge-intensee intensiiiteeivve QA (GPQA-D), IOA also provides consistent 1-2 point improvements. In sum, IOA delivers robust gains across tasks and student SLMs. with especially strong advantages on math and coding when distilled from DeepSeek-RI. These results highlight the effectiveness of IOA's pedagogical design transferring both fnslluction-folloiinng and reasoning abilities.
3B, covering instruction-followt
Ablation Study Table 3 presents the ablation results when removing the identifier, organizer, or step,
adapter modules from IOA. Specifically, the "-Identifier" removes the causing all knowledge modules to be treated uniformly and preventing the framework from focusing on the student's 's actual weaknesses; ". Organizer" disables the dependency-drie driven curriculum and presents identified deficient modules in random order, removing prerequisite-wwan sequencing; 4, Adapter" eliminates the adaptive synthesis mechanisms in Section (3.4] prompting the teacher to generate data without difficulty representation adjustments and thereby producing samples that
8

---

Published as a conference paper at ICLR 2026
Table 3: Ablation study various evaluation benchmarks for LLaMA and Qwen models. OpenAl ol is taken as the teacher LLM. * indicates statistically significant.
Model
Method
DollyEval
VicunaEval
GSMBK
MATH
AIME2024
HumanDival
MBPP
LiveCndellench
GPQA D
Full (OA
38.16*
36.85
55,79
15.53*
6.29
20.64"
47.36
26,94*
13,75
Qwen22.53B
Ideniiferr
36.72
34.91
56.23
14.92
5.01
39.87
46.95
26.13
13.21
Orgunizer
37.04
35.22
51.68
13.77
5442
40.21
47.05
26.34
13.08
Adapaer
37.61
36.05
54.72
15.01
6.12
36.48
42.67
25.18
13.25
IOA
36,88
34.81
$2.07*
14,02*
5.47"
57.981
44.25
24.06
12.15
L1
Idemitice
35.47
33.26
50.73
13.61
5:23
57.25
43.32
25.47
11.85
Organizer
33.74
12.48
37.51
43.61
23.64
11.72
Adapier
36
34.09
$1.26
13.73
5.32
34.42
39.58
21.11
11.79
of
Tourery
31.5
2113
800 000.00E-.
33.5
24g
25.3
35.5
35.
1.45
0..0
6:35
0.20
0:25
0.30
0.70
0.75
0.00
0.05
0.00
0.55
1.00
ROUGELL (Qoan2255338
ROUGEs, (LL.NA3.2-30)
Pas (Qwen22.5-88)
Peaa (1LUM632-00)
Figure 3: The hyperparameter robustness analysis for three critical hyperparameters Jr T2PD: Tmastery.
may exceed the student's cognitive capacity. For Qwen2.5-3B and LLaMA3.2-3B the complete IOA consistently achieves the highest scores, while excluding any component leads to noticeable degradation. Removing the identifier primarily hurts instruction-following benchmarks (DollyEval, VicunaEval), since emrrr reasoning tasks such as GSM8K and MATH most severely, reflecting the importance of structured curriculum in learning mooblem-solving skills. Finally, removing the adapter causes the largest drops on coding tasks (HumanEval, MBPP, LiveCodeBench), highlighting the necessity of representation adaptation for generalizing in program synthesis. The consistent performance decline across all tasks when any component is removed demonstrates that each stage of IOA contributes uniquely and positively to the overall distillation performance.
longer be effectively localized. Removing the organizer impacts
Hyperparameter Robustness Figure 3 reports the effect of three hyperparameters: synthesis data amount Jj for each seed data, proximal development zone threshold TZPD. and progressive learning mastery threshold Tmastery. on averaged ROUGE-L over instruction-tune tuning tasks (blue, left y-axis) and averaged Pass @1 over reasoning tasks (orange, right y-axis). The vertical red lines indicate our standard setting (J,=10, 72PD=0.15, Tmaattery00.900 improve and then plateau around E [30, 40], showing diminishing returns; Qwen2.5-3B benefits slightly more than LLaMA3.2-33 3.B.BB. Varying TZPD yields clear non-monotonic trend with peak near 0.15: too small under-challenges the learner, while larger thresholds overshoot the step size and hurt transfer. For Tmastery. performance rises the requirement becomes stricter and peaks around 0.90-0.95, after which excessive strictness leads to a mild drop, likely due to overfitting and slower progression. Across all sweeps, Qwen2.5-3B maintains consistent advantage yet follows the same trends as LLaMA3.2-3B. indicating that IOA is robust and easy to tune.
As J; increases from 5 to 50, both metrics
Time-Consumption Analysis To address concerns about end-to-end efficiency, Figure 4 compares the averaged wall-clock time of IOA with representative data-synthesis baselines under our standard setting. IOA (Ours) requires 11.00 on Qwen2.5-3B and ~ h2.2 on LLaMA3.2-33 outperforming heavier pipelines such as Lion (12.1/13.05 h; 29.1% / 6.2% faster) and CasCoD (11.6/12.7 h; xx5.2% / 3.9% faster), and remaining competitive with MADA (11.8/12.6 h; R36.8% 3.2% faster). While Self-Instruct is the fastest (9.8/10.7 h), it lags notably in quality, so its time is not directly comparable to IOA's stronger results (Ta- bles 2 We attribute this efficiency to three key design choices in IOA. First, the Identifier narrows the synthesis scope by diagnosing knowledge gaps and selecting only
End-co-end Time
Ower2 5
LLa4433 38
1255
3228
11.01
11.8
133N
Coubu
Figure 4: Time consumption comparison between our IOA and baselines.

---

Published as a conference paper at ICLR 2026
Table 4: Comparison between IOA and other recent distillation methods. Bold and underline indicate the best and second- best results, respectively. indicates statistically significant.
Model
Method
DollyBivl
VicunaLival
GSMBK
MATI
AIME2024
HumanDvvll
MBPP
LiveCodellench
GPQA D
ABKD
37.85
36,01
55.56
14.31
5.62
36.36
44.87
25.04
12,99
DssLLLM-22
38.28
30.80
50.25
15.07
6.23
38.14
45.03
25.93
13,47
Qwen2.5.7B
38.62
37.20
29.12
18.16
26.46
13.73
SuperCorrect
38.46
36.98
56.24
16.51
7.97
35.95
45.05
24.99
15,05
30.55
37.61
57.76
14.49
6.32
38.46
46.57
26.14
13.79
TOA (Ours)
41.64
39,97
63.85
19.64
10.89
47,07
54,94
32.30*
17.08
Table 5: Averaged performance across 9 tasks under the moderately strong teacher model setting.
Model
Self-Instruct
LaMini
Lion
DSS
CasColD
CounterDistil
Star-Agents
MADA
IOA (Ours)
Qwen2,5:3B
20.25
22.18
25.02
25.45
27.11
26.07
27.65
28.32
31.19
the most essential missing components, thereby reducing teacher calls and training volume. Sec- ond, the Organizer enforces a dependency-awarre curriculum with controlled difficulty progression, preventing wasted effort on overly challenging samples before prerequisites are mastered. Finally, the Adapter promotes structured and simplified representations that reduce unnecessary verbosity, resulting in more compact supervision and improved token efficiency during training. Together, these mechanisms enable IOA to achieve stronger performance while maintaining competitive or even reduced wall-clock time compared to more complex multi-round or adversarial pipelines.
4.3 SUPPLEMENTARY RESULTS AND ANALYSIS
Comparison with Other Distillation Methods To enhance comparison completeness, also compare with several recent distillation methods beyond those in main experiments: Vanilla White- box Distillation: ABKD (Wang al.) (0225b) and DistiLLM-2 (Ko al. 2025); Only-Policy Logit Distillation: GKD (Agarwal et al.] 2024); RL-based Distillation: SuperCorres (Yang et al.] 2025); Curriculum- based Distillation: POCL (Liu & Zhang 2025). To ensure tokenizer matching necessary for white-box distillation, we take
Qwen-14B and Qwen2.5-7B as the teacher model and student model, respectively. According to corresponding results in Table IOA stably outperforms various recent baselines, demonstrating the effectiveness of IOA even against white-box distillation and ao--poliyy/currculum-bbe
methods.
Distillation on Moderately Strong Teacher Models To evaluate whether our IOA frameworks still performs well under moderately strong teacher model setting, we employ ReepSeek-RI--istte --it------llleeeen -wenn 7B the teacher and Qwen2.5-3B results on Table 5] we can observe that IOA continues to provide consistent improvements over baselines, although the absolute gain naturally smaller compared to large-gap settings.
the student conduct the experiments. From the averaged
Seed Data Analysis To explore the influence of seed data quality and quantity on performance, conduct experiments under two setting: a) We first filter and downsample the high-quality OpenThoughts3-1111 1.2M dataset (Guha et al. 2025) to 3K examples, then use this set as the seed; b) We take samples in currated Dxeed base templates and synthesize 9K samples as the seed. Note in our default setting of main experiments, we directly employ Dxeed with 3K samples as the seed. Especially on the challenging AIME2024 dataset, improving see data quality (setting a) achieves 27.52, much higher than default setting of 6.29; increasing seed data quantity (setting b) achieves 11.83, moderately higher that default setting. Thus, we conclude that improving seed quality (coverage and distribution) can bring more obvious performance gain than solely increasing amounts.
CONCLUSIONS AND FUTURE WORKS
In this work, we introduce a pedagogically inspired framework IOA, for data synthesis, in which the prioritized critical knowledge is progressively adapted and delivered to student language models through curriculum structuring and knowledge representation adaptation. Drawing analogies from human learning theories, our approach demonstrates that carefully designed synthetic data can substantially improve the effectiveness and efficiency of language model distillation, as validated by the extensive experiments across multiple instruction following and reasoning benchmarks. In the future, we plan to further integrate insights from cognitive science with language models to establish more systematic principles for curriculum design in LIM distillation.
10

---

Published as a conference paper at ICLR 2026
ETHICS STATEMENT
This work focuses on improving the effectiveness of language model knowledge distillation through pedagogically inspired data synthesis. All experiments were conducted using publicly available large language models (e.g., LLaMA, Qwen) and openly released benchmarks for cestruction-oolooinng reasoning, coding, science QA tasks. To mitigate risks of misuse, our framework emphasizes synthetic data generation without reliance sensitive or personally identifiable information. We acknowledge that more capable student models distilled from powerful teacher models could potentially be misapplied in harmful contexts; however, believe that contributions primarily advance research toward more resource-efficeent and accessible AI, reducing costs and environmental footprint.
REPRODUCIBILIIYY STATEMENT
We provide detailed descriptions our methodology, including the Identifier Caattfiier-Oggnizzer--r Organizer-Adaple framework, experimental setups, baselines, and evaluation protocols in the main text and appendices. Hyperparameters, prompting templates, and seed data construction processes are documented to enable replication. All models used are publicly available (e.g., LLaMA, Qwen families), and the benchmarks employed (e.g., DollyEval, VicunaEval, GSM8K, MATH, HumanEval) are open- sourced. To further support reproducibility, we have provided code in the Github repository https //github. hol.thub.com/Be htt.:///.ooooo.o.aa...oooo// Bww./a/o/wIoi
ACKNOWLEDGMENTS
This work supported by the Early Career Scheme (No. CityU 21219323) and the General Research Fund (No. CityU 11220324) of the University Grants Committee (UGC), the NSFC Young Scientists Fund (No. 9240127), and the Donation for Research Projects (No. 9229 164 and No. 9220187).
REFERENCES
Asma Ben Abacha, Wen-wai Yim, Yujuan Fu, Zhaoyi Sun, Meliha Yetisgen, Fei Xia, and Thomas Lin. Medee: A benchmark for medical error detection and correction in clinical notes. arXiv preprint arXiv:2412. 19260, 2024.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.085: 087744 2023.
Rishabh Agarwal, Nino Vieillard, Yongchao Zhou, Piotr Stanczyk, Sabela Ramos Garea, Matthieu Geist, and Olivier Bachem. On-policy distillation of language models: Learning from self- generated mistakes. The twelfth international conference on learning representations, 2024.
Jacob Austin, Augustus Odena, Maxwell Nye. Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoe Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2 108, 077307 2021.
Benjamin Bloom. Learning for mastery, instruction and curriculum. regional education laboratory for the carolinas and virginia, topical papers and reprints, number 1. Evaluation comment, 1(2):n2, 1968.
Hao Chen, Abdul Waheed, Xiang Li, Yidong Wang, Jindong Wang, Bhiksha Raj, and Marah I Abdin. On the diversity of synthetic data and its impact on training large language models. arXiv preprint arXiv:2410.1522 1522222 2024a.
Mark Chen. Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained code. arXiv preprint 10XVv:2107.03374 2021.

---

Published as a conference paper at ICLR 2026
Xin Chen, Hanxian Huang, Yanjun Gao, Yi Wang, Jishen Zhao, and Ke Ding. Learning to max- imize mutual information for chain-of thunuhh distillation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 6857-6868, 2024b.
Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed April 2023), 2(3):6, 2023.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 14168, 2021.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXhv:2507.0626. 2025.
Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. Free dolly: Introducing the world's first truly open instructiontuned llm. 2023.
Thomas Cormen, Charles Leiserson, Ronald L Rivest, and Clifford Stein. Introduction to algorithms. MIT press, 2022.
Chengwei Dai, Kun Li, Wei Zhou, and Songlin Hu. Improve student's reasoning generalizability through cascading decomposed cots distillation. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 15623-15643, 2024.
Tao Feng, Yicheng Li, Li Chenglin, Hao Chen, Fei Yu, and Yin Zhang. Teaching small language models reasoning through counterfactual distillation. In Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 1831-5842, 2024a.
Yunzhen Feng, Elvis Dohmatob, Pu Yang, Francois Charton, and Julia Kempe. Beyond model collapse: Scaling up with synthesized data requires reinforcement. In ICML 2024 Workshop on Theoretical Foundations of Foundation Models, 2024b.
James Flemings and Murali Annavaram. Differentially private knowledge distillation via synthetic text generation. In Findings of the Association for Computational Linguistics ACL 2024, pp.
12957-12968. 2024.
Matthias Gerstgrasser, Rylan Schaeffer, Apratim Dey, Rafael Rafailov, Tomasz Korbak, Henry Sleight, Rajashree Agrawal, John Hughes, Dhruv Bhandarkar Pai, Andrey Gromov, et al. Is model collapse inevitable? breaking the curse of recursion by accumulating real and synthetic data. In First Conference on Language Modeling, 2024.
Yuxian Gu, Li Dong, Furu Wei, and Minlie Huang. Minillm: Knowledge distillation of large language models. In The Twelfth International Conference on Learning Representations, 2024.
Etash Guha, Ryan Marten, Sedrick Keh, Negin Raoof, Georgios Smyrnis. Hritik Bansal, Marianna Nezhurina, Jean Mercat, Trung Vu, Zayne Sprague, et al. Openthoughts: Data recipes for reasoning models. arXiv preprint wwxiiv25006041778 2025.
Daya Guo, Dejian Yang, Haowei Zhang. Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint a2X1 2025.
Shivam Gupta and Sushrut Karmalkar. Efficient knowledge distillation via curriculum extraction. In ICLR 2025 Workshop Deep Generative Model in Machine Learning: Theory, Principle and Efficacy, 2025.
Xintong Hao, Ke Shen, and Chenggang Li. Maga: Massive genre-audience reformulation to pretraining corpus expansion. arXiv preprint 250i:25502.042255, 2025.
12

---

Published as a conference paper at ICLR 2026
Changyi He, Yifu Ding, Jinyang Guo, Ruihao Gong. Haotong Qin, and Xianglong Liu. Da-kd: Difficulty-wware knowledge distillation for efficient large language models. In Forty-second International Conference Machine Learning, 2025.
Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge neural network. arXiv preprint arXiv:1503.02533 2015.
Cheng-Ye Yu Hsieh, Chun-Liang Li, Chih-kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alex Ratner, Ranjay Krishna, Yu Lee, and Tomas Pfister. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. Findings of the Association for Computational Linguistics: ACL 2023, pp.
Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720, 2024.
Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Stoica. Livecodehench: Holistic and contamination free evaluation of large language models for code. In The Thirteenth International Conference on Learning Representations, 2025.
Yuxin Jiang, Chunkit Chan, Mingyang Chen, and Wei Wang. Lion: Adversarial distillation of proprietary large language models. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 3134-3154, 2023.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 4163 4174, 2020.
Jongwoo Ko, Sungnyun Kim, Tianyi Chen, and Se- Young Yun. Distillm: Towards streamlined distillation for large language models. International Conference on Machine Learning, pp- PMLR, 2024.
Jongwoo Ko, Tianyi Chen, Sungnyun Kim, Tianyu Ding, Luming Liang, Ilya Zharkov, and Se- Young Yun. Distillm-2: A contrastive approach boosts the distillation of llms. In sorty-second International Conference on Machine Learning, 2025.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Chen Liang, Simiao Zuo, Qingru Zhang, Pengcheng He, Weizhu Chen, and Tuo Zhao. Less is more: Task-aware layer-wise distillation for language model compression. In International Conference on Machine Learning, pp.
PMLR, 2023.
Lingyuan Liu and Mengxiang Zhang. Being strong progressively! enhancing knowledge distillation of large language models through a curriculum learning framework. arXiv preprint arXiv:2506.05695, 2025.
Mathematical
Association of America. American invitational
mathematics ex-
amination (aime), 2024. URL htttps://maa.rr//meeeee http://aaa...m//aaaaaaaaa competitions
-aatnaalmmaatmaaattema www.seeeeeiiiannionnaaaaaaa Accessed: August-
23-2025.
Meta AI. Introducing Llama 3.1: Our most capable models to date, July 2024a. URL https //ai www.mmeta..m ccom/bblo//metee 11amala---------- [Online; accessed August-23-2025).
Meta AL. Llama 3.2: Revolutionizing
edge ai and vision with open, cus- URL https://ai www...eta...m www/boooooo.oooooooooooooooooooo
tomizable models, September 2024b.
Ilama- wow.enton-t---ee.eeeeecoeee..... 23-2025].
[Online; August-
13

---

Published as a conference paper at ICLR 2026
Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes, Sahaj Agarwal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, et al. Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11004 2023.
Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707 2023.
Mihai Nadas, Laura Diosan, and Andreea Tomescu. Synthetic data generation using large language models: Advances in text and code. arXiv preprint /4X2: 2025.
Abhishek Panigrahi, Bingbin Liu, Sadhika Malladi, Andrej Risteski, and Surbhi Goel. Progressive distillation induces an implicit curriculum. In The Thirteenth International Conference on Learning Representations, 2025.
Shishir G Patil, Huanzhi Mao, Fanjia Yan, Charlie Cheng-Jie Ji, Vishnu Suresh, Ion Stoica, and Joseph E Gonzalez. The berkeley function calling leaderboard (bfel): From tool use to agentic evaluation of large language models. In Forty-second International Conference on Machine Learning, 2025.
Akshara Prabhakar, Zuxin Liu, Ming Zhu, Jianguo Zhang, Tulika Awalgaonkar, Shiyu Wang, Zhiwei Liu, Haolin Chen, Thai Hoang, Juan Carlos Niebles, et al. Apigen-mt: Agentic pipeline for multi-tum data generation via simulated agent-human interplay. arXiv preprint arXiv:250.003601 2025.
Qwen. Qwen2.5: A party of foundation models, September 2024. URL https://qwent github. h./blogg/wen2/ [Online; accessed August-23-2025).
Qwen. Qwq-32b: Embracing the power of reinforcement learning, March 2025. URL https
// //qwenlm.gite github. ww.blog./o/oooo/o qwq 32b/ [Online; accessed August-23-2025j.
David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang. Julien Dirani, Julian Michael, and Samuel R Bowman. Gpqa: graduate-level google-proof q&a benchmark. In First Conference on Language Modeling, 2024.
Mohamed El Amine Seddik, Suei- Chen, Soufiane Hayou, Pierre Youssef, and Merouane Debbah. How bad is training on synthetic data? statistical analysis of language model collapse. arXiv preprint arXiv:22404.05: 05090, 2024.
Zhiqing Sun, Hongkun Yu, Xiaodan Song, Renjie Liu, Yiming Yang, and Denny Zhou. Mobilebert: compact task-agnostic bert for resource-limited devices. arXiv preprint arXiv:2004.02994 2020.
Yaya Sy, Christophe Cerisara, and Irina Illina. Lillama: Large language models compression via low-rank feature distillation. arXiv preprint 167 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An instruction-following llama model, 2023.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 23X1v 2023.
Lev Semenovich Vygotsky and Michael Cole. Mind in society: Development of higher psychological processes. Harvard university press, 1978.
Chengyu Wang, Junbing Yan, Yuanhao Yue, and Jun Huang. Distilqwen2. 5: Industrial practices of training distilled open lightweight language models. arXiv preprint arXiv:2504. /5027, 2025a.
Guanghui Wang, Zhiyong Yang, Zitai Wang. Shi Wang, Qianqian Xu, and Qingming Huang. Abkd: Pursuing a proper allocation of the probability mass in knowledge distillation via a----diverennee In Forty-second International Conference on Machine Learning, 2025b.
14

---

Published as a conference paper at ICLR 2026
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self generated instructions. In Proceedings of the 6l Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 13484-13508, 2023.
Minghao Wu, Abdul Waheed, Chiyu Zhang, Muhammad Abdul-Mageed, and Alham Aji. Lamini- Im: A diverse herd of distilled models from large-scale instructions. In Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 944-9444999 2024.
Ling Yang, Zhaochen Yu, Tianjun Zhang, Minkai Xu, Joseph E Gonzalez, Bin CUI, and Shuicheng YAN. Supercorrect: Advancing small Ilm reasoning with thought template distillation and self- correction. In The Thirteenth International Conference on Learning Representations, 2025.
Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. r-bench: A benchmark for tool-agent-user interaction in real-world domains. arXiv preprini arXiv:2406.1 /204,, 2024.
Longhui Yu, Weisen Jiang, Han Shi, YU Jincheng, Zhengying Liu, Yu Zhang, James Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, 2024.
Chen Zhang, Yang Yang, Jiahao Liu, Jingang Wang, Yunsen Xian, Benyou Wang, and Dawei Song. Lifting the curse of capacity gap in distilling language models. In Proceedings of the 61 Anmual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 4535-4553, 2023.
Han Zhao, Haotian Wang. Yiping Peng, Sitong Zhao, Xiaoyu Tian, Shuaiting Chen, Yunjie Ji, and Xiangang Li. 1.4 million open-source distilled reasoning dataset to empower large language model training, arXiv preprint arXiv:2503. 19633, 2025.
Hang Zhou, Yehui Tang, Haochen Qin, Yujie Yang, Renren Jin, Deyi Xiong, Kai Han, and Yunhe Wang. Star-agents: Automatic data optimization with IIm agents for instruction tuning. Advances in Neural Information Processing Systems, 37:4575 4597, 2024.
Qingqing Zhu, Xiuying Chen, Pengfei Wu, JunFei Liu, and Dongyan Zhao. Combining curriculum learning and knowledge distillation for dialogue generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 1284-1295, 2021.
Xuekai Zhu, Daixuan Cheng, Hengli Li, Kaiyan Zhang, Ermo Hua, Xingtai Lv, Ning Ding, Zhouhan Lin, Zilong Zheng, and Bowen Zhou. How to synthesize text data without model collapse? arXiv preprint arXiv:2412.116688 2024.
15

---

Published as a conference paper at ICLR 2026
CONTENTS
Introduction
2
Related Works
3
Methodology
Problem Formulation
3.2
Identifier: Knowledge Deficiency Diagnosis and Targeting
3.3
Organizer: Progressive Curriculum Design with Mastery Learning
3.4
Adapter: Knowledge Representation Adaptation for Cognitive Alignment
6
3.5
Overall Knowledge Distillation Framework with Data Synthesis
Experiments
Experiment Setup
4.2
Main Results and Analysis
4.3
Supplementary Results and Analysis
10
5
Conclusions and Future Works
10
Pedagogical Foundations and How They Shape IOA
18
Theoretical Background
18
Operationalization in 1OA
18
Why These Principles Improve Distillation
18
A.4
Practical Hyperparameter Guidance
19
A.5
Summary: From Principles to Mechanisms
19
Seed Data Construction
19
Detailed Introduction to Baselines
20
Detailed Introduction to Evaluation Benchmarks and Metries
22
Implementation Details and Hyperparameter Settings
22
Extensive Experiments on More Student Models
23
Evaluation on Agentic Tasks
24
Ablation and Analysis of GennerateRmmddialDaaaa
25
Additional Hyperparameter Robustness Analysis
26
Robustness to Hierarchical Knowledge Decomposition
27
16

---

Published as a conference paper at ICLR 2026
K
Exploration Combining On-Policy Distillation and IOA
27
Knowledge Distribution of Seed and Synthesis Datasets
27
M Knowledge Hierarchy Generation Prompts
28
Knowledge Representation Adaptation Prompts
29
Practical Prompting Guidelines for Data Synthesis
29
0.1
Relationship to Appendix H
29
0.2
Prompting Framework Overview
30
0.3
System Prompt Template
30
0.4
User Prompt Template (Per Curriculum Stage)
31
0.5
JSON Output Schema
................................
31
0.6
Remedial and Bridging Prompts
32
0.7
Illustrative Mini-Templates
................................
32
0.8
Implementation Notes and Reproducibility Checklist
32
Case Studies
32
Distinguishing Intrinsic Difficulty, Prerequisite Failure, and Cognitive Load
36
Explanations for Extreme Hyperparameter Settings
37
Analysis of Catastrophic Forgetting in Mastery-Based Progressive Learning
37
Limitation Analysis
37
Declaration of LLM Usage
38
17

---

Published as a conference paper at ICLR 2026
A PEDAGOGICAL FOUNDATIONS AND HOw THEY SHAPE 1OA
A.1 THEORETICAL BACKGROUND
Bloom's Mastery Learning (Bloom 1968). Mastery learning emphasizes criterion-refereneed learners advance only after demonstrating sufficient mastery of prerequisite material; otherwise they receive targeted remediation. In our framework this principle appears as stage-wise advancement gates that require the student model to reach a high performance ratio relative to the teacher before moving on, i.e. the mastery gate with threshold Tmastery.
progress:
Vygotsky's Zone of Proximal Development (ZPD) VVygotsky & Cole, 1978). ZPD emphasizes introducing tasks just beyond the learner's current independent capability while providing scaffolds, thereby controlling the "step size" of difficulty. In IOA this appears bounded difficulty increments between adjacent curriculum stages regulated by the ZPD threshold TZPD-
Together, these principles motivate our design criteria: topological curricula, bounded difficulty increments, and stage-wise advancement rules, operationalized throughout the pipeline.
A.2 OPERATIONALIZATION IN 1OA
We diagnose fine- ggainer knowledge gaps and build a dependency graph over modules that remediation targets concepts whose mastery most strongly enables downstream progress. This includes thresholds for recognizing mastery of prerequisites (Thigh = 0.9, Tlow 0.7) and dependency inclusion threshold (Tdep 0.3).
Identifier - "What to teach":
Organizer + "When to teach": We construct topological curriculum over the dependency graph so that prerequisite knowledge learned before dependents, forming stages 51, ....... Drawing on ZPD, we bound difficulty increments between consecutive stages using:
Adfiitly (8; + 8i+1) S TZPD avg difficulty(#:).
with TZDD = 0.15, ensuring each step stays within the student's learnable zone.
Organizer > Mastery gating: Following Bloom, require criterion-reffereneed mastery before
progression:
Ps(k) RER; (k)
min
Tmastery:
with Tmastery 0.9, triggering remedial synthesis if unmet.
Adapter + "How to teach": To keep each stage within the ZPD and to facilitate mastery, we scaffold representations via: abstract--o-comerrte coneretization, (ii) reasoning decomposition into atomic steps with verification, (iii) cognitive- load management via controlled instance complexity, (iv) template- based solution formats for stable scaffolds, and (v) linguistic simplification with explicit discourse markers. Full prompt templates for these adaptations appear in Appendix N O]
A.3 WHY THESE PRINCIPLES IMPROVE DISTILLATION
In this part, we clarify why pedagogical principles in IOA can benefit language model distillation, even though human learning and neural optimization differ substantially.
1. Curriculum principles reshape the optimization landscape through training-data distribution. LLMs are trained by gradient descent. and the gradients they receive are determined entirely by the distribution of training examples. Mastery Learning and ZPD act precisely on this distribution: they regulate which examples appear each point in training and how difficulty progresses. Although these ideas originate from human education, in IOA they function data-level inductive biases that smooth the optimization trajectory, reduce gradient variance, and prevent the student from being exposed to examples far beyond its current capacity. These effects are known improve convergence stability in neural models.
2. Controlling difficulty prevents gradient domination by unsolvable examples. If a student model is repeatedly trained on examples that are too difficult, the loss surface is dominated by high-error regions that produce large gradients but little useful signal. ZPD's bounded difficulty
18

---

Published as a conference paper at ICLR 2026
step (Eq. 8 prevents this mismatch by ensuring that each batch of synthesized data remains within a solvable range relative to the student's competence. This avoids gradient explosion, reduces ineffective updates, and keeps the optimization process in a learnable region.
Mastery thresholds align the training sequence with knowledge dependencies. Advancing to new knowledge only when the student has achieved sufficient mastery on prerequisite modules (Eq. 9 prevents underfitting foundational skills. For neural networks, this mechanism reduces interference from partially learned concepts and promotes representation consolidation before new content is introduced. Such staging is a known driver of sample efficiency in curriculum and self-paced learning frameworks.
4. Structured ordering reduces destructive interference between tasks. The Identifier's prerequi- site graph ensures that modules that support one another are learned in an order consistent with their dependency relations. For neural architectures, this reduces conflicting gradients between related sub-tasks and enables shared internal representations develop gradually. This effect parallels observations in multi-task and progressive learning that ordering tasks by dependency improves transfer and reduces forgetting.
5. Adaptive representation scaffolding improves gradient informativeness. The Adapter modifies the representation format of synthesized data to better match the student's expressive capacity (Sec. 3.4). This increases the alignment between the student's hypothesis class and the target signals receives, which reduces loss ambiguity and yields more informative gradients. In gradient-descemt terms, this provides better-conditioned optimization problem.
Overall, these pedagogical principles improve distillation because they induce a structured, capacity- aware training distribution that reduces gradient variance, prevents exposure unsolvable examples, respects knowledge dependencies, and aligns data representations with the student's capabilities. Their benefit arises not from human-like cognition but from the way they shape the optimization landscape through curriculum design.
A.4 PRACTICAL HYPERPARAMETER GUIDANCE
In our standard configuration we set T2PD 0.15 and Tmastery 0.90. Hyperparameter sweeps show non-monotonic trend for TZPD with peak near 0.15 (too small under-challenges: too large overshoots), and performance for Tmastory peaks around 0.90-0.95 before very strict values slow progress.
A.5 SUMMARY: FROM PRINCIPLES TO MECHANISMS
Bloom's Mastery Learning + Stage gating + remedial loops (advance only after demonstrating criterion-level competence).
Vygotsky's ZPD Bounded difficulty increments scaffolding (keep steps learnable; adapt representations and reduce load).
Curriculum theory + Prerequisites - Topological sequencing over dependency graph to respect knowledge structure.
These mechanisms instantiate educational theory in concrete synthesis and training rules, yielding the IOA pipeline that explicitly answers what teach (Identifier), when to teach (Organizer with ZPD-bounded progression and Bloom's mastery gates), and how to teach (Adapter with scaffolds and cognitive alignment).
B SEED DATA CONSTRUCTION
To anchor the synthesis pipeline, curate compact seed dataset Dwoed specifically aligned with four core domains: instruction following, mathematical problem solving, code generation. and academic knowledge reasoning. The design emphasizes coverage, difficulty stratification, and cleanliness.
19

---

Published as a conference paper at ICLR 2026
Instruction Following We collect ~880 prompts from open instruction datasets (e.g.. Dolly OpenAssistamt uudder permissive licenses) and complement them with author-authored tasks targeting underrepresented skills such multi-tumn dialogue, style transfer, and long- mor-form summarization. All items are filtered to avoid overlap with instruction-tuning evaluation benchmarks such as Dolly Evaluation, Vicuna Evaluation, MT-Bench and AlpacaEval.
Mathematical Problem Solving We construct ~900 problems spanning multiple difficulty levels:
Elementary to undergraduatt-levee sampled from openly licensed K-12 and college math reposi- tories (arithmetic, algebra, calculus).
Graduate-level and Olympiad-styyl problems: re-authored from public-domain collections (e.g.. International Mathematical Olympiad archives) and graduate problem sets from open courseware (e.g.. MIT OCW D. All problems are paraphrased and re-contextualized to avoid leakage into evaluation sets such as GSM8K or MATH.
Synthetic templates: 200 programmatically generated algebra and number theory problems with parameterized difficulty.
Code Generation We curate 700 tasks, combining both toy and realistic programming challenges:
Algorithmic and snippet-level tasks: author-written, inspired by programming textbooks (e.g.
"Introduction to algorithms" (Cormen al. 2022)) and tutorials.
Project-level code excerpts: drawn from permissively licensed open-source repositories on GitHub 4 We select self-contained functions, modules, or scripts that perform meaningful tasks (e.g., data parsers, configuration utilities, numerical algorithms), avoiding benchmark-style tasks from Hu- manEval or MBPP.
Academic Knowledge Reasoning We compile ~600 items across science, engineering, and social sciences:
Undergraduate- factual and reasoning questions: curated from open educational resources (e.g., Wikibooks OpenStax textbooks).
Graduate-level or applied science/engineering problems: adapted from open-access lecture notes and course materials (e.g.. control systems, thermodynamics, molecular biology). Problems paraphrased and checked to avoid overlap with GPQA subject pools.
Cleaning & Partitioning Across domains, we enforce strict de dd-ulliccaiion and contamination checks: items with n-gram or semantic similarity to evaluation benchmarks (e.g., Dolly Evaluation, MATH, HumanEval) are excluded. Additional filtering removes toxic privacy-sensitive content. The dataset split into train/validation an 8:2 ratio, with validation reserved for probe tasks. This splitting operation is conducted on the granularity of knowledge modules in the knowledge hierarchy.
Statisties The final Dieed consists ~3000 items: ~000 instruction, ~-00 math (including graduate/Olympiad-less
~700 code (including project-level snippets), and ~600 academic reason- ing (with university- science/engineering)) Despite its compact scale, the dataset provides broad, stratified coverage that anchors synthesis while enabling fine- grained evaluation.
C DETAILED INTRODUCTION TO BASELINES
To help better understand the baseline methods used in experiments, we provide the corresponding detailed introductions as follows:
Self-Instruct (Wang et al. 2023): Self-Instruct introduces semi-automated pipeline for creating respoos data without heavy human annotation. Starting from a small seed set of
"haps://ocw.mit.edu
'nttp://githuu..com
hhttps//wwwwwwii
"haps://openstax.org/
20

---

Published as a conference paper at ICLR 2026
manually written tasks, the framework iteratively prompts pretrained language model to generate new instructions and corresponding imput output instances. Low-quality or redundant samples are filtered using heuristic rules, and the resulting synthetic dataset is then used to finetune the same model. This bootstrapping process enables large-scale data synthesis that aligns pretrained models follow instructions more effectively through supervised distillation from their own generations.
LaMini (Wu et al., 2024): LaMini proposes a large-scale data distillation framework that creates instroction-response pairs by first collecting diverse queries from publicly available instruction datasets and community sources, then unifying and normalizing them into a standardized format. A strong teacher model is prompted to generate high-quality responses for these queries, yielding a massive synthetic corpus covering varied domains and task types. Smaller student models are then finetuned on this distilled dataset, allowing them to inherit cntrructionfolloviiii ability from the teacher while operating with significantly fewer parameters.
Lion (Jang et al. 2023): Lion introduces an adversarial distillation framework that synthesizes training data by querying a proprietary teacher model with carefully designed prompts. Instead of relying on static seed instructions, Lion iteratively generates diverse and challenging instruc- tion-response pairs, where the student's weaknesses are identified and exploited to craft adversarial prompts. The teacher model provides outputs for these prompts, forming synthetic dataset that di- rectly targets areas where the student underperforms. The student then finetuned on this evolving dataset, enabling knowledge distillation from the teacher through closed-loop, adversarial data generation process.
Distilling Step-by-Step (DSS) (Hsieh et al. 2023): DSS leverages the reasoning ability of large language models to produce not only task labels but also intermediate natural language rationales via thain-tf prompting. These rationales, paired with the predicted labels, are treated as synthetic supervision and used to train smaller student models under a multi-task framework, where the student learns both to predict final answers and to generate rationales. By enriching the distilled dataset with step-by-step explanations, this approach provides denser task knowledge than label-only distillation, allowing compact models to inherit reasoning skills from larger teachers without requiring the teacher model at inference time
CasCoD DDai et al. 2024): CasCoD proposes a curriculum-based data synthesis framework for distilling large language models. It first generates synthetic instruction-response pairs using teacher model, but instead of treating all data equally, the method organizes these pairs into a structured curriculum. The curriculum is built by ranking samples according to difficulty and causal relevance, starting from simpler instances and gradually moving toward harder ones. The synthetic dataset is then used to train student models in a staged manner, ensuring that knowledge distillation from the teacher is both effective and stable.
CounterDistill (Feng et al. 2024a): CounterDistill introduces a counterfactual distillation frame- work that augments training data by systematically editing task instances with the help of large teacher model. The method first applies topic word extraction and syntactic analysis to identify causal features in the input text, which are then masked and replaced generate counterfactual examples that closely resemble the original inputs but yield different labels. To further enrich supervision, the teacher model produces multi-view chain-of- chain-of-thought rationales: positive rationales explaining why the correct answer is valid, and negative rationales refuting each incorrect option. By combining factual and counterfactual data with diverse reasoning paths, the approach synthe- sizes a rich distillation dataset that helps student models learn causal reasoning patterns and avoid spurious correlations
Star-Agents
(Zhou et al. 2024): Star-Agents formulates data synthesis for distillation in multi- agent framework. A set of specialized teacher agents, each with distinct roles such as planner, executor, or critic, are orchestrated collaboratively generate high-quality instruction-responss data. The interaction among agents allows them to refine prompts, critique outputs, and iteratively improve the synthetic dataset. This diverse and self-improving corpus is then used distill knowledge into student model, enabling it to acquire richer capabilities than what could be achieved from aggett data generation alone.
MADA WWag et al. 2025a): MADA introduces multi-perspective distillation framework that synthesizes training data by prompting a teacher model to generate responses under diverse reason- ing styles perspectives. For each input instruction, the teacher produces multiple complementary answers-suce as step-by-step explanations, concise summaries, or alternative solution strate- gies-whick are then aggregated to form richer supervision signal. The student model is distilled
21

---

Published as a conference paper at ICLR 2026
this multi-view synthetic dataset, allowing it capture broader reasoning patterns and avoid overfitting to a single response style.
D DETAILED INTRODUCTION TO EVALUATION BENCHMARKS AND METRICS
As mentioned in the Section 4.1 we evaluate the language model distillation performance from two perspectives: instruction following and reasoning, considering they most attractive LLM capabilities currently Here, we supplement detailed descriptions of these benchmarks:
Dolly Evaluation (Conover et al., 2023): A benchmark derived from the Dolly dataset, covering diverse nostruction-oolloiing tasks including open-ended question answering. brainstorming, and text generation. It measures how well model follows natural instructions without additional context.
Vicuna Evaluation (Chiang et al., 2023): A widely used benchmark for cntrraction-followiiii models, consisting of 80 turn conversations. Model outputs are compared with reference responses from strong open-source models, making it proxy for alignment and usability.
GSM8K (Cobbe et al. 2021): A grade-school math word problem dataset with 8.5k questions. evaluates arithmetic reasoning and step-by-step problem-solvingg often requiring multi-hop logical
inference.
MATH (Lewkowyez al. 2022): ; A large-scale benchmark of high-school competition-level
math problems across algebra, geometry, calculus, and number theory. It tests deeper mathematical reasoning beyond GSM8K.
The 2024 American Invitational Mathematics Examination dataset, consisting of challenging competition-style math problems. It pushes models toward precise reasoning under high difficulty, closer to Olympiad-leve tasks.
AIME2024 (Mathematical Association of America, 2024):
HumanEval (Chen al. 202T): A code generation benchmark of 164 hand-crafted Python programming problems. Models are required to generate correct, executable solutions given only natural language descriptions.
MBPP (Austin et al., 2021): The "Mostly Basic Programming Problems" dataset of 974 crowd- sourced Python programming tasks. It evaluates code synthesis ability on short, self-contained tasks that resemble cisjinee--t--nntemmdiaatt coding exercises.
LiveCodeBench (Jain et al. 2025): : A dynamic, automatically updated benchmark for code
generation, pulling from real-world programming questions. is designed to prevent test-set contamination and provides
realistic measure of coding capability.
(Rein et al. 2024); A subset of the GPQA benchmark that focuses high- quality, difficult graduate-level questions across science and engineering. It evaluates deep domain knowledge and expert-level reasoning.
Furthermore, we provide the explanations the evaluation metries: ROUGE-L and Pass
ROUGE-L: text-overlap metric that measures the longest common subsequence (LCS) between generated and reference answers. It captures fluency and content overlap, and is commonly used for summarization and instruction-follooiing evaluation.
Pass@k: k: A probabilistic evaluation metric that measures the likelihood of obtaining correct answer within & independent attempts. Formally, given multiple sampled generations, @ is defined as the probability that least one of the outputs matches the ground truth solution. Originally popularized in code generation tasks, this metric has been adopted more broadly as robust measure of reasoning performance, as it captures both the accuracy of individual attempts and the benefits of diverse solution exploration. A higher Pass @ k indicates that model is more capable of eventually producing a correct reasoning path, even not on the first try.
E IMPLEMENTATION DETAILS AND HYPERPARAMETER SETTINGS
All experiments are conducted on multiple clusters, each equipped with 8x Nvidia HGX H20 GPUs (96GB memory), Intel Xeon Platinum 8358 CPUs, and 1.5 TiB RAM. We implement
22

---

Published as a conference paper at ICLR 2026
Table 6: Comparison among different data synthesis-based distillation approaches various evalua- tion benchmarks for remaining student models. OpenAI ol is taken as the teacher LLM. Bold and underline indicate the best and second-best results, respectively. indicates statistically significant.
Model
DoilyEval
VicunaEval
GPQA-D
Undistileed
26.12
25.66
Sest-tast
34.68
31.66
47.89
9.22
3.67
28.83
39.27
19.44
10.48
LaMini
36.44
33.45
49.48
10.06
3.75
30.03
40.49
20.02
10.85
Lion
37.23
34.92
52.56
11.53
4.33
32.15
42.14
22.34
11.66
Qwen2.5-7B
37.78
35.75
53.77
CasCoD
36.21
DounterDistil
36,76
35.38
54,04
13.29
5.47
34.35
43.35
23.89
12.25
Star Agents
38.58
37.00
55.45
14,57
5.93
35.91
44.63
25.13
12.87
MADA
38.92
37.40
56.24
15:25
6.24
36.59
45.18
25.66
13.13
(Ouns)
Undistileed
Self Iast-nt
36,38
33.26
50,09
10:42
4.17
30,43
40,97
20.54
11.28
LaMini
38.14
35.05
51.68
11.26
4,25
31.63
42.19
21.12
11.65
Lion
38.93
36.52
94.76
12.73
4.83
33.75
43.84
23.44
12.46
Qwen2.5-14B
37.35
CasCoD
CounterDistil
38,46
36.98
56.24
14.49
5.97
35.95
45.05
24.99
13.05
Star-Age Star-Agents
40.28
38.59
57.65
15.77
6.43
37.54
46.33
26.23
13.67
MADA
40.62
38.99
58.44
16.45
6.74
38.19
46.88
26.76
13.93
(Ouns)
Undistileed
23.57
Self Instractt
32.22
29.44
43.57
7.81
2.98
26.44
36.22
17.25
9.54
LaMini
33.57
31.25
45.25
8.53
3.23
27.65
37.34
17.92
9.91
Lion
34.71
32:46
48.63
10.09
3.76
29.64
38.93
20.16
10.72
LLaMA3.1-8B
33.17
49.75
CasCoD
33.53
DountenDistill
31,99
32.72
50,06
11.93
1.85
31.55
40.11
21.63
11.28
Star Agents
36.49
34.23
51.14
12.81
5.02
32.76
41.04
22.61
11.64
MADA
37.12
34.81
51.82
13.26
5.08
33.68
41.53
23.02
11.87
(Ours)
method in PyTorch with HuggingFace Transformers, enabling mixed precision training and gradient checkpointing for efficiency. For data synthesies, set J; the uniform value 10 for main experiments. During the training phase, we use the AdamW optimizer with 3g = 0.9, 82 = 0.95, weight decay 0.01, and gradient clipping 1.0. A cosine learning rate schedule with 3% warm-up is applied, setting the peak learning rate to 2e-5 for full-parameter fine-tuning (3B models) and 1e-4 for LoRA-based tuning (7/8/14B models). Training is performed with an effective global batch size of 128 using gradient accumulation, and maximum context length of up to 4096 tokens, depending on the task (4096 for reasoning, 2048 for instruction-follovvnn). We train up to epochs over the stage's ' synthesized data (or until minges, Ps(k) 2 0.9), then advance to next unit. All reported results are averaged over five runs with different random seeds. We further conduct paired 1-tests between our method and the perffrrrrornng baseline, confirming statistical significance at the 99% confidence level (p 0.01). The code has been provided in the Github repository htttp:://gitie com/ http:/oowwaaHHHToo IIOA
(k)
F EXTENSIVE EXPERIMENTE ON MORE STUDENT MODELS
In addition to the experiment results on Qwen2.5-3B and LLMMA322-BB as provided in the main text, we conduct more extensive experiments on student models of Dwen2.5-7B. Qwen2.5-14B. and LLLMA3.1-888 The corresponding results with OpenAl ol and DeepSeek-RI as teacher models are provided in Table Tad respectively. Across all settings. IOA consistently achieves the best performance, outperforming strong multi-agent, cascade, and adversarial data synthesis baselines by clear margin. For example, with ol as teacher, IOA boosts Qwen2.5-7B (undistilled) to 59.99 and HumanEval from 25.66 to 43.84, while similar improvements hold for larger Qwen2.5-14B and 1-- students. Scaling the student further amplifies the benefits, with IOA gains increasing steadily from 7B to 14B (c.g.. GSM8K: 59.99 + 62.16; HumanEval: 43.84 45.47). Moreover, teacher choice shapes the performance profile: R R1-trained students
GSM8K from 41. 1.44
23

---

Published as a conference paper at ICLR 2026
Table 7: Comparison among different data synthesis-based distillation approaches various evalua- tion benchmarks for remaining student models. DeepSeck-RR RI is taken as the teacher LLM. Bold and underline indicate the best and second-best results, respectively. indicates statistically significant.
DoilyEval
VicunaEval
LiveCdeBench
GPQA-D
Undistileed
25.92
34
35.58
Sest-tast
34.38
31.46
48.79
9.72
3.97
30.03
40.27
20.24
11.08
LaMini
36,14
33.25
50.38
10.56
4.05
31,23
41.49
20.82
11.45
Lion
36.93
34.72
53.46
12.03
4.63
34.35
43.14
23.14
12.26
Qwen2.5-7B
54.67
CasCoD
36.01
DounterDistil
36,46
35.18
54.94
13.79
5.77
36.55
44.35
24.69
12.85
Star Agents
38.28
36.90
56.35
15.07
6.23
38.14
45.63
25.93
13.47
MADA
38.62
37:20
57.14
15.75
6.54
39.19
46.18
26.46
13.73
(Ouns)
Undistileed
37.28
Self Iest-nn
36,08
33.06
51.00
10,92
4.47
31.63
41,97
21.34
11.88
LaMini
37.84
34.85
52.58
11.76
4.55
32.83
43.19
21.92
12.25
Lion
38.63
36.32
55.66
13:23
5.13
35.95
44.84
24.24
13.06
Qwen2.5-14B
37:15
CasCoD
CounterDistil
38.16
36.78
57.14
14.99
6.27
38.15
46.05
25.79
13,65
Star-Agent
39.98
38.40
58.55
16.27
6.73
39.74
47.33
27.03
14.27
MADA
40.32
38.80
59.34
16.95
7.04
40.79
47.88
27.56
14.53
(Ouns)
Undistileed
23.37
16.04
Self Instractt
31.92
29.24
44.47
8.31
3.28
27.64
37.22
18.05
10.14
LaMini
33.27
31.05
46.15
9.03
3.53
28.55
38.34
18.72
10.51
Lion
34.41
32:26
49.53
10:59
4.06
30.84
39.93
20.96
11.32
LLaMA3.1-8B
32.97
50.65
CasCoD
DounterDistill
31.69
32.52
50,96
12:43
5.15
32.75
41.11
22.13
11.88
Star AgentA
36.19
34.03
52.04
13.31
5.32
33.96
42.04
23.41
12.24
MADA
36,82
34.61
52.72
13.76
5.38
34.88
42.53
23.82
12.47
(Ours)
particularly strong on reasoning and code (e.g., Qwen2.5-7B GSM8K: 61.78, HumanEval: 46.48), while ol yields slightly higher instruction-follie forlloonn scores, showing that IOA effectively transfers the strengths of distinct teachers. Finally, the observed improvements generalize across model backbones (Qwen vs. LLaMA) and task categories (instruction following and reasoning including competition math, code, and science QA). Taken together, these results demonstrate that IOA not only effective but also scalable and broadly generalizable across different student model sizes and structures.
G EVALUATION ON AGENTIC TASKS
In addition to experiments on instruction following and reasoning benchmarks employed in the main text, further evaluate the performance of our proposed IOA framework agentic tasks, which has attracted more and more attention recently. Specifically, we select r-Bench (Yao et al. (024)) and BFCL (Patil et al. 2025 benchmarks. r-Bench evaluates agents' ability interact with simulated human users and domain-specific APIs under realistic rules (e.g., retail and airline customer service), emphasizing consistency and reliability via the pass@ k metrie. BFCL benchmarks LLMs' function- calling skills across single-turn, crowd-sourced, multi-tum, and agentic tasks, with a distinctive abstract syntax tree-based validation method that enables scalable and deterministic correctness checking. Especially, for agentic capability distillation, we additionally augment the seed synthesis multi-turn user-API interactions,
pool with agentic-oriented exemplars (tool-useds wtt-uuuunctiooncaallll and task decompasition patterns) so that the distilled data explicitly covers the behaviors targeted by r-Bench and BFCL. As shown in Table IOA consistently achieves the highest performance across all student models and both teacher LLMs, while MADA is the strongest among prior baselines. Concretely, withn OpenAl ol as teacher. IOA improves Qwen2.5- from 44.83 (undistilled) to 59.02 on r-Bench and from 42.55 to 56.48 on BFCL, and similar margins hold for larger Qwen2.5- 14B and LLaMA3.1-88 students. With DeepSeek-RR R1 as teacher, IOA yields further gains on agentic skills, particularly on BFCL which emphasizes function-calling correctness (e.g., Qwen2.5-14BB
24

---

Published as a conference paper at ICLR 2026
Table 8: Comparison among different data synthesis-based distillation approaches agentic bench- marks 7-Bench and BFCL. OpenAl ol and DeepSeek-Re RII are taken as teacher LLMs. Bold and underline indicate the best and second-best results, respectively. For clear illustration, Qwen2.5 and LLaMA3. 1/3.2 are abbreviated as Q and L, respectively. indicates statistically significant.
LLM
SLM
Task
Undistilled
Self-Instrect
LaMini
Lion
DSS
CisCoD CounteDiitiil
Star-Agents
MADA
10A(Ours)
0:3B
reBnch BFCL
41.22
47.36
48.15
49.28
50.14
50.66
49.87
51.72
52.48
$5.63*
38.97
45.16
46.24
47.05
46.93
52.11
-eRench
44.83
50.27
51.36
52.57
53.41
53,96
53.27
54.85
55.72
59.02
Q-78
BFCL
42.55
47.68
48.72
49.91
50.68
51.19
50.53
52.36
53.14
56.48
of
Y- Y-Hench
46.72
52.31
53.42
54.56
55. 38
55.94
55.11
56,92
S7.86
61.14
Q-14B
BFCL
44.37
49.63
50.58
51.76
52.53
53.09
52.41
54.28
55.16
$8.52
L-SB
r-Bench
40.08
46.55
47.63
48.42
49.01
48.16
50.67
53.84
BFCL
43.18
44.26
45.09
44.85
46.38
50.16"
L-38
Y- Hench
36.47
41.25
42,03
43.19
44.07
44.63
43.78
45.52
46.34
49.28
BFCL
34.12
39.27
40.15
41.23
42.06
42.64
41.77
43.26
44.08
46.87
r- BeBcch
42.35
48.46
49.27
50.42
51.26
51.84
51.05
52.96
53.81
$7.09
Q-38
BFCL,
39.78
45.12
46.08
47.25
48.09
48.68
47.93
49.52
50.46
53.77
Beench
45.94
51.42
52.37
54.41
55.03
54.28
56.11
$7.03
600.44
Q-7B
BFCL
43.67
48.75
49.73
50.92
51.74
52.33
51.57
53.42
54,36
$7.62*
r. Bench
47.88
53.41
54,52
55.73
56.54
57.16
56.42
58.37
59.22
62.58*
Q-14B
BFCL
45.51
$0.86
51.81
53.02
53.84
$4.48
53.73
55.58
56.44
59.83*
L-8B
y- Bench
41.23
46.91
47.72
48.93
49. 73
50.35
49.61
51.42
52.31
$5.66
BFCL
44.57
16.59
16.42
52.54
L33
T- r-Bench
37.69
42.43
43,26
44.48
45.36
45.94
45.18
46.92
47.76
$0.98
BRCI
35.27
40.44
41,36
42.57
43.39
43.98
43.23
44.87
45,73
48.92
Table 9: Supplementary Ablation study various evaluation benchmarks for LLaMA and Qwen models. OpenAl ol is taken as the teacher LLM. * indicates statistically significant.
Model
Method
DollyEval
VicumaEval
GSMSK
MATH
AIME2024
HuranEval
MBPP
LiveCodeBench
GPQA-D
Qwen2.5-38
FOl
38.16*
36.83*
55.79-
15.53*
6.29*
40.64
47.86*
26.94*
13,76*
Generaeeemociallbaa
37.42
35.08
53.25
14.35
5.84
39.92
46.74
26.18
13.28
LLaMA32-38
Full IOA
36.88*
34.81
$2.07
14.02
5.4*
37.0*
44.25*
24
12.15
36.15
33.92
50.42
13.25
5.08
37.31
43.28
23.41
11.76
improves from 45.51 to 59.83). These results highlight three consistent trends: (i) effectiveness IOA provides large and robust improvements over both undistilled models and prior distillation methods; (ii) scalability larger students (7B + 14B) consistently benefit more; (iii) generalization - IOA adapts to different teachers and backbones while excelling on new task types beyond standard instruction following and reasoning. Taken together, these findings confirm that IOA not only distills basic knowledge and reasoning ability effectively, but also enhances students' agentic capabilities in realistic interactive and function-calling scenarios.
H ABLATION AND ANALYSIS OF GENERATEREMEDDAALDAT
Ablation on GennerateRmmedialDaaaa To isolate the contribution of Mastery-Basee Progressive Learning, we ablate the GenerateRemedialDatt component while keeping the dificient knowledge identification, dependency- -driven curriculum and the adaptation mechanisms unchanged. As shown in Table removing remedial data consistently lowers performance across all benchmarks and across both Qwen2.5-3B and I.LAMA32--3B student models. The full IOA pipeline yields gains of +2.54 and +1.6 points on GSM8K, +1.18 and +00.7 points on MATH, and +0.45 and +0.39 points on AIME2024, respectively. Similar improvements are observed for code-generation tasks such as HumanEval, MBPP, and LiveCodeBench. These results indicate that a fixed curriculum alone is insufficient; targeted remedial examples help the student close the residual gaps that remain after the first pass through a module.
How often does GENERATE TEMEDIALDAT. occur in practice? In our implementation. remedial data generation is invoked only when the student fails to meet the mastery criterion for a module (Eq. 9 Empirically, this condition triggered infrequently. Across all experiments reported in the main paper,
25

---

Published as a conference paper at ICLR 2026
Fifeet Teop
27.0
26.0
27.00
26.75
Aug mouutoo-4
Pog Peroog
18.3
10.30
10.00
15.79
35.3
0.13
0.20
0.20
6.50
0.35
0.80
0.90
3.00
0.73
0.50
Trg
7am
Tae
37.30
37.25
31.0
Dossag
26,75
Passgg
36.30
36,00
35,75
3.20
0.23
0.33
0.40
0.35
0.83
0.23
Telp
ROUGE L 10wcnn2.5 5
381
ROUGE I
Possg] (0wen2
38
Possgi (LLDMAS 38)
Figure 5: The hyperparameter robustness analysis for other five critical hyperparameters: Tgaps Thigh Nows Tdep. in the Identifier component of our IOA framework.
we observe that the majority (86-92% depending on the student model) knowledge modules are mastered in single pass without requiring any remedial iteration. Among the remaining cases, most require exactly one remedial generation step, and only small fraction (< 3% modules) invoke more than once. This behavior arises naturally from the structure of IOA. The dependency-drine driven curriculum ensures that modules are presented an order aligned with prerequisite relations, so by the time module is introduced, the student has already consolidated the foundational knowledge needed to learn it efficiently. Moreover, the bounded difficulty increment enforced by ZPD (Eq. {) prevents the student from being exposed tasks that are far beyond its current competence, reducing the likelihood that mastery fails on the first attempt.
Overall impact. Together with the ablation results, these observations suggest that GemerateReme- dialData plays a targeted rather than pervasive role in IOA: it is invoked only when necessary and typically only once per module, yet yields consistent improvements on both reasoning and coding benchmarks. This demonstrates that Mastery- Mastery-Based Progressive Learning provides an effective and computationally efficient mechanism for correcting residual weaknesses that would otherwise remain unaddressed in a single-pas, curriculum.
I ADDITIONAL HYPERPARAMETER ROBUSTNESS ANALYSIS
To further explore the robustness of remaining hyperparameters Tgap- Thigh: Tow Tdep- and a in the Identifier component, we conduct additional robustness experiments by varying each hyperparameter value while keeping all others fixed. Figure S the results for both Qwen2.5-3B and LLaMA3.2- 3B student models.
Across all five hyperparameters, observe consistent pattern: IOA maintains stable performance over a broad range of values, with degradation occurring only at extreme settings. For Tgap and Tdop. performance improves as the dependency threshold increases from very small values and plateaus around 0.25-0.35, indicating that overly permissive thresholds introduce noisy spurious dependencies while overly strict thresholds remove useful prerequisite structure. Similarly, Thigh exhibits broad optimum between 0.85 and 0.95, reinforcing that IOA benefits from strict but not extreme high-m migttrystrry criterion. Here, a stricter threshold (c.g., increasing Thigh from 0,9 to 0.95) remedial iterations, but the additional computational overhead is modest,
does lead slightly typically within 3-5% of the total training time, and does not obviously affect the practicality IOA.
For Tlow, both models show declining performance when the threshold is too high (c.g., 0.8), this causes insufficient separation between masterete and poorly-ma mastered modules. Performance stabilizes the default value of 0.7 and remains robust 0.75. Finally, the severity parameter
26

---

Published as a conference paper at ICLR 2026
Table 10: Exploration the combination of TOA and on-policy ditillation method: GKD. DeepSeek- RI RI-Distil-Qwen- 14B and Qwen2.5-7B are taken as the teacher and student model, respectively, Bold indicates the best results. indicates statistically significant.
Model
Method
DollyEval
VicunaEval
GSMSK
MATH
AIME2024
HumanEEval
LiveCodeBench
GPQA-D
39,97
63.95
47.67
17.08
Qwen2.5-7B
GKD
38
37.20
39.19
13.73
10AGGKD
42,88*
38.92
68.43*
20.78
11.62
49.25
56.38
34.51"
17.86
demonstrates a smooth unimodal trend with an optimum near 0.7; lower values under-emphasize deficient modules, while overly large values may suppress important but less-challenging knowledge.
Overall, these results indicate that IOA is not sensitive to precise hyperparameter choices: each hyperparameter has a wide region where performance is stable, and the defaults used in the main paper lie near the center of these regions. This further confirms that the Identifier component is controlled by coarse stability regulators rather than fine-grained task-specific tuning knobs.
J ROBUSTNESS TO HIERARCHICAL KNOWLEDGE DECOMPOSITION
To examine how sensitive IOA is to the quality of the hierarchical knowledge decomposition extracted from the teacher model, we conduct set of controlled perturbation experiments. Specifically, we introduce structural noise by (i) randomly merging a fraction of neighboring knowledge modules into coarser units, and (ii) randomly splitting some modules into finer modules. These perturbations directly disturb the dependency graph and the scope of each module, thereby testing whether IOA critically relies on a perfectly decomposed hierarchy.
Across both Qwen2.5-3B and LLaMA3.2-3B students, we observe that IOA's performance degrades only mildly under moderate levels of perturbation. Random merging causes slight reduction in rea- soning tasks due to less granular prerequisite structure, while random splitting occasionally increases the number small modules requiring mastery. However, drop remains small (typically < 2% across benchmarks), and IOA continues to outperform baselines that do not use hierarchical decomposition at all.
settings, the relative performance
These findings indicate that IOA does not depend on perfectly accurate knowledge hierarchy; instead, it is robust to reasonable structural noise. The Identifier's performance-basee dependency scoring and the Organizer's mastery gating help stabilize the curriculum when the underlying decomposition is perturbed. This confirms that IOA benefits from the hierarchical structure but is not overly sensitive its exact form.
K EXPL.ORATION ON COMBINING ON-POLICY DISTILLATION AND IOA
To explore whether IOA is complementary to on-policy distillation, we conduct study combining IOA with GKD (Agarwal et al. 2024). Following the setting the supplementary experiments of main text, employ model respectively, to ensure the on- an-policy logit distillation feasible. As shown in Table [0] GKD alone underperforms IOA across all benchmarks, but incorporating GKI) top of IOA yields further improvements over IOA on nearly every evaluation task, including GSM8K (63.95 - 65.43), MATH (19.64 20.78), AIME2024 (10.89 11.62), and HumanEval (47.67 49.29 29). This suggests that IOA and on-policy distillation operate on different aspects of the language model distillation process. Specifically, IOA provides structured, capacity-awar curriculum for what and when learn, while GKD offers fine-graines corrective feedback on student rollouts from the objective side. Therefore, such two methods can be combined to produce complementary gains. This result confirms that IOA is compatible with and can further enhance on-policy distillation methods.
Rick--RR---ill-OO Qwen--4Wennneeeeee 14B and Qwen2.5-7B as teacher and student
L KNOWLEDGE DISTRIBUTION OF SEED AND SYNTHESIS DATASETS
Figure 6 compares the knowledge composition of the original seed set with the dataset synthesized by TOA. Note that the basis of the standard seed dataset in Appendix B we supplement 500
27

---

Published as a conference paper at ICLR 2026
Knowledge Distribution in Seed Dataset
Knowledge Distribution Synthesis Dataset (IOA)
14.29%
17.28%
22.86%
17.14%
12 62%
31.47%
25.71%
20.00%
24. 31%
Instruction Following
Generation
Agentic Function Calling)
Problem Solving
Academic Knowledge Reasoning
Figure 6: The knowledge distribution of original seed dataset and synthesis dataset by IOA.
items of tool using and function calling data corresponding to Appendix The seed set concentrates more on instruction following and math problem solving, with shares of 22.86% and 25.7 1, followed by code generation (20.00%). academic knowledge reasoning (17. ), and agentic/tool use (14.29%). IOA reshapes this mix toward capabilities where small LMs exhibit larger deficits: math increases to 31.47% (+5.76 pp), code to 24.31% (+4.31 pp), and agentic to 17. 17.28% (+2.99 pp), while instruction following decreases to 14.32% (-8.54 pp) and academic knowledge to 12.62% ( (-4.52 pp). Overall, 13.06 percentage points are reallocated from generic instruction/encyclopedii QA to reasoning- code-, and tool-centric data. This targeted yet diversified shift is produced by IOA's pipeline-whie which favors domains with larger teacher-student disagreement and uncertainty (i.e., math, code, and agentic behaviors). Consistent with this redistribution, IOA delivers the strongest empirical gains on reasoning, coding, and agentic benchmarks (Table] 2
7 indicating that the diagnosis module steers synthesis toward the student's 's most pronounced weaknesses while maintaining balanced coverage across knowledges.
M KNOWLEDGE HIERARCHY GENERATION PROMPTS
To facilitate the reproduction of hierarchical knowledge decomposition in Section 3.2 corresponding prompt as follows.
we provide the
Prompt 1: Knowledge Hierarchy Generation
Please decompose the following learning domain into a hierarchical structure of fine-grained
knowledge modules:
Domain: {domain]
Description: (description)
Please refer to above description and create 25-35 knowledge modules organized by category (not
limited above mentioned the description), each with:
Unique ID in format "category/modulle"
Category name
Descriptive module name
Difficulty level
Return as JSON array. Example format: ("id": "algebra/linear equations", "cate-
gory": "Algebra", "name": "Linear Equations", "difficulty": "introductory"), ("id": "alge-
"etuuudtrtticeuattooo"."""" "category") "Algebra", "name": "Quadratic Equations", "difficulty": "intermediate")
28

---

Published as a conference paper at ICLR 2026
N KNOWLEDGE REPRESENTATION ADAPTATION PROMPTS
To operationalize the five adaptation dimensions described in Section 3.4 we design explicit prompt- ing templates for the teacher model. These templates ensure that the synthesized data is adapted to the cognitive capacity of student models.
Prompt 2: Absrrcct Concept Concretization
Explain the abstract concept of [CONCEPT] using a concrete analogy or real-world example (e.g., speed of a car for derivatives). Avoid formal definitions at first, and gradually transition the symbolic or mathematical expression.
Prompt 3: Complex Reasoning Decomposition
Solve the problem step by step. Break down the reasoning into small sub-steps: (1) Extract relevant information; (2) Identify relationships; (3) Formulate equations; (4) Solve step by step; (5) Verify the solution. Provide each step explicitly.
Prompt 4: Cognitive Load Management
Reformulate the problem into a simpler version of the same type (c.g., start with a 2 x 2 system of equations before moving to larger systems). Ensure each sub- sub-problem is self-contained and introduce incremental difficulty only after demonstrating mastery.
Prompt 5: Representation Format Optimization
Present the solution consistent, structured format using the following template: Step 1: [Action]; Step 2: [Action]; Step 3: [Action]. Use the same template across multiple examples highlight structural patterns while varying the surface details.
Prompt 6: Linguistic Complexity Reduction
Rewrite the explanation [PROBLEM] in simpler language. Use short, direct sentences. Replace advanced terms with simpler synonyms where possible. Use clear connectors such as "first", "next", "therefore". Ensure the reasoning remains correct but linguistically accessible.
These prompts are applied in practice to generate pedagogically adapted synthetic data for each curriculum stage, ensuring that the student model receives content that is both comprehensible and incrementally challenging.
O PRACTICAL PROMPTING GUIDELINES FOR DATA SYNTHESIS
0.1 RELATIONSHIP TO APPENDIX H
Appendix O provides the canonical, atomic prompt templates for the five adaptation dimensions described in Section 3.4 (1) Abstract Concept Concretization, (2) Complex Reasoning Decomposi-
tion, (3) Cognitive Load Management, (4) Representation Format Optimization, and (5) Linguistic Complexity Reduction. These templates specify what kind of representation adaptation the teacher
LLM should apply.
This section complements Appendix operationalizing these atomic templates into full prompting protocols that can be executed across curriculum stages. Concretely, we (i) integrate the five templates into static System Prompt that encodes global pedagogical constraints, (ii) supply stage-specific User Prompt carrying curriculum context and difficulty bounds, and (iii) enforce a machine-checkable JSON output schema for automatic filtering, verification, and training.
29

---

Published as a conference paper at ICLR 2026
O.2 PROMPTING FRAMEWORK OVERVIEW
We adopt two-layer prompting stack to synthesize pedagogically adapted distillation data from teacher LLMs. The stack separates global, invariant constraints (System Prompt) from stage-specific context (User Prompt), and couples generation with machine-checkable output contract.
System Prompt (static): Encodes the five adaptation dimensions from Appendix N as hard constraints applicable to all synthesis calls. These constraints ensure consistent representation adaptation regardless of curriculum stage.
User Prompt (per stage): Carries the current curriculum stage and its context: target knowledge units, prerequisite set, and the difficulty/size caps determined by the Organizer. It may further include the student's ss baseline ratio (relative to the teacher) and the requested number of items, enabling the teacher to calibrate instance complexity and coverage at this stage.
Execution protocol. For each curriculum stage 8;, we: (1) instantiate the static System Prompt once; (2) fill stage-specific User Prompt with (i) stage_id, knowledge -unge. prereqs, Alfficulty/ wwwuulyssizze.e size.caps, and num. examples; and (ii) any stage-level guardrails such as "start from concrete analogy then transition to symbolic form," "explicit step-by-step reasoning." and "self- veeff-verffication.. (3) issue synthesis call and collect candidates; (4) run automatic checks (schema validation, verification success, difficulty bounds, stage alignment); (5) discard and regenerate any failing items until the target count is met. We optionally apply semantic deduplication to maintain diversity across items while keeping template consistency.
Output contract. The teacher LLM must return a JSON array that conforms to the schema in Appendix 0.5 (all required keys present, types correct). Each item must include: the targeted module and prereqs, the problem (analogy - formal), stepwise solution with canonical final answer, an independent verification routine, and adapter flags that document how the five dimensions were applied. Any item that fails verification, violates caps, breaks the schema is discarded and regenerated. This contract enables downstream automatic filtering and reliable training set construction.
Remedial and bridging hooks. mastery is not achieved for s;, we trigger the Remedial prompt to generate simpler, tightly scoped items on the weak sub-skills; upon mastery, trigger the Bridging prompt to gently increase complexity to the next difficulty notch while preserving the five adaptation requirements (see Appendix 1.6). This keeps difficulty increments bounded and aligned with the Organizer's ' pacing.
Reproducibility notes. We log the exact System/User prompts and random seeds, enforce stable stepwise formatting, and reject items with missing verification, out-of-ran -range difficulty, or stage misalignment (see Appendix (O.8. These practices make the prompting framework robust and replicable across runs and stages.
O.3 SYSTEM PROMPT TEMPLATE
Users can copy and paste the following contents System Prompt:
Prompt 7: System Prompt Template
You are a teacher LLM generating pedagogically adapted synthetic data for a student model (SLM). Your goal is to align knowledge representation with the student's cognitive capacity.
Strietly enforce the following adaptation requirements (see Appendix for the canonical templates): 1) Abstract Concept Concretization: begin with concrete analogies before formalism. 2) Complex Reasoning Decomposition: present explicit, small-step reasoning. 3) Cognitive Load Management: start minimal and increase difficulty gradually. 4) Representation Format Optimization: use a consistent stepwise template. 5) Linguistic Complexity Reduction: prefer simple words, short sentences, and clear connectors (e.g., "first", "next", "therefore").
If reasoning verification fails, discard the example and regenerate. All outputs MUST follow the JSON schema provided by the user prompt.
30

---

Published as a conference paper at ICLR 2026
0.4 USER PROMPT TEMPLATE (PER CURRICULUM STAGE)
Users need to fill the placeholders below and then use as the User Prompt at stage 8;²
Prompt 8: User Prompt Template (Per Curriculum Stage)
Target Domain: {DOMAIN}
Curriculum Stage: {STAGE_ID)
Knowledge Units: MODDLLSS
Prerequisites: {PREREQS}
Student Baseline (relative teacher): (BASELINE RATIO)
Please generate {NUM) new synthetic examples adapted to this stage.
Requirements:
Obey the five adaptation dimensions (Appendix N).
Cognitive load constraints: max problem size {SIZE. max symbolic/arith. complexity [COMPLEXITY CAP).
Provide a problem that transitions from concrete analogy to symbolic form.
Provide a full solution with explicit step-by-ster reasoning and verification.
Output MUST conform to the JSON schema below.
O.5 JSON OUTPUT SCHEMA
Listing I: Schema (all keys required unless marked optional)
"module": "<knowledge unit, e.g.,
'>",
"prereq": ("<prereql>", "<prereq2>"),
5
"difficulty_tag": "Kintroductory interdiate advanced>", "problem"; "<text: concrete analogy symbolic formulation>", "solution":
7
"steps": ["Step 1: ...", "Step 2: ...", "...",,
"Final_answer". "<canonical answer>", "verification"; "<independent check; describe or show tests" b
11
12
'adapter_flags"; {
13
"concretization". true, "decompositionn" true, "cognitive_load": {
14
15
16
"scale": "<e.g., 2x2 system' or small input sizer>", "notes": "<what simplified and why> } "format_templatee" "Stepwise-3", laimpliffee__lnnuueee" true
18
20
,,
22 "metadata":
"stage_id"" "(STAGE_ID]
24
"seed_sstiy style_ref";; ref": "(SEED SET ID)" // optional: link to seed style
26
31

---

Published as a conference paper at ICLR 2026
O.6 REMEDIAL AND BRIDGING PROMPTS
Prompt 9: Remedial (when mastery not achieved)
The student did NOT achieve mastery for {STAGE MODULLES Generate (NUM) sim- plified remedial examples focusing ONLY on these weak sub-skills: SWEAK.SUBSKILLS.LO
Constraints:
Reduce linguistic and structural complexity further.
Keep instance size minimal; remove distracting details.
Maintain explicit step decomposition and verification.
Use the same JSON schema.
Prompt 10: Bridging (after mastery)
The student ACHIEVED mastery for (STAGE IDY/(K MODULES). Generate {NUM} bridging examples with SLIGHTLY increased complexity (only one notch higher scale/coefficients/ccn
straints).
Constraints:
Bounded difficulty increments; do NOT skip levels.
Keep the five adaptation requirements. Use the same JSON schema.
O.7 ILLUSTRATIVE TEMILTEMS
Mathematics (Quadratic Roots via Discriminant).
Problem scaffold (analogy - formal): Intuitive change/area analogy, then define a, b, compute À = 122 4ac, apply the quadratic formula, and check by substitution.
Step template: Step Identify coefficients - Step 2 Compute discriminant - Step Apply formula > Step 4 Verify by substitution.
Programming (String Processing).
Problem scaffold: "Assembly line" analogy for per-character processing + pseudocode - implementation.
Verification: Minimal runnable unit tests (small, then slightly larger inputs).
O.8
IMPLEMENTATION NOTES AND REPRODUCIBILITY CHECKLIST
Hard filtering:
Discard any item whose verification fails or missing.
Schema validation: Enforce the JSON schema keys/types; reject items with missing keys or out-of-range difficulty.
Stage alignment: Ensure examples only permitted knowledge units and prerequisites for stage 8c: apply gentle difficulty increments across stages.
Style stability: Keep the same stepwise format across examples to facilitate pattern extraction by the student model.
Data logs: Record the exact prompts (System/User) and random seeds enable replication.
P CASE STUDIES
To illustrate how our pedagogically inspired synthesis framework operates in practice, we provide five representative synthetic data cases. Each example is instantiated in the unified JSON schema (Appendix OO) and annotated with the adapter flags (Appendix(N) IN.. Beyond presenting the raw cases, also analyze how they reflect methodological design choices: (i) grounding abstract knowledge
32

---

Published as a conference paper at ICLR 2026
in concrete analogies, (ii) stepwise decomposition with verifialle outputs, (iii) explicit control of dif- ficulty and cognitive load, and (iv) curricular progression across introturcry/mmeemdddeeeevvvvaaa stages.
Listing 2: Algebra: solving 2x2 linear system (introductory stage).
2 "module": "Algebra/iinnerr-ppptttoom"
"prereq": ["A:ithmettic/Inteeerrs" "Algebra/Variable-
Manipulation"),
5
"difficulty_tag"" "introductory",
"problem": "You and your friend together have apples, and you have 1 more than your friend. Translate this into equations (x
7, x - y 1) and solve for both x ( your apples) and (your friend's apples) "solution":
"steps":
"Step 1: Translate the story into equations: x + y and
y 1.",
10
"Step 2: Add the two equations to get 2x = 8, thus x 4."
11
"Step 3: Substitute back into x + = 7, thus = 3.",
12
"Step 4: Verify: x - y
1, which is correct."
13
},
"final_answer": "x 4, y 3", "verification": "Both equations are satisfied with the solution."
15
16
17
"adapter_flags": {
"concretization": true, "decomposittion": true, "cognitive_load"; { "scale": "2x2 system", "notes": "Small integer coefficients to reduce difficulty" h "format_tempil "Stepvise-3", "simplified__aanueee"" true
19
22
},
24
"metadata"; "stage_id"" _d"; "Math-S1", "seed staee__tyle_ee"" "Seed- Math-C01"
25
26
This case demonstrates our concretization principle: everyday scenarios (apples) act as a bridge to formal algebra. The solution is broken into atomic steps (translation - elimination - substitution - verification), ensuring decomposability and minimizing working memory burden. The cognitive load flag restricts complexity to 2x2 system with integer coefficients, aligning with the Zone of Proximal Development by keeping the entry an introductory level.
Listing 3: Calculus: derivative at point (intermediate stage).
"module": "eaaccolts//errvvatiieeAAA--oo "prereq": ("Algebraa/uuntioons"" "Limits/Intuition""
5 "difficuultytag": "intermediate", "problem"; "Connect the idea of car's speedometer: { instantaneous velocity) to the derivative of f(x) = 3x"2 2x at x = 4. Provide both symbolic differentiation and numerical approximation check.", "solution":
33

---

Published as a conference paper at ICLR 2026
ssteps":
9
"Step 1: Analogy to formal concept: instantaneous velocity
for derivative at a point.",
10
"Step 2: Compute derivative: fr (x) 6x 2, thus f' (4) =
24 2 =2.""
11
"Step 3: Approximate numerically; with h - 0.001, (f(4th)
/h 22."
12
},
13
saanal_nsswrr; "22", "verification"; "Symbolic result and numerical approximation are consistent."
14
15
16
aedapter_flagaee "concretization"s true, "decomposition": true, "cognitive_ioad"" { "scale": "single derivative", "notes":
18
Quadratic function only, avoids chain rule" }, "formatttemplate"; "Stepwise-3", "simplified lanquugge,_,, true
22
"metadata": "stage_id" "Math-S2", stele_syyle_ref "Seed- Math-012"
25
Here we illustrate bridging: the analogy of speed directly scaffolds the abstract idea of a derivative. The dual representation (symbolic derivative and numerical approximation) provides two-path validation mechanism, strengthening student confidence. The cognitive load is carefully capped (quadratic only, single point evaluation), so that the task constitutes manageable step up from algebra.
Listing Programming: string normalization (introductory stage).
2 "module": o "prereq": ["Programiig//yrtoo///sscl))
"difficulty_tag": "in"":"iitttonn "introductory",
"problem"; "Implement function normalize normalize_spaces// (s) that removes leading/trailing spaces and ensures words are separated by exactly one space.",
7
"solution":
"steps":
9
"Step 1: Strip leading and trailing whitespace.",
"Step 2: Split string into words by whitespace.",
11
"Step 3: Join words with a single space.",
"Step 4: Write minimal unit tests for empty string, single word, and multiple spaces."
14
"final_answer" "def
(s: str) -> str
parts s.split return \" \" jojn (parts) "verification': "assert normalize_ "paceesspces a b () == la
15
b'; assert pormalize_spaces ("')) '1; assert normalize_spaces xx) == 'x'"
16
b,
"adapter_flags": {
18
"concretization": true, "decomposition": true, "cognitive_ioad"; { "scale": "small input size", "notes": No regex advanced constructs" },
20
34

---

Published as a conference paper at ICLR 2026
"format "format_templatee template"; "Stepvise-3", @simplified_Lannuaalle lanquage": true
22
|,
24
"metadata": "stage_id"; "Code-S1"; "seed_style_ref" "Seed- Code-010"
25
This case embodies machine-verifiable decomposition: steps specified as "strip split rejoin," each independently testable. By attaching unit tests inside the JSON, case enforces explicit verifiability, key design for reducing hallucination in distilled student models. Moreover, cognitive load is constrained by avoiding regex or hase-case logic, enabling an accessible intro programming entry.
Listing 5: Instruction Following: polite rescheduling email (intermediate stage).
"module": "
"prereq": ["Instruction/Tone Poliieeeessreeeeeisttteesssss "Instruction/ Constraint-Trakking"
5 "difficulty_tagg: "intermediate",
"problem"; "Write short, polite email in English to reschedule 30-minute 1:1 meeting originally on Wednesday 10:00. Suggest moving it to Thursday or Friday the same time, explain that you have medical appointment conflict, and include an apology and closing.", "solution":
"steps":
"Step 1: Identify required elements: recipient, reason, apology, alternatives, closing.",
10
"Step 2: Draft the email under 120 words with polite tone.
"Step 3: Ensure all checklist items are present."
12
},
"final_answer" "Subject: Request Reschedule Our 1:1Ini nHi <Name>, \nI have à medical appointment that conflicts with our Wednesday 10:00 meeting, Could reschedule our 30-minute 1:1 to Thursday 10:00 or Friday 10:00 this week? Sorry for the inconvenience, and appreciate your flexibility, An \n\nThanks, \n<Sender>", "verification": "Checklist satisfied: (reason, apology, two alternatives, recipient, courteous closing)."
14
},
"adapter_flags": "concretization"s true,
17
"decomposition": true, "cognitive_loadd" { "scale"; "short-form template", "notes";
19
"Fixed structure reduces planning load" }, forrat_template": "Stepwise-3", "simpliified__aaguueee"" true
20
22
}
"metadata"; "stage_id"" id": "IF-S2", sseed_style_ref" "Seed-I7 -023"
]
This illustrates how format templates and constraint- based verification function within our framework. Instead of relying vague stylistic evaluation, the output is judged against structured checklist
35

---

Published as a conference paper at ICLR 2026
(recipient present, apology included, alternatives given, courteous tone). The scaffold reduces ambi- guity and fosters transferable patterns instruction-followee tasks. Difficulty is set to intermediate level: learners must combine multiple constraints but still within short-form template.
Listing 6: Physics: Ohm's law in series circuit (advanced stage).
2
"module": hBppaccc//Leetraccy///ooo-----rr "prereq": ("Physics/Units""
"difficulty_tag"" "advanced",
"problem": "Two resistors R1 = 2 ohms and R2 - 3 ohms are in series with total voltage 1CV. Compute the total current and the voltage drop across each resistor.", "solution":
"steps":
9
"Step 1: Compute total resistance RT R1 + R2 5 chms
10
"Step 2: Apply Ohm's Law: I = V / RT 10 / 5 = 2A.",
11
"Step 3: Voltage drops: V1 = I Ri 4V; V2 = I R2 = 6V
12
"Step 4: Verify: VI V2 = 10V, consistent with supply."
14
"final Frinal_answer'; "Current = 2A; Voltage drops = (V1 4V, V2
6V)
15
"verification"; "Equation balance and units are consistent."
}
17
addater__lags";
18
"concretization" true, "decompositionns true, "cognitive_load"" { "scale"; "two resistors", "notes"; "No
20
parallel or AC cases" 1 "format_template": "Stepwise-3", "simplified_languaee"" true },
21
23
"metadata": "stage_id": "Sci-s3", "seed_style_ref' "Seed- Sci-005 }
26
This highlights incremental difficulty escalation: after algebraic prerequisites and prior exposure to circuit basics, students are introduced to multi-component reasoning. The analogy (water flow) grounds the physics concept, while the verification step enforees dimensional analysis and conserva- tion (V1 + V2 V). This aligns with our curricular progression principle, as it sits at the advanced level but still keeps complexity bounded (two resistors only).
Q DISTINGUISHING INTRINSIC DIFFICULTY, PREREQUISITE FAILURE, AND COGNITIVE LOAD
Equ. 8 in the main paper uses the student performance Ps( signal for controlling the difficulty increments across consecutive learning stages. We clarify that (s(k) is not interpreted as the intrinsic difficulty of knowledge module. Instead, it represents the student-per purrevveerreeived difficully, i.e., how challenging a module for the current student model its present mastery level. This interpretation aligns with Vygotsky's Zone of Proximal Development, where difficulty is defined relative to the learner's current capability rather than as an absolute property of content. By treating Ps (k) as a proxy for developmental appropriateness, the difficulty constraint in Equ. (qu.Benn ensures that the progression of modules remains within the student's effective learning region.
This distinction is important because task difficulty and model performance not equivalent in general: a model may fail either because the module is inherently challenging or because prerequisite knowledge is missing. Our IOA framework already separates these cases. Missing foundational
36

---

Published as a conference paper at ICLR 2026
knowledge is handled by the dependency graph extracted during the Identifier stage and enforeed through the curriculum ordering; a module is never introduced until all of its prerequisites have been mastered (as shown in Equ. (7). Therefore, when Equ. B evaluated, the student has already satisfied prerequisite knowledge, and a low (s(k) at that point reflects excessive cognitive load rather than missing dependencies.
R EXPLANATIONS FOR EXTREME HYPERPARAMETER SETTINGS
To further clarify the behavior of IOA under extreme hyperparameter choices, we provide here additional explanations for the two edge cases: Tiaatery 1.0 and TZPD = 0.
Case 1: Tmasterr = 1.0. Requiring perfect mastery before advancing to the next module is generally infeasible in black-box distillation. Even small stochastic variation or inherent capacity gaps between the teacher and student may prevent the student from ever matching the teacher's performance exactly, causing GenerateRemmeialDaata as also reflected in Figure 3 pushing Tmastery too close to 1.0 yields diminishing performance gains while substantially increasing computational overhead due to additional remedial iterations. This aligns with Bloom's Mastery Learning principle, which advocates high but not absolute mastery thresholds (typically 80-95%). The default range used in IOA (0.550.995) follows this pedagogical recommendation and offers favorable balance between stability and efficiency.
be repeatedly invoked without convergence. Empirically,
Case 2: T2PD = In Equ. S TZpD controls the maximum allowable increase in student-perceiver difficulty between consecutive stages. Setting T2PD 0 does not eliminate difficulty control; instead, enforces a zero-increase constraint, requiring each new module to be no more difficult than the previous one. This results in non-increasing even reversed difficulty progression, directly contradicting Vygotsky's Zone of Proximal Development, which emphasizes gradual upward difficulty growth to maintain productive learning. Consistent with this interpretation, the second subfigure of Figure 3 shows that performance degrades significantly as TZDD - O, reflecting that the student no longer exposed to appropriately challenging modules. Therefore, maintaining TZPD is essential for pedagogically aligned and effective curriculum.
Overall, these observations confirm that extreme settings of Tmastery or T2PD break key pedagogical assumptions underlying IOA and lead to undesirable optimization behavior. This further validates the ranges adopted in our main experiments.
S ANALYSIS OF CATASTROPHIC FORGETTING IN MASTERY-BASED PROGRESSIVE LEARNING
To examine whether Mastery-Based Progressive Learning induces catastrophic forgetting, we monitor the stability of knowledge acquired in carlier curriculum stages. Specifically, we track the ratio Ps(k') Pr(K] for subset of previously mastered modules k' & 8; training progresses into later stages. Empirically, we do not observe systematic degradation. The monitored ratios remain stable or exhibit slight upward trends, indicating that the student's ' competence on earlier modules is preserved rather than overwritten. This suggests that the progression mechanism IOA does not cause the student to forget previously mastered knowledge. A key reason for this stability is the structure of the curriculum itself. Since later modules are sequenced according to prerequisite dependencies, the student naturally reuses carlier knowledges when learning more advanced modules. This repeated utilization acts as an implicit rehearsal mechanism, reinforcing foundational knowledge without the need for an explicit replay buffer. Furthermore, the mastery gating ensures that modules not advanced prematurely. reducing gradient instability that commonly contributes to forgetting. In summary, across all experiments we evaluated, the Mastery- Progressive Learning procedure does not induce catastrophic forgetting, and the student's 's performance on previously mastered modules remains well presereeee throughout training.
T LIMITATION ANALYSIS
While our proposed IOA framework demonstrates consistent gains across instruction-followiii and reasoning benchmarks, several limitations remain.
37

---

Published as a conference paper at ICLR 2026
Evaluation scope. Our experiments focus small- to mid-sized models (e.g., 3B parameters in Qwen and LLaMA families) and a limited benchmark suite. This restricts claims of scalability to larger architectures, long-context reasoning, or safety-critical domains. Future work should broaden both task coverage and model diversity.
Dependence seed data. The identifier relies compact seed (~3,000 items across four domains). Although de-duplication and filtering are applied, the diagnostic ability may still be biased toward these domains and linguistic styles, limiting generalization.
Teacher reliance. IOA assumes access to strong teacher models (e.g., OpenAl ol, DeepSeek-RI I). Mis-calibration systematic biases from these teachers can propagate into the deficiency diagnosis and curriculum design. Reproducibility is also challenged by reliance on proprietary, black-box systems.
Heuristic sensitivity. Key thresholds (c.g., for deficiency, dependency, mastery) are set empirically, While effective in our setting, they may need re-tuning for new domains or students, and could introduce brittleness when score distributions shift.
synthesis, and iterative -eriiin com- putationally intensive, requiring multi-GPU clusters. This may hinder adoption by smaller labs, motivating work on lighter-weight teachers or more efficient curriculum construction.
Compute overhead. The full
U DECLARATION OF LLM USAGE
During the preparation of this manuscript, large language models were employed exclusively as auxiliary tools for language-related purposes. Specifically, LLMs were used to (1) refine the clarity and fluency of sentences; (2) polish grammar and style for improved readability; and (3) suggest alternative phrasings without altering the original technical content. No parts of the research design, data collection, algorithm development, experiment execution, or result analysis relied on LLMs. All conceptualization of the methodology. implementation details, and interpretation of results were conducted independently by the authors. The authors confirm that L.M assistance did not contribute the generation of novel scientific ideas, data, claims, and that the core intellectual contributions of this work remain solely those of the authors.
38


Reviews:
[{"review_id": "ayNOFEawQ1", "reviewer": "Reviewer_EkXd", "summary": "The research introduces IOA (Identifier-Organizer-Adapter), which is a pedagogically inspired framework for knowledge distillation from large language models to smaller models. This framework brings educational theories (Bloom's Mastery Learning and Vygotsky's Zone of Proximal Development) to life in the form of a three-stage pipeline that helps to systematically identify knowledge gaps, organize curriculum progression, and adapt knowledge representations. The results obtained from several benchmarks clearly show the advantages of the proposed method over baseline distillation methods.", "questions": "1. What happens if the teacher LLM produces an inconsistent or incomplete knowledge hierarchy? How sensitive is IOA to decomposition quality?\n\n2. How does the framework handle cyclic knowledge dependencies in the graph if they ever happen?\n\n3. What happens with τ_mastery = 1.0 (perfect mastery required) or τ_ZPD = 0 (no difficulty control)?\n\n4. How does IOA perform when the teacher is only moderately stronger than the student (e.g., 7B teacher → 3B student)?\n\n5. What's the gain from the topological+ZPD curriculum over a simple easy-to-hard ordering by task difficulty or skill-based decomposition?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Novel pedagogical concepts that map educational principles (Bloom's Mastery Learning, Vygotsky's ZPD) to concrete distillation mechanisms that distinguish this work from other ad-hoc data synthesis approaches.\n\n2. Comprehensive experimental validation with extensive benchmarks via testing with multiple teacher and student models.\n\n3. Thorough ablation studies and appendices demonstrating each component contributes meaningfully with extended experiments like hyperparameter robustness, additional student models, and agentic tasks, which greatly support reproducibility.\n\n4. Honest limitations section acknowledging theoretical gaps and practical constraints", "weaknesses": "1. While the pedagogical inspiration is appealing, the paper lacks proper justification for why these specific educational principles should transfer to neural network learning, as humans learn through sparse, interactive experiences with semantic understanding while LLMs optimize loss surfaces through gradient descent over massive corpora. \n\n2. Critical hyperparameters (τ_gap=0.3, τ_high=0.9, τ_low=0.7, τ_dep=0.3, α=0.7, τ_ZPD=0.15, τ_mastery=0.9) appear empirically tuned rather than principled. The paper states these are \"empirically set\" but provides limited ablation on sensitivity. Figure 3 shows some robustness, but some clarifications are needed for justification. (for example, why is τ_mastery=0.9 optimal rather than 0.85 or 0.95?)\n\n3. No proof for the mastery-based progression converges in algorithm 1 (lines 11-13). The worst-case number of remedial iterations is not mentioned anywhere in the paper.\n\n4. The difficulty constraint (Equation 8) assumes \"P_S(k) represents the average difficulty level\" but performance ≠ difficulty. A model might also fail due to missing knowledge, not cognitive overload.\n\n5. A minor mistake in figure 1 caption: \"Anology\" → \"Analogy\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HStvbv40kF", "reviewer": "Reviewer_GGqR", "summary": "The paper introduces a pedagogically-inspired framework for knowledge distillation in LLMs. Unlike prior methods, which treat knowledge transfer as a straightforward supervised fine-tuning (SFT) procedure, this work conceptualizes distillation as a systematic, learning-aware process, guided by established educational principles. The proposed framework, IOA (Identifier–Organizer–Adapter), operates in three stages: it first diagnoses knowledge deficiencies in the student model, then sequences knowledge delivery into a progressive curriculum, ensuring mastery of foundational concepts before introducing more complex material, and finally adapts and simplifies knowledge representations to align with the capacity of the student.", "questions": "See weaknesses section", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The work introduces a pedagogy-inspired perspective to knowledge distillation, framing it as a systematic learning process rather than a straightforward supervised fine-tuning task on synthetic data generated by the teacher LLM. This represents a novel conceptual contribution.\n\n- Extensive experiments across instruction-following, reasoning, and coding benchmarks demonstrate substantial performance gains over baseline distillation methods, highlighting the effectiveness of the proposed pipeline. The ablation studies further show each component’s necessity.\n\n- The paper is well-written and clearly structured.", "weaknesses": "- The framework relies on multiple heuristics (e.g., thresholds for knowledge gaps, mastery gating, module decomposition, curriculum chunking). While some sensitivity analyses are provided, it remains unclear how well these heuristics generalize across tasks or domains, potentially requiring careful manual tuning.\n\n- Although the paper evaluates against several strong synthetic-data baselines, it does not include comparisons with the most recent knowledge distillation methods such as ABKD, DistillM‑2, or SuperCorrect.\n\n- Certain references (e.g., lines 54, 352, 952) are missing the publication year.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "15ZN5mQ1mJ", "reviewer": "Reviewer_wP76", "summary": "The paper proposes a method for creating synthetic data for knowledge distillation from large to small language models in a black-box setting. The authors propose a pedagogically-inspired framework from LLM knowledge distillation, which consists of knowledge identification, organization, and adaptation. It identifies knowledge deficiencies in the student, organizes the knowledge using a curriculum, and generates and adapts the training data so that the student can better internalize it. The experiments show that IOA outperforms baseline synthetic-data-based distillation methods.", "questions": "- Section 3.2 lacks implementation details, particularly related to the probe task creation. How do you assign validation examples from D_{seed} to knowledge modules k? The paper states that the probe tasks P_k come from the validation split, but doesn't explain whether this mapping is manual, fully automatic, or something in between. Then, is the hierarchy D generated end-to-end for each training run separately? If so, how is the validation example mapping done? Finally, it appears that the prompts corresponding to the generation of the hierarchy D are missing\n- The ablations in Table 3 could be explained in more detail. What do -identifier, -organizer, and -adapter mean? For instance, how does the organizer module work if you do not have identified knowledge modules? In addition, if the authors have sufficient computational resources, I would also like to see an ablation without GenerateRemedialData (if none of the ablations in Table 3 already correspond to this).\n- How do you estimate the values of the conditionals in Equation 3? It seems like you would need multiple checkpoints with PS(k_i)/PT(k_i) above 0.9 and below 0.7 for the conditionals to make sense, and for that, you'd need many training runs. Are these included in the wall-clock time estimations of Figure 4? More generally, which steps of the pipeline are included in Figure 4?\n- The method is used on a well-curated D_{seed}. Do you have any intuition whether the performance would translate to a random public sample or a larger D_{seed}, for which human curation is unscalable?\n- An on-policy logit-distillation baseline with a smaller teacher (e.g., from the DeepSeek-R1-Distill family for matched tokenizers) would strengthen the case for the paper in stating that black-box distillation can be used as a general alternative for white-box (logit-based) distillation. Furthermore, on-policy distillation and IOA could perhaps be combined for even greater performance.\n\nOverall, the discrepancy between the code and what is written in the paper should be clarified before this paper is ready for acceptance.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 1, "presentation": 2, "contribution": 2, "strengths": "- Strong results in the self-instruct-like synthetic data generation setting, outperforming other synthesis-based distillation approaches.\n- Clear, theoretically grounded motivation with a connection to how humans learn, which makes a lot of intuitive sense.\n- Extensive experiments with multiple benchmarks, student models, and a particular focus on statistical significance. The method provides consistent improvements across diverse tasks (instruction following, math, code).\n- The \\tau_{ZPD} method peaking at around 0.10-0.15 is an interesting finding, showing that the training data should neither be too difficult nor too easy for the student models for optimal distillation results.", "weaknesses": "- The released code appears to be non-functional and doesn't implement the described pipeline. For instance, the function decompose_knowledge returns hard-coded modules instead of deriving from the data. Key components are mocked. This doesn't correspond to what is written in the paper, and the paper cannot be reproduced.\n- Some methodological details could be better described in the paper (see my questions below).\n- IOA depends on many hand-selected/tuned hyperparameters, which might affect generalizability. The method doesn't seem overly sensitive to the ablated hyperparameters (J_i, \\tau_{ZPD}, \\tau_{mastery}), but some other hyperparameters like the threshold m for the selection of knowledge modules, \\alpha for severity, and \\tau_{high}, \\tau_{low}, and \\tau_{gap} are underanalyzed. \n- IOA consistently outperforms synthesis-based baselines in controlled settings, but there's a significant performance gap to the state-of-the-art distilled models (IOA: 6.3 % on AIME2024, DeepSeek-R1-Distill-Qwen-!.5B: 28.99 % on AIME2024). DeepSeek-R1 distillation likely uses orders of magnitude more training prompts than IOA's seed set, making the numbers incomparable. However, Figure 3 (left) indicates that IOA gets limited benefits from increasing the number of synthetic samples J_i. Hence, a question remains: would IOA maintain its relative advantage if scaled to a much larger seed set, which appears necessary for results close to SOTA? A compute-matched baseline with a larger open-source dataset (e.g., OpenThoughts [1]) that is filtered/sampled to match IOA's total compute budget (e.g., 30k-50k prompts) and trained with standard SFT would indicate how much value the curriculum adds.\n- The problem framing in the intro can be questioned. The intro states that \"more recent distillation methods in the LLM era discard the need for white-box access.\" This recency-based characterization is questionable. White-box distillation approaches are an active research direction and are used in practice (see e.g., DistiLLM-2 [2] and Qwen3 technical report [3]). The intro states that \"most merely treat distillation as a simple two-stage process: generating synthetic data with LLMs and then training SLMs on it\". Among the benchmark methods in Tables 1 and 2, at least Lion, Star-Agents, and MADA are not simple two-stage methods. Hence, claiming that \"previous works overlook the fact that KD should be a systematic learning process\" can be questioned. Furthermore, the work could more clearly acknowledge the line of research on on-policy distillation [4], which is built on the idea of the teacher correcting the mistakes generated by the student. Also, the Orca paper that argued that SLMs shouldn't simply learn to replicate the outputs of LLMs ([5]) is closely related to this work (especially the idea that the knowledge representations should be adapted to the learner's cognitive level), and you could consider citing it. Hence, the paper's novelty is mostly the overall combination, not necessarily the individual components.\n\n[1] Guha, E., Marten, R., Keh, S., Raoof, N., Smyrnis, G., Bansal, H., ... & Schmidt, L. (2025). OpenThoughts: Data Recipes for Reasoning Models.\n\n[2] Ko, J., Chen, T., Kim, S., Ding, T., Liang, L., Zharkov, I., & Yun, S. Y. (2025). Distillm-2: A contrastive approach boosts the distillation of llms. ICML\n\n[3] Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., ... & Qiu, Z. (2025). Qwen3 technical report. arXiv preprint arXiv:2505.09388.\n\n[4] Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., & Bachem, O. (2024). On-policy distillation of language models: Learning from self-generated mistakes. ICLR\n\n[5] Mitra, A., Del Corro, L., Mahajan, S., Codas, A., Simoes, C., Agarwal, S., ... & Awadallah, A. (2023). Orca 2: Teaching small language models how to reason. arXiv preprint arXiv:2311.11045", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dWX6O7fSpF", "reviewer": "Reviewer_HaWf", "summary": "The paper proposes to distill knowledge from close-sourced LLMs to small language models in a three-stage data synthesis pipeline with Knowledge Identifier, Organizer, and Adapter, inspired by pedagogical principles. Specifically, the Knowledge Identifier locates knowledge deficiencies of student models, which the Organizer leverages to schedule a curriculum in which the synthesized data is partitioned for progressive learning. The Organizer is inspired by Bloom’s Mastery Learning Principles and Vygotsky’s Zone of Proximal Development. Following the two components, a knowledge Adapter is proposed to rewrite the training data to match the cognitive constraints of student models. Experiments conducted with both LLaMA and Qwen series models as student models show consistent improvements of the proposed framework over baseline distillation methods.", "questions": "1. Line 195: Any analysis on the value of $\\Delta(k)$ over datasets? Why 0.3 is a good threshold?\n\n2. What's the intuition behind using $P_T$ in eq.3? As eq.3 is just a measurement of the implicit dependency between two knowledge modules in the student model, would it be more memory efficient if we drop $P_T$?\n\n3. What's the Prerequisites function in eq.7? How do you define the prerequisite knowledge?\n\n4. How often is the $\\tau_\\text{mastery}$ in eq.9 sufficed without generating remedial data? I think figuring out this is significant to know the additional computation led by the Mastery-Based Progressive Learning.\n\n5. Do you observe catastrophic forgetting in the Mastery-Based Progressive Learning process? To be more specific, do you have studies on $\\frac{P_S(k’)}{P_T(k’)}, k’ \\not\\in s_i $?\n\n6. Line 267-291: How to measure the  SLMs’ cognitive constraints in ADAPTER. Do you assume the cognitive ability of SLM increases during training or remains static?\n\n7. Figure 3 shows that it is consistently better when you use $\\tau_{\\text{mastery}}=0.95$ instead of your default setting 0.9. Why not set 0.95 as your default setting? Or how much more computation is incurred by increasing $\\tau_{\\text{mastery}}$ from 0.9 to 0.95 becauseof recursively generating remedial data? This question also relates to question 4.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The proposal of the knowledge identifier is novel and sound. The definition of the knowledge deficiency can be used beyond knowledge distillation in analyzing the capabilities of language models.\n\n2. A good leverage of Bloom’s Mastery Learning Principles to construct curriculum data that incentivize progressive learning of the small models.\n\n3. The paper is structured and written in a clear way.", "weaknesses": "1. It seems to me that the paper lacks in comparison to previous work on improving knowledge distillation with curriculum learning, either in the related work section or the experiments. As the paper proposes this 3-stage knowledge distillation pipeline, which is closely relevant to curriculum learning, it would be considered incomplete by me if there is no comparison to works in curriculum learning.\n2. The difference between Table 1 and Table 2 is whether using OpenAI o1 or DeepSeek R1 as the teacher model. I feel the information in these two tables overlaps. As Table 2 also takes large space, it would be better if we could incorporate Table 1 and 2 into one table and include other exciting analysis results from other dimensions into the main text.\n3. The relationship between related works in  \"Language Model Distillation\" to this work\n4. Typo, line80: stag- -> stage", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)