Title: Learning to Solve Complex Problems via Dataset Decomposition
Venue: NeurIPS 2025
Accepted: yes
Keywords: language models

decomposition

math reasoning
Abstract: Curriculum learning is a class of training strategies that organizes the data being exposed to a model by difficulty, gradually from simpler to more complex examples. 
This research explores a reverse curriculum generation approach that recursively decomposes complex datasets into simpler, more learnable components.
We propose a teacher-student framework where the teacher is equipped with the ability to reason step-by-step, which is used to recursively generate easier versions of examples, enabling the student model to progressively master difficult tasks. We propose a novel scoring system to measure data difficulty based on its structural complexity and conceptual depth, allowing curriculum construction over decomposed data. 
Experiments on math datasets (MATH and AIME) and code generation datasets demonstrate that models trained with curricula generated by our approach exhibit superior performance compared to standard training on original datasets.

Learning to Solve Complex Problems via Dataset Decomposition

Abstract

Curriculum learning is a class of training strategies that organizes the data being exposed to a model by difficulty, gradually from simpler to more complex examples. This research explores a reverse curriculum generation approach that recursively decomposes complex datasets into simpler, more learnable components. We propose a teacher-student framework where the teacher is equipped with the ability to reason step-by-step, which is used to recursively generate easier versions of examples, enabling the student model to progressively master difficult tasks. We propose a novel scoring system to measure data difficulty based on its structural complexity and conceptual depth, allowing curriculum construction over decomposed data. Experiments on math datasets (MATH and AIME) and code generation datasets demonstrate that models trained with curricula generated by our approach exhibit superior performance compared to standard training on original datasets.

1 Introduction

When teaching language models (LMs) to solve mathematical problems, one common solution is to apply supervised learning on a dataset of problems with worked-out solution steps. This is, to some extent, in stark contrast to how humans are educated and generally taught to solve problems, e.g. with a curriculum , where a teacher presents simpler concepts first, to build a foundation for later tackling more challenging tasks.

We build upon this intuition and, assuming access to a teacher model that can reason step-by-step, we propose a way of 'decomposing' a dataset of mathematical problems into a hierarchy of problems with different difficulty levels. Our approach, DECOMPX, enables smaller LMs to progressively acquire elemental skills before tackling more intricate ones. Although we focus on mathematical datasets such as MATH [Hendrycks et al., 2021] and AIME [of America, 2024], as well as code generation datasets such as [Chen et al., 2021], we think the approach might be applied to a broader range of domains in the future.

Transposing this strategy to train machine learning models, e.g. via curriculum learning [Elman, 1993, Bengio et al., 2009], has historically yielded mixed results. We speculate that one of the reasons is that approaches largely relied on ranking by difficulty examples within a dataset [Wu et al., 2020, Lalor and Yu, 2020]. Not only is accurately estimating difficulty challenging, but datasets may also lack the diversity or granularity needed to isolate and teach the fundamental 'skills' required for complex problem-solving. Recent studies suggest that curriculum learning is more effective when 'decomposed' datasets that isolate atomic skills are available to the learner [Lee et al., 2024].

Our dataset decomposition approach starts with a dataset of mathematical questions and step-by-step solutions. The main idea is that a teacher model can generate simpler problems by looking at sub-steps in the solution. Every sub-step is by definition simpler than the original problem and thus can be used as the answer for a latent, simpler problem. Crucially, this process can be recursively iterated: we can now ask a teacher model to create a step-by-step solution to the sub-problem and recursively generate simpler problems in the same manner until a stopping criterion - such as recursion depth or problem simplicity - is met. Given the strong inductive bias of step-by-step reasoning, the teacher is encouraged to work out solutions to progressively simpler problems. This process creates a tree of sub-problems for each question in the original dataset.

Weuseour dataset decomposition to train small LMs to do mathematical reasoning and code generation via supervised fine-tuning (SFT). Even when decomposed problems are presented i.i.d. to the models, our approach improves the ability of small LMs to learn from a small set of examples. We see further gains when sub-problems are presented in the order of increasing complexity.

To facilitate curriculum construction, we assign difficulty scores to sub-problems based on their structural complexity. We tag sub-problems and construct a graph of tags by connecting each subproblem's tag to its parent problem's tag. The graph synthesizes tag relationships across the entire dataset. The simplicity of a problem is then computed both using the "depth" of the tags associated with a given problem and the number of sub-problems it generates.

Apart from the dataset augmentation and curriculum construction aspect, one of the useful byproducts of our method is the potential for creating a 'cartography' of a given dataset [Swayamdipta et al., 2020]. Our induced graph of skills can be used to gain more insights into coverage of the dataset and how additional samples might increase the coverage, or even how two seemingly different datasets are related to each other.

2 Related Work

Data Augmentation Data Augmentation, synthetically generated from an LLM can be seen as a form of distillation [Hinton et al., 2015, Kim and Rush, 2016]. This approach has been especially successful in distilling reasoning capabilities into smaller models [Mitra et al., 2023, Li et al., 2023, Mitra et al., 2024]. This approach has also been applied to generate mathematical reasoning traces. For instance, Self-Taught Reasoner [Zelikman et al., 2022] generates and then learns from its own chain-of-thought reasoning steps. MuggleMath [Li et al., 2024] and MetaMath [Yu et al., 2024b] both amplify diversity by evolving problem queries and sampling multiple reasoning traces, and by applying paraphrases and backward reasoning transformation, respectively. MuMath [Yin et al., 2024] further enriches examples through multiple rewrites using several 'perspectives', e.g., paraphrase, symbolic reformulation. MathGenie [Lu et al., 2024] back-translates a small seed set through a generator-verifier loop to create high-fidelity new problems. PersonaMathQA [Luo et al., 2024] adds 'persona diversification' plus self-reflection to generate richer problem contexts. Beyond direct data augmentation, Shridhar et al. [2023] generates subquestions and solutions to distill the knowledge from the teacher model to students, Huang et al. [2025] synthesizes QA pairs by extracting key points from problems. To simultaneously address the quality, diversity, and complexity of the dataset, Davidson et al. [2025] proposes Simula, a unified framework for generating and evaluating synthetic data. However, existing data augmentation techniques somewhat neglect prior knowledge and difficulty. To address this gap, we propose hierarchically decomposing problems with multi-step augmentation so models progressively acquire elemental skills.

Curriculum Learning Bengio et al. [2009], Elman [1993] originally propose to apply curriculum to train LMs. Wu et al. [2020] shows mixed results when applying curriculum learning methods based on example difficulty. In the past, many have studied different curriculum learning strategies to either increase the context length or more efficiently train LMs [Press et al., 2020, Nagatsuka et al., 2021], with unconvincing results. More recently, studies started to investigate synthetic data creation associated with curriculum learning, yielding promising results for training small LMs for code [Naïr et al., 2024]. Work on TinyStories [Eldan and Li, 2023] shows that carefully synthetically curated datasets can be helpful in teaching tiny LMs basic English proficiencies. These works align with ours in the hypothesis that the synthetic data creation of easier examples can be used as a driving factor underpinning a successful curriculum. In the robotics domain, [Florensa et al., 2017] proposes a 'reverse' learning strategy that starts RL training from the goal state and gradually guides the policy

𝑞

𝑐𝑜𝑡

𝑎

𝑆𝑡𝑒𝑝1

𝑆𝑡𝑒𝑝2

𝑆𝑡𝑒𝑝3

𝑆𝑡𝑒𝑝4

𝑞1

𝑐𝑜𝑡1

𝑎1

split

split

𝑆𝑡𝑒𝑝11

𝑆𝑡𝑒𝑝12

𝑆𝑡𝑒𝑝13

verify

depth = 1

depth = 2

create

question + tag

create

question + tag

𝑡𝑎𝑔

𝑡𝑎𝑔1

𝑡𝑎𝑔11

solve

𝑐𝑜𝑡11

Graph of tags and

associated problems

𝑡𝑎𝑔1

𝑡𝑎𝑔11

𝑡𝑎𝑔

to learn to reach the goal from a set of start states increasingly far from the goal. This approach also builds a curriculum with increasing difficulty levels (in a guaranteed way) by reversing the original task. However, our reasoning domain is different because it allows us to leverage the compositionality of language to decompose the reasoning tasks at much meaningful abstraction levels.

3 DECOMPX: Dataset Decomposition

In this section, we propose a recursive algorithm, DECOMPX, that decomposes complex math problems into simpler subproblems based on their underlying reasoning steps. Our algorithm consists in two phases: first, we decompose each example into a hierarchical structure of sub-problems; second, we connect these sub-problems at a dataset level using a tagging approach and we build a graph of tags, useful to infer the difficulty of every sub-problem. Finally, we describe how we use the difficulty score in our curriculum learning procedure.

3.1 How to generate subproblems? - Recursive Problem Decomposition

We propose a recursive dataset decomposition framework that constructs verified, grounded subproblems from multi-step solutions. Each sub-problem corresponds to a clear, atomic mathematical reasoning operation, explicitly linked to a core mathematical concept and validated for correctness. We assume access to a teacher model  (a large LM, we use GPT-4o in our experiments), which we assume is proficient at the task of the dataset of interest.

Concept Tagging. For each step 𝑠 𝑖 , we extract an atomic concept tag 𝑡 𝑖 by querying the teacher model to identify the most specific mathematical concept that governs the reasoning step.

Step Extraction. Given a solution trace 𝑐𝑜𝑡 for a math problem 𝑞 , we first decompose it into at most 𝑘 reasoning steps. This is achieved by prompting the teacher model to segment the text based on conceptual granularity: 𝑐𝑜𝑡 ↦ [ 𝑠 1 , 𝑠 2 , … , 𝑠 𝑘 ] , where each step 𝑠 𝑖 introduces a new and distinct mathematical operation.

Subproblem Generation. Given the original problem 𝑞 , a step 𝑠 𝑖 , and its tag 𝑡 𝑖 , we generate a new subproblem 𝑞 𝑖 (i.e. a question) grounded in the original context ( 𝑡 𝑖 , 𝑞 ) . Then, we ask the teacher model to solve the generated problem 𝑞 𝑖 step-by-step leading to a solution 𝑐𝑜𝑡 𝑖 and final answer 𝑎 𝑖 .

Verification. To assess that the answer to the generated sub-problem 𝑞 𝑖 is correct and answerable, we also ask the teacher model to solve 𝑞 𝑖 without access to the original context ( 𝑡 𝑖 , 𝑞 ) resulting in an answer ̂ 𝑎 𝑖 . Given that 𝑎 𝑖 has been generated with access to the original context, it is more likely to be correct. Therefore, we compare the resulting numerical answer ̂ 𝑎 𝑖 to 𝑎 𝑖 using a symbolic verifier  . Only subproblems satisfying  ( 𝑎 𝑖 , ̂ 𝑎 𝑖 ) = True are accepted. Otherwise, the subproblem is regenerated with a retry budget of 𝑅 attempts.

Solve for x in the equation: x2-4x+GCD(16,12)=0

Greatest

First note that gcd(16,12)=4.

Common

Greatest

The equation becomes x²-4x+4=0,

which is the perfect square.

Perfect

Square

with multiplicity two.

x-2=0

Hence x=2

Remainder

Theorem

Chinese

Common

Divisor

Divisor

gcd(16,12)=?

if(x²-4x+a) is a perfect square,

find the value of a.

if a²=0, find a

Solve for x, x+4=0

Factor

Prime

Exponentiation

Square

Subtraction

16=2^4,

2^2=4

a=4/2=2

Root

Modular

Multipli

Factor

Prime

12=2^2x3

Exponentiation

Division

Arithmetic

cation

Find prime factors of 6

3^2=？

6/3=?

Perfect

Addition

Square

6=2*3

Multipli

cation

3*3=9

业

Multipli

cation

2*3=?

4*3=?

InferredGraphofConceptTags

Decompositionofanexample

Recursive Expansion. The sub-problem reasoning 𝑐𝑜𝑡 𝑖 contains further multi-step reasoning by construction. Therefore, we can apply the process recursively by decomposing 𝑐𝑜𝑡 𝑖 using the same procedure up to a maximum depth 𝐷 . This produces, for every example in the dataset, a nested structure:

where each child 𝑞 𝑖,𝑗 corresponds to a problem grounded in the constituent sub-step 𝑗 of 𝑐𝑜𝑡 𝑖 .

In summary, the final output is a dataset composed of sub-problems, associated concept tags, subproblem depth, reasoning steps and answers (see Figure 1, left). We also illustrate the whole workflow with concrete problems in Figure 2.

Algorithm 1 Recursive Dataset Decomposition

3.2 How difficult is a certain (sub)problem? - Concept Dependency Graph

We construct a directed acyclic graph (DAG) over concepts derived from the recursive decomposed problem-solving process above. This graph encodes the prerequisite relationships between mathematical operations across all the examples in the dataset, thus enabling curriculum design.

The construction algorithm proceeds as follows. We initialize an empty directed graph  . We add the ensemble of tags across examples to the node set  . A parent tag 𝑡 𝑝 has a child tag 𝑡 𝑐 if they are obtained while decomposing the same example and their depth differ by 1. For every parent tag 𝑡 𝑝 and child tag 𝑡 𝑐 , we insert an edge ( 𝑡 𝑝 , 𝑡 𝑐 ) ∈  , unless 𝑡 𝑝 = 𝑡 𝑐 .

Let  = (  ,  ) denote the Concept Dependency Graph , where  is a set of concept tags (such as 'GCD' or 'Square Root'), and  ⊆  ×  is a set of directed edges representing prerequisite relationships between concepts. A directed edge ( 𝑢, 𝑣 ) ∈  indicates that concept 𝑣 depends on concept 𝑢 .

Nodes with zero in-degree in  represent foundational concepts. Formally, the set of root nodes is defined as  = { 𝑣 ∈  ∣ in-degree ( 𝑣 ) = 0} . To quantify concept difficulty, we define a depth function 𝑑 ∶  → ℕ such that:

This depth serves as a proxy for the reasoning complexity required to apply concept 𝑣 .

3.2.1 Concept Clustering via Embedding Similarity

Our generated tags include variations such as "GCD" and "Greatest Common Divisor" that refer to the same mathematical concept. To unify these concepts and minimize semantic redundancy in our graph, we use unsupervised clustering on tags' embedding representations.

To cluster tags, we employ a greedy clustering algorithm with a predefined similarity threshold 𝛿 . Specifically, we sequentially select unassigned tags as cluster representatives and group all other tags exceeding the similarity threshold under them. Formally, the mapping from original tags to cluster representatives 𝜏 is concisely defined as:

Specifically, each concept tag 𝑡 ∈  is mapped to a dense vector representation 𝜙 ( 𝑡 ) ∈ ℝ 𝑑 using a pre-trained LM. We then compute pairwise cosine similarities between all tag embeddings. Denote, the similarity as 𝑆 ( 𝑖, 𝑗 ) .

where  rep ⊆  represents the set of selected representative tags.

After clustering, we relabel the nodes in the concept dependency graph  accordingly. Edges are rewired based on these updated labels, explicitly removing any self-loops arising from clustering:

This results in a refined concept dependency graph  ′ = (  ′ ,  ′ ) , where each node uniquely represents a distinct conceptual skill.

3.2.2 Difficulty Measurement via Structural and Conceptual Features

The difficulty of a data sample stems from both the number of mathematical operations it includes and the complexity of the concepts involved. For example, consider solving the following three problems: a) 1+1 ; b) 1+1+...+1 ; (sum of thousands of 1); c) sqrt(3) + (5/76)**2 . c) is harder than P1 due to conceptual complexity (the complexity of operations involved) while b) is harder than a) because of structural complexity, the number of atomic operations involved.

While the Concept Dependency Graph provides a data-driven way to gauge a concept's complexity by measuring the depth of the concept's tag in the graph, it may fall short in accounting for the structural

complexity of a specific reasoning step that involves that concept, such as the number of sub-problems it can be decomposed into, or how important the sub-problem is to solve other problems in the dataset. To address this, we introduce a composite difficulty score that combines both conceptual and structural factors for a more comprehensive characterization.

Given a reasoning step 𝑠 (or a data sample 𝑞 before the first iteration of decomposition) in the recursive decomposition tree (e.g., step 1 in Figure 1), we compute:

structural complexity SC ( 𝑠 ) : the number of direct children of the reasoning step, reflecting its structural branching factor. In the example in Figure 1, SC ( step 1 ) = 3 because step 1 can be further decomposed into three lower-level reasoning steps (i.e., step 11 , step 12 , and step 13 ).

conceptual depth CD ( 𝑠 ) : the maximum depth of the concept tag in the Concept Dependency Graph  that is associated with this reasoning step. In the example in Figure 1, CD ( step 1 ) equals the depth of tag 1 in  .

Therefore, the overall difficulty score 𝓁 ( 𝑠 ) ∈ ℝ + is defined as a weighted combination of these two terms:

where 𝛼 1 , 𝛼 2 ∈ ℝ + are tunable coefficients that balance structural complexity and conceptual depth.

3.3 Curriculum Learning via Difficulty Measurement

To leverage the difficulty scores 𝓁 during training, we implement a staged curriculum learning framework where the model is exposed to data in increasing order of difficulty. This approach enables the model to first acquire capabilities on simpler sub-problems before attempting harder examples.

Let  ′ = { 𝑞, 𝑐𝑜𝑡, 𝓁 ( 𝑞 )} be the decomposed dataset where the question in each sample is annotated with our difficulty score. We partition  ′ into 𝐾 non-overlapping subsets  ′ 1 , … ,  ′ 𝐾 based on the quantiles of difficulty:

where { qb 0 , qb 1 , … , qb 𝐾 } are the quantile breakpoints computed from the score distribution. We adopt an easy-to-hard curriculum, where the model is trained sequentially from  ′ 1 to  ′ 𝐾 . Each stage is trained with early stopping to avoid overfitting and to allow controlled progression:

where 𝜃 𝑖 denotes model parameters after stage 𝑖 . Wesplit the training budget across the full curriculum, so that experiments presented in this paper are compute matched for a given setting.

4 Experiments and Results

In this section, we describe our evaluation procedure used to validate the effectiveness of DECOMPX.

4.1 Experimental Setup

Models We adopt the Qwen2.5-1.5B [Qwen et al., 2024] and Qwen3-4B-Base [Qwen et al., 2025] models as our student models. Our dataset decomposition is driven by GPT-4o and o4-mini for mathematical reasoning and code generation, respectively, which we leverage as our teacher models.

Datasets Our setup uses MATH [Hendrycks et al., 2021] and the American Invitational Mathematics Examination (AIME). MATH [Hendrycks et al., 2021] is a benchmark of competition math problems of varying difficulty. We evaluate on the same 500 samples in the prior work [Lightman et al., 2023]. AIME contains challenging mathematical competition problems. For training, we use AIME '24 as training set and AIME '25 as test set. Both datasets contain 30 problems that were used in the AIME in 2024 and 2025, respectively. More details can be found in Appendix A.

To further demonstrate the generalization capability and versatility of our method, we conduct additional experiments in a non-mathematical domain that demands complex reasoning: coding. Specifically, we use the CodeForces-CoTs dataset [Penedo et al., 2025] (competitive programming solutions in C++) from HuggingFace Open-R1 [Hugging Face, 2025] for training. This dataset contains solutions generated by DeepSeek-R1 [DeepSeek-AI et al., 2024], where long and complex reasoning traces are common. For evaluation, we employ the HumanEval benchmark [Chen et al., 2021] (Python function completion), which serves as an explicitly out-of-distribution scenario.

Training Details Follow the fine-tuning setup in the previous work [Muennighoff et al., 2025] we train each model for 5 epochs with a batch size of 16. We train the models using bfloat16 precision with a learning rate of 10 -5 , warmed up linearly for 5% and then decayed to 0 over the rest of the training, following a cosine schedule. We use the AdamW optimizer [Loshchilov and Hutter, 2019]. Unless otherwise specified, we evaluate with a temperature of 0 (greedy decoding) and measure accuracy (equivalent to pass@1). The results are averaged over three different training seeds. Our experiments are conducted on NVIDIA A100 GPUs with 80GB VRAM.

Baselines We compare our dataset decomposition and curriculum learning method with a set of baseline systems: (1) closed-weights models such as GPT-4o; (2) open-weights models; (3) Supervised Fine-Tuning (SFT), which uses the training set of the original datasets; (4) direct distillation from the same teacher model and the same original datasets. (4) Data augmentation methods including MetaMath [Yu et al., 2024a] and MuggleMath [Li et al., 2024]: We applied MetaMath's augmentation pipeline for mathematical datasets by rephrasing questions as well as generating answers in four augmentation types (rephrasing, self-verification, answer-augmentation and backward reasoning).

4.2 Main Results

We present our main results, evaluated across different benchmarks, and compared with baseline approaches. We summarize the findings below.

DECOMPX improves performance across different benchmarks. We start our analysis by investigating performance on smaller LLMs. We see that across two different base models and datasets, DECOMPX shows consistent gains over standard baselines; Table 1 presents results using Qwen2.51.5B finetuned and evaluated on MATH. We note that it achieves a 2.4% improvement over SFT and a 13.6% relative gain over SFT-MetaMATH-Aug. It also demonstrates the advantages of DECOMPX over direct distillation. Table 2 reports test accuracy on AIME2025 obtained from fine-tuning Qwen3-4B-Base on the AIME2024 data. Again, DECOMPX performs well, showing improvements of 10% over SFT and 8.9% over SFT-MetaMATH-Aug. It even outperforms Qwen2.5-72B-Instruct, a significantly larger model, using only 385 training samples. Overall, these results highlight the effectiveness of our method and validate the benefit of structured decomposition and curriculum learning in mathematical reasoning tasks. Finally, we note that DECOMPX is better than MetaMATH on both of the benchmarks, which is used for generating augmented data for finetuning. MuggleMath also underperforms our approach despite leveraging substantially more training data and richer prior knowledge from the seed datasets (MATH and GSM8K [Cobbe et al., 2021]). These results further support the advantage of our structured, decomposition-based curriculum. Last but not least, Table 3 demonstrates the advantage of our methods compared to the baselines on the code generation tasks.

Performance reduced during post-training for traditional SFT. Surprisingly, we find that SFT on mathematical datasets may reduce the performance compared to the base model in our experiments. This behavior is especially visible on AIME (Table 2), suggesting that the model may be prone to overfitting given the small dataset size.

DECOMPX shows better generalisation. S1 [Muennighoff et al., 2025] is a reasoning model obtained via supervised finetuning based on Qwen2.5-32B-Instruct using 1,000 samples. Although this sample-efficient baseline achieves strong performance on MATH-500 and AIME2024, it does not perform as well on AIME2025. The model is over-parameterized relative to the amount of signal in the data. which means that the dataset contains less information than the model can represent. In contrast, DECOMPX outperforms S1 even with a much smaller base model (4B) on AIME2025. Model trained only on decomposed data generated from AIME2024 performance suggests its effectiveness in generalizing to unseen mathematical tasks. SFT on AIME 2024 fails to generalize to another year of AIME, whereas our data decomposition can help model better to learn the features that stay predictive not only for in-domain generalisation, but also under the distribution shift. With curriculum learning added to DECOMPX, it can reshape the training trajectory so that the model first captures

universal mathematical skills, then gradually adapts itself against wording and distribution drift. We see evidence that curriculum learning can lead to better generalization and help model competitive performance on AIME 2025 even when the other strong baselines do not.

Curriculum learning is beneficial. In regular SFT, the examples are randomly shuffled. However, with the sample difficulty measurement yielded by DECOMPX, we can create a learning curriculum, starting with the easiest samples and progressively moving towards harder ones. This explores whether difficulty measurements are useful in forming a curriculum without changing the set of training examples. From Table 1 and Table 2, we find that even when training on the same set of examples, difficulty measurements are useful for improving performance, compared to training with a random sample order. Indeed, curriculum yields a 0.8% and 3.4% on MATH and AIME datasets for DECOMPX, or relative improvements of 1.6% and 25.6% respectively over randomly ordered sampling.

4.3 Case Study

Table 4 presents both the original data sample from the MATH training set (Left) and three corresponding generated data samples (Right), which are decomposed based on the original example. Using the difficulty measurement defined earlier, we compute the difficulty scores and categorize the samples into different levels. In both the MATH and AIME datasets, the difficulty scores range from 2 to 20. We define three levels of difficulty: low (scores around 2 to 6), medium (around 6 to 10), and high (from 10 up to 20). As shown in Table 4, we successfully decompose complex problems into simpler subproblems and effectively quantify the difficulty of each subproblem.

5 Discussion and Conclusion

In this work, we propose a novel curriculum learning approach via recursive dataset decomposition, enabling smaller language models to progressively master mathematical reasoning tasks. Our experiments on math benchmarks (MATH and AIME) show significant performance improvements over baseline methods, highlighting the effectiveness of our structured decomposition and difficulty-scoring strategies. In the future, we plan to improve DECOMPX so it could be effectively used for generating more general datasets in a broader range of domains. However, this will require extensive study on teacher models' capability and reliability in task decomposition in other specific domains.

Implications and future work. Our work provides stronger evidence for self-improvement. While our current work adopts a 'large teacher to small student' setup, the method was designed to leverage the previous generation of models not only as data consumers but also as data generators. The key idea is to create a high-quality synthetic curriculum that teaches complex concepts by shaping data for learning, rather than simply augmenting the dataset.

Original Data

Decomposed Data

Problem:

Simplify

3 4 +3 2

3 3 -3

.

Express your answer as a

common fraction.

Solution:

The common factor

of 3 in the numerator and the

denominator can be factored

out:

3 4 +3 2

3 3 -3

= 3(3 3 +3)

3(3 2 -1)

= 3 3 +3

3 2 -1

.

Now compute: numerator is

27 + 3 = 30

, denominator is

9 - 1 = 8

, so

30

8

=

15

4

.

Generated Data with Lower Difficulty

Difficulty Score: 4.0

Problem:

What is the value of

3 3

?

Solution: To solve for 3 3 , we multiply 3 by itself three times: 3 × 3 = 9 , then 9 × 3 = 27 . So, 3 3 = 27 .

Tag:

Exponentiation

Generated Data of Medium Difficulty

Difficulty Score: 10.0

Problem: What is the greatest common divisor (GCD) of the numbers 30 and 8?

Solution: To find the greatest common divisor (GCD) of 30 and 8, we need to identify the largest number that divides both 30 and 8 without leaving a remainder. 1. List the factors of 30:- 1, 2, 3, 5, 6, 10, 15, 30 2. List the factors of 8:- 1, 2, 4, 8 3. Identify the common factors of 30 and 8:- The common factors are 1 and 2. 4. The greatest of these common factors is 2.

Tag:

GCD Calculation

Generated Data with Higher Difficulty

Difficulty Score: 18.0

Problem:

Simplify the expression

3 3 +3

3 2 -1

.

Solution:

Wefirst compute the powers

3 3

= 27

,

3 2

= 9

. Next, substi-

tute into the expression:

27+3

9-1

. Perform addition and subtraction

30

8

.

And simplify the fraction

30

8

=

15

4

. Final Answer is

15

4

.

Tag:

Fraction Simplification

This design naturally supports scenarios where teacher and student have similar capacity: the benefit comes not from the size gap but from the teacher's ability to organize and distill knowledge into teaching-oriented examples. In the weak-to-strong generalization setting, this can be achieved as long as the weaker model is already capable of producing clean and well-structured decompositions. We leave this as a promising future direction, especially at scale, where our proposed hierarchical data decomposition and curriculum learning could have even greater impact.

Limitations and broader impacts. This work proposes using stronger LLM teachers to recursively generate simpler data that builds up a curriculum for training student models. It assumes access to strong enough teacher models that are capable of understanding a math problem, decomposing the problem into meaningful sub-tasks, and faithfully describing the sub-tasks. As mentioned above, for tasks beyond math, where the reasoning path to solve a task is less divisible in an obvious way, the teacher models may face challenges in generating the curriculum. This may require the design of better scaffolding and/or the use of more advanced teacher models. Moreover, LLMs have been shown to hallucinate in various ways, our method is inherently vulnerable because LLM usage is at the core of the system design. For example, a hallucinating or fabricating teacher model may generate inaccurate reasoning chains and decompose them in the wrong ways. The student models trained on the generated curriculum in such a way may result in poor performance or learn unexpected behaviors due to the suboptimality of the curriculum. Finally, we acknowledge that our work is not yet at a stage to be used in many real-world tasks, especially in domains involving high-risk decision making such as law enforcement, legal, finance, or healthcare.

References

Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Andrea P. Danyluk, Léon Bottou, and Michael L. Littman, editors, Proceedings of the 26th Annual International Conference on Machine Learning (ICML 2009) , volume 382 of ACM International Conference Proceeding Series , pages 41-48, Montreal, Quebec, Canada, 2009. ACM. doi: 10. 1145/1553374.1553380. URL https://doi.org/10.1145/1553374.1553380 .

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168 , 2021.

MarkChen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian, Clemens Winter, Philippe Tillet, Felipe Petroski Such, Dave Cummings, Matthias Plappert, Fotios Chantzis, Elizabeth Barnes, Ariel Herbert-Voss, William Hebgen Guss, Alex Nichol, Alex Paino, Nikolas Tezak, Jie Tang, Igor Babuschkin, Suchir Balaji, Shantanu Jain, William Saunders, Christopher Hesse, Andrew N. Carr, Jan Leike, Josh Achiam, Vedant Misra, Evan Morikawa, Alec Radford, Matthew Knight, Miles Brundage, Mira Murati, Katie Mayer, Peter Welinder, Bob McGrew, Dario Amodei, Sam McCandlish, Ilya Sutskever, and Wojciech Zaremba. Evaluating large language models trained on code. 2021.

Tim R. Davidson, Benoit Seguin, Enrico Bacis, Cesar Ilharco, and Hamza Harkous. Orchestrating synthetic data with reasoning. In Will Synthetic Data Finally Solve the Data Access Problem? , 2025. URL https://openreview.net/forum?id=VOoeogZbMb .

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, et al. The llama 3 herd of models, 2024. URL https://arxiv.org/abs/2407.21783 .

DeepSeek-AI, :, Xiao Bi, Deli Chen, Guanting Chen, Shanhuang Chen, Damai Dai, Chengqi Deng, Honghui Ding, Kai Dong, Qiushi Du, Zhe Fu, Huazuo Gao, Kaige Gao, Wenjun Gao, Ruiqi Ge, Kang Guan, Daya Guo, Jianzhong Guo, Guangbo Hao, Zhewen Hao, Ying He, Wenjie Hu, Panpan Huang, Erhang Li, Guowei Li, Jiashi Li, Yao Li, Y. K. Li, Wenfeng Liang, Fangyun Lin, A. X. Liu, Bo Liu, Wen Liu, Xiaodong Liu, Xin Liu, Yiyuan Liu, Haoyu Lu, Shanghao Lu, Fuli Luo, Shirong Ma, Xiaotao Nie, Tian Pei, Yishi Piao, Junjie Qiu, Hui Qu, Tongzheng Ren, Zehui Ren, Chong Ruan, Zhangli Sha, Zhihong Shao, Junxiao Song, Xuecheng Su, Jingxiang Sun, Yaofeng Sun, Minghui Tang, Bingxuan Wang, Peiyi Wang, Shiyu Wang, Yaohui Wang, Yongji Wang, Tong Wu, Y. Wu, Xin Xie, Zhenda Xie, Ziwei Xie, Yiliang Xiong, Hanwei Xu, R. X. Xu, Yanhong Xu, Dejian Yang, Yuxiang You, Shuiping Yu, Xingkai Yu, B. Zhang, Haowei Zhang, Lecong Zhang, Liyue Zhang, Mingchuan Zhang, Minghua Zhang, Wentao Zhang, Yichao Zhang, Chenggang Zhao, Yao Zhao, Shangyan Zhou, Shunfeng Zhou, Qihao Zhu, and Yuheng Zou. Deepseek llm: Scaling open-source language models with longtermism, 2024.

Ronen Eldan and Yuanzhi Li. Tinystories: How small can language models be and still speak coherent english?, 2023.

Carlos Florensa, David Held, Markus Wulfmeier, Michael Zhang, and Pieter Abbeel. Reverse curriculum generation for reinforcement learning. In Conference on robot learning , pages 482-495. PMLR, 2017.

Jeffrey L. Elman. Learning and development in neural networks: the importance of starting small. Cognition , 48(1):71-99, 1993. doi: 10.1016/0010-0277(93)90058-4.

Nathan Habib, Clémentine Fourrier, Hynek Kydlíček, Thomas Wolf, and Lewis Tunstall. Lighteval: A lightweight framework for llm evaluation, 2023. URL https://github.com/huggingface/ lighteval .

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https://arxiv.org/abs/2103.03874 .

Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network, 2015. URL https://arxiv.org/abs/1503.02531 .

Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024.

Yiming Huang, Xiao Liu, Yeyun Gong, Zhibin Gou, Yelong Shen, Nan Duan, and Weizhu Chen. Key-point-driven data synthesis with its enhancement on mathematical reasoning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 24176-24184, 2025.

Hugging Face. Open r1: A fully open reproduction of deepseek-r1, January 2025. URL https: //github.com/huggingface/open-r1 .

Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, MarieAnne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mixtral of experts, 2024.

Yoon Kim and Alexander M Rush. Sequence-level knowledge distillation. In Proceedings of the 2016 conference on empirical methods in natural language processing , pages 1317-1327, 2016.

John P. Lalor and Hong Yu. Dynamic data selection for curriculum learning via ability estimation. In Findings of ACL: EMNLP 2020 , pages 545-555, 2020. URL https://aclanthology.org/ 2020.findings-emnlp.48 .

Jin Hwa Lee, Stefano Sarao Mannelli, and Andrew Saxe. Why do animals need shaping? a theory of task composition and curriculum learning, 2024. URL https://arxiv.org/abs/2402.18361 .

Chengpeng Li, Zheng Yuan, Hongyi Yuan, Guanting Dong, Keming Lu, Jiancan Wu, Chuanqi Tan, Xiang Wang, and Chang Zhou. Mugglemath: Assessing the impact of query and response augmentation on math reasoning. Association for Computational Linguistics , 2024.

Liunian Harold Li, Jack Hessel, Youngjae Yu, Xiang Ren, Kai-Wei Chang, and Yejin Choi. Symbolic chain-of-thought distillation: Small models can also 'think' step-by-step. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 2665-2679, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.150. URL https://aclanthology.org/2023.acl-long.150/ .

Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step, 2023. URL https://arxiv.org/abs/2305.20050 .

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.

Zimu Lu, Aojun Zhou, Houxing Ren, Ke Wang, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathgenie: Generating synthetic data with question back-translation for enhancing mathematical reasoning of llms. Association for Computational Linguistics , 2024.

Jing Luo, Run Luo, Longze Chen, Liang Zhu, Chang Ao, Jiaming Li, Yukun Chen, Xin Cheng, Wen Yang, Jiayuan Su, et al. Personamath: Enhancing math reasoning through persona-driven data augmentation. arXiv preprint arXiv:2410.01504 , 2024.

Arindam Mitra, Luciano Del Corro, Shweti Mahajan, Andres Codas, Clarisse Simoes Ribeiro, Sahaj Agrawal, Xuxi Chen, Anastasia Razdaibiedina, Erik Jones, Kriti Aggarwal, Hamid Palangi, Guoqing Zheng, Corby Rosset, Hamed Khanpour, and Ahmed Awadallah. Orca-2: Teaching small language models how to reason. arXiv, November 2023.

Arindam Mitra, Luciano Del Corro, Guoqing Zheng, Shweti Mahajan, Dany Rouhana, Andres Codas, Yadong Lu, Wei-ge Chen, Olga Vrousgou, Corby Rosset, Fillipe Silva, Hamed Khanpour, Yash Lara, and Ahmed Awadallah. Agentinstruct: Toward generative teaching with agentic flows. arXiv, July 2024. URL https://www.microsoft.com/en-us/research/ publication/agentinstruct-toward-generative-teaching-with-agentic-flows/ .

Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling, 2025. URL https://arxiv.org/abs/2501.19393 .

Koichi Nagatsuka, Clifford Broni-Bediako, and Masayasu Atsumi. Pre-training a BERT with curriculum learning by increasing block-size of input text. In Ruslan Mitkov and Galia Angelova, editors, Proceedings of the International Conference on Recent Advances in Natural Language Processing (RANLP 2021) , pages 989-996, Held Online, September 2021. INCOMA Ltd. URL https://aclanthology.org/2021.ranlp-1.112/ .

Marwa Naïr, Kamel Yamani, Lynda Lhadj, and Riyadh Baghdadi. Curriculum learning for small code language models. In Xiyan Fu and Eve Fleisig, editors, Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 4: Student Research Workshop) , pages 390-401, Bangkok, Thailand, August 2024. Association for Computational Linguistics. ISBN 979-8-89176-097-4. doi: 10.18653/v1/2024.acl-srw.44. URL https://aclanthology.org/ 2024.acl-srw.44/ .

Mathematical Association of America. Aime, February 2024. URL https:// artofproblemsolving.com/wiki/index.php/AIME_Problems_and_Solutions/ .

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. Gpt-4 technical report, 2023.

Guilherme Penedo, Anton Lozhkov, Hynek Kydlíček, Loubna Ben Allal, Edward Beeching, Agustín Piqueres Lajarín, Quentin Gallouédec, Nathan Habib, Lewis Tunstall, and Leandro von Werra. Codeforces cots. https://huggingface.co/datasets/open-r1/codeforces-cots , 2025.

Ofir Press, Noah A. Smith, and Mike Lewis. Shortformer: Better language modeling using shorter inputs, 2020.

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report, 2024. URL https://arxiv.org/abs/2412.15115 .

Qwen, :, An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, Chujie Zheng, Dayiheng Liu, Fan Zhou, Fei Huang, Feng Hu, Hao Ge, Haoran Wei, Huan Lin, Jialong Tang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei Li, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang, Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025. URL https://arxiv.org/abs/2505.09388 .

Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models. Findings of the Association for Computational Linguistics , 2023.

Swabha Swayamdipta, Roy Schwartz, Nicholas Lourie, Yizhong Wang, Hannaneh Hajishirzi, Noah A. Smith, and Yejin Choi. Dataset cartography: Mapping and diagnosing datasets with training dynamics. arXiv preprint arXiv:2009.10795 , 2020. doi: 10.48550/arXiv.2009.10795. URL https://arxiv.org/abs/2009.10795 .

Xiaoxia Wu, Ethan Dyer, and Behnam Neyshabur. When do curricula work? CoRR , abs/2012.03107, 2020. URL https://arxiv.org/abs/2012.03107 . ICLR 2021.

Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, et al. Gemma 2: Improving open language models at a practical size, 2024. URL https://arxiv.org/abs/2408.00118 .

Shuo Yin, Weihao You, Zhilong Ji, Guoqiang Zhong, and Jinfeng Bai. Mumath-code: Combining tooluse large language models with multi-perspective data augmentation for mathematical reasoning. arXiv preprint arXiv:2405.07551 , 2024.

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024b. URL https://arxiv.org/abs/2309.12284 .

Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models, 2024a. URL https://arxiv.org/abs/2309.12284 .

Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah D. Goodman. Star: Bootstrapping reasoning with reasoning, 2022. URL https://arxiv.org/abs/2203.14465 .

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Guidelines:

Justification: Our abstract and intro reflect the contributions accurately.

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

Question: Does the paper discuss the limitations of the work performed by the authors?

2. Limitations

Answer: [Yes]

Justification: See Section 5.

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

Guidelines:

The authors are encouraged to create a separate "Limitations" section in their paper.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

3. Theory assumptions and proofs

Answer: [NA] .

Justification: We do not have any theoretical results in our paper.

The answer NA means that the paper does not include theoretical results.

Guidelines:

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

All assumptions should be clearly stated or referenced in the statement of any theorems.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

4. Experimental result reproducibility

Theorems and Lemmas that the proof relies upon should be properly referenced.

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Justification: We explain our experimental setup in Section 4.1.

Answer: [Yes]

Guidelines:

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

The answer NA means that the paper does not include experiments.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(d) Werecognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-weights models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

5. Open access to data and code

Answer: [Yes]

Guidelines:

Justification: We will release the code when the paper gets accepted. Our implementation is based on open-source training and evaluation scripts and is reproducible.

The answer NA means that paper does not include experiments requiring code.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

6. Experimental setting/details

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

The answer NA means that the paper does not include experiments.

Justification: See Section 4.1.

Guidelines:

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

7. Experiment statistical significance

The full details can be provided either with the code, in appendix, or as supplemental material.

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Guidelines:

Justification: See Table 1 and Table 2.

The answer NA means that the paper does not include experiments.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

8. Experiments compute resources

Answer: [Yes]

Guidelines:

Justification: See Section 4.1.

The answer NA means that the paper does not include experiments.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

9. Code of ethics

Answer: [Yes]

Guidelines:

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Guidelines:

Justification: See Section 5.

The answer NA means that there is no societal impact of the work performed.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

11. Safeguards

Answer: [NA]

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained LMs, image generators, or scraped datasets)?

Justification: Our paper focuses solely on the mathematical dataset and does not pose a high risk of misuse.

The answer NA means that the paper poses no such risks.

Guidelines:

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

12. Licenses for existing assets

Answer: [Yes]

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Justification: We have cited related work appropriately.

The answer NA means that the paper does not use existing assets.

Guidelines:

The authors should cite the original paper that produced the code package or dataset.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

The authors should state which version of the asset is used and, if possible, include a URL.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

13. New assets

Answer: [NA]

Guidelines:

Justification: Our paper does not release new assets.

The answer NA means that the paper does not release new assets.

The paper should discuss whether and how consent was obtained from people whose asset is used.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

14. Crowdsourcing and research with human subjects

Answer: [NA]

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Justification: We did not conduct any crowdsourcing and/or research with human subjects. Guidelines:

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Guidelines:

Justification: We did not conduct any user studies requiring IRB approval.

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Justification: We use GPT-4o as the teacher model for distillation. See Section 3 and Section 4.1.

Answer: [Yes]

Guidelines:

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Technical Appendices and Supplementary Material

Contents

A Details of subset of MATH training data

In Table 5, we present the original and sampled sizes of the MATH dataset used in our experiments, broken down by subject domain and difficulty level. Sampling was performed uniformly at random within each group to ensure representative coverage of the various topics and levels.

B Examples of Concept Dependency Graph

Figure 3 presents the concept dependency graphs constructed during the data decomposition process. We observe that an atomic mathematical operation, such as Addition , has many edges linking it to more advanced operations.

TheEx

of

Vinatgd Bar

C Zero-shot Small Model Performance

We partitioned the decomposed AIME2024 dataset into five equal-sized bins (quintiles) based on our proposed difficulty measurement, shown in Table 6. This measurement is derived from the concept dependency graph, designed to reflect the conceptual complexity of each problem. We then evaluated the zero-shot performance of the Qwen3-4B-Base model on each difficulty tier.

Our results shown in Figure 4 demonstrate a clear inverse correlation between difficulty score and model accuracy: the model achieves the highest accuracy on problems with the lowest difficulty scores (Quintile 1), and its performance degrades as the difficulty increases, reaching the lowest accuracy on the highest difficulty tier (Quintile 5). This performance trend validates the effectiveness of our concept dependency graph-based difficulty metric in capturing the relative hardness of mathematical problems.

Q1 (Easiest)

Q2

Q3

Q4

Q5 (Hardest)

Difficulty Quintile

20

30

40

50

60

70

80

Accuracy (%)

72.29%

68.67%

65.06%

64.63%

26.83%

Trend

D Tag Clustering Details

We summarize the tag information identified in both the MATH dataset (Table 7) and the AIME dataset (Tables 8 and 9).

E Examples of Decomposed Data

We apply our data decomposition method to the S1 dataset and demonstrate the two-layer recursive decomposition process for a sample data point in Table 10.

Table 10. Decomposed S1 data sample. Black text denotes the original data; yellow text represents the first level of decomposition; and blue text represents the second level of decomposition.

{Question:"Given a rational number, write it as a fraction in lowest terms and calculate the product of the resulting numerator and denominator. For how many rational numbers between 0 and 1 will 20! be the resulting product?",

Solution:"Let the rational number between 0 and 1 be 𝑟 . When written as a fraction in lowest terms, let it be 𝑎 𝑏 , where 𝑎 and 𝑏 are positive integers, gcd( 𝑎, 𝑏 ) = 1 , and 0 < 𝑎 < 𝑏 .We are given that the product of the numerator and denominator is 20! , so 𝑎 × 𝑏 = 20! .We need to find the number of pairs ( 𝑎, 𝑏 ) such that 𝑎 × 𝑏 = 20! , gcd( 𝑎, 𝑏 ) = 1 , and 𝑎 < 𝑏 .Let 𝑁 = 20! . The prime factorization of 𝑁 is 𝑝 𝑒 1 1 𝑝 𝑒 2 2 ⋯ 𝑝 𝑒 𝑘 𝑘 , where 𝑝 𝑖 are distinct prime numbers.The number of distinct prime factors of 20! is the number of primes less than or equal to 20. These primes are 2, 3, 5, 7, 11, 13, 17, 19. Thus, there are 8 distinct prime factors, so 𝑘 = 8 .Since 𝑎 × 𝑏 = 𝑁 and gcd( 𝑎, 𝑏 ) = 1 , each prime power factor 𝑝 𝑒 𝑖 𝑖 of 𝑁 must be entirely a factor of 𝑎 or entirely a factor of 𝑏 .The number of ways to distribute the prime power factors between 𝑎 and 𝑏 is 2 𝑘 , where 𝑘 is the number of distinct prime factors of 𝑁 . Each distribution corresponds to a unique pair ( 𝑎, 𝑏 ) such that 𝑎 × 𝑏 = 𝑁 and gcd( 𝑎, 𝑏 ) = 1 .The number of such pairs is 2 𝜔 ( 𝑁 ) , where 𝜔 ( 𝑁 ) is the number of distinct prime factors of 𝑁 .Since 20! is not a perfect square, for every pair ( 𝑎, 𝑏 ) such that 𝑎 × 𝑏 = 20! and gcd( 𝑎, 𝑏 ) = 1 , we have 𝑎 ≠ 𝑏 . The pairs can be grouped into ( 𝑎, 𝑏 ) and ( 𝑏, 𝑎 ) , where 𝑎 ≠ 𝑏 .The number of pairs with 𝑎 < 𝑏 is equal to the number of pairs with 𝑏 < 𝑎 .The total number of pairs is 2 𝜔 (20!) . The number of pairs with 𝑎 < 𝑏 is 1 2 × 2 𝜔 (20!) = 2 𝜔 (20!)-1 .The number of distinct prime factors of 20! is 𝜔 (20!) = 8 .The number of pairs ( 𝑎, 𝑏 ) such that 𝑎 × 𝑏 = 20! , gcd( 𝑎, 𝑏 ) = 1 , and 𝑎 < 𝑏 is 2 8-1 = 2 7 = 128 .Each such pair ( 𝑎, 𝑏 ) corresponds to a unique rational number 𝑎 𝑏 between 0 and 1 in lowest terms, with the product of the numerator and denominator equal to 20! .Final Answer: The final answer is 128 "} Solution: { ReasoningStep: "Let the rational number between 0 and 1 be 𝑟 = 𝑎 𝑏 in lowest terms with gcd( 𝑎, 𝑏 ) = 1 , 0 < 𝑎 < 𝑏 , and 𝑎 × 𝑏 = 20! ". Tag:"Reducing a fraction to lowest terms", Question:"In the context of finding rational numbers 𝑟 = 𝑎 𝑏 in lowest terms with 𝑎 ⋅ 𝑏 = 20! , suppose you start with the unreduced fraction 7 20!∕7 . Reduce this fraction to lowest terms.", Solution:{ ReasoningStep: "We have the fraction 𝐹 = 7 20!∕7 . Since 20! contains 7 2 (because ⌊ 20∕7 ⌋ = 2 ), the number 20!∕7 is still divisible by 7.", Tag:"Exponent of a prime in a factorial (Legendre's formula)", Question:"In the context of reducing the fraction 7/(20!/7), use Legendre's formula to find the exponent of the prime 7 in the prime factorization of 20!.", Solution:" 1. Legendre's formula states that the exponent of a prime p in n! is given by the sum of ⌊ 𝑛 ∕ 𝑝 𝑘 ⌋ 𝑓𝑜𝑟𝑘 = 1 , 2 , ..𝑝 𝑘 > 𝑛. 2. Here, n=20 and p=7: - For k=1: ⌊ 20∕7 ⌋ = 2 - For k=2: ⌊ 20∕49 ⌋ = 0 - Higher powers of 7 exceed 20, so their terms are zero. 3. Summing these gives 2 + 0 = 2." },

{

Tag:"Prime Factorization",

ReasoningStep:"Set 𝑁 = 20! . Its prime factorization has 𝑘 = 8 distinct primes (2, 3, 5, 7, 11, 13, 17, 19).",

Question:"How many distinct prime numbers appear in the prime factorization of 20!?",

{ ReasoningStep:"By definition, 20! = 1×2×3×...×20, so every prime p 20 divides 20!.",

Solution:

Tag:"Definition of factorial",

Solution:"By definition, n! = 1×2×3×...×n, which multiplies all integers from 1 up to n. For n = 20, the integers are 1, 2, 3, . . . , 20. Counting these gives 20 numbers being multiplied.",

Question:"In the definition of factorial, 20! is written as the product 1×2×3×...×20. How many integers are multiplied together in this product?",

{ ReasoningStep:"The primes less than or equal to 20 are 2, 3, 5, 7, 11, 13, 17, and 19, and there are no others in that range.",

Question:"Among the integers from 1 through 20 inclusive, how many are prime numbers?",

Tag:"Primality Testing",

Solution:"1. By definition, a prime number is an integer greater than 1 that has no positive divisors other than 1 and itself.

3. The primes in this range are 2, 3, 5, 7, 11, 13, 17, and 19.

2. We test each integer from 1 to 20: - 1 is not prime by definition. - 2 is divisible only by 1 and 2 → prime. - 3 is divisible only by 1 and 3 → prime. - 4 is divisible by 1, 2, 4 → not prime. - 5 is divisible only by 1 and 5 → prime. - 6 is divisible by 1, 2, 3, 6 → not prime. - 7 is divisible only by 1 and 7 →prime. - 8 is divisible by 1, 2, 4, 8 → not prime. - 9 is divisible by 1, 3, 9 → not prime. - 10 is divisible by 1, 2, 5, 10 → not prime. - 11 is divisible only by 1 and 11 → prime. - 12 is divisible by 1, 2, 3, 4, 6, 12 → not prime. - 13 is divisible only by 1 and 13 → prime. - 14 is divisible by 1, 2, 7, 14 →not prime. - 15 is divisible by 1, 3, 5, 15 → not prime. - 16 is divisible by 1, 2, 4, 8, 16 → not prime. - 17 is divisible only by 1 and 17 → prime. - 18 is divisible by 1, 2, 3, 6, 9, 18 → not prime. 19 is divisible only by 1 and 19 → prime. - 20 is divisible by 1, 2, 4, 5, 10, 20 → not prime.

4. Counting them gives a total of 8 primes.",

Question:"When prime factorizing 20!, we include every prime number that is less than or equal to 20. How many distinct prime numbers appear in the prime factorization of 20! ?",

{ ReasoningStep:"Therefore, the prime factorization of 20! includes exactly these 8 distinct primes.", Tag:"Prime Factorization",

Solution:"1. A prime divides 20! exactly if it is 20, since 20! is the product of all integers from 1 to 20.

3. Counting these gives 8 distinct primes in the factorization of 20!." },

2. The primes 20 are 2, 3, 5, 7, 11, 13, 17, and 19.

{ ReasoningStep:"Since gcd( 𝑎, 𝑏 ) = 1 , each prime power 𝑝 𝑒 𝑖 𝑖 of 𝑁 must go entirely to 𝑎 or 𝑏 , giving 2 𝑘 pairs ( 𝑎, 𝑏 ) .", Tag:"Fundamental Theorem of Arithmetic", Question:"Let N be 20! and write N in its prime factorization N = p 𝑒 1 1 ⋅ 𝑝 𝑒 2 2 ⋅ … ⋅ 𝑝 𝑒 𝑘 𝑘 . Using the Fundamental Theorem of Arithmetic, in how many ways can you choose a pair of positive integers (a, b) such that ab = N and gcd(a, b) = 1 by assigning each prime power 𝑝 𝑒 𝑖 𝑖 entirely to either a or b?", Solution:{ReasoningStep:"By the Fundamental Theorem of Arithmetic, 20! factors into prime powers corresponding to each prime p 20.", Tag:"Prime Factorization", Question:"In the prime factorization of 20!, what is the exponent of the prime 2?", Solution:"1. By Legendre's formula, the exponent of a prime p in n! is ⌊ 𝑛 ∕ 𝑝 ⌋ + ⌊ 𝑛 ∕ 𝑝 2 ⌋ + ⌊ 𝑛 ∕ 𝑝 3 ⌋ + . . . until the terms are zero. 2. For p = 2 and n = 20: ⌊ 20∕2 ⌋ = 10 ⌊ 20∕4 ⌋ = 5 ⌊ 20∕8 ⌋ = 2 ⌊ 20∕16 ⌋ = 1 ⌊ 20∕32 ⌋ = 0 (and all higher powers give zero) 3. Sum these: 10 + 5 + 2 + 1 = 18.", }, {ReasoningStep:"The primes 20 are 2, 3, 5, 7, 11, 13, 17, 19, so there are k = 8 distinct prime powers in the factorization of 20!.", Tag:"Listing primes up to a given limit", Question:"In the prime factorization of 20!, how many distinct prime bases (i.e., prime numbers 20) appear?", Solution:" 1. To find the distinct prime factors of 20!, we list all prime numbers that are less than or equal to 20. 2. A prime number is a number greater than 1 with no positive divisors other than 1 and itself. 3. Checking the integers from 2 up to 20, the primes are: 2, 3, 5, 7, 11, 13, 17, 194. Count these primes: there are 8 of them.", },

{ReasoningStep:"To form a pair (a, b) with ab = N and gcd(a, b) = 1, each prime power 𝑝 𝑒 𝑖 𝑖 must go entirely to a or entirely to b.", Tag:"Fundamental Theorem of Arithmetic", Question:"Let N = 20! and write its prime factorization as 𝑁 = 𝑝 𝑒 1 1 ⋅ 𝑝 𝑒 2 2 ⋅ … ⋅ 𝑝 𝑒 𝑘 𝑘 . By the Fundamental Theorem of Arithmetic, to form pairs of positive integers (a, b) with ab = N and gcd(a, b) = 1, each prime power p 𝑒 𝑖 𝑖 𝑚𝑢𝑠𝑡𝑔𝑜𝑒𝑛𝑡𝑖𝑟𝑒𝑙𝑦𝑡𝑜𝑎𝑜𝑟𝑒𝑛𝑡𝑖𝑟𝑒𝑙𝑦𝑡𝑜𝑏.𝐹 𝑜𝑟𝑎𝑠𝑖𝑛𝑔𝑙𝑒𝑝𝑟𝑖𝑚𝑒𝑝𝑜𝑤𝑒𝑟 p 𝑒 𝑖 𝑖 , in how many ways can you assign it to either a or b so that gcd(a, b) = 1?", Solution:" 1. The Fundamental Theorem of Arithmetic ensures N's prime factorization is unique. 2. To keep gcd(a, b) = 1, you cannot split 𝑝 𝑒 𝑖 𝑖 between a and b. 3. Therefore, for that prime power, you have exactly two choices: - Assign 𝑝 𝑒 𝑖 𝑖 to a (so exponent in a is e 𝑖 , 𝑖𝑛𝑏𝑖𝑠 0) , -𝑜𝑟𝑎𝑠𝑠𝑖𝑔𝑛 p 𝑒 𝑖 𝑖 to b (so exponent in a is 0, in b is e 𝑖 ) . 4. No other distributions are allowed without violating gcd(a, b) = 1.", }, {ReasoningStep:"Each of the k = 8 prime powers has 2 choices, so by the multiplication principle there are 2 𝑘 = 2 8 ways.", Tag:"Fundamental Counting Principle", Question:"Suppose the prime factorization of 20! contains 8 distinct prime powers. If you assign each prime power entirely to either integer a or integer b, in how many ways can you make these assignments by the Fundamental Counting Principle?", Solution:"1. There are 8 prime powers to assign. 2. Each prime power has 2 independent choices: assign it to a or to b. 3. By the Fundamental Counting Principle, multiply the number of choices for each prime power: Total ways = 2∙2∙ . . . ∙2 (8 times) = 2 8 = 256.", }], }, {ReasoningStep:"As 20! is not a perfect square, 𝑎 ≠ 𝑏 , so half the pairs have 𝑎 < 𝑏 , yielding 2 𝑘 -1 = 2 7 = 128 .", Tag:"Divisor Pairing Principle", Question:"Using the Divisor Pairing Principle, in how many ways can we write 20! as a product ab of two positive integers with gcd(a,b)=1 and a<b?",

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a curriculum learning strategy where a \"teacher\" Large Language Model (LLM) recursively decomposes complex mathematical problems into simpler, hierarchically structured sub-problems. Each sub-problem is tagged with a concept and verified. A novel scoring system quantifies the difficulty of these sub-problems based on their structural complexity (number of sub-steps) and conceptual depth (derived from a \"Concept Dependency Graph\" that maps prerequisite skills). Smaller \"student\" LLMs are then trained on this decomposed dataset in stages of increasing difficulty. Experiments on MATH and AIME datasets show that this approach significantly improves the performance and generalization of student models compared to standard training and other data augmentation methods. The method's success hinges on the teacher model's capability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Novel Methodology: The paper proposes a novel \"reverse curriculum generation\" approach that recursively decomposes complex problems into simpler, learnable components using a teacher-student framework. This hierarchical decomposition is a key innovation.\n\n2. Sophisticated Difficulty Scoring: It introduces a unique system for measuring data difficulty based on both \"structural complexity\" (number of sub-problems) and \"conceptual depth\" (derived from a \"Concept Dependency Graph\" of skills). This allows for more granular curriculum construction.\n\n3. Dataset Insights: A byproduct of the method is the potential to create a \"cartography\" of a dataset, offering insights into skill coverage and relationships between concepts.\n\nWeaknesses:\n\n1. Reliance on a Capable Teacher Model: The entire framework heavily depends on the availability and proficiency of a strong \"teacher\" LLM (like GPT-40) that can accurately reason, decompose problems, and generate correct sub-tasks. The quality of the generated curriculum is tied to the teacher's capabilities.\n\n2. Incomplete Evaluation Scope: The paper primarily evaluates the method's effectiveness on base student models (Qwen2.5-1.5B and Qwen3-4B-Base). It does not include experiments applying the DECOMPX training methodology directly to reasoning models of comparable sizes to see if similar or enhanced gains are achieved on reasoning models.\n\n3. Limited Comparison with State-of-the-Art Distillation: This approach shares some similarities with knowledge distillation, as both leverage capability transfer from larger models to enhance smaller ones. However, experimental results demonstrate that neither the 4B nor 1.5B parameter models can match the performance of the DeepSeek-R1-Distill-Qwen-1.5B model, which is directly distilled from Deepseek-R1.\n\nQUESTIONS:\n1. This method involves a more complex procedure than direct distillation and underperforms compared to distillation using a reasoning model. What advantages does it offer over directly distilling with a reasoning model?\n\n2. Is this method effective for reasoning models? Can it further enhance their performance?\n\n3. How dependent is this method on the capability of the Teacher Model? Can it still achieve good results when the Teacher Model itself performs poorly?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-04"}

comments: SUMMARY:
The authors propose a new method for generating curriculum (for curriculum learning), in which the teacher model recursively breaks down task into increasingly simple components for the student model to learn the curriculum in increasing difficulty. A new scoring system is proposed for measuring data difficulty during curriculum construction. Experiments are conducted on math datasets.

STRENGTHS AND WEAKNESSES:
Strengths:
The writing is clear. The proposed method is convincing. I appreciate the case study in the submission.

Weaknesses:
I don’t think the experiment results are strong enough to demonstrate the superiority of the proposed method yet. First of all, only one other method is compared with in the experiments. More importantly, while I understand the reason behind some of the current choices of # data in Table 1 and 2, it would make more sense to have this number leveled across DecompX and comparison experiments. For example, both DecompX and DecompX+Curriculum use 4500 in Table 1 while MetaMATH-Aug only uses 2638 and SFT only 360. This makes the higher score of DecompX less convincing.

QUESTIONS:
Besides the issues noted weaknesses section, it would also be good to see if this method offers improvement on non-math datasets. The results in this submission currently run a bit thin.

Could you also explain how you implemented Line 8 of Algorithm 1 in your experiments (i.e., what auto-rater you used and how consistency is checked)?

An interesting but not necessary thing would be to see if the proposed method can still offer big improvement when the teacher and student models have similar size, which could show some promise of self-improvement.

LIMITATIONS:
Limitations have been sufficiently discussed in the conclusion section.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-29

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a curriculum learning approach built upon the concept of distillation/data augmentation. Applied to math problem solving, the approach entails decomposing math problems into subproblems based on reasoning steps generated by LLM. These subproblems are generated recursively and organized into a tree structure, tagged with difficulty levels using structural and conceptual features. Empirical results show marginal improvements over alternative methods on the training set but demonstrate significant gains on unseen datasets (AIME 2025) with encouraging generalization promise.\n\nSTRENGTHS AND WEAKNESSES:\nQuality: This paper is well written, and the presentation of the proposed methods is both clear and sound, supported by solid experimental results. The work would be further strengthened by a deeper analysis of the results—for example, by examining failure cases to better understand the limitations of the approach. Additionally, incorporating human validation of the generated tags and difficulty levels would further enhance the credibility of the decomposition approach. \n\nClarity: Overall, the authors have done a decent job in presenting their ideas. The paper is clearly written, and the inclusion of illustrative examples and detailed algorithm descriptions supports comprehension. The use of a case study to demonstrate the decomposition process is particularly appreciated, as it offers concrete insight into how the proposed method operates in practice. \n\nSignificance: The paper presents a knowledge distillation approach to curriculum learning, demonstrated through the task of mathematical problem solving. While the authors suggest that the method can be generalized, a more detailed discussion on its broader applicability would be valuable. For instance, the approach appears well-suited for procedure-based problem solving, but its effectiveness in other domains remains unclear. Additionally, the definition of difficulty in the paper seems primarily tied to the number of steps required to complete a task. However, in some cases, difficulty also involves the need for non-obvious insights—such as re-framing a problem to allow for an elegant, minimal solution. This may help explain why, despite significant efforts (including the proposed enhancements), performance on solving new AIME problems remains relatively low.\n\nOriginality: The paper is situated in the literature of data augmentation/distillation with LLM as well as curriculum learning. The related work is detailed and relevant. The paper mentioned that the idea of curriculum learning with synthetic data is not entirely novel. I wish the authors could articulate better the contribution of this paper in the context of other similar methods, such as Tiny Stories, understanding it is from different domains.\n\nQUESTIONS:\n1. Have you analyzed failure cases, particularly on challenging problems like AIME, to better understand the limitations of your approach?\n\n2. Have you considered human evaluation to validate the generated tags and difficulty levels, ensuring they align with human intuition?\n\n3. How well do you expect your method to generalize beyond procedural domains like math? do you have insights on what types of tasks is it best suited for and what are not suitable for?\n\nLIMITATIONS:\nThe limitation was lightly discussed. I wish the authors could also discuss the social impact of this work. While it is interesting to design AI to be as smart as possible, the authors could also discuss how this approach can be used to improve math education, for example, using the approach to automatically generate scaffolding, determine the difficulty level, and tag the problem domains are very interesting contributions to math education and beyond.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces DECOMPX, a novel curriculum learning approach that recursively decomposes complex problems into simpler, more learnable components. \nThe authors propose a teacher-student framework where a powerful teacher model (GPT-4o) generates easier versions of examples by reasoning step-by-step. \nThe method also includes a novel scoring system for data difficulty based on structural complexity and conceptual depth, enabling curriculum construction over the decomposed data. \nExperiments on MATH and AIME datasets demonstrate improved performance compared to standard training baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The core idea of recursively decomposing complex problems into simpler sub-problems, coupled with extracting and organizing \"concept tags\" into a dependency graph, is novel.\n* Models trained with DECOMPX exhibit better generalization, particularly on the AIME2025 dataset which represents an out-of-domain distribution shift from AIME2024.\n* The proposed scoring system, combining structural complexity and conceptual depth, provides a principled way to quantify problem difficulty within the decomposed dataset.\n\nWeaknesses:\n* The experiments appear to primarily focus on the decomposition of relatively short CoTs, lacking an in-depth exploration of how the recursive decomposition method performs and scales with long CoT solutions, which are common in complex mathematical reasoning problems and large reasoning models like DeepSeek-R1 and Qwen3.\n* While the paper highlights improvements on AIME, the AIME2024 training set used consists of only 30 problems. This is an extremely small dataset for fine-tuning language models in mathematical reasoning, making the observed performance gains potentially sensitive to this limited data size. The generalizability of conclusions drawn from such a small training set to broader mathematical reasoning tasks remains a concern.\n* The comparison with MetaMath-Aug, particularly with only 114 data points for AIME, appears to be a relatively weak baseline. The paper could benefit from comparing against a wider range of more recent and robust data augmentation methods in mathematical reasoning.\n\nQUESTIONS:\n* Could the authors provide more details on the specific prompts used for the teacher model during \"Step Extraction\" and \"Concept Tagging\"? How is the initial set of concept tags defined or discovered, and is there any pre-defined ontology or does it emerge purely from the teacher model's responses?\n* The description of training sets is somewhat vague. Could the authors clarify if Table 1 and Table 2 both utilize AIME2024 as the training set for their respective models, or if Table 1 uses the MATH dataset's training portion? Why DECOMPX use less data than SFT baseline in Table1 but more data in Table2. A detailed introduction of the training data for each experiment would be beneficial.\n* How is it guaranteed that every extracted \"step\" will always have a well-defined, verifiable answer? Some intermediate steps in a human-like reasoning chain might involve logical inferences, variable assignments, or reflections that do not directly yield a answer that can be \"solved\" or \"verified\" independently as a sub-problem. How does the current verification mechanism handle such cases, or are these types of steps implicitly ignored?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}