Curriculum Reinforcement Learning from Easy to Hard Tasks Improves LLM Reasoning
Authors: Shubham Parashar
Shurui Gui
Xiner Li
Hongyi Ling
Sushil Vemuri
Blake Olson
Eric Li
Yu Zhang
James Caverlee
Dileep Kalathil
Shuiwang Ji
Venue: ICLR 2026 Poster (2026)
Keywords: LLM
Reinforcement Learning
Post Training
Primary area: foundation or frontier models, including LLMs

Abstract: We aim to improve the reasoning capabilities of language models via reinforcement learning with verifiable rewards (RLVR). Recent RLVR post-trained models like DeepSeek-R1 have demonstrated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RLVR alone to improve reasoning on inherently difficult tasks is less effective due to sparse rewards. Here, we draw inspiration from curriculum learning and propose to schedule tasks from easy to hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner. Empirically, we observe that, although easy tasks are important initially, fading them out through appropriate scheduling is essential in preventing overfitting. Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive finite-sample complexity bounds and show that when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Experiments across diverse datasets and models demonstrate that E2H Reasoner substantially enhances LLM reasoning. Code is available at - https://github.com/divelab/E2H-Reasoning

Published as a conference paper at ICLR 2026
CURRICULUM REINFORCEMENT LEARNING FROM EASY TO HARD TASKS IMPROVES LLM REASONING
Shubham Parashar' Shurui Gui Xiner Li Hongyi Ling' Sushil Vemuri
Blake Olson' Eric Li" Yu Zhang' James Caverlee Dileep K Shuiwang Ji
*Department of Computer Science & Engineering, Texas A&M University
DDeaartment of Electrical & Computer Engineering, Texas A&M University
ABSTRACT
We aim to improve the reasoning capabilities of language models via reinforcement learning (RL). Recent RL post-trained models like DeepSeek-RI have demon- strated reasoning abilities on mathematical and coding tasks. However, prior studies suggest that using RL alone to improve reasoning on inherently difficult tasks is less effective. Here, we draw inspiration from curriculum learning and pro- pose to schedule tasks from easy hard (E2H), allowing LLMs to build reasoning skills gradually. Our method is termed E2H Reasoner: Empirically, we observe that, although easy tasks are important initially, fading them out through appro- priate scheduling is essential in preventing overfitting, Theoretically, we establish convergence guarantees for E2H Reasoner within an approximate policy iteration framework. We derive fmite-sample complexity bounds and show that, when tasks are appropriately decomposed and conditioned, learning through curriculum stages requires fewer total samples than direct learning. Our code is publicly available at https://gtttb.ccm/dilll///ILLL Reasonnig
INTRODUCTION
Large Language Models (LLMs) have demonstrated reasoning capabilities in tasks such as multi-step arithmetic, planning, and code generation. However, the notion of reasoning in LLMs remains vague. with some works equating it to generating intermediate steps during problem solving (Wei et al. 2022; Cobbe et al., 202T). This view, although intuitive, blurs the line between genuine reasoning and surface-level pattern recognition (Stechly et al. 2024 Valmeekam et al. 2023). Therefore, we adopt a view focusing on generalization, defining reasoning as the ability to extract principles from simpler tasks and apply them to harder ones. Supporting this capability requires training methods that go beyond imitation and help models learn underlying moblem-solving strategies.
In this direction, the success of DeepSeek R1 (Guo et al. (025) and OpenAl ol (Jaech et al. 2024) shows that reinforcement learning (RL) based post-training enhances reasoning. RL uses task-specific rewards based output correctness, unlike supervised fine-tuning (SFT), which trains models to imitate fixed ouput-output examples (Zelikman et al. 2022). However, RL struggles on harder tasks
on which pre-trained models have low zero shot performance (Shao et al. 2024: Zeng et al. 2025) and since rewards are granted only for correct answers, resulting in sparse learning signals.
To address the sparse reward problem, curriculum learning has been applied reinforcement learning (CRL) by structuring training from easier to harder tasks (Bengio et al.) 2009). This idea has recently been extended LLM post-training (Team et al.] [025 Bereovich al. 2025). However, these initial efforts primarily rely on simplified strategies, such as training on easy tasks before switching to hard ones after a fixed number of iterations. In contrast, we introduce E2H Reasoner (E2H), a CRL approach with probabilistic scheduler that gradually shifts focus from easy to hard tasks, enabling even LLMs to develop core reasoning abilities and generalize more complex problems. We show that L.MMs can learn tasks they initially failed in the zero-shot setting (Fig. D. Empirically, E2H achieves otate-of-the-art performance across five reasoning tasks, i.e.
"qqual contribution
Correspondence Dileep Kalathil <dileep.kalathil wew.e.tmm.ccuu.... and Shuiwang Ji <sji @tamu.edu>

---

Published as a conference paper at ICLR 2026
pawsk
10 1.51 2.JMA
Base LLaMt 8.2
L
<ardeeer>> Lm for able to find in equationn <innswer
0.0
255
ethemboF Reasoner ELL 8.2 380
(a) Pass ko on Countdown
subtrect da
ger get
EOH Ia
22H
get get
pawgk
subttrccting also addingg adtiing get anding get <ancwar-89 28 30v/answer
get
(b) @ on Blocksworld
(c) LLaMA-3. 3.2 reasoning
Figure I: (a, b) Reinforeement lcarning (RL) based post- bost-training is believed to improve accuracy at low ls values in pass @+ k evaluation (Guo et al. 2025 Yue et al. 2025). we show that E2H Reasoner, a curriculum based RL (CRL) approach, enables LLMs to solve tasks they previously could not, outperforming base models even higher k. LLaMA 3.2 3B reasoning trace for Countdown (Gandhi al. 2024) after E2H Reasoner post-training.
Pretraining Distribution
Pretraining Distribution
Pretraining Distribution
Pretraining Distribution
Pretraining Distribution
Pretraining Distribution
Trivial Tasks
Trivial Tasks
Trivial Tasks
Trivial Tasks
Trivial Tasks
Easy Tasks
Easy Tasks
Easy Tasks
Easy Tasks
Easy Tasks
Texk Davtbutio
Medium
Medium Tasks
Medium Tasks
Medium Tasks
Medium Tasks
Hard Tasks
Hard Tasks
Hard Tasks
Hard Tasks
Hard Tasks
Hard Tasks
Figure 2: Task Decomposition of Easy Hard Reasoner (E2HH). E2H first decomposes the training dataset into tasks of increasing difficulty, namely trivial, easy, and medium, to help the LLM acquire core skills. As training progresses, E2H schedules harder tasks accordingly. See Section 3.2 for scheduling details.
Blocksworld (Valmeekam et al., 2024) and Countdown (Gandii et al. 2024), as well as three
arithmetic reasoning benchmarks (eendrycks et al. 2021 Agarwal et al. 2021 Cobbe et al. (221I) On the theoretical front, we provide a comprehensive analysis of CRL through the lens of Approximate Policy Iteration, establishing convergence guarantees for the final performance gap and finite-sample complexity bounds. Importantly, we prove that with a well-designed curriculum, CRL requires fewer total samples than direct learning, aligning with and supporting our empirical observations.
2 BACKGROUND AND RELATED WORK
Reasoning in Large Language Models remains loosely defined, with interpretations varying by tasks and contexts Yu et al. (2024). Prior work describes it as generating logical chains of thought Wei
al. 2022); Wang et (2023), ), performing multi-step deductions (Saparov He 2023 Yao et al.
2023 Ling et al. 2025), or simulating human-lite like problem solving (Parashar et al. 2025). These views, lack clear boundaries between reasoning and pattern recognition. To address this gap, we are inspired by prior work that frames reasoning as generalization abstraction (Valmeekam et al.)
2024 2023), building on the idea that reasoning involves learning core principles and applying them
broadly (Stechly et al. 2024 Huang et al. 2024).
Post-training of Large Language Models has emerged as a popular approach to improve rea- soning Snelll et al. 2025). These methods are grouped into Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) based techniques. In SFT, the model is trained to imitate outputs from carefully curated human- -ike reasoning examples (Zelikman et al.) 2022 Muennighoff et al.) 2025). However, studies have shown that SFT can lead models to overlit to surface-leve patterns (Chu et al. 2025), limiting generalization. In contrast, RL-bbased post-training uses task-specific rewards and
2

---

Published as a conference paper at ICLR 2026
updates the model through policy optimization algorithms (Schulman et al. 2017 Rafailov et al. 2023 Shao et al. 2024). instead of imitation. This approach has shown greater potential in improving reasoning performance, demonstrated by the success of models fine-tuned with RL (Guo et al.) 2025 Jacch et al. 2024). Still, for inherently difficult tasks that L.LMs struggle to solve in zero-shot settings, post-training with RI. alone has been insufficient Shao et al. (2024); Zeng et al. (2025).
Curriculum Learning organizes tasks by increasing difficulty to promote smoother and more effective training (Bengio al., 2009) Graves et al., 2017). In the context of RL, has been applied to help agents acquire complex behaviors by first mastering simpler tasks (Narvekar et al. 2020
Tajwar et al. 2025). Recently efforts have been made to investigate how curriculum-
RL can
enhance reasoning and generalization in LLMs (Qiu et al. 2025 Bae et al., 2025; Zeng ct al. 2025).
Others improve learning by removing examples that are too easy or too hard Bae et al. (2025 Yu et al. 2025), or by maintaining balanced mix of task difficulties Zeng et al. (2025). Similarly, Chen et al. 2025) and Foostrr al. (2025) adaptively sample problems with a 50% solve rate to maximize the GRPO advantage during training. Other recent efforts implement manual curricula switching from easy to hard tasks after a fixed number of training iterations (Xie et al. 2025 Team et al.) 2025
Bercovich et al. 2025). In contrast, our work schedules tasks probabilistically from easy to hard, improving reasoning and generalization to out-of -it-of-distributionn tasks.
METHOD
RL for LLM reasoning. We formulate the reasoning process of LLMs as a RL problem defined AS, P,r;7), where S is the state space, A is the finite action space, P: S x A + A(S) is the transition kernel, r S x A [0, Rmax] is the reward function, and 7 E (0, 1) is the discount factor. The state space consists of all valid token prefixes, where each state 8t (zg, #1.... ..) is sequence of tokens from the vocabulary Σ. The action space is the vocabulary itself, A = E. A policy Ta corresponds to the pre-trained I.M,, which defines distribution over the next token conditioned the current prefix: (o( (x)) = po 1 1 rt). We adopt the tags used in (Guuo et al. (025), ie., <think> </think> and <answer> </answer>, distinguish intermediate reasoning from final answers. The reward function is sparse: r(s, s') 0 for all intermediate states, and r(s, a, s') r(y) only when the final predicted answer y wrapped by <answer> </answer> is completed. The goal is to optimize the policy to maximize the expected cumulative reward, encouraging the model to generate correct and well-formatted answers through effective reasoning.
over discounted Markov Decision Process (MDP) M= =
3.1 TASK DECOMPOSITION FOR RL POST-TRAINING
While SFT provides strong supervision signals, Chu et al. 2025 suggest that reasoning and general- ization ability are more effectively enhanced through RL-based post-training. However, applying RL techniques similar to SeepSeek-R--Zeere (Guo et al. 2025 to leam complicated reasoning tasks remains challenging. In this work, we analyze these challenges in two key aspects, the distribution gap and reward design.
Challenge 1: Distribution gap. Learning tasks that exceed the base LLM's reasoning capabilities introduce significant learning challenges. These challenges are often caused by non-trivial distribu- tion gap between the model pre-training source data distribution de and the target data distribution dg. As shown in Fig. 2 because rewards are only given for correct outputs. large distribution shifts can lead to low accuracy and sparse reward signals (Shao et al.) 2024, Zeng et al. 2025). Moreover, fitting the model a single target distribution can lead to overhtting and memorization, undermining the model's generalization and reasoning ability.
Challenge 2: Reward design. Challenging reasoning tasks often require LLMS to combine multiple skills to arrive at solution. While designing fine-grained, step-by-step reward function could potentially guide the model effectively, such design is generally task-specific and labor-intensive. For example, computer science student is supposed to learn basic mathematics and linear algebra before learning machine learning. Similarly, a typical Countdown (Gandhi et al.) 2024) task involves skills like basic arithmetic, estimating the distance to the goal, and backtracking. While it is possible to include a supervision signal for each intermediate step, doing so is not sustainable and generalizable across diverse tasks.
3

---

Published as a conference paper at ICLR 2026
To overcome these RL learning challenges, we propose task decomposition by splitting training data into subsets of increasing difficulty, based on either human annotations or model performance. This aligns with curriculum learning, where we interpolate between the pre- pre-training distribution dg and K target task distribution dx via intermediate stages d= reducing distribution shift and improving training stability (see Fig.2). 2). From the reward design perspective, decomposing tasks by difficulty breaks complex skill acquisition into simpler steps. For instance, the 6-number Countdown task involves using six integers and four operators (+, x, +) to reach a target number, requiring the model to perform arithmetic, estimate distances to the goal, and backtrack effectively. In contrast, a 2-number problem focuses primarily on arithmetic, allowing the model to build foundational competence before scaling harder variants. This avoids handcrafted reward shaping and improves transferability.
For tasks like Blocksworld, MATH, and Countdown, we use human-aligned difficulty signals such as plan length, labeled difficulty, or operand count (see Table For others like AQuA and GSM8K, where such annotations are unavailable, we automatically estimate difficulty based on model error rates under CoT prompting and group examples accordingly (see Figs. and 6). We generate 20 responses per question using I-shot CoT prompting, with prompts and in-context examples from (Hao) ct al. 2024 Parashar et al. (025). and assign difficulty by grouping questions based quartiles of the error rate distribution.
In this work, we tackle the challenge of learning hard reasoning task by partitioning the training distribution into four difficulty levels, namely, trivial, easy, medium, and hard, then we adapt our pre-trained LLM these tasks in sequential curriculum. In Section 3.3 we provide a theoretical justification that, under fixed sampling and training resources, learning step by step leads to better performance than training directly on the hard task.
Cosine Sampler
3.2 TRAINING SCHEEDLERRS FOR LLM REASONING ABILITY
0.40
Task 1
While task decomposition simplifies RL post-training, traditional curriculum learning poses two main challenges, mainly task forget- ting and overfitting, due to the rigid progression through tasks in a fixed task order.
0.35
Task 2
Probabiltt
0.33
Task
lask 4
0.25
0.20
0.15
Task forgetting and overfitting. The first challenge, task forgetting, refers the degradation in performance on earlier (easier) tasks as the model adapts to later (harder) tasks. According to the traditional model generalization literature (Arjovsky et al.. 2019; Gulrajani & Lopez-Paz 2021), the task distribution shifts are considered as explicit signals for the model generalization direction; scheduling. thus, retaining strong performance across all task distributions is essential for generalization. There- fore, the task forgetting will undermine the model's generalization capability, i.e., the reasoning ability. Task overfitting, the second challenge, arises when the model overfits to trivial tasks and prefers simplistic patterns or short answers that bypass meaningful reasoning. This phenomenon is called reward hacking (Laidlaw al. 2025), where the model exploits shortcuts easy tasks and fails to learn harder ones, resulting in poor reasoning performance. To address both challenges, we explore different training sampling techniques, forming four different scheduling strategies as follows.
Challenges:
0.10
200
400
600
800
1000 [-T
t-0
t-T/3
t-27/3
Step
(U)
Figure 3: Illustratioo of cosine
Traditional scheduling. We first formulate the traditional sequential curriculum learning sampler with T training steps as (t, k) = 1 when Tk < t < Tk+1, otherwise, O, where & denotes the current training step; for K tasks k I..... K, Tk denotes the threshold when the curriculum learning proceeds to the k-tth stage, while = 0 and Tx+1 T. The output of the sampler denotes the probability of sampling data from the k-th task; therefore, at the step, the sampling distribution ... (4, K)].
will be (Smaa (,, 1), Sma (1, 2)....
Balanced scheduling. To avoid forgetting, the simplest way is to mix all data with different difficulties together and sample randomly, which can be considered as a trivial case of curriculum learning. Alternatively, this can be interpreted as the default behavior of any policy optimization algorithm (Shao et al. 2024 | Schulman et al. 2017). where training occurs without considering task difficulty. This balanced sampler can be written as: Statanced (t, k) = K: where each task difficulties have the same probability to be selected at each training step. Although this is an efficient way to avoid forgetting, this sampling introduces harder tasks too early, leading to sparser rewards and suboptimal CRI..
4

---

Published as a conference paper at ICLR 2026
Teak
Bak Tesk 2
lak
Toak
04
+
200
1000
202
400
200
450
32g
27
Step 112
Stop (3)
Step (0)
Gaussian sampler
0.25
0L5
0.75
Figure 4: Gaussian Sampler. (a) This figure represents the Gaussian sampling process. (bed) These figures denote the sampling probabilities of different tasks changing along the training steps with different Gaussian sampler hyperparameters.
Cosine scheduling (E2H-C). To alleviate both the reward sparsity and forgetting issues, we propose non- non-parametric scheduling strategy, namely, cosine sampling. This strategy can be written as: Sossiee (t. k) a₂ (K - k - 1) + (1 oz) k, and a₁ = 0.5 (1 + cos( (7)), where the resulting probabilities need to be normalized before sampling. Intuitively, this cosine sampler sets both the initial and ending sampling probabilities simply according to their ordinal number and interpolates the intermediate probabilities using a cosine function. As shown in Figure the easiest task has the highest probability of being sampled at the beginning, and has the lowest probability of being sampled at the end.
Gaussian scheduling (E2H-G). Although cosine scheduling addresses reward sparsity and forgetting, the parameter-fre free design limits flexibility handling issues like trivial task overfitting and fine- grained control over different learning stages. Empirically, while adding the trivial task can boost the model performance, is also easy for the model to overfit to trivial tasks. To overcome this challenge, we propose Gaussian scheduling strategy inspired by the Gaussian mixture model (Reynolds et al.. 2009).
As shown in Figure (a), in one-dimensional space, we assume the data distributions of tasks follow Gaussian distributions with the same variance a. The means of the adjacent task's Gaussian distributions are assumed have the same distance 1, i.e., f4 k Then the sampling probability is defined as the likelihood of a given position 2, belonging to different task Gaussian distributions, similar the expectation maxximizatiinnnn algorithm (McLachlan & Krishnan 2008). Therefore, the Gaussian scheduling strategy can be expressed as:
SCaussian (t, k)) exp (- (x++ - 12x)3 [1x)²
and It = - (K-1),
( 20
where we ignore the normalization term, and the probabilities will be normalized for sampling. In this sampling scheduler, only have two hyperparameters, i.e,, and B. While the variance controls the sampling concentration, 3 controls the sampling position xyy moving speed. When is smaller, the training is more focused, more similar traditional curriculum learning. When 3 < 1, the sampling process will assign fewer training steps focusing on easier tasks and train harder tasks longer, avoiding easy task overfitting, As shown in Figure (bed), we use three typical hyperparameter settings for this Gaussian sampler, i.e., B = 0.25 a = 0.75, 3 0.75 a 0.25, and 0.5 0.5.
3.3 THEORETICAL ANALYSIS
In this section, we analyze our curriculum reinforcement learning framework for LLMs under the lens of Approximate Policy Iteration (API). We follow the theoretical structure of Chen & Maguluri (2025); (Scherrerrrrrrre (2014), adapting it to our curriculum setting. Specifically, we show how sequentially solving interpolated curriculum distributions enables convergence guarantees and improved sample complexity bounds, under function approximation errors. Our analysis considers action-value functions and explicitly tracks the impact of approximation and distribution shift across curriculum stages. All proofs are presented in Appendix
3.3.1 THEORETICAL SETUP
Recall our MDP M (S, A, P,r,7). We denote the policy space as II, where each policy π E II is a mapping from states to distributions over actions, म : S A (AA. In our curriculum setting, we introduce sequence of MDPs (M&) K where each MDP shares the same state and action spaces
5

---

Published as a conference paper at ICLR 2026
but may differ in reward functions and/or transition dynamics. Each MDP induces state visitation distribution dę under the optimal policy for that curriculum, with {d ) interpolating between an easy source distribution di and the final hard task dx. Let πą. be the learned policy at curriculum step k, and let k be the optimal policy under the final target task. The goal is to learn a sequence of policies {TT Jemi such that the final policy т performs well under the target distribution dx-
The action-valus E E y'r(st, as)lsg = S, au = a, a ~ T(118)). 8141 ~ P(-[st,a)) Let T be the Bell- man operator for policy т, defined as (=)(s.a) == r(s, a) -(sss((sss)))a st) (Q(s',a')] We define the optimal Bellman operator T, TQ(s, a) == r(s.a) a) + + s.a) [max, ((s(,a')). The fixed point of the optimal Bellman operator is TQ" = =Q-
function of
policy
is defined
(T(s, a)
The Approximate Policy Iteration (API) framework is generalization of classical policy iteration that accommodates function approximation and inexact updates. serves as a foundational tool for analyzing practical reinforcement learning algorithms including actor-critic and deep RL methods. The API algorithm alternates between two stages. At iteration k, the algorithm performs two algorithmic steps. (i) Policy Evaluation: Given policy Tk. compute an approximate estimate Qx of its action-valle value function Q**, (ti) Policy Improvement: Update the policy to Tk+11 which is greedy or approximately greedy with respect to Q+- In the context of CRL., API offers a natural framework to study how sequentially adapted policies, defined over interpolated curriculum distributions, evolve toward the optimal policy and shape the final policy π--
We adopt the following API assumptions from Chen & Maguluri 2023); ; Scherrer (2014), adapted to the curriculum setting. Let Qk := Q be the action-value function for the policy Ti at curriculum k.
Approximate Policy Evaluation. At each iteration k; a function approximator Q6 is used to estimate Qk. The estimated Q-function Q satisfies || QF |l I 62.
Approximate Greedy Policy Improvement. Let T4+1 be an eg-gredyy policy with respect Qx: Eg-pa [QTH: (s, (k))))))))) 2 Egeup max, Qx(s, a) Ck2 for sampling distribution (4 at step
Thi1 k Le.. ITQ - 11x0 <e.-
Distribution Mismatch (Concentrabilltyy) Let μ4 be the sampling distribution for policy improve- ment at step k and de the true task distribution. We assume the distribution mismatch between μ*x da(a)
and dy is bounded by Ck, ie., supses px(e) < Ck.
Bounded Curriculun Drift. For the weighted norm, ||Q] /E [max, Q(s, a)a]. The deviation between successive optimal Q-funetions satisfies |Q Qillax bounded for all k:
3.3.2 CONVERGENCE GUARANTEE
Let Q be the optimal Q-function for the final task under distribution dx. and let Q be the Q-function of the policy learned at the final step. Define the performance loss of the final policy compared to the optimal target policy as: εK := 1|Q Q l|
Theorem 3.1 (CRL Performance Guarantee). Let T the number of API policy updates within each task. B > O is tunable parameter for stepsizes specified in Chen & Maguluri (2025). Under the approximate greedy update and evaluation error assumptions above, the final performance gap EK satisfies:
2y(1 (7(1-77) 7)-
27 B(1
K-1
EK
S
Σ k=1
(T
6y
+
Σ
1 - Qi llxx
7/2
0% 1loo is the per-task Bellman error, d, is the evaluation error, and en is where Th : |1Qi
absorbed in 77--
The first term represents the convergence bias of the actor, and goes to zero geometrically fast as T' goes to infinity. The second term captures accumulated evaluation errors, which involves the stochastic error due to sampling and the error due to function approximation The third term captures the error introduced by the policy update rule, which can be made arbitrarily small by using large enough B. Alternatively, we can use geometrically increasing stepsizes, in which case the third geometric rate. The last term captures the deviation in optimal Q-functions
term goes

---

Published as a conference paper at ICLR 2026
across curriculum stages, representing the cumulative gap between intermediate cerriculum-optinee values and the final optimal value, which we refer to as the curriculum approximation error. This decomposition highlights the dual effect of CRL in improving sample efficiency (small (g.Eg) and ensuring smooth interpolation (small |[ 1x
3.3.3 FINITE SAMPLE APPROXIMATION ERROR ANALYSES
In this section, we perform finite-sample analysis. Considering the critic, i.c., how to obtain an estimate of Q", we can estimate the Q-function Q" of given target policy using TD-Jearning In TD-learning especially when the acttin space size is large, the use of function approximation natural. In linear function approximation, a set of basis vectors is selected to approximate the target value function Q" using linear combinations of the basis vectors. Let c E [|S|||| be the matrix of basis vectors & [01, Ød). Then, the goal is to find from the linear subspace = &w E J the "best" approximation of the Q-function Q", where E Rd is the weight vector.
Following Section 3 and 4 of Chen & Maguluri (025)) derive the inite- theorem for CRL. Theorem A.I and its analyses are provided in Appendix [A.1.I] The result of Theorem [A.] suggests that smoother curriculum trajectories, together with smaller approximation errors under finite samples, can lead to improved final policy performance. These effects are closely tied to the choice of stepsize and the number of updates per iteration.
Following Chen & Maguluri 2025). let K.S.A E I (|]AAx(S|]LA be diagonal matrix and let K.S.A.min be the minimal diagonal entry. Js is the number of critic updates per policy iteration at curriculum The bootstrapping parameter n is chosen such that "o = 7²/ K.S.A.min 1. L the parameter defined by 1+ (1/Pmux,)) Eapprox,K == sup, || - "w II is the critic's 's function approximation error, |Qx - Qill is the gap between the curriculum subtask and the final task, and Amin is the
mininum eigenvalue of the positive definite matrix &7 The term N2, -k=1 (1->),
24E in
Theorem is the function approximation error and equals zero when using complete basis. term of similar form is present in all existing work studying RL with function approximation (Munos)
2003 Agarwal al. 2021). Based on Theorem A. we next derive the sample complexity result.
Theorem 3.2 (Sample Complexity). For a given accuracy level > O, to achieve EllQK
QT [oo] I e e N2,1 + Σk-1 - Qillax the total number of samples (i.e., the integer TE k= Jk. required across all K curriculum stages is of the order:
log (1/ea) Ō ( (11-)) - 70) 33
Lin
2
7)7(1
where EN is the target accuracy for curriculum k. and Ek=1 Ex
Let McRI. = McBI==
Me be the total number of samples needed by CRL with K curriculum steps to achieve error eon the final task, and let Mimet = Mx be the number of samples needed by direct learning, where m > 1 represents the relative difficulty factor of direct learning. Under geometric error and L3 allocation EN EK e-- K and Li TH for curriculums, we have:
(e* ()) -1
McRL < MDrrect
< m 1
1 1 (e * 1)2
The geometric error and L. allocation reflect the curriculum gradually increases in difficulty while allowing larger errors in earlier stages. Since the function fx) 1-x² monotonically decrease for x. > when K is integer larger than 1, the final condition can be reasonably satisfied, e.g., with K = = 1.4, and 1.8 in practice.
The first half Theorem 3.2 analysis highlights the dual benefit curriculum design in CRL. First, by constructing intermediate distributions .i....,dx-- close dx. the curriculum error term ||Qr - 2 can be made small. Second, easier curriculum tasks improve estimation accuracy and yield more stable approximate greedy updates, enhancing sample efficiency and policy improvement. For the second half of Theorem 3.2] the final condition in Eq. is satisfied when curriculums are appropriately learned, the allocation of accuracy targets eg is gradually optimized across curriculum
7

---

Published as a conference paper at ICLR 2026
Table Impact of task decomposition LLM post-training Trivial and easy examples help the model leam core principles that enable success on harder tasks.
Blocksworld
Coundown
Trivial
Easy
Med
Hard
0OD
Trivial
Easy
Med
Hard
0OD
Trivial
Easy
Med
Hard
OOD
Hard
0.0
0.0
0.0
0.00
0.0
0.0
43.9
16.4
18.1
6.5
82.3
64.7
53.4
38.2
20.6
Med Hard
2,0
0.0
12.9
47.8
19.2
68.9
58.1
42.5
21.0
Easy
55.5
79.3
21.2
42.5
21.5
Trivisl + Fasy Mod
Hord
98.0
83.
21.1
2.6
96.1
64.9
28.8
18.1
9.3
87.2
72.0
61.6
46.3
25.7
Table 2: Effect of scheduling strategies LLM post-training. We compare balanced scheduling, traditional curriculum learning (CL), and our proposed E2H Reasoner variants, namely, E2H-G and E2H-C. CoT is reported as reference.
Blocksworld
Counddown
MATH
Trivial
Easy
Med
Hard
OOD
Trivial
Easy
Med
Hard
OOD
Trivial
Easy
Med
Hard
GOD
0.0
16.0
27.9
8.2
Balanced
98.0
100
84.5
26.3
5.3
96.1
64.9
28.8
18.1
9.2
87.2
72
61.6
46.3
25.7
CI.
46.0
100
45.2
5.8
0.7
57.7
85.8
57.2
31.5
12.6
86.2
71.5
62.4
46.7
28.6
E2H-G (0.25.
51.4
7.3
72
47.9
E2H-G (0.5, 0.5)
100
100
34.5
10.5
0.7
97.9
87.2
70.4
41.0
19.2
85.3
71.7
62.5
48.7
27.6
R2H-G (0.75, (.25)
98.0
93.3
17.9
2.0
0.0
95.7
56.0
28.8
17.1
10.2
86.0
72.4
62.0
46.7
26.3
E2H-C
100
15.5
0.0
96.7
64.0
25.9
15.8
6.4
84.6
69.5
47.6
steps, and the designed curriculums with increasing difficulties effectively bridge the gap between source and target distributions. This mathematical derivation shows CRL requires fewer total samples than direct learning on the final task, aligning with experimental observations (see Appendix
4 EXPERIMENTS
We conduct experiments investigating the following research questions. RQ1: What role does task decomposition play in RL-based post-training? RQ2: How does task scheduling impact the learning process? RQ3: Can small-scale LLMs learn to reason on hard tasks?
4.1 EXPERIMENTAL SETUP
We evaluate method diverse set of reasoning and planning tasks, covering both datasets with and without human-annotale annotaaeed difficulty levels. For datasets with difficulty labels, such as Blocksworld (using plan length) (arth)(Valll (Valmeekam et al. 2023). Countdown (using number of ) et al. 2025), and MATH (using problem levels) (Hendrycks et al. 1). we use the provided annotations and define difficulty, namely GSM8K (Cobbe et al. 2021) and AQuA (Ling et al. 2017a). we estimate difficulty using the zero shot error rate of the base model. Specifically, for each question in the training set, we generate 20 responses and compute the error rate as Number of Correct Responses and bucket the samples into trivial, easy, medium, and hard, based on quartiles Fig. More dataset details are in Appendix C We conduct experiments on Qwen 2.5/1.5B Instruct (Yang et al. 2024), LLaMa 3.2 3B Instruct (Grataafioi et al., (2024). More models are included in Appendix G.I research questions RQI and RQ2.
-iti----iittttttioonn (OOD) split to assess generalization. For datasets without explicit
We use Qwen 1.5B for
4.2 BASELINES
(CoT) performance for all models. All post-training experiments use GRPO (Shao et al. (024) as the reinforcement learning algorithm (see Appendix E for implementation). GRPO, by default, employs balanced scheduling over all tasks, which we use as our baseline and refer to as GRPO in Table We also assess whether models can learn directly from the most challenging examples by training only on the Hard and OOD subsets. In addition. we evaluate traditional curriculum learning (CL) as used by Team et al. (2025); Bercovich et al. (2025). We compare against Self-Evolve (Chen al. (205), an adaptive curriculum baseline that samples problems with a 50% solve rate to maximize learnability. Finally, we provide comparison against SFT in Appendix
First, report the Chain of Thought (Wei al. 2022)
4.3 EXPERIMENTAL RESULTS
We conduct experiments addressing the research questions listed above. We examine how task decomposition impacts LLM post-training using Qwen- 1.5--1.5B-Insstue (RQ1) with a balanced scheduler (Table T). We find that including trivial and easy examples helps the model build core skills. This
8

---

Published as a conference paper at ICLR 2026
Table 3: Results of E2H Reasoner across three models on Blocksworld Valmeekam et al. 2023), , Count- down (Gandhi et al. 2 and MATH(Hendrycck et al. (2021). Our method consistently improves performance especially on HARD and OOD tasks, demonstrating effective reasoning. results on more models are in Ap- pendix G.T
Best numbers are in bold and second-best are underlined.
Blocksworld
Countdown
MATH
Trivial Facy Med Hard
00D
Trivial
Easy
Mod
OOD
Trivial Easy
ODD
CoT
4.0
0.0
0.0
0.D
0.0
16.0
5.6
1.7
0.1
0.1
40.1
27.9
22.7
17.6
8.2
GRPO (AIl)
83.3
21.1
720
25.7
GRPO (Hard)
0.0
0.0
0.0
0.0
0.0
0.0
43.9
16.4
18.1
6.5
82.3
64.7
53.4
38.2
20.6
Qwen 1.5B Instract
GRPO (OOD)
0.0
0.0
0.0
0.0
0.0
3.1
23.1
18.1
11.3
5.3
37.0
21.1
15.0
8.5
3.7
12.6
Self Evolve
100
70.2
13.8
65.3
34.2
17.8
70.6
62.6
48.6
26.1
F2H-G
98.0
100
95.3
32,9
7.3
97.9
87.2
70.4
41.0
19.2
85.3
11.7
62.5
48.7
27.6
EZH-C
100
15.5
28.6
CoT
24.0
0.0
1.2
1.0
0.0
37.1
4.6
0.3
0.0
0.0
65.9
44.5
35.2
24.1
13.6
GRPO (AIl)
100
100
94.1
38.9
13.3
89.5
47.9
65.9
47.0
36.0
10.2
GRPO (Hard)
0.0
0.0
0.0
40.5
33.8
3.0
9.7
22.7
14.
10.3
7.5
LLMMa 3.2 38 Instrnct GRPO (OOD)
0.0
0.0
0.0
0.0
0.0
8.0
D.3
0.0
0.0
0.0
63.6
39.0
$1.3
19.0
7.6
CL
100
0.0
0.0
0.0
17.2
36.0
22.7
11.2
4.1
74.1
54.1
43.9
28.0
12.5
Self-Evvolve
100
91.1
35.8
37.9
27.5
18.5
79.1
61.4
E2HIG
100
100
98.8
44.1
17.6
95.0
89.9
73.3
46.5
24.3
78.7
58.4
46.4
32.3
14.5
F2H-C
100
0.0
55.3
0.0
74.8
60.6
15.8
Table 4: Performance of E2H Reasoner GSM8K and AQuA, where difficulty splits de- rived from error rates due to the absence human lahels. Fig. shows these splits, and Table T5 con- firms robustness to the number of splits..
Figure 5: GSM8K difficulty distribution based error rates. Difficulty groups used for training are derived from quartiles this distribution.
Rate Distribution by Difficulty Bucket
Qwen 1.5H Instruct
500
Medium
GSM8K
AQUA
Number Oueemtre
500
Trivial Easy Med Hard Aug Trivial Easy Med Hard Avg
400
CaT
90.2
87.3 76.5 38.1 67.7
70,8
$1.3 20,8
2.6-40.9
GRPO
95.8
23.0
200
CL
98.0 97.2 85
52.2 78.6
88.8
72.2
18.4 58.6
Self-Pvolve
28.1 95.3
87.0
57.3
94.4
75.0
40.3
61.6-64.2
200
E2H-G
97.6
89.0
$1.8 78.7
90.2
43.0
34.2 66.1
100
E2H-C
98.0
95.3
83.9
75.7
86.1
48.6
26.3
62.5
0.6
Error Rate
enables effective transfer from simpler to harder tasks and better OOD performance, consistent with our view of reasoning as learning core principles and applying them to harder tasks.
Next, we examine how scheduling impacts post-training (RQ2) in Table. 2 Balanced scheduling serves as a strong baseline but lacks structure, leading to suboptimal learning. CL schedules tasks in fixed difficulty order, which can cause forgetting of earlier tasks or overfitting to easier ones. In our experiments, easier tasks help initiate learning due to their dense rewards, but overexposure hinders generalization to harder, sparse-reward tasks. We address this through our two schedulers, i.e., E2H-C and E2H-G (see Section (.22). For tasks like MATH, where models perform reasonably well across all difficulty levels (as seen with CoT in Table the cosine scheduling in E2H-C is effective, beginning with a focus on trivial and easy tasks and gradually shifting toward harder ones. However, on tasks like Blocksworld, where rewards are sparse since harder examples are more challenging, this leads to overfitting and degraded performance. E2H-G addresses this issue by using a Gaussian schedule that quickly decays the sampling probability of trivial and easy tasks. As shown in Fig. 4 provides enough exposure early on to support initial learning while rapidly shifting focus to harder examples. This prevents overfitting and improves generalization in sparse- reward settings where E2H-C struggles.
Next, we investigate whether models can learn to reason directly from difficult examples (RQ3). As shown in Table [ the answer is largely no. For instance, Qwen-2.5 1.5B when trained on Level 5 MATH directly, underperforms the CoT baseline on the model! This failure highlights the need for CRL methods for LLM reasoning. To this end, we compare with CRL method, Self-Evolve 3). Empirically, we find that its reasoning performance varies significantly across models and tasks. Self-Evolve during training, samples problems at a 50% success rate maximize learnability and gradually reduces easier ones once this threshold is surpassed. However, being only halfway proficient at solving a problem is often not enough. For example, learning calculus requires strong

---

Published as a conference paper at ICLR 2026
Table 5: Ablation E2H on GRPO DAPO. E2H improves overall performance over both baselines and
yields consistent gains when combined with DAPO, indicating that the two approaches are complementary.
Blocksworld
Countdown
MATH
Method Variant
Triv.
Easy
Med.
Hard
OOD
Triv.
Easy
Med.
Hard
OOD
Triv.
Easy
Med.
Hard
OOD
Baseline
98
100
83.3
21.1
2.6
96.1
64.9
28.8
18.1
9.2
87.2
72.0
61.6
46.3
25.7
GRPO
EZH-G
98
100
95,3
32.9
7.3 97.9
87.2
70.4
41.0
19.2
85.3
71.7
62.5
48.7
27.6
E2H-C
100
100
15.5
0.0
0.0
96.7
64.0
25.9
15.8
6.4
84.6
69.6
63.0
47.6
28.6
Baseline
100
100
89.3
24.3
3.4 96.5
67.3
35.7
22.0
10.9
85.6
73.1
63.3
47.9
28.3
DAPO
F2H-G
100
100
98.8
46.7
9.3
99.9
90.6
82.4
59.4
30.1
86.3
72.7
64.4
49.1
28.5
E2H--C
100
100
96.4
44.1
8.7
96.5
64.1
26.0
16.7
8.7
86.3
72.3
64.5
50.2
29.2
Table 7: Pass Performance on AIME24 Table 6: Ablation of E2H on GRPO vs DAPO on and OlympiadBench for models trained on MATH.
GSM8K and AQuA.
AQUA
GSMKK
Model
Method
AIME24
OlympladBench
Method Variant
Triv. Easy Mod. Hard Avg Triv. Easy Med.
Hard Avg
6a Pass
Pass
Baseline
95.8
84.1
49,9
GRPO
3.3
10.0
7.3
33,3
GRPO
E2H-G
E2HIC
90.2
86.1 72.2
81.9 43.0
26.3
34.2
66,
97.6
98.0
94.7
83.9
89.0
46.6
78.7
Qwen 1.58
Self Dvolive
E2H-G
6.
16.7
8.7
39.3
37.0
E2H-C
6,7
16.7
10.0
40.0
Baseline
94.4 72.2 48.6
31.5
65.7
97.0
98.0
87.5
52.0 79.0
DAPO
E2H-G
95.8
81,9 51.3
28.9
69.3
99.1
97.3
89.3
53.2 80.1
GRPO
0.0
3.3
3.7
22.0
E2H-C
95.8 75.0
55.5
368
69.6
98.5
97.0
88.1
78.0
LLaMA 3B
Self Evolive
E2H-G
3.3
10.0
5.3
28.6
29.3
E2H-C
6.7
10.0
6.0
30.0
command of school mathematics. In addition, implementing Self-Evolve also requires careful tuning of hyper-parameters for different models and datasets.
In contrast, E2H Reasoner guides learning scheduling tasks from easy to hard, improving generaliza- tion, as reflected in stronger OOD performance (RQ3). This is further supported by the results, where E2H Reasoner shows better performance over baselines as task difficulty rises. Note that for E2H-G we report the best numbers out of our 3 parameter settings with extensive results in Table 13 and does not need extensive hyper-paramedt parameter tuning. Finally, E2H Reasoner remains effective even without human difficulty labels by using error rates as proxy for difficulty (Table
In Table 5 we compare against DAPO, which builds on GRPO by filtering out problems that are either too easy (all rewards 0) or too hard (all rewards 1). In principle, if DAPO were allowed to resample up to N times (where N is the dataset size) before forming each batch, it would always find the perfect batch, but this is computationally infeasible for training. Our E2H Reasoner mitigates this issue by gradually focusing on harder task groups through probabilistic difficulty- based scheduling. As shown in Figure 7 combining E2H with DAPO reduces the the fraction of batches with zero advantage during training, indicating that it selects prompts whose overall difficulty is better aligned with the model's current competence. Taken together, these results show that E2H and DAPO are complementary: E2H shapes the difficulty conditioned sampling distribution that DAPO resamples from, and their combination yields the strongest performance across benchmarks.
Finally, we conduct an extended evaluation of our models trained on the MATH dataset using E2H and vanilla GRPO on two challenging evaluation benchmarks, AIME24 (Veeraboina) (023) and OlympiadBench (He et al. 2024). E2H demonstrates stronger generalization to both tasks in Table
5 CONCLUSION
We introduce the E2H Reasoner (E2H), a CRL-based method for LLM post-training. E2H enables models to learn tasks they initially failed at by scheduling tasks from easy to hard. E2H challenges the assumption that small LLMs cannot reason and demonstrates strong empirical performance supported by theoretical analysis, offering convergence guarantees and improved sample efficiency over direct RL. In summary, E2H provides a scalable, theoretically grounded, and practical method for LLM reasoning.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Alekh Agarwal, Sham M Kakade, Jason D Lee, and Gaurav Mahajan. On the theory of policy gradient methods: Optimality, approximation, and distribution shift. Journal of Machine Learning Research, 22(98):1-766 2021.
Martin Arjovsky, Léon Bottou, Ishaan Gulrajani, and David Lopez-Paz. Invariant risk minimization. arXiv preprint arXiv:1907.0289) 2019.
Sanghwan Bac, Jiwoo Hong, Min Young Lec, Hanbyul Kim, Jeong Yeon Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning, arXiv preprint 0333:0 2025.
Yoshua Bengio, Jéröme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ICML, pp. 41-48, 2009.
Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-iv,, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeh Nabwani, et al. Llama-nemotron: Efficient reasoning models. arXiv preprint arXiv:2505.0004:
2025.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for I1m reasoning. arXiv preprint arXiv:2505. 14970, 2025.
Zaweei Chen and Siva Theja Maguluri. An approximate policy iteration viewpoint of actor-critic algorithms. Automatica, 179:1 12395, 2025.
Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V Le, Sergey Levine, and Yi Ma. Sft memorizes, generalizes: A comparative study of foundation model post-training. arXiv preprint 10VVv:25011176611 2025.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchrr Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv preprint 11.1:2110..1168, 2021.
Thomas Foster, Anya Sims, Johannes Forkel, Mattie Fellows, and Jakob Foerster. Learning to reason at the frontier of learnability. arXiv preprint arXiv:2502. 2025.
Kanishk Gandhi, Denise H Lee, Gabriel Grand, Muxin Liu, Winson Cheng, Archit Sharma, and Noah Goodman. Stream of search (sos): Learning to search in language. In CoLM, 2024.
Kanishk Gandhi, Ayush Chakravarthy, Anikait Singh, Nathan Lile, and Noah D Goodman. Cognitive behaviors that enable self-improving reasoners, or, four habits of highly effective stars. arXiv preprint arXiv:2503.013007 2025.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21778. 2024.
Alex Graves, Mare G Bellemare, Jacob Menick, Remi Munos, and Koray Kavukcuoglu. Automated curriculum learning for neural networks. In ICML, pp. 1311-1320, 2017.
Ishaan Gulrajani and David Lopez-Paz. In search of lost domain generalization. In ICLR, 2021.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in LLMs via reinforcement learning. arXiv preprini arXiv:2501. 2025.
Shibo Hao, Yi Gu, Haotian Luo, Tianyang Liu, Xiyan Shao, Xinyuan Wang. Shuhua Xie, Haodi Ma, Adithya Samavedhi, Qiyue Gao, Zhen Wang, and Zhiting Hu. LLM reasoners: New evaluation, library, and analysis of step-by-ster reasoning with large language models. In First Conference on Language Modeling, 2024. URL http:://ppneeeeeee...eeoommmdddd

---

Published as a conference paper at ICLR 2026
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang. Jie Liu, Lei Qi, Zhiyuam Liu, and Maosong Sun. OlympiadBench: A challenging benchmark for promoting AGI with olympiad problems. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 3828-3850, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10.18653/v hotp://////2224..aooooog....... 211. URL URL http://///daaatooooyyoo/000000-0
bilingual multimodal scientific
Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. In NeurIPS, 2021.
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou. Large language models cannot self-correct reasoning yet. In ICLR, 2024.
Aaron Jacch, Adam Kalai, Adam Lerer, Adam Richardson, Ahmed El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel, Alex Carney, et al. Openai ol system card. arXiv preprint arXiv:2412.16720. 2024.
Cassidy Laidlaw, Shivam Singhal, and Anca Dragan. Correlated proxies: A new definition and improved mitigation for reward hacking. ICLR, 2025.
Hongyi Ling, Shubham Parashar, Sambhav Khurana, Blake Olson, Anwesha Basu, Gaurangi Sinha, Zhengzhong Tu, James Caverlee, and Shuiwang Ji. Complex LLM planning via automated heuristics discovery. arXiv preprint arXiv:2502. 19295, 2025.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In ACL, pp. 158-1672 2017a.
Wang Ling, Dani Yogatama, Chris Dyer, and Phil Blunsom. Program induction by rationale genera- tion: Learning to solve and explain algebraic word problems. In ACL, pp. 158-167, 2017b.
Geoffrey McLachlan and Thriyambakam Krishnan. The EM algorithm and extensions. John Wiley & Sons, 2008.
Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannaneh Hajishirzi, Luke Zettlemoyer, Percy Liang. Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint UrXiv:2501.19395 2025.
Rémi Munos. Error bounds for approximate policy iteration. In ICML, pp. 560-567, 2003.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal of Machine Learning Research, 21(181):1-55 2020.
Shubham Parashar, Blake Olson, Sambhav Khurana, Eric Li, Hongyi Ling, James Caverlee, and Shuiwang Ji. Inference-time computations for LLM reasoning and planning: A benchmark and insights. arXiv preprint arXiv:2502.115221 2025.
Chenhao Qiu, Qianglong Chen, Jintang Li, Caiyu Wang, Runsen Hua, Minghui Li, Shengshan Hu, and Yechao Zhang. WISDOM: Progressive curriculum synthesis makes LLMs better mathematical reasoner, 2025. URL http:://peneeveew.e//ooumm///iie Did
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. NeurIPS, 36:53728-5377 53741, 2023.
Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663):3, 2009.
Abulhair Saparov and He He. Language models are greedy reasoners: A systematic formal analysis of chain-of-thouu thought. In ICLR, 2023.
12

---

Published as a conference paper at ICLR 2026
Bruno Scherrer. Approximate policy iteration schemes: A comparison. In ICML, pp. 2014.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint vrXivv1707.06344 2017.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.0338 03300, 2024.
Charlie Victor Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling LLM test-time compute optimally can be more effective than scaling parameters for reasoning. In ICLR, 2025.
Kaya Stechly, Karthik Valmeekam, and Subbarao Kambhampati. Chain of thoughtlessness? an analysis of cot in planning. In NeurlPS, 2024.
Fahim Tajwar, Yiding Jiang, Abitha Thankaraj, Sumaita Sadia Rahman, Zico Kolter, Jeff Schneider, and Russ Salakhutdinov. Training generally curious agent. In Forty-second International Conference on Machine Learning, 2025. URL http:://pppmeees....rrrooorrooda
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi k1. Scaling reinforcement learning with LLMs. arXiv preprinl wrXiv22501.25590 2025.
Karthik Valmeekam, Matthew Marquez, Sarath Sreedharan, and Subbarao Kambhampati. On the planning abilities of large language models-a critical investigation. NeurIPS, 36:75993-76005.
2023.
Karthik Valmeekam, Kaya Stechly, and Subbarao Kambhampati. Lims still can't plan; Irms? preliminary evaluation of openai's ol on planbench. arXiv preprint arXiv:2409.13375 2024.
Hemish Veeraboina. Aime problem set 2983-2024, 2023. URL http::///www.kggglcco///ussss
harpaiiiinaaiimemprole-e-eeeeeee 1983- 2024
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V Le, Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In ICLR, 2023.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, et al. Chain-of-thoughh prompting elicits reasoning in large language models. NeurlPS, 35: 14824-24837 2022.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing LLM reasoning with rule-basee reinforcement learning. arXiv preprint 2rXiv:2502..4777 2025.
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint 2411::4412.11155. 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. NeurlPS, 36:1 1809- 11822, 2023.
Fei Yu, Hongbo Zhang, Prayag Tiwari, and Benyou Wang. Natural language reasoning, a survey.
ACM Computing Surveys, 56(12):1 2024.
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, et Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14477 2025.
Yang Yue, Zhiqi Chen, Rui Lu, Andrew Zhao, Zhaokai Wang. Shiji Song, and Gao Huang. Does reinforcement learning really incentivize reasoning capacity in Ilms beyond the base model? arXiv preprint 2504 13837, 2025.
13

---

Published as a conference paper at ICLR 2026
Eric Zelikman, Yuhuai Wu, Jesse Mu, and Noah Goodman. Star: Bootstrapping reasoning with reasoning. NeurlPS, 11115476-154888 2022.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint ar arXiv:2503. 18892, 2025.
APPENDIX
A PROOFS
We now present the proof' for the main theoretical results stated above. Our analysis closely follows derivation style Chen & Maguluri (2025); Scherrer (2014), extending to the curriculum setting.
A.1
PROOF OF THEOREM 3.1
Proof. We aim to derive tight bound for the final performance enror in task Tk:
EK == 11 Q llax
where Qk is the optimal Q-function for the final task Tk. and Q is the Q-function under the learned policy æk-
We begin by analyzing the policy performance error under API. For each task k, we define:
?h 11Q Q 1l.0.
By norm monotonicity, we have:
1 - Q Ilax S 10k - QK 1boo
We proceed by recalling the result from Theorem 2.1 Chen & Maguluri applied with step size B:
2025) for fixed task k
T
1 - OF lloo 19 - 2 1lac +
2y
8x +
27
3(1
7)
t=0
where & is the approximate value evaluation error curriculum step 7 |Q 2 |l reflects
initialization n. and B > 0 is the API soft update step size parameter. The finite geometric series
T 1
T-1-t
1=
has closed-form expression
T-1 Σ 1
1 - xT
T-1
for
7 +
and
Σ
72 = 1, for 1.
1=0
1=0
Thus simplify with
1 --
Σ
Since we denote:
10 - -Thh 11o0;
m
14

---

Published as a conference paper at ICLR 2026
then, the bound becomes:
27(1 - (-(1-1))
27
S 7m
Th +
6x + B(1 - 7)
(1 71-
We isolate 7):
m/ (1 - (1
2y
1 (2 (y(1-17) --7)
24
8k
7 7k 1
6k + 3(1 7)2
(1 71)
3(1 - 7)2
1 72²
(1 (1-
We want to bound:
11 Q llaxi
using triangle inequality:
11 Q7 Il 2 =1QQ% - - 1l S 7K +
1Q Villax
Let us now plug in the bound for 7)x
(2y(1- -7 7)2
2-
K-1
11 - Q lldk I 1 1-77
(7)1 8k + 3(1 7)2
+
1 - Qillax
Am
The distribution mismatch is defined by C₁ ==
If d2 is constructed by curriculum to smoothly interpolate toward dx, and both dx and da are supported on the same growing state space. In our analysis, we use:
dx
de
ac
10 2T" C -1 0% 11o,
where ||- 1lax is the L2 norm under the distribution dx and || 1m is the sup norm (worst-case).
In general, for any probability distribution 1r, and any function f.
|lfllea()) ( (())
1/2
S |11|b
So, fact, we have the reverse:
11l1lax $ 11 11
If we do want to write
IIllax C ll
then the tightest possible value for C₁ is exactly 1, and any value Cs may be possible depending on the support f under dx. This holds when later curriculum stages subsume earlier ones, a design principle in CRL. Hence, using C2 simplifies bounds without loosening them unnecessarily.
Because curriculum proceeds sequentially, and assuming that earlier tasks are easier and learned more accurately,
sum the per-step bounds for curriculum steps:
(27(1- 1p(11-7) -") 5x + B(1 B(1-7)3) 1 1-
1
(
2y
Ex 1|Q
+
Σ 1 Qillax
(1-7)² 7)2
k=1
This completes the proof.
Optionally, assuming 7" is small for large T. 1/(1 (1 (T) <
leads to:
K
(1(1-7²))
47
K
EK x k=1 (1 71-7)2
8n +
Σ
1 - Qillax
3(1
7)3
15

---

Published as a conference paper at ICLR 2026
A.1.1 THEOREM A.I
AND PROOF
Theorem A.1 (Finite-Sanil Sannle Guarantee). Consider the sequence of tasks {Tk] k=1 where the final task Tx has optimal action-value function QK. Suppose that лk is the final policy produced by CRI. and assumptions hold for all curriculum stages. Following Chen & Maguluri (2025), let KSAA E TR. [I be a diagonal matrix and let K.S..iin be the minimal diagonal entry. Ju is the number of critic updates per policy iteration at curriculum k. The bootstrapping parameter n is chosen such that To == 7 /===""///""s /KS.Amii < 1. Then, when using constant stepsize a satisfying
1K
alta + n 1n+11= S
( (- Amiin
we have for all JR 2 ta +n+
130L2
2 Expprro
27 2 (1
E [ QTH S
TK [
QTh 11x
7)
7)
N2.1
Na.2
K
6(1 (1 - Tc) A 3 ((o+ta)))
K 70Lx/a(ta + n 1))2/2
+
(1 - 7) (1 -70) 1/2 1/2
Amni (1 - 76)(1 - 7)3
min
N2.3
N2.4
278K
K
+
1Q Qillax
7)
No
Curriculan discreponcy
where .) and .): are generalized importance sampling factors, La the parameter defined by 1 + (7iPmax, )" E sup,, 1|Q.; P II is the critic's 's function approximation error, Ebias, == maNOSIST m (1 X(s)) ( =x,b(-|s)]l, is the importance sampling bias at curriculum k, 11Q Qilloo is the gap between the curriculum subtask and the final task, and Amin is the mininum eigenvalue of the positive definite matrix Ks
The terms N1 and N3 are the same as appeared in Thm. 3.1 and together capture the error in actor update. Term N2,1 is the function approximation error and equals zero when using complete basis. A term of similar form is present in all existing work studying RL with function approxima- tion (Munos] 2003 Agarwal et al. 202T). Term N2,2 is the bias introduced by generalized importance sampling factors .) and pl: and 22 0 when c(s, a) p(s. a) (als)/m((a)s) N2.3 represents the convergence bias in the critic and goes to zero geometrically fast as the inner loop iteration index JR goes to infinity. The term N2.4 represents the variance in the critic and is propor- tional to Vata = O( log(1 (og)),)),, thus arbitrarily small given small enough stepsize a. Together, (N2,)im1~4 correspond to the second term in Thm. 3.1 Finally, the curriculum approximation error is the same as the last term in in Theorem 3.1 This result suggests that smoother curriculum trajectories and smaller approximation errors under finite samples, which is closely related with the stepsize and number of updates per iteration, can lead to better final policy performance.
Term
Proof. We proceed by mathematical induction across the curriculum steps. Let T&, denote the final policy after T' iterations curriculum step k. Base case: For the first curriculum step (k: 1), we directly apply Theorem 4.1 from Chen & Maguluri (0025):
E E Q [00] S 7711Q: - -7
(3)
27 Papproz,1
+ 272 FoiaAS11
+ (1 - 7)
(4)
(1 7)
6(1 - (1 - (1-7)))mmin (x) (1J
(5)
(1- 31 76.) 1/2 min
1/2
70L, (ar(la + n 1))
2
(6)
Amin (1 7c)(1 7)
(1 7)
16

---

Published as a conference paper at ICLR 2026
Inductive step: Assume that for curriculum step k. we have:
[||[Q - Q [lo0] S TT111 - Q 100
27 Eapprous
+ 27 Exiass
+
Σ
7)2
(1 7)
f=1
6(1 - (1 -e))min (x) 0:) -1, (ta++++
(1 - (1- 7/)2 X
70L + n + 1)]
27/3
+
+
Amin (1 7c)(1 7/3
(1 - 7)
11Qx 2j1ld
For curriculum step h + 1, we initialize with policy TH. and apply Theorem 4.1 from Chen & Maguluri (2025):
Ell1Q 1 [10] S s1111 - 1l
2 2appppros + 2 Ebsas,kt1
+
(1 7)-
(1 71
6(1 - (1 To) Pe) 02) (1U/et1-(to+:::
(1- 2022 1/2
70L441/a(ta + +
2yB
Amin (1 - (1 - 73 - (1-
We need to relate ||Q741
Q ol to our induction hypothesis. By triangle inequality:
1[Q++ 1 1l < <1Q++1 Qx + ||Q,
11 + +1
Q ll
|0 Qulloo + |1 T.TT
|100
||rk
1k41
1 1-7
where the last inequality follows from the performance difference lemma with respect rewards. For curriculum learning, we design the reward functions to satisfy ||r Tetilloo S 8, for some small 5, Thus:
70,
||Q - T4.T ol < 114+1 Qalloc + 11Qx
1-7
17

---

Published as a conference paper at ICLR 2026
Substituting our induction hypothesis:
70p
Bl|[Q++1 Qu ool / Bl11Q Qk|lo0 + +1Qu -Q% 10
1 +
1 1-7
+ 27 Eapproos 27 Exiaaki
(1-7)²
(1 (1-
+ 6(1 101-(11 (1- Am)m
(fan+))))
(1- - 1/2
70Lx+1/a(ta + n2 1))1/2
2y
+
Amin - 76)(1 7)3
3(1 7)2
T 8,
7 7/Q0++1 Qullx + +~"
2x 21ool
+
1 1-7
27 2yEappoo. (1 (1-
27 2y2nEina,
+
(1-
6(1 101-(11 (1- %oo Amin -
14a
- 73 - 2/2
70Lx+1 (a(to + 12 1))1/2
27/3
+ (1 7)2
Amin 7c)(1 7)3
Applying the induction hypothesis:
B||[Q++1 QUR+1.7 [10] S S7
Qulloo + + 7 Th11 Q70
k
(2Papproo
2 Enian.j
+ t Σ
(1 7)-
(1 7)
1=1
+ 6(1 - (1 - To) Amin (x) - (to +n-
(1 - 31 1/2 1/2
70L, (a(ta n 1/
27
+
Amin (1 70) 77)3
3(1 - 7)
k-1
T+ 8r
+
T
1Q - -gllds + Σ
1 1-7
j=1
27 Eapprrar,kt1
27 Fsiissk+1
+
71-
(1 (1-7)4
6(1 - 71- ( (1)))))1 ((1ott1((ootttt)
+
(1 (1- 31 - 1/2 1/2
70L (a(1a + n + 1//
2
+ (1 7)
Amin (1 7c) 7)
18

---

Published as a conference paper at ICLR 2026
Simplifying:
Bll[Q++1 - Quetii²1 [bo] S 7 ( [|Q - -Qr 11:00
k+1
27 27Boppron
27 Ebias
+
7)2
(1 7)
j=1
6(1 - (1 Tom a) ) (tutntt]
(1 - 7) 2/)) 1/2
70L, + n + 1)11/2
27/3
+
+
(1 - 7)2 Amin (1 - 7c) 7)
1k++1 - Qillda41
where we used the fact that
k-1 Σ
à Σ 1Q - 171la4+
1Q Qilldet + +
l1Q2 - Qilld -
2=1
due to the triangle inequality and the curriculum design. By induction, for the final curriculum step K, we have:
E| Q** lboo S Tk121 -QT
2 27Eappror.
71- + 2>² Ebias,k
(1 - 7)-
6(1 (1 - Nom a) - (1at+11
(1 -1- (1 7o) 1/2 1/2
70Ls(a(ta + +
2//3
+
Amin (1 - 76) (1-11-1) 7)3
(1 - 71
K-
1 - Qkellax
k=
This completes the proof.
A.1.2 PROOF OF THEOREM 3.2
Proof. From the Finite-Sample Theorem for CRL, we aim to control all error terms to achieve the desired accuracy. We allocate the total error budget across the K curriculum steps, with ég being the error allocation for step k, such that Lkml S e. For each curriculum k, we need to control the following terms:
00 S 4 TK - 1
-000
(8)
6(1 - (1 7e)
6k
(1 31 7o) 1/221/2
70L (a(la 1)j2/2
Ek
(10)
Amiin (1 - (1 - 7)3
27,3
(11)
(1 7)
19

---

Published as a conference paper at ICLR 2026
We solve each of these constraints. For the first constraint, need:
1 K
T2 logy
4||Qi Q7
Since |[Q] - 1lo -
S T-g. we have:
c(( (11-7)
log 1/EE) log(1/E)
T 2 K
log
4
=
K
For the second constraint, we need:
<x(1 - -3)1-11 Vo. 1//2 1/2
(1 - - Vo Mmin H (1ot-())
24
thus by taking log,
Ex Ex(1- - 7)) - -10) 1/2
2/22 x min
2 2/ (la + n + 1)] log(1 - 7o) S log
24
thus
2 2log (1-=)) 1/2
J& (1a +
log(1 (1 - Joo mmin
Using log(1 2)) xe x for small x, and atta + n + +1) (1-7))ama
we have:
1302
2og
24
(1(1-)) (1-77)))))))3))) 7/2 1/2
Jk 2 (ta +
min
(1 Yc) Mmin
(<(1-n)* (1-7)) 1/2
2 log
24
= (1a + + 1) +
(1 Vo)
For the third constraint, we need:
cg(1 - 71 2
atta + +
4900L
[ - 70 - 760) 2
I
For the fourth constraint, we need:
8 <x 7)2
8y
Combining these constraints, the dominant factor in the sample complexity comes from the third constraint, which gives us:
a aso ( 71 2
Lt + n + 1)
The mixing time f. is O(log(1 1/a)), which gives us:
2/ L (log(1//))
81 -
92) 22 Amin
a 0
(1/x) + n +
This implies:
a a= ( [1 (1 - 71 (171--)000 22 N
1 log(1 (1/a)
( 7) -re) 2>2
alog(1/a) =
12
20

---

Published as a conference paper at ICLR 2026
Using the Lambert W function, solve for az
( - 71 (1- (70))am
θ
112
L log (1(1-1))(1-7e) (2
6 ( 7) 7e)² 22 Xaain
where Ô hides logarithmic factors. The number of samples required for curriculum step ls is:
M Jk T
1og(1/ex)
(og(1/c)
0
(ta + n + 1)+
(1
aminn
K
log(1/a) + ennt1
(og(1/EN) log(1/t)
0
log(1/e)
K
K(1 Vo) mmnn
log(1/e) log
llog(11/e)
[[1-7)011
K
log(1/ (gg()/E)) log L log
1.2
J2X 2111 (1(1-1)/11 [1)1--)) wiia
+
Kct (1 -%7) 33
mai
|lg((1/E))
log(1/EX) log(1/e)L3
+
K
1 Ke 71 7o)
Ammn
The second term dominates, giving us:
logg((1/e))) (1/Ex) log log(1/e)LZ
M = Ô
(12)
Kez(1 - 7) 7a) X
The total number of samples across all curriculum steps is:
K
McRL =
Σ
MR
k=1
log" (1/ex)log(1/))))
Σ δ
Ke 7) 3233
k=1
= ò log (1/c
log² (1/cx)
2 71 7o) 23
Assuming 6 = x for all & (uniform error allocation), we get:
log(1/e)
K
Oog2(K/e)L3
McRL Ō
K
(E/K)((L 7
Vo) 13
X
log(1/e) L/e) log (K/e)K
=
L
2(1 61 7c) 323 min k=1
log (1/e()
K
L2
21 81 3 Nanin
k=1
Adding the dependence on the bootstrapping parameter n,, we have:
K McRI =
log (L/ER),
2 ((1
.
Linn
(13)
t
71 To)
31 Amin
21

---

Published as a conference paper at ICLR 2026
This completes the proof for the first half of the Theorem. Now move to prove the second half of the Theorem.
The sample complexity for the final task K is:
llo 2 (1/EK)
C Li (1-7)7(1- (6))
Mr =
KK
2
70)
Amin
We can factor this term out of the total sum:
(1
L logo/l//////// (I/Ea) cx
MCRL MM
L log (1/ck)
Define the curriculum efficiency factor:
K Σ
L log (1/Ex) Ex 1 log (1/ex) oz
CEF=14
This factor represents the ratio total curriculum sample complexity to the sample complexity of just the final task. We make two structure assumptions about our curriculum design, geometric error allocation and Lx progression, é& = Ex K and Lı Lk/(k - where e > 1. This reflect curriculum that gradually increases in difficulty while allowing larger errors in earlier stages. For designed curriculum where early tasks are simpler than later ones, > 1.
For the error ratio term:
2
Ex (eK
ck- = =E -((K-K)
2
6k
and for the LE ratio term:
1
-2)
L
(Lx 1KK 1k2
For the logarithmic term:
log(1/<)) log(11((x)) - kk= log(1/<x) (K k:) log(e).
Substituting these expressions into CEF:
K- Σ
(k) [log(1/e (K 2
-(K--K)
CEF =
log²(1/eax)
k=1
Since e > 1 and K2 k (K- - k)) log(e) 2 0. Thus we have
[lo[[/ex) (K k)) (e) (el)-2(KKN)
CEF =1
log (1/()/
K-1 51+
(el) (k)
The sum is a geometric series:
(el))(1-K) 1 (e])2(K-1) (el)2(1-K3
(el) (el)-2(K 2(K
Σ =
(el)²
1 (-(ee))
CRL more sample-efficient than direct learning when:
MCRL Mirrcct
22

---

Published as a conference paper at ICLR 2026
Since MERL Mr CEF and for comparable direct learning approach, MDirect Mx where m represents the relative difficulty factor of direct learning, our condition becomes:
CEF m.
Thus for McRs. Direct, substituting expressions becomes:
McRL = MK CEF & Mx (1+ (el) (
< MK m = MDirect
1 1 (el)²
Thus
(el)2(1-X) -1
McRI. < MDirect
< m 1.
(el)2
This completes the proof.
B SAMPLE EFFICIENCY GAINS WITH CRL
Our theoretical analysis (see Sec 3.3 establishes that curriculum reinforcement learning (CRL) attains target performance while requiring fewer hard samples than non-curriculum RL methods. To show this empirically, count how many training samples from each difficulty level are during post-training. All methods are trained for 1600 iterations with an effective batch size of 8, totaling 12,800 samples, which allows direct comparison of their sample efficiency under the same budget. For simplicity, we perform this analysis on Blocksworld, shown Table Our empirical results show that CRL methods between 2.5 3x more sample efficient than non-CRL. methods that train exclusively on hard samples, while also achieving better performance, underscoring the importance of curriculum design for post-training.
Table 8: Consistent with our theoretical guarantees, CRL methods (E2H-C, E2H-G) attain strong performance while requiring substantially fewer hard samples than non-curriculum baselines. For example, E2H-G uses 3580 hard samples versus 12800 GRPO (HARD). Training exclusively on OOD (GRPO-OOD) trained on 12800 OOD samples performs poorly (see Table
Number of Training Samples
Method
Trivial Easy Medium Hard OOD
Total
GRPO (All)
3200 3200
3200
3200
12800
GRPO (HARD)
0
0 12800
0 12800
GRPO (OOD)
0
12800
12800
E2H-C
3200 3200
3200
3200
0 12800
E2H-G (0.5, 0.5)
1628 2997
4595
3580
12800
C DATASET DETAILS
In this section we provide details of the datasets used for evaluation. We categorize the datasets into two categories, datasets that contain human annotated difficulties and others that do not.
C.I DATASETS HUMAN ANNOTATED DIFFIULTIES
Blocksworld (Valmeekam et al. 2023) is dataset used to evaluate the planning capabilities of LLMs. Each task involves transitioning from an initial block configuration to a target configuration, which requires LLMs to generate a sequence of actions, plan to achieve the goal. Tasks become more difficult as the required number of steps increases, since the model must reason over longer sequences and maintain correct intermediate states. To study this. group tasks into four in-distribution difficulty levels: Trivial with I step, Easy with steps, Medium with steps, and Hard with 6 steps. Additionally, we include out-of-distribution (OOD) split with 8-step plans test generalization
23

---

Published as a conference paper at ICLR 2026
Table 9: Difficulty splits for datasets with human annotated difficulty levels. Each dataset categorized based on -ase-specific properties, specifically, plan length for Blocksworld, number of operands for Countdown, and problem level for MATH.
Difficulty
Blocksworld
Countdown
MATH
(Plan Length) (Num. Operands) (Problem Level)
Trivial (T)
Easy (E)
2
3
Medium (M)
Hard (H)
5
OOD
beyond the training distribution. Trivial tasks are especially simple because the model only needs to predict one correct action out of four possible choices to complete the plan. This setup allows the LLM to first grasp fundamental planning mechanics, which can then be leveraged learn more complex multi-step tasks.
(aandhi et al. 2024) is task where the model must reach target value by combining given numbers using basic arithmetic operations. While the original dataset uses four numbers per instance, we extend it to create range of difficulty levels based the number of input numbers, mainly. Trivial (2), Easy (3), Medium (4), Hard (5), and OOD (6). As the number of inputs increases, the space of possible operation sequences grows rapidly, making it harder for the model identify the correct combination and order of operations. In contrast, the trivial setting is extremely simple, requiring just one operation between two numbers to reach the target, allowing the model to first leam basic arithmetic before scaling to more complex multi-step problems.
Countdown
MATH (Hendrycks et al.) (022)) is benchmark of 7,500 training and 5,000 test problems covering high-school level mathematics, with each problem labeled from Level (easiest) to Level 5 (hardest). The dataset covers topics such as algebra, geometry, number theory, and probability, including step- by-step solutions. We create difficulty based setup using the existing labels, specifically, Trivial (Level 1), Easy (Level 2), Medium (Level 3), Hard (Level 4), and OOD (Level 5). As difficulty increases, problems require understanding. Trivial problems are typically short and rely on basic techniques, making them ideal for teaching foundational reasoning. We use Levels 1 through for training and treat Level 5 as out-of-distributi to assess generalization to the most difficult problems.
complex reasoning, multi-step solutions, and deeper mathematical
We provide the summery of the difficulty splits for each dataset in Table
C.2 DATASETS WITHOUT HUMAN ANNOTATED DIFFICULTIES
GSM8K (Cobbe al. 2021 is a dataset of high gigh-qualtty linguistically diverse grade school math word problems designed to evaluate multi-step arithmetic reasoning. Each problem typically requires between two and eight steps involving basic arithmetic operations such as addition, subtraction, multiplication, and division. To assess performance across varying difficulty levels, we create a 4-way split into Trivial, Easy, Medium, and Hard, based on model eттоr rates, as described in Section. 4 and illustrated in Figure.
AQuA (Ling et al., 2017b) is a dataset of algebraic word problems with multiple-choice answers and detailed rationales, designed test arithmetic reasoning and symbolic manipulation. For our experiments, we randomly sample 5,000 problems for training. Similar to GSM8K, we use model error rates to define four difficulty levels. Trivial, Easy, Medium, and Hard (see Figure. (
Since both GSMSK and AQuA lack explicit difficulty annotations, we construct difficulty splits using model error rates and do not include out-of diitrr-distribution (OOD) category for these datasets.
24

---

Published as a conference paper at ICLR 2026
Error Rate Distribution by Difficulty Bucket
500
Virial
Easy
Number of Quesions
400
Hand
200
200-
100-
0.2
0.4
Error Rate
Figure 6: AQuA Difficulty distribution based on error rates, grouped into quartiles. Difficulty groups used for training derived from quartiles of this distribution.
D MODEL LICENSE INFORMATION
For our experiments, we LLaMA 3.2 3B Instruct, Qwen 2.5 1.5B Instruct, and Qwen 2.5 3B Instruct. Qwen 2.5 1.5B Instruct is released under the Apache 2.0 License, Qwen 2.5 3B Instruct under the Qwen Research License, and LLaMA 3.2 3B Instruct under the Meta Community License.
E IMPLEMENTATION DETAILS
In this section, we provide implementation details for E2H Reasoner. We begin by describing the hardware setup, followed by training parameters, and finally the inference settings used to report the final results.
E. HARDWARE DETAILS
For our experiments, we use up to three 80GB NVIDIA A100 GPUs, particularly for the 3B parameter models. One GPU is dedicated to vLLM for fast inference, while the remaining used for model training. On average our experiments require anywhere between 16-18 hours for training.
E.2 TRAINING DETAILS
We use GRPO as the policy optimization algorithm, combined with parameter-efffcient fine-tuning using LoRA. All models trained for up to 1600 GRPO steps, with hyperparameters listed in Table I0 We adopt the reward design from Guo et al. (2025). where outputs with the correct format receive partial rewards, and full rewards are given only when both format and answer are correct.
For fair comparison, the same configuration is used in our SFT experiments. All training is con- ducted using the TRL library from Hugging Face, averaged across 3 seeds. For the Self-Evolh baseline (Chen et al. 2025), we use the same default
as reported by authors. For DAPO YYu
al.
2025
F.3
DIFFICULTY-BASED TRAINING SPLIT CREATION FOR GSM8K AND AQUA
Since both GSM8K and AQuA lack explicit difficulty annotations, we construct difficulty splits based on model error rates. We use CoT prompting with I-shot in-context learning (ICL) to generate answers for each question. The prompts and ICL examples have been directly adopted from prior works (Hao et al.) 2024 Parashar et al. 2025). The model is queried 20 times per question, and the error computed as the fraction of incorrect responses relative to the ground truth. Based on this, samples with 0-25% % error rate are labeled as trivial, 25-50% as easy, 50-75% % as medium, and 750100% hard.
25

---

Published as a conference paper at ICLR 2026
Table 10: Training hyperparameters used for GRPO post- post-training LoRA adaptation.
Component Parameter
Value
learning rate
le-6
Ir [_scheduler__tpp
cosine
per srr_ddvice__tge _arlee__riiin__ battss_tttnaaahhh___eeaeea size
2
catpiinnttaccmmlatttoonnn steps
4
true
GRPO
max steps bfl6
1600
true
uf32
true
num_generation
beta
0.001
use_vllm
true
vllm_gpu_mmee
0.2
r
32
alpha dropout
64
LoRA
0.1
parget_module
q_proj. v_proj
task task_type
CAUSAL LM
E.4 INFERENCE DETAILS
For inference, we use temperature of 0.0 in all experiments to ensure deterministic outputs and reproducibility, For pass@ evaluations, we enable sampling with a temperature of 0.7. top p of 0.9, and top_k of 50, where top_k controls the number of candidate tokens considered at each decoding step.
F COMPARISONS WITH SFT
We compare vanilla supervised fine-tuning (SFT) with RL-based post-training methods (see Ta- ble [Tp. The results show that SFT performance varies significantly across tasks. For instance, in Blocksworld (Valmeekam et al. 2023), where most problems involve fewer than four blocks, SFT performs well the LLM learns effectively due to the small gap between the training and distribu- tions. This aligns with prior work suggesting that SFT is most effective when the training data is high quality and closely aligned with the downstream task (Muennighoff et al. 2025 Chu et al.) 2025). In contrast, on tasks like Countdown, where the problem distribution much broader, SFT performs notably worse than RL-based post-training. This gap is especially clear in the out-of-distribttion (OOD) setting, where SFT fails solve any Countdown OOD examples, unlike in Blocksworld, where its performance generalizes more successfully. We highlight these two tasks to illustrate how SFT performance can vary across distributions, in contrast to the more consistent behavior of RL-based post-training methods.
G ADDITIONAL ANALYSIS AND ABLATIONS
G.1 RESULTS ON MORE LLMs
Additional results on more LLMs are reported in this subsection.
G.2 RESULTS ACROSS GAUSSIAN PARAMETERS
In this section, expand on the results presented in the main paper by reporting all three parameter
settings of Gaussian scheduling in Table Similarly, for Qwen 2.5 1.5B Instruct we include the results for all Gaussian scheduling variants on GSM8K and AQuA in Table 14
26

---

Published as a conference paper at ICLR 2026
Table 11: Comparison of RL-basee post-training and supervised tine-tuning (SFT). performance varies noticeably across tasks, highlighting its inconsistency in generalizing across domains.
Blocksworld
Countdown
Models
Methods
Trivial Easy Med Hard OOD
Trivial Easy Med Hard OOD
SFT
100.0 97.8 88.1 55.3
16.5
97.4
41.8
14.2
4.6
0.0
GRPO
98.0 100.0 84.5
26.3
5.3
96.1
64.9
28.8
18.1
9.2
Qwen 2.5/ 1.5B Instruct
E2H-G
98.0 100.0 95.3
32.9
7.3
95.2
84.1 48.1
28.1
14.2
E2H-C
100.0
100.0 15.5
0.0
0.0
96.7
64.0 25.9
15.8
6.4
SFT
100.0
100.0 94.0
62.5
27.8
99.9
63.9
30.8
13.3
0.0
GRPO
100.0
100.0 88.1
38.8
14.6
99.6
89.3 73.4
41.7
9.4
Qwen 2.5/3B Instruct
E2H-G E2H-C
100.0
100.0
96.4
53.3
23.2
98.5
90.8
71.0
43.5
19.4
100.0
100.0 94.1
52.0
22.5
100.0
90.4 69.7
35.7
12.6
SFT
100.0
97.8 96.4
72.3
37.8
100.0
66.6
27.2
12.1
0.0
GRPO
100.0
100.0 94.1
38.9
13.3
99.9
89.5
71.6
47.9
2.7
LLaMA 3.2/ 3B Instruct
E2H-G
100.0
100.0
98.8
44.1
17.6
95.0
89.9
73.3
46.5
24.3
E2H-C
100.0 0.0 0.0
0.0
0.0
100.0
55.3
0.0
0.0
0.0
Table 12: Results on Qwen-2.5 3B Instruct.
Blocksworld
Countdown
MATH
Trivial Easy Med Hard OOD Trivial Easy Med Hard OOD Trivial Easy Med OOD
Qwen 2.5 3B Instruct
CoT
1.0
7.1
15.9
0.5
0.0
68.0
55.8
48.8
GRPO (All)
100
38.3
14.6
99.6
89.3
73.4
41.7
9.4
90.1
78.5
71.9
57.4
GRPO (Hard)
50.0
42.2
86.9
72.4
48.3
1.6
49.5
38.7
27.7
12.6
90.6
78.4
70.5
58.0
36.0
GRPO (OOD)
0.0
13.3
9.5
0.0
0.0
0.4
25.1
21.2
13.8
8.5
91.3
78.1
70.0
58.7
36.8
CL
100
100
77.3
42.1
20.5
39.0
91.0
72.9
38.7
12.5
90.6
78.9
72.5
57.2
36.6
Self-Evolve
100
42.5
99.6
91.4
79.8
56.8
29.3
92.2
70.6
E2H-G
100
100
96.4
53.3
23.2
98.5
90.8
81.2
60.7
32.3
90.4
81.0
71.5
58.2
37.2
E2H-C
100
100
94.1
52.0
22.5
100
90.4
69.7
35.7
12.6
90.4
80.0
73.4
58.9
38.1
G.3 CHOICE OF NUMBER OF DIFFICULTY SPLITS
The number of difficulty splits can be treated as a hyperparameter in E2H Reasoner, particularly for datasets without human-annotated difficulty labels such GSM8K (Cobbe et al. 2021) and AqUA (Ling et al.] 2017a). We set this value to 4, consistent with the range of 3 to 5 used in the curriculum learning literature (Bengio al. 2009). As shown in Table 15 our method remains robust across different choices of this parameter.
G.4 DAPO + E2H
Combining E2H with DAPO consistently lowers the fraction of advantage-zeru batches (see Fig. 7 indicating that the curriculum helps select more informative batches with difficulty better aligned to the model's competence.
G.5 QUALITATIVE ANALYSIS OF SCHEDULING TECHNIQUES
We provide qualitative analysis of scheduling techniques Table 16 highlighting their respective strengths and weaknesses. The results show that different schedulers suited to different tasks and model scales. Inspired by the effectiveness E2H Reasoner, we hope this motivates the community explore sophisticated scheduling strategies for L.LM post-training.
27

---

Published as a conference paper at ICLR 2026
Table 13: Effect scheduling strategies LLM post-training, compare balanced scheduling, traditional curriculum learning (CL), and our proposed E2H Reasoner variants, namely. E2H-G and E2H-C. CoT is reported as reference.
Blocksworld
Countdown
MATH
Trivial Easy Med Hard OOD Trivial Fasy Med Hard OOD Trivial Easy Med Hard OOD
Qwen 2.5 3B Instruct
CoT Balanced
24.0
15.9
0.5
0.0
33.4
20.0
100.0 100.0
38.8
14.6
99.6
86.9
57.8
23.6
7.8
90.1
71.9
57.4
36.8
100.0 100.0
42.1
20.5
39.0
91.0
72.9
38.7
12.5
90.6
72.5
57.2
36.6
E2H-G (0.25, 0.75)
100.0 100,0 96.4
53.3
23.2
98.5 90.8
71.0
43.5
19.4
90.4
81.0
71.5
58.2
37.2
E2H-G (0.5, 0.5)
100.0 100,0 89.2
32,9
10.6
99.6
89.3
73.4
41.7
9.4
91.1
78.2
70.5
56.6
35.0
E2H-G (0.75, 0.25)
100.0 100.0 42,9
3.9
2.0
100.0
87.8
52.3
16.7
6.5
90.0
79.3
70.6
57.1
35.9
E2H-C
100.0 100.0
94.1
52.0
22.5
100.0
90.4 69.7
35.7
12.6
90.4
80.0
73.4
58.9
38.1
LLaMA
Instruct
CoT
24.0
0.0
1.2
1.0
0.0
37.1
4.6
0.3
0.0
0.0
65.9
44.6
35.2
24.1
13.6
Balanced
100.0
100.0
94.1
38.9
13.3
99.9
89.5
71.6
47.9
2.7
65.9
47.0
36.0
22.0
10.2
CL.
100.0
0.0
0.0
0.0
0.0
17.2
36.0
22.7
11.2
4.1
74.1
54.1
43.9
28.0
12.5
E2H-G (0.25, 0.75)
100.0 100.0
98.8
44.1
17.6
95.0
89.9
73.3
46.5
24.3
78.7
58.4
46.4
32.3
14.5
E2H-G (0.5, 0.5)
100.0 100.0
88.1
26.3
9.6
94.4
86.7
65.1
27.9
2.3
66.1
45.0
36.2
24.4
11.4
E2H-G (0.75, 0.25)
100.0 97.8
75.0
21.1
98.9
89.4
48.6
0.7
0.0
63.8
48.2
35.6
24.0
8.6
F2H-C
100.0 100.0 15.5
96.7
64.0
25.9
6.4
84.6
69.6
63.0
47.6
28.6
Table 14: Expanded results for Qwen 2.5 1.5B
Instruct showing all E2H-G (Gaussian scheduling) variants on GSM8K and AQuA. We provide comparison against Balanced and E2H-C (Cosine scheduling).
AQuA
GSM8K
Trivial Easy Med Hard Avg Trivial Easy Med Hard Avg
Balanced
95.8
68.0 48.6
21.0 63.3
99.0 95.3 84.1 49.9 77.1
E2H-C
86.1 72.2 48.6
26.3
62.5
98.0 95.3 83.9 46.6 75.7
E2H-G (0.25, 0.75)
93.0 70.8 40.2
23.6 61.4
98.5 96.3 83.6 51.2 77.6
E2H-G (0.5, 0.5)
90.2
81.9 43.0
34.2 66.1
97.6
94.7 89.0
51.8 78.7
E2H-G (0.75, 0.25)
83.3 79.1 54.5
26.3 64.1
98.0 97.3 85.9 47.7 77.1
DAPO+E2N-C
Advantage Zero Batches
DAPO
0.4
0.3
0.2
0.1
200
600
1000
1200
1400
1600
Training Steps
Figure 7: Fraction of advantage-zero batches over training steps for DAPO, DAPO+E2H-G, and DAPO+E2H-C.
H LLM USAGE
LLMss used for text-refining purposes only.
28

---

Published as a conference paper at ICLR 2026
Table 15: Results on GSM8K (Cobbe al. 2021 across different Gaussian parameter splits. The main experiments use the 4-split setting: below we show that our method is robust to the number of splits.
Method 3 Splits 4 Splits Splits
E2H-C 79.1
75.7
76.1
E2H-G 78.6
78.7
78.5
Table 16: Qualitative comparison of scheduling strategies, outlining their strengths and weaknesses context of LLM post traintrainnn.
Scheduling Strengths
Weaknesses
1. Assumes uniform difficulty across the dataset. 2. carameter-free and casy use.
1. Default in LLM post-training.
Balanced
2. Often fails to improve reasoning hard tasks.
Simple and parameter- free.
Can cause overfitting easy tasks. 2. May lead to forgetting earlier tasks.
Traditional
2. Easy to implement.
1. Parameter- free and simple to apply. May overfit easy tasks when rewards for hard tasks are sparse.
F2H-C
2. Suitable for tasks with similar zero-shot performance across difficulty levels.
1. Effective across tasks and models.
E2H1-G
Requires tuning hyperparameters.
2. Enables fine-gratn fine-grained control.
I
LIMITATIONS
Despite the strengths of E2H Reasoner, it has certain limitations. Our approach uses simple and intuitive probabilistic schedulers, specifically based on Gaussian and cosine functions, which do not adapt during training. While these choices are effective, our results suggest that incorporating adaptive strategies, such as advantage-baser based scheduling, could offer further improvements. Comparisons with adaptive curriculum methods that aim to maximize learnability reveal important insight: maximizing learnability does not always lead to stronger reasoning. This outcome depends on the structure and difficulty of the problems within the dataset. Combining our method with adaptive approaches presents promising direction for future work.
J SOCIETAL IMPACTS
Our work introduced E2H Reasoner, a curriculum-baat -based reinforcement learning (RL) approach designed to enhance the reasoning capabilities of small-scale large language models (LLMs). This advancement holds significant societal implications. Enhanced reasoning in LLMs can improve eeciiiionmmainn processes in critical domains such as healthcare, education, and legal systems, where nuanced understanding is paramount, Moreover, by empowering smaller models, E2H Reasoner promotes broader accessibility to advanced AI capabilities. Although our tasks focus on reasoning, future extensions involving real-world interaction could pose risks, including potential misuse of language models.
29


Reviews:
[{"review_id": "ga1EHEdwut", "reviewer": "Reviewer_kbdX", "summary": "This paper proposes E2H Reasoner, a curriculum reinforcement learning (CRL) approach for enhancing LLM reasoning capabilities. It decomposes complex tasks into easier subtasks, uses probabilistic schedulers (cosine and Gaussian) to gradually shift from easy to hard tasks during RL post-training, and provides empirical improvements on benchmarks like Blocksworld, Countdown, and arithmetic tasks, achieving SOTA results. Theoretically, it analyzes CRL via approximate policy iteration, proving convergence and reduced sample complexity compared to direct learning.", "questions": "1. Advantages of Task Decomposition Over Traditional Curriculum Learning: What specific advantages does your task decomposition offer compared to standard curriculum learning (e.g., fixed-stage switching)? Is there a theoretical comparison on generalization, perhaps extending your API framework?\n\n\n2. Inconsistencies in Model Trends in Figure 1(a): In Figure 1(a) for Countdown, why do Qwen 1.5B and LLaMA 3.2 3B show inconsistent relative performance trends under E2H vs. base models (e.g., one benefits more at low k)? Could this relate to architectural differences or training artifacts?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method creatively combines task decomposition with probabilistic scheduling in CRL, addressing rollout inefficiencies in difficult reasoning tasks by building skills incrementally, which makes intuitive sense and extends prior RL post-training like DeepSeek-R1.\n\n2. Theoretical analysis provides finite-sample bounds and convergence guarantees, grounding the approach in approximate policy iteration.\n\n3. Well-structured presentation with illustrative figures (e.g., task decomposition in Fig. 2, schedulers in Figs. 3-4) and precise definitions of reasoning as generalization; methods and experiments are logically sequenced.", "weaknesses": "1. Risk of Overfitting in Task Decomposition: Decomposing hard tasks into varying difficulty levels may cause repeated exposure to similar knowledge patterns across subtasks, increasing overfitting risks, especially if subtasks overlap significantly without explicit regularization.\n\n\n2. Lack of Implementation Details for Reproducibility: Key details are missing, such as prompts used for automatic difficulty estimation (e.g., in AQuA/GSM8K) or exact hyperparameters for task grouping, raising concerns about replicating the reported effects.\n\n\n3. Limited Scope of Experiments: Evaluations focus on relatively lower-difficulty tasks like Blocksworld and arithmetic benchmarks; lacks experiments on highly challenging ones like AIME, LCB, or agent-based tasks, limiting evidence for broader applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XNIGqhdZcn", "reviewer": "Reviewer_mKj1", "summary": "This paper proposes using a curriculum learning approach where they schedule tasks from Easy to Hard during RLVR which shows better performance at the end of training. They provide convergence guarantees for this algorithm in an approximate policy iteration framework and derive finite-sample complexity bounds which show the this is more sample efficient than training without any curriculum.", "questions": "Please look at the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper proposes a simple method of using curriculum learning. The curriculum implicitly assumes some grouping of tasks, but they also  show that the grouping is not necessary because tasks can be clustered just using pass rates of the initial model. They also compare with different baselines and the empirical results seem sound.", "weaknesses": "The only weakness that comes to mind is not comparing with DAPO [1] which also has an implicit curriculum because the model keeps filtering prompts that are either too easy or too hard. Could the authors compare with DAPO as well and show results on the benchmarks? \n\nAlso the paper doesn't cite Paprika [2] which also proposes a curriculum when tasks can be grouped. \n\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale (https://arxiv.org/abs/2503.14476)\n\n[2] Training a Generally Curious Agent (https://arxiv.org/abs/2502.17543)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "He7TE9O7K5", "reviewer": "Reviewer_rLJH", "summary": "This paper introduces the E2H Reasoner, a reinforcement learning (RL) method for large language models (LLMs) inspired by curriculum learning. The method schedules tasks from easy to hard during training, aiming to accelerate LLM learning. Theoretical analysis shows that curriculum-based reinforcement learning (CRL) requires fewer total samples than directly training on the final task, and experimental results are generally positive.", "questions": "See weakness", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The paper provides theoretical justification for why CRL can achieve sample efficiency, requiring fewer total samples than direct learning on the final task.\n2. The experimental results are sound and well-presented.", "weaknesses": "1. The idea of using curriculum learning to improve RL efficiency is not novel. The paper acknowledged prior work—e.g., Chen et al., Foster et al., Bae et al., Zeng et al. which used curriculum learning ideas. The paper should also cite Yu et al. (DAPO: An Open-Source LLM Reinforcement Learning System at Scale). \n2. In the experimental results, E2H does not consistently outperform baselines such as GRPO or Self-Evolve. \n3. The paper does not clearly articulate the advantages of E2H over adaptive filtering methods such as DAPO or Self-Evolve. In fact, adaptive filtering—where pass rate determines sample filtering—has several appealing properties: \n1) Model-dependent difficulty: “Easy” and “hard” samples are relative to the specific model; what is easy for one model may be hard for another. Thus, classifying samples by difficulty a priori can be problematic. \n2) Lack of synchronization: Without adaptive scheduling, the scheduler and the model may become misaligned—for instance, the scheduler may advance to harder tasks before the model is ready.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)