CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs
Authors: Yongcheng Zeng
Zexu Sun
Bokai Ji
Erxue Min
Hengyi Cai
Shuaiqiang Wang
Dawei Yin
Haifeng Zhang
Xu Chen
Jun Wang
Venue: ICLR 2026 Poster (2026)
Keywords: Reinforcement learning
LLM reasoning
Curriculum learning
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We propose a curriculum learning method CurES for LLM reasoning, which is a systematic and theoretical investigation into how to improve the training efficiency of LLMs.

Abstract: Curriculum learning plays a crucial role in enhancing the training efficiency of large language models (LLMs) on reasoning tasks. However, existing methods often fail to adequately account for variations in prompt difficulty or rely on simplistic filtering mechanisms to select prompt datasets within a narrow criterion range, resulting in significant computational waste. In this work, we approach the problem from the perspective of reinforcement learning gradient optimization, offering a systematic and theoretical investigation into how to improve the training efficiency of LLMs. We identify two key factors influencing training efficiency: the selection of training prompts and the allocation of rollout quantities across different prompts. Our theoretical analysis reveals that the sampling distribution of prompts dictates the convergence rate of gradient descent, while the allocation of the rollout quantity influences the consistency and stability of overall gradient updates. Based on these insights, we propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead. Experiments demonstrate that our CurES outperforms Group Relative Policy Optimization (GRPO) by $\textbf{+3.30}$ points and $\textbf{+4.82}$ points with 1.5B and 7B models, respectively, and exceeds the best prior sample efficient methods by $\textbf{+2.12}$ points on average across eight math reasoning benchmarks. Additionally, CurES exhibits faster convergence compared to baselines, including GRPO.

Published as a conference paper at ICLR 2026
CURES: FROM GRADIENT ANALYSIS TO EFFICIENT CURRICULUM LEARNING FOR REASONING LLMs
Yongcheng Zengl.2, 2,, Zexu Sun Bokai Ji Erxue Min Hengyi Cai²,
Shuaiqiang Wang', Dawei Yin Haifeng Zhangl 1.2.t, Xu Chen Jun Wang
IIstitttee Automation, Chinese Academy of Sciences
Sccooo of Artificial Intelligence, University of Chinese Academy of Sciences Baidu Inc. UUniversity College London
$ Gaallnng School of Artificial Intelligence, Renmin University of China
sunzexu082 82 gma com, zengyongchengg22222 acc.c
ABSTRACT
Curriculum learning plays crucial role in enhancing the training efficiency of large language models (LLMs) on reasoning tasks. However, existing methods often fail to adequately account for variations in prompt difficulty or rely sim- plistic filtering mechanisms to select prompt datasets within a narrow criterion range, resulting in significant computational waste. In this work, approach the problem from the perspective of reinforcement learning gradient optimization, of- fering systematic and theoretical investigation into how improve the training efficiency of LLMs. We identify two key factors influencing training efficiency: the selection of training prompts and the allocation of rollout quantities across dif- ferent prompts. Our theoretical analysis reveals that the sampling distribution of prompts dictates the convergence rate of gradient descent, while the allocation of the rollout quantity influences the consistency and stability of overall gradient up- dates. Based on these insights, we propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead. Experiments demonstrate that our CurES outperforms Group Relative Policy Optimization (GRPO) by +3.30 points and +4.82 points with 1.5B and 7B models, respectively, and exceeds the best prior sample efficient methods by +2.12 points on average across eight math reasoning benchmarks. Additionally, CurES exhibits faster convergence compared to baselines, including GRPO.
Q GitHub: hutps://gitubbccom//xx/u///u/Ccu
INTRODUCTION
Although Reinforcement Learning with Verifiable Reward (RLVR) (Guo et al.) 2025| Lambert et al., 2024 Team al. 2025) has emerged as a powerful paradigm for reasoning tasks Large Language Models (LLMs), prevailing approaches often rely on uniform sampling strategies that treat all train- ing instances identically (Zeng et al., 2025 Xie et al., 2025). This paradigm fails to account for the inherent heterogeneity in prompt difficulty and the varying training utility that different prompts offer. Consequently, computational resources are inefficiently allocated, being wasted either on triv- ial prompts that yield diminishing returns or on excessively challenging examples where the model shows negligible progress.
Recent works have investigated progressive training curricula that partition the process into several hand-crafted stages of increasing difficulty (Luo et al. 2025 Song et al. 2025). However, such par- titioning is overly coarse and struggles to align with the evolving capabilities of reasoning models during training. Other approaches apply online data filtering by generating and then pruning sam-
ples YYu al. 2025 Bae et al. 2025
Lin al. 2025). Yet, this paradigm does little to conserve
"Equal Contribution.
"orresponndng authors.

---

Published as a conference paper at ICLR 2026
Theoretically Optimal Update
CurES
P0+4 1.
CurEs Promet S Sampling
CurES Rollout Quantity
Park)
Paix) 0.50
Difficulty
0.25
Palk) 0.25
2(0)) Eu-p
(o Po(Ki]
0.5
SPEED-RL BLED and GVM
Method
Accorncy
Accuracy
Peli)
SPEED Promiee Baampiingg
GRPO
Ta
Difficulty
PoiN) O.
GVM
Quantity
SPEED
palx) 0. 25
CurES
Figure 1: Illustration of our theoretical and practical contributions. The first part presents our the- oretical analysis, which establishes the relationship between the gradient efficiency and models' question-answeriin accuracy, denoted po (z)). Building upon these insights, we develop CurES, practical method that initially estimates pa (x) using small rollout quantity, then reallocates prompt sampling probabilities and rollout quantities based on the estimated accuracy. We progressively en- hance the confidence of these accuracy estimates through posterior estimation. The figure further contrasts CurES with existing approaches, highlighting differences in managing prompt sampling distributions of Speed-RL RL (Zhang et al. (2025) and rollout quantities of GVM (Yao et al. (2025).
computational resources and instead teads to suboptimal sample efficiency. Addirional studies have begun exploring dynamic computation reallocation across prompts with minimal overhead (Yao) al. 2025 Zhang al. 2025 Shi al., 2025). Nevertheless, these techniques address only iso- lated facets of training acceleration, without fully accounting for the problem's 's inherent dynamism.
In this work, we first analyze the efficiency of training optimization for reasoning models from the perspective of gradients, elucidating its close relationship with the sampling probability distribution of prompts and the allocation rollout quantities across these prompts. Our analysis reveals that the prompt sampling distribution directly influences the speed of gradient descent, while the alloca- tion rollout quantities affects the consistency and stability overall gradient updates. Leveraging these insights, propose CurES, practical training method. CurES first estimates prompt diffi- culty via models' question-anse anwworring accuracy, then reallocates prompt sampling probabilities and rollout quantities accordingly. During training, the confidence in these accuracy estimates is progres- sively refined through posterior estimation based on previously sampled data, thereby improving the robustness of the allocation process. Figure I iillustrates the overall approach, and contributions are summarized below:
We provide a theoretical analysis from the gradient perspective, elucidating the intrinsic relationship between training optimization efficiency and prompt sampling distribution, as well as the allocation rollout quantities across prompts.
Guided by the theoretical analysis, we propose practical training method that integrates Bayesian posterior estimation, achieving enhanced efficiency and stability in reasoning model training with minimal computational overhead.
Experimental results show that our approach achieves state-of of- the-hhhrrr art performance across diverse mathematical reasoning datasets and model scales, accelerating convergence by up to 5.5 times compared to uniform sampling basclines like GRPO.
2 RELATED WORKS
Gradient Analysis in Optimization. Gradient analysis plays a pivotal role in understanding and improving optimization processes (Ruder) 2016 Bottou al. 2018 Yang al. 2024). A com-
2

---

Published as a conference paper at ICLR 2026
mon application involves leveraging gradient analysis to reduce variance in iterative optimization and enhance training stability (Medyakov et al.] [025 Yuan et al. 2024). Additionally, it facili- lates adaptive learning rate adjustments by dynamically scaling updates based on gradient histories
(Dereich et al. 2024 Chen et al.) 2024). Gradient analysis also guides data selection and curricu- lum learning strategies, enabling dynamic adjustment of training data difficulty (Yao et al. 2025 [i] et al. 2024bb. Furthermore, it aids in detecting anomalous samples to improve data quality by iden- tifying outliers in the gradient space (Chhabra al., 2024). As gradients are directly tied to model optimization, they provide the most immediate insights into training dynamics. Theoretical analy- sis of gradients enables predictions of convergence rates and bounds, ensuring robust optimization guarantees (Zhao & Xu 2024). In this work, we examine the interplay between model optimization and sample selection from a gradient perspective, deriving methods to enhance training efficiency.
Curriculum Learning and Data Selection in RLVR. Effective data selection is critical for op- timizing RLVR training, yet designing curricula that align with the dynamic capabilities of LLMS remains challenging. Progressive training curricula, such those proposed in (Luo al. 2025 Song al. 2025), partition training into hand-crafted stages of increasing difficulty. However, these static approaches often fail to adapt the evolving proficiency of models during training. Self-evolving curriculum learning (Chen et al.) (025) addresses this by selecting sub-datasets of appropriate dif- ficulty based on the model's current performance. Nevertheless, such approaches still operate the dataset granularity, resulting in a relatively coarse-grained curriculum mechanism. Online data fil-
tering methods, such as those in (Yu et al. 2025 Bae et al. 2025) Lin al., 2025 Xu et al.) 2025). generate and prune samples to focus on high-impact data, yet since they still require executing the overall rollout generation phase, they fail effectively reduce rollout-induced computational overhead, resulting in wasted sampling budget. Recent efforts have explored dynamic computa- tion reallocation to prioritize prompts with higher training utility (Yao et al.) [025 Zhang et al.] 2025 Shi et al.) 2025). However, these methods address only specific aspects of training accelera- tion. Moreover, these dynamic computation allocation approaches often require explicit estimation of dataset-related parameters such prompt difficulty. which could introduce additional and po- tentially unnecessary overhead, One direction to mitigate this is to incorporate Bayesian posterior estimation techniques (Qu et al. 2025a Wang al., 2025 Qu et al., 2025b) to guide prompt lection or active sampling. thereby reducing computational cost. Our work proposes CurES, which first systematically analyzes the correlation between gradient efficiency and problem difficulty from a gradient perspective. This analysis then informs our curriculum learning strategy, enabling the dy- namic adjustment of prompt sampling probabilities and rollout quantities. Concurrently, to minimize computational overhead, we integrate Bayesian posterior estimation to optimize parameter estima- tion cost, resulting improved overall training efficiency with minimal computational burden.
PRELIMINARIES
Reinforcement Learning with Verifiable Reward (RLVR) represents a specialized reinforcement learning paradigm tailored for reasoning tasks, where reward signals can be deterministically ver- ified through programmatic means. This approach is particularly well-suited for domains such as mathematical reasoning, code generation, and logical deduction, where correctness criteria are ob- jectively defined (Lambert et al. 2024] Guo el al. (0255] Team et al. 2025). Formally, given a policy model Te and a prompt distribution p, the RLVR objective aims maximize the expected reward while constraining policy updates within a trust region:
(1) Here, Aa,u (z. v) = r(x, v) - B By [r(x, v)] denotes the advantage function, lold represents the policy parameters from the previous iteration, and 5 defines the trust region boundary that prevents excessive policy divergence.
((0) = - -
[A (2, y)], subject to Eg-D | (T
(-1x)]
< 8.
A key characteristic of RLVR is its reward formulation. Unlike preference-bacee RLHF that relies on subjective human judgments, RLVR employs a verifiable reward function defined as:
1, if y is the correct answer for x
r(x v) = o, otherwise
(2)
The straightforward reward function design partially mitigates the issue of reward hacking.
3

---

Published as a conference paper at ICLR 2026
4 METHODOLOGY
In this section, we introduce CurES, a novel method designed to enhance the training efficiency of Reasoning LLMs. We begin by establishing theoretical connection between gradient optimization efficiency and two key factors: the sampling distribution of prompts and the allocation rollout quantities across these prompts. Based on this analysis, CurES first leverages the estimation of the model's question-ans timal sampling strategy and rollout quantity allocation. By leveraging Bayesian posterior estimation, we progressively refine the confidence in these accuracy estimates using historical sampling data, ensuring robust and adaptive resource allocation with minimal computational overhead.
accuracy to assess prompt difficulty, which is then used to guide an op-
4.1 PROMPT DIFFICULTY CAPS OPTIMIZATION POTENTIAL
To facilitate subsequent derivations, we define the question difficulty as the model's accuracy in answering the question. Given a policy model Ty and a binary reward function r(æ; y) in Eq. (2) the expression for the model's question-ansee answering accuracy pa is given by:
po (x) Byywy [r(2,, y)]-
To investigate how prompt difficulty influences model gradient updates, first consider the follow- ing optimization problem for a given prompt .::
min ((x; 0) min - -Ey-ra (-1z) [ (z, 8]] s.t. DKL (T (-1x) (=+(1==)) S 8.
(4)
Here, separately analyze the impact of different prompts the loss function and theoretically examine how varying prompt difficulty levels affect
training efficiency.
We employ the Lagrange multiplier method to solve the above problem. First, set A = Poed d and reformulate the problem as follows:
d' argmin L(x; Oold + d) + A(DKL (Te ((=a)](Taaaa((( 8).
(5)
By performing first-order Taylor expansion on the loss function L(æ; 0) and second-order Taylor pansion on the KL divergence term DRL. (TA. the following equation:
(- (-)=))] followed by simplification, we derive
> add F(x; A6,
d" argmin L(x; Pota)
VoL(x;
d+ 16=6aa
(6)
where F(x; 0) is the Fisher Information Matrix, a metric quantifying the information that observed data provides about parameter estimates in probabilistic models. In reinforcement learning, it pri- marily serves to construct more reasonable parameter update directions, thereby improving the effi- ciency and stability of policy optimization. Here, F(x; f) is represented as
F(x; 0) EyysT [vo log no (y(a)) ) Vo log []
Through mathematical derivation to address the problem of Eq. (6) we obtain the following results:
d' d-= F (r;; V Vol(r;
(8)
0=0.a
V vaL(x;0)
18=0m F F'1 Oold) VeL(I; 0) 8=8ma
X
28
With Eq. and Eq. we derive the expression for the update of the loss function:
6o d) L(x; Oowa)l
v 2]V
F- (x;(2; Bola) V VoL(x;
(10)
10-0at
According to the definition, the binary reward function r(z, y) serves an unbiased estimator the model's 's of the Cramér-Rao inequality, derive the following fundamental result:
accuracy po (z). i.e., (z) = Eysar [r(z, v)]. Through the application
[ax; ( Osta d) L(x; Osta) S 22ppo.a (x)) (1 Po.a (x)).
(11)
4

---

Published as a conference paper at ICLR 2026
Therefore, for the loss function [(0), its optimization potential exhibits the following relationship with prompt difficulty:
[C(oss + d) [(Onaa) Ez-p (28pu (x) poua (x))
(12)
This demonstrates that the convergence rate of the model's loss function is intrinsically related to the difficulty of the prompt dataset, which is quantified by the model's answering accuracy. To accelerate training. the sampling distribution should assign varied probabilities to prompts based on difficulty while maintaining a balance with exploration. Thus, we seek the optimal sampling distribution p under the entropy maximization constraint. Concretely, we address the following problem:
N S.t.
max Enpp
pa.u (x)) (V 20po (3) p1
+ aH(()),
p(.)) = (13)
Solving the aforementioned problem, we obtain the optimal sampling distribution follows:
exp CEpPVVenan (Vpa..(z)) (2) (1 paase (z))/T
p² (x)
(14)
Ex. exp (VPo.. (x') (1 (=)=(=
where
a hyperparameter. For the theoretical proof please refer Appendix A.I
V25
4.2 CLOSING THE GAP WITH THEORETICAL BOUND
In the previous section, we derived an upper bound on the gradient update for a given prompt. However, due the high computational cost of the natural gradient method, it is often avoided in practice, and the theoretical result is instead used to guide prompt sampling. During actual gradient updates, aim to closely approximate the theoretical efficiency limit within trust region bounded by KL divergence constraint of ò. Specifically, after sampling a batch of m prompts, we seek to optimize operations to approach the bound. Within the curriculum learning framework, we consider optimizing the allocation of rollout quantities across prompts under a fixed total rollout budget of N minimize the following loss function:
min E [((() -((eu)) - (- (EExpp
[V 28po (z) (1 Pe (z))
s.t.
nu
N. (15)
Here, i denotes the updated model parameters obtained from Gold after applying the practical gradi- ent update, i.e.:
To Adas Te. (1/ (z:))
0 = Aoa V)L() p- (())
Σ
(16)
T4
Where 77 is the learning rate and n, denotes the number of sampled rollouts for question 2,. We assume that 1y is chosen such that the policy update remains within KI. divergence constraint of 5.
For convenience, we denote = VoL(0) س= 9 = voL(() 0= By simplifying the loss func- tion, we can show that the optimization problem reduces to the following:
min D TV(0)g.
s.t. Σ N
(17)
The theoretical gradient direction g is typically unknown, and we seek to control the uncertainty of the estimator in all possible directions. Therefore, instead minimize the total variance Tr(V(g)),
which corresponds to uniformly reducing the variance in all directions. This approach is a widely adopted technique for variance estimation(Bottou al. 2018 Papini et al. 2018 Wang al. 2013 In other words, we consider the following optimization problem:
min Tr(V())), s.t. Σ n N.
(18)
5

---

Published as a conference paper at ICLR 2026
By expanding the variance of the aforementioned gradient, we isolate the rollout quantities n; to facilitate analysis:
m Tr V
VyrtTos (h (ys 0osal)
m S.t.
min
n4
N.
(19)
m72
74
Ao,a (Zi, Uj)) By applying the Lagrange multiplier method to solve the above problem, we obtain the optimal solution as follows:
where (1, y: 8)
Vono (x)
mew (35
V n4 Σ, N, o: - TT (Venrew (h (13
(20)
The remaining challenge is computing d, By expanding the variance and noting that the ad- vantage function can be evaluated based on whether the rollout y is correct, i.e. Agma (z. y) = I(y is correct for z) Pa.. (z), derive the following symmetric computational form:
Tr
(Vy-Toue (h (v, Iz; Dosa))
=Pa. (2x) (1 Pa (xx)))
E
V log Te
IP]
youte
+ (x:e) (1 Peas (2+4)
E
V log (
(21)
you
-z: Pe (x:)))
E
Ve log To (B)
E
Vo log To (3)22))
yousTe
The optimized formula decomposes the variance estimation problem into two categories based on answer correctness, integrating it with the prompt difficulty estimation from Section 4.1 By lever- aging algebraic operations on prompt difficulty and policy gradients, reuses difficulty estimates from sampling and transforms variance estimation into more tractable form. The theoretical proof provided in Appendix A.2
4.3 PROMPT DIFFICULTY ASSESSMENT AND ALGORITHMIC IMPLEMENTATION
Estimating prompt difficulty is crucial for both sampling questions and allocating rollout quanti- ties. However, difficulty changes dynamically during policy training, making accurate estimation challenging. A straightforward approach is to add pre-evaluation step before each sampling, but this increases computational overhead and fails to leverage new samples for posterior estimation to improve confidence. To address this, propose a Bayesian inference framework that decomposes rollout into a multi-stage mini-bated process. This refines the posterior estimation of the dataset, dynamically adjusting the sampling distribution based on updated difficulty assessments.
Specifically, as the model Thw rollouts on a prompt x; multiple times, the number of correct answers follows binomial distribution with success probability Pe,, (z;). We can assume that (=... (4;) fol- lows a Beta distribution, the conjugate prior of the binomial distribution, which is a widely adopted technique in Bayesian inference (Kruschke, 2010 Qu al., 2025a):
(x;) Beta(on((:)) (o(()))
(22)
where on (24) and 8o(x;) can be interpreted as the counts of correct and incorrect answers during sampling, which can be initialized using a small batch of sampled data for cold-start estimation.
Since the Beta distribution is conjugate the binomial likelihood, the posterior distribution remains Beta-distributed after observing new samples. Let ox-- (xx,) and 3 (a;) denote the cumulative counts of correct and incorrect answers for prompt 2; up to step t 1. If. step t, mini-batee generates n, answer with correct, the posterior distribution for Pe.. (zx) after steps is:
(z;;) = (2,) + s,
(x,,)) = (1, + T4 Pe.. (2:) (2ta,
(23)
(24)
To reduce randomness, we use the mean of the Beta distribution to estimate prompt difficulty in our experiments.

---

Published as a conference paper at ICLR 2026
Algorithm I: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs (CurES)
1: Input: initial policy model Te; reward function r(z, y); prompt dataset D { number of iterations T; prompt batch size m; learning rate 7) parameter 7; total rollout budget N; parameter estimation sample size N'; number of steps per iteration M. 2: for each t E 1, I.....T T do
3: Parameeter Estimation Stage
4: for each 2; E "D, do
5: Sample & N' rollouts (y.,1..... "/.kk) ~ (-124-
6:
Compute rewards r(zy, Bt.j).
7:
Initialize counts and difficulty estimations p(r;) according to Eq. (22)
8:
Initialize sampling probabilities p" according to Eq. (14)
9: Compute gradient contribution according to Eq. (21).
10: end for
11:
> Large-scale Training Stage
12:
for 1, M do
13:
Update the old policy model new f- Tg.
14:
Sample batch of prompts B [ with replacement according p².
15:
Obtain rollout quantities n, for x; E B according to Eq. (20).
for each 2; B do
17:
Sample k = rollouts (...1...., 3ink } ~ Teau Teau(-1a)) (-12;).
Compute rewards r(z,, Vt.j).
19:
Update counts and difficulty estimations p(x;) according to Eq.
23
end for
21:
Update sampling probabilities o" according to Eq. (14
22:
Update policy mo by applying RI. training.
23:
end for
24:
end for
25:
Return mo-
This estimation approach enables modeling and estimating the difficulty of each prompt with mini- mal overhead. However, the model's performance evolves during training, the estimation process susceptible to distribution shift, which becomes more pronounced with increasing training steps. To mitigate this issue, we adopt straightforward solution inspired by GVM. Specifically, the dataset into 7' non-overlapped subsets and perform iterative training on these subsets. We train the model for fixed training steps of M in every iteration. The estimations of prompt difficulty and gradient variance are reset when a new iteration begins. This method effectively alleviates distribu- tion shift without introducing significant computational overhead. Moreover, the iterative process allows the model to adaptively adjust its sampling allocation based on its own evolving capabilities throughout training. For further details, please refer to Algorithm[] In each iteration of training, our algorithm operates in two phases. The first phase performs lightweight parameter estimation stage, where for each prompt n,, we draw a small sample nary estimates of its prompt difficulty and corresponding gradient variance. The second phase is the large-scale training stage, where each step we sample a batch of m prompts and collect a total of N rollouts, corresponding average of N/m rollouts per prompt.
divide
rollouts. This enables us to obtain prelimi-
EXPERIMENTS
In this section, present comprchensive experimental results and analysis of our CurES with other baselines. Our experiments focus on the following research questions:
RQ1: Does CurES outperform other related baseline methods across various benchmarks?
RQQ2: How does CurES adapt its sampling strategy to accelerate learning efficiency?
RQ3: Does CurES enhance sampling efficiency compared to other baseline methods?
Training Details. We employ VERL (Sheng et al. (025) as our training framework and initialize our policy using Qwen2.5-Math models (1.5B and 7B parameters). For the training dataset, utilize

---

Published as a conference paper at ICLR 2026
Qwen2.5-Mathh 3-1-5--Mthh----- 15B-GRPO
Owen2.5-Math-1 1.5-R-
Qwen2.5- 1aan225--Mtt----BBR MM-Mt----RPPPP
0.000
0.850
0.82
Average Accuraey
0.775
0.00
0.590
0.78
0.755
0.775
0.700
0.675
0.723
0.72
0.650
0.075
0.700
0.67%
0.000
0.70
0.68
GVM
CurEs
150
Step
Step
Step
Figure 2: Comparison of learning curves between CurES and GVM across different backbone mod- els. CurES consistently outperforms GVM under the same number of training steps, demonstrating more efficient utilization of samples.
Owen2.5-Matb-11.5
Qwen223-Math-7B
Eixtmmaed Accuracy
8.
0.8
Neration
Iteration
Figure 3: The evolution of the estimated accuracy distributions for the Qwen2.5-Math-1 1.5B (left) and 7B (right) models across 15 iterations. Each violin shows the distribution accuracy across samples: the width reflects density, the central line marks the median.
Mumina-Math (Li al. 2024a), partitioning into subsets following GVM (Yao et al. 2025). We conduct iterative training across these subsets, resulting in 15 training iterations. At the beginning of each iteration, we perform rollouts per prompt to establish an initial difficulty distribution and an assignment of rollout quantities under total sample budget of 8 x 1024. During training, we sample prompts according to the difficulty distribution with replacement and conduct rollouts according to the assigned rollout quantities. To make fair comparison with GVM, we train 10 steps in each
iteration. We employ GRPO (Shao et al.) 2024) and REINFORCE++ (RPP) (Hu et al. 2025) as advantage estimators for all methods. The learning rate is set to constant x 10-6.
Evaluation Benchmarks. To evaluate the complex reasoning capabilities, we choose a broad set of challenging reasoning benchmarks, including MATH500 (Hendrycks al. (02T). AIME 2024 and 2025 (Li et al.] 2024a), AMC 2023 (Li et al. 2024a), GSMSK (Cobbe et al. (021). Gaokao- EN 2023 (Zhang et al.] (023). Mineva (Lewkowycz al. 2022) and Olympiadßench (He et al.) 2024). These benchmarks comprehensively evaluate mathematical reasoning capabilities. Since AIME 2024, 2025 and AMC 2023 are highly challenging competition benchmarks, which are of limited sizes of test samples, we present the results averaged over 16 runs.
Baselines. To demonstrate the reasoning ability of our CurES, we compare it with many strong base- line methods: GRPO (Shao et al. 22024), RPP (Hu et al. 2025), Speed-RL (Zhang et al.) (2025) and GVM (Yao et al. 2025). Specitically, GRPO and RPP are commonly used in training mathematical problem solving models. Sp is an adaptive online RI. curriculum that selectively chooses samples of intermediate difficulty to maximize learning efficiency (i.e., samples whose accuracy is not or 1). GVM is prompt-specific dynamic sample allocation strategy designed to minimize stochastic gradient variance under a computational budget constraint.
5.1 OVERALL PERFORMANCE (RQ1)
We present the learning curve of Owwe2.55-Mtt-1155 and 7B models trained with different meth- ods in Figure Across all configurations, CurES exhibits higher progressive and final accuracy compared to GVM. This advantage originates from two key differences: (i) CurES adaptively allo- cates prompt sampling probabilities based on estimated success rates, which our theoretical analysis confirms enhances training efficiency; (ii) while GVM monotonically decreases rollout allocation as accuracy increases, CurES allocates more rollout budget to prompts of moderate difficulty in Figure 4 resulting in consistent training gradients and improved training stability.
8

---

Published as a conference paper at ICLR 2026
Table I: Quantitative results of different methods across various datasets. The best and second best results in bold and underlined.
Pass@1
Average 16
Method
Avg.
MATH500
GSM8K
GAO23
MINERVA
OLYM
AIME24
AIME25
AMC23
Qwen2.5-Maie
40.20
43.90
25.19
11.40
21.04
1.67
1.67
14.84
20.00
+GRPO
73.80
86.43
48.83
27.94
35.41
8.54
6.67
45.47
41.64
+RPP
64.80
82.94
42.08
21.32
29.19
4.17
3.33
39.06
35.86
+Spced-RL-GRROO
68.80
85.67
47.14
27.40
35.56
12.08
6.88
47.19
41.34
+Spped-RR RPP
65.80
85.67
48.31
27.94
36.30
12.58
10.08
47.34
41.75
+GVM-GRPO
74.80
84.23
48.83
27.21
35.56
10.21
11.25
50.47
42.82
+GVM-RPP
75.40
84,00
49.61
24.63
35.56
11,46
6.04
50.94
42.21
+CurES- GRRPO
77.20
85.97
51.43
31.62
37.33
13.33
10.42
52.19
44.94
+CurES-RPP -PP
75.40
85.82
51.43
28.31
37.04
12.71
11.46
50.94
44.14
Qwen2.5-Mathe
60.20
72.40
44.68
22.79
30.81
7.92
1.88
27.19
33.48
+GRPO
80.00
91.43
51.43
31.99
38.37
20.00
10.00
57.50
47.59
+RPP
81.20
91.89
55.58
39.71
40.00
18.54
11.67
62.81
50.18
SSpeed-RL-GRYY
82.80
88.70
55.58
29.41
42.37
20.21
11.46
60.16
48.84
+Speed-RL
78.60
91.81
53.77
37.13
42.96
17.29
12.08
62.81
49.56
81.60
91.28
34.03
32.72
42.67
23.54
15.00
64.31
50.64
+GVM-RPP
81.60
90.07
55.32
29.04
40.30
17.50
8.33
53.44
46.95
(CurES-GRPO
84.80
92,27
56.62
37.87
43.56
24.58
15.21
64.38
52.41
-CuPES-RPP
81.80
91.89
54.55
33.09
40.59
23.33
12.92
58.75
49.62
Iteration
Iteration
Iteration
40
10
x:0.5
15
15
Rallout Quantity
30
20
30
25
25
25
20
20
20
1.5
10
0.0
0.2
0.4
0.5
0.8
1.0
0.0
0.2
0.4
2.6
0.5
1.0
0.0
0.2
0.6
0.8
Accuracy
Accuracy
Accuracy
Figure 4: Allocation of rollout quantities with respect accuracy in CurES at different training iterations. CurES concentrates more rollouts on moderately difficult prompts.
To demonstrate the effectiveness of our CurES, we compare with representative baselines that trained for the same number of steps. The main results demonstrated in Table [ The sults clearly demonstrate that CurES consistently outperforms GVM and other baselines with both GRPO and RPP advantage estimators. Across both model scales, CurES establishes state-of-the- art results on several datasets and consistently matches or surpasses the strongest baselines across all settings, confirming the superior generalization ability of our CurES.
5.2 SAMPLING BEHAVIOR (RQ2)
Figure illustrates the evolution of the difficulty distribution for both the Qwen2.5-Math-1155 and 7B models throughout the training process. At iteration 1, the estimated accuracy broadly dis- tributed. As training progresses, this distribution shifts toward higher values and becomes more con- centrated, indicating that the models are effectively learning and mastering the presented samples. This shift also underscores the importance of redistributing prompt sampling probabilities, as the models' success rates on problems are primarily bimodal, concentrated at high and low values. Such redistribution enhances training efficiency, whereas uniform sampling followed by answer genera- tion and accuracy-based filtering significantly reduces efficiency under this bimodal distribution.
Meanwhile, another view of how the CurES method dynamically adjusts the rollout quantities as- signment during training is presented in Figure 4 The trend lines of all iterations approximate "bell-shaped" distribution, with prompts intermediate accuracy allocated more rollout quantities, as anticipated given their high efficiency. Furthermore, the training procedure continues, the dis- tribution becomes progressively sharper and narrower, indicating that CurES dynamically increases rollout quantities for moderately difficult prompts. This pattern aligns with the observation in Fig- ure 3, which shows a gradual reduction moderately difficult prompts as the model improves. By adaptively increasing rollout quantities, CurES compensates for their diminishing presence, ensur-

---

Published as a conference paper at ICLR 2026
Qwen-Math--155BB -5--CrFF-GGRPO
Qwen-Math- 1.5.hl1---B--Cr
Average Accuracy
0.74
0.72
0.72
0.70
0.70
0.00
0.50
0.66
0.66
0.64
0.64
0.62
15
Step
Step
N'
N W-4,po n=10
W-+3, neB
N'=16, mm8
Figure 5: Performance convergence of CurES on MATH500 with different sampling configurations.
5.5x
Average Accuracy
0.773
0.750
Mmmming My
0.750
1.75x Faster
0.700
0.675
3.000
Qwen2 2-ann2---aahhhh1 -8S-GRPO
0.650
Qwen2.5-Hait -88-h-
SoudS-Quwen-SS-Maatte S8--APO
CurfS-Quen2.5- Saa.eis---t-th-----00
300
200
350
400
300
800
0
50
100
150
200
250
300
330
Step
Step
Figure 6: Efficiency comparison of CurES against baselines on MATH500. The gray dashed lines mark the number of steps CurES and each baseline required to reach the highest average accuracy within the total training duration.
ing they remain a substantial portion of each training batch. By coupling difficulty bbaed prompt sampling with the adaptive sample sizing, CurES sustains an abundance of informative prompts and therehy maximizes performance gains per step.
5.3 EFFICIENCY ANALYSIS (RQ3)
To analyze the effect of different combinations of parameter estimation sample size (N') and training-phin phase sample budgets coefficient (n), indicating training-phase sample budget of N m X on the convergence of model performance, we conduct a series of experiments, as shown in Figure 5 A larger N' leads more accurate initial accuracy estimation, while a larger n pro- vides a greater computation budget. The results show that increasing either N' or does not yield proportional performance benefit relative to the increased computational cost. This finding confirms that the core mechanism of CurES efficiently guides the model toward high-yield learning samples without the need for extensive computational overhead. In practice, relatively small values of N' and are already sufficient for effective training, highlighting the high sample efficiency of our method.
We also provide a direct comparison of CurES against GRPO and RPP in Figure 6 The plots show the learning curve of cach method over training steps. CurES-GRE achieves the same peak perfor- mance as the GRPO in just 5.5 x fewer steps. Similarly, CurES-RIE -PPP reaches its peak performance 1.75x faster than the RPP baseline. The remarkable sample efficiency is direct consequence of CurES's s ability to consistently provide the model with optimally challenging samples.
6 CONCLUSION
In this paper, we propose CurES, a theoretically grounded curriculum learning algorithm for RLVR. By linking gradient efficiency accuracy, our approach adaptively prioritizes training prompts of optimal difficulty and dynamically allocates rollout budgets. Beyond the theoretical analysis, our algorithmic design leverages Bayesian framework to track prompt difficulty in a lightweight yet adaptive manner. Specifically, we model the success rate of each prompt instance with a Beta dis- tribution, which naturally incorporates prior information and posterior updates as new rollouts are observed. Combined with the derived sampling distribution and variance-based rollout quantity al- location, this Bayesian mechanism ensures that both question selection and sample budgeting adapt dynamically to the evolving policy, thereby maximizing training efficiency in practice. Experiments on a wide range of mathematical reasoning benchmarks show that CurES consistently outperforms strong baselines in both accuracy and convergence speed, demonstrating superior sample efficiency.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Sanghwan Bae, Jiwoo Hong, Min Young Lee, Hanbyul Kim, Jeong Nam, and Donghyun Kwak. Online difficulty filtering for reasoning oriented reinforcement learning. arXiv preprint arXiv:2504.03380 2025.
Léon Bottou, Frank E Curtis, and Jorge Nocedal. Optimization methods for large-scale machine learning. SIAM review, 60(2):223-311, 2018.
Shuang Chen, Changlun Zhang, and Haibing Mu. An adaptive learning rate deep learning optimizer using long and short-term gradients based on g-l fractional-order derivative. Neural Processing Letters, 5662):106, 2024.
Xiaoyin Chen, Jiarui Lu, Minsu Kim, Dinghuai Zhang, Jian Tang, Alexandre Piché, Nicolas Gontier, Yoshua Bengio, and Ehsan Kamalloo. Self-evolving curriculum for Ilm reasoning. arXiv preprint arXiv:2505. 14970, 2025.
Anshuman Chhabra, Bo Li, Jian Chen, Prasant Mohapatra, and Hongfu Liu. Outlier gradient analy- sis: Efficiently identifying detrimental training samples for deep learning models. arXiv preprint arXiv:2405.03869 2024.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, et al. Training verifiers to solve math word problems. CoRR, abs/21 14168, 2021. URL httpss///arxivoog/////1111111111
Steffen Dereich, Arnulf Jentzen, and Adrian Riekert. Learning rate adaptive stochastic gradient descent optimization methods: numerical simulations for deep learning methods for partial differ- ential equations and convergence analyses. arXiv preprint arXiv:2406.144 14340, 2024.
Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint 2wxXiv:2501.29444 2025.
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. arXiv preprint arXiv:2402.14008, 2024.
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2 103.03874 2021.
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rihf algorithm with robustness to both prompt and reward models, 2025. URL http///s///arriiorg/////2200
John K Kruschke. Bayesian data analysis. Wiley Interdisciplinary Reviews: Cognitive Science, 1(5); 658-676, 2010.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brah- man, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. T," ulu 3: Pushing frontiers in open language model post-training. arXiv preprint 2rXiv22411.55124 2024.
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ra- masesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solle et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Huang, Kashif Rasul, Longhui Yu, Albert Q Jiang, Ziju Shen, et al. Numinamath: The largest public dataset in ai4maths with 860k pairs of competition math problems and solutions. Hugging Face repository, 13:9, 2024a.
Xinyu Li, Wenqing Ye, Yueyi Zhang, and Xiaoyan Sun. Grace: Gradient- active learning with curriculum enhancement for multimodal sentiment analysis. In Proceedings of the 32nd ACM International Conference Multimedia, pp. 5702-5711, 2024b.

---

Published as a conference paper at ICLR 2026
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. Cppo: Accelerating the training of group relative policy optimization- based reasoning models. arXiv preprint arXiv:2503.22342, 2025.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, et al. Deepscaler: Surpassing ol ol-preview with a 1.5 model by scaling rl. Notion Blog, 2025.
Daniil Medyakov, Gleb Molodtsov, Savelii Chezhegov, Alexey Rebrikov, and Aleksandr Beznosikov. Variance reduction methods do not need compute full gradients: Improved ef- ficiency through shuffling. arXiv preprint arXiv:2502. 14648, 2025.
Matteo Papini, Damiano Binaghi, Giuseppe Canonaco, Matteo Pirotta, and Marcello Restelli. Stochastic variance-reduced policy gradient. In International conference on machine learning, pp. 4035-4035. PMLR, 2018
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Björn Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models? arXiv preprint arXiv:2507.04632, 2025a.
Yun Qu, Qi Cheems Wang. Yixiu Mao, Yiqin Lv, and Xiangyang Ji. Fast and robust: Task sampling with posterior and diversity synergies for adaptive decision-maker, in randomized environments. arXiv preprint arXiv:2504.19133 2025b.
Sebastian Ruder. An overview of gradient descent optimization algorithms. arXiv preprint
arXiv:1609.044 04747, 2016.
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient RLHF framework. In Proceed- ings of the Twentieth European Conference on Computer Systems, EuroSys 2025, Rotterdam, The Netherlands, 30 March 2025 3 April 2025, pp. 1279-1297. ACM, 2025.
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetun- ing via adaptive curriculum learning. arXiv preprint arXiv:2504.05520, 2025.
Mingyang Song, Mao Zheng, Zheng Li, Wenjie Yang, Xuan Luo, Yue Pan, and Feng Zhang. Fastcurl: Curriculum reinforcement learning with stage-wise context scaling for efficient train- ing rl-kike reasoning models. arXiv preprint arXiv:2503. 2025.
Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du, Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with llms. arXiv preprint arXiv:2501.1259) 2025.
Chong Wang, Xi Chen, Alexander J Smola, and Eric Xing. Variance reduction for stochastic gradient optimization. Advances in neural information processing systems, 26, 2013.
Qi Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predictive task sampling for efficient and robust adaptation. arXiv preprint arXiv:2501.11033 2025.
Tian Xie, Zitian Gao, Qingnan Ren, Haoming Luo, Yuqian Hong, Bryan Dai, Joey Zhou, Kai Qiu, Zhirong Wu, and Chong Luo. Logic-rl: Unleashing Ilm reasoning with rule-based reinforcement learning. arXiv preprint arXiv:2502.14700 2025.
Yixuan Even Xu, Yash Savani, Fei Fang, and J Zico Kolter. Not all rollouts useful: Down- sampling rollouts in Ilm reinforcement learning. arXiv preprint arXiv:2504.13811 2025.
Hongru Yang, Bhavya Kailkhura. Zhangyang Wang, Yingbin Liang, al. Training dynamics of transformers to recognize word co-occurrence via gradient flow analysis. Advances in Neural Information Processing Systems, 37:46047-4611 17, 2024.
12

---

Published as a conference paper at ICLR 2026
Jiarui Yao. Yifan Hao, Hanning Zhang, Hanze Dong, Wei Xiong, Nan Jiang, and Tong Zhang. Optimizing chain-of tf-uuuttttthtt reasoners via gradient variance minimization rejection sam- pling and RL. CoRR. abs/2500.02391, 2025. doi: 10.4850/ARXXI.:555000222 URL https: (/dbi.org/10.48550// 10044455500/XX1225555022235
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, al. Dapo: An open-source Ilm reinforcement learning system at scale. arXiv preprint arXiv:2503.14476. 2025.
Huizhuo Yuan, Yifeng Liu, Shuang Wu, Xun Zhou, and Quanquan Gu. Mars: Unleashing the power of variance reduction for training large models. arXiv preprint arXiv:2411. 2024.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- zoo: Investigating and taming zero reinforcement learning for open base models in the wild. arXiv preprint arXiv:2503. 10022503.88899 18892, 2025.
Ruiqi Zhang, Daman Arora, Song Mei, and Andrea Zanette. SPEED-RL: faster training of reasoning models via online curriculum learning. CoRR, 0000//25000000066, 2025, doi: 10.48550/ARXIV. 2506.09016. URL https://doi.org/ 00444485505XXX.22500000000
Xiaotian Zhang, Chunyang Li, Yi Zong. Zhengyu Ying, Liang He, and Xipeng Qiu. Evaluating the performance of large language models on GAOKAO benchmark. CoRR, abs/2305. 12474, 2023. doi: 10.48550/ARXIV.2205..222 12474. URL https:://///// http::/www......0.0aa/a.........
Hongru Zhao and Jinchao Xu. Convergence analysis and trajectory comparison of gradient descent for overparameterized deep linear networks. Transactions on Machine Learning Research, 2024.
13

---

Published as a conference paper at ICLR 2026
A MATHEMATICAL DERIVATIONS
Lemma (Cramér-Ras Rao Inequality) Let {po(x), Q} be Cramér-Rao regular family with parameter space ④ C R where the Fisher information matrix I(0) is non-singular Let g (g) (0), ... ,9, (0)) for S k, and assume the partial derivatives dgi (6) /00g exist for all 1,.... and = 1, ................................ k. Suppose T(X) is an unbiased estimator of g(0) with finite second moment. Denote G(0) = Veg(O), then we have
Vo(T(x)) 2 (OG))
(25)
A. PROMPT DIFFICULTY CAPS OPTIMIZATION POTENTIAL
Given an individual sample z, we first consider the optimization problem as follows:
min [(x;; 0) = min - -Byono(-la) [A (x, v)],
(26)
s.t. (TI,(TO (- (-1x)]l=o0-(a))) S 6
We define f Oota + and rewrite this constrained optimization problem via the Lagrange multiplier method:
d' argmin L(r; + d) + A(DKL (TTea (-1x)] +a) -5)-
(27)
d
Using the Taylor expansion formula, we have:
d' argmin L(x; Gold + d) + A(DKL(TTa.) (=))) ((x)))) - 8)
arrgmin O(x;) A6
dd (Te
0=0wrd
(28)
We first compute the first-order and second-order derivatives of the KL divergence term:
V V DKL (Tosia (-Ix)]| 88
VoByNtoN Dog Ten (1)22) [
VO
Dog To (y(x=)] la-
0=
Ey
Vo log To [2|2))
10=0.a
VoTo(Ul>))
E Bynttous
To(()
0=0as
(29)
V (2)
V.
Σ
nep <
V
-0.
14

---

Published as a conference paper at ICLR 2026
V DKL aT (: (-)
Vy
[log (2) [z)]
0000
Ey
Vi log NT (3)
0=0as
VOTO(yl>)
Ey By
V
(T No (r)
VoTo(yl=)) T
(30)
Eg Taaa
T (y)
V (y
(
V (o
(VoTo (yla))
E sy
+ Byrree
To. (3)xz)
(y(a))
P==Pus
-
(vo lologae no (x) log (v)l)))
lmmaus
=F(;; Bola).
where F(x;0)
Ve log (el log (x)l)))) is termed the Fisher information matrix. E Ey
Therefore,
d' = argmin ((x; ( + VoL(x;;) 0=0m
+ dd F(x:0os)) - 26.
(31)
To find the minimum, we take the derivative of the right- hand side and set it to zero:
VoL(x; 0)
+ AFF(a;; foaadd )d = O.
(32)
8=0ms
So we have
-
F F (2; Ooia VoL[(()
(33)
0-00s
We now derive the critical point of the constraint condition:
d F(x; flaadd ze (KL (am((a))) =
(34)
By simplifying, we obtain
[oL(x;
1- 1 (x; VoC(; (1);0]0)00000 =
(35)
2 X2
Therefore, we have obtained the critical value of A:
X VoL(x;0)
8==8m (x; 00t) VoL(I; 0) A=
(36)
28
In this case, the change of the loss function is computed as
C(x; Oat + d) -(x; VoL(z;0)
8=
VoLxx 10-0A0
1 A
1)T P-1(xx
Doas VoL(I; 1)
(37)
2 O)T 10= P-1 (x; Osta) V vaL(x; 0) B=Bus
15

---

Published as a conference paper at ICLR 2026
Furthermore,
V (e 0-00s
Vo
By to [Aau (2, v) B=0as
(oTo( (r(2,)) v) - Byrroou (r(2,)))) l=ma
Σ (omo(y) (r(z, la
(38)
Vol (r(z, v))
-VoPo (=)
where po ( is the model's question-answering accuracy.
Consider that r(x, y) is unbiased estimator of pe (I), according to the Cramér-Rao inequality, obtain:
(C(Oala + d) L((oma)) | Pxp (x; fala d) [(x; Bota)]
< Exap
(L:;; Ool + d) ((x; Onta) |]
Ezup
28 20VoL(x; 0=0 F (x; VoL 0=
< Ezzp
v oo (r (r(,r,)))
Exup
280pe (x) (1 Po (x))
(39)
This indicates that the optimization potential of the loss function inherently related to the diffi culty of the prompt itself. To balance the trade-off between exploration and exploitation, we derive the optimal sampling distribution by solving the following objective function under the maximum entropy constraint with the hyperparameter
max Egop [V 20po.u (2) (1 Paa (x)) + aH(())
N
(40)
s.t.
Σ p(xx))
1, p(xs) 2 0
1=1
To find the optimal distribution p. we employ the method of Lagrange multipliers. The objective function becomes:
N
N
max J max Σ plz;)) 28pam (Iz) (1 - Pa.u (I²)) Σ log j=1
1
p(z;)
j=1
(41)
where y is the Lagrange multiplier associated with the normalization constraint.
Taking the partial derivative of J with respect to p(x,)::
0J
(8pp (1 Pa (xz)) aloo (logg((z;) + 1) p.
(42)
Ōp(z,)
Then set the derivative zero:
/ (20) Po (x;)) a (1o0 +1) O.
(43)
Solving for log p(x;):
log p(zz) / 20pp (Iz) (1 Pe (r,)
(44)
a
So we have:
220po (2)p Pa (r,))
11
p(x;) = exp
(45)
16

---

Published as a conference paper at ICLR 2026
Let exp (-1 a), then Eq. 45 becomes
/20po (zz) (1
Pe (z₂))
p(z;)) cexp
(46)
a
Using the constraint Σ)=1 p(zz)
1. we can obtain
c=
(47)
/20po (1oppo (t,))
Σj exp
So the optimal distribution is
28pea (x) Po (x))
exp
p" (x)
(48)
26pe (x' (1 (e'))
Ex' exp
With the substitution
V25* the distribution finally becomes:
exp (V)e.. (1 pa.. (I)))/)
(x(x)
(49)
Ex exp ( (VPomu (x') po.. (x 1)/T
A.2 CLOSING THE GAP WITH THEORETICAL BOUND
In the previous proof, we established a lower bound for single-step gradient descent within a ò-local trust region constrained by KL divergence, i.c.,
C(Bou + d) L(Bols) Expop [x::; Bodd + d) L(x;; Goaa)l
= Egpo
20VoL(x;0)
1= Fx Gold VeL(x; 0) ())
(50)
2 -Ego
an (r(x,y)))
= -E
V2ōpea (x) (1 Pa p())(())
However, due the high computational cost of the natural gradient method, it is often avoided in practice, and the theoretical result is instead used to guide prompt sampling. During actual gradient updates, we aim to closely approximate the theoretical efficiency limit within trust region bounded by KI. divergence constraint of &. Specifically, after sampling batch of prompts, we seek optimize operations approach the bound. Within the curriculum learning framework, we con- sider optimizing the allocation of rollouts across prompts under fixed total rollout budget of N to minimize the following loss function:
min E [C(0) L(Mun)) - (-Ezmp [V 28po... (2) (1)
Po... (x))
S.t.
N. (51)
Here, i denotes the updated model parameters obtained from las after applying the practical gradi- ent update, i.e.:
m
no (v( TR.a (Mj (x;)
0 = 0sss Vo 2(0)
ABas
(52)
T1:
Where 7 is the learning rate and n; denotes the number of sampled rollouts for question E,. We assume that 1) is chosen such that the policy update remains within KI. divergence constraint of 5.
17

---

Published as a conference paper at ICLR 2026
In fact, ((0) is an unbiased estimator of [()), that is:
[(0
E
To (x:))
E
Anaa (aras mab
aaw (Ug (2:4)
1=1
My EDD
1
TO To(y Aen (1 H1)
E Bampp-yy~a.. (1
m
T4
Towe (2 (20))
D,EDD
(53)
(1((2))))
B BusrppyyTa (-12x)
Ana (1 w)
To (ul
To (U/or)
Ex-partoa (12))
Aoca (20, v)
Tow (3/120)
-Benp.y
[Aaul (2,, 8)
-[(0)
By applying the Taylor expansion, we obtain:
[(0) - L(fora) ~V 0=0.0 - Pola) -
(54)
=- -V)() 10=0.0 VoL(o) 1= (0000)
where
Val(0) = E P.pppyymaa (-12r)
Anasa (2c, 1) Vo log To To [x)]
(55)
m m
VOTO (y)l()) ARna (2 311))
VaL(0)
Σ
(56)
Tea (y3 (x;)
My My
For convenience, we adopt the following notation:
g VoL(O) 9 -vo((0) 0=
(57)
A Ahho = Egaup
25 pa (x) (1 Pa.u (x))
(58)
Therefore, the original problem can then be simplified as follows:
E (C()) L(Dota) (-Eamo (V 228pu (x) (1 po (x))
=E (-ngg T 9 + Athoo)
']
(59)
2 E[g o) tig Elg g] + 2ho0
Because L(0) is unbiased estimator of L(0), we have
E
VoL(0)]
VolE [((0)] VoL(O)
(60)
E g
E
V
L()
VoE
[(0) Vol(o)) =9
(61)
Therefore, we can obtain
E
9 g] = TE [ gg
(62)
Now, regarding the first item:
[
0)
Eig 99 9) 9
_Elgg 1
(63)
18

---

Published as a conference paper at ICLR 2026
B[gg V(0)
| T V(g) 99
(64)
Hence, the original problem is equivalent to the following formulation:
E (C(O) _((ola) (-Bemp [v 128pmau (x) (1 po (x))
(65)
=n² (1 V()g + (0 (T=x)² 2n/e (g + +
Since we aim minimize the gap from the theoretical update lower bound by reallocating the rollout quantities per question under a total sampling budget of N, the simplification of the objective function reveals that this problem only affects the first term V(g)g. Thus, the original optimization problem is equivalent to the following:
min g VV()g.
s.t.
N.
(66)
1=1
The theoretical gradient direction is typically unknown, and we seek control the uncertainty of the estimator in all possible directions. Therefore, we instead minimize the total variance Tr(V(g)). which corresponds to uniformly reducing the variance in all directions. This approach is a widely adopted technique for variance estimation(Bottou al. 2018 Papini et al. 2018 Wang et al. 2013 In other words, we consider the following optimization problem:
min Tr((()),
s.t.
Σ
N
(67)
1=1
Since each yj is independently draws from Thw we can conclude that:
m
1
(uso( (i) Aasu (xrs Mz)
V())=
V
77
72
Tew (U, (x:))
ED
(68)
VOTO(UJ (x;)) 0= Anas (vis, (vi, Mi)
Thi
Tha (1) (.))
MyED,
Let (1,,910) 0:
Vomo(y me)
Ao (Iis Vj), have
Jeeu (1l (r.))
V
Me 3ig Oold)
m
M/ Meu
1=1
771
(VyNTo (h (e: Patal)
(69)
14
772
72
(y. 2e. Bata))
y
Therefore, for the total variance Tr(V(0)):
Tr
V Vy-res (h (us xi;; 0onal)
Tr(V(O))
(70)
m2
14g
E=1
19

---

Published as a conference paper at ICLR 2026
Then dive into calculating the value of Tr (V(h(y, ([1p//(yx))) n4; doa))))))):
Tr
(V you (h(y, 0oia))
Tr VOTTO V
-ua
Abss
(Te. (2)(2)))
V Votolylaz 1=0= 0=
VOTO (o)o(() o= Anau
Tr Bag
Anas
To (y) (4i)
Toaa (2)(3r)
(71)
-Byssaoss
[Tr Voloogooyll Vo log To (ulos) lo= Abaul
Tr BByrroN
Ve (og) Anul Byrroon [V log
-Eynsoss
l= Ve log To T (1 (0= A [Tr (Ve log To (2)
Tr EBy-ree
V log To (y nu Aouel Eng-srers Vo log Ne (y [.)
new Ansul)
=Bynross
Vo log To 2 10=0mu1l Abue] ByrnTees Vo log No (y)
Consider the advantage function defined as:
As (x, v) rzz, y) Byrro (r(z, 8)).
(72)
We classify the rollouts into two categories based on whether the final answer is correct or not:
(V Yynnoos (h (g,, zz: (ola))
=P(r 1 [r:))
E
Vo log No (r 1=) 3 (1 Po (2:
y you
+ P(r 0(z;))
E
log To (1) (po. (xx)))
y
y
- P(r = 1|xx))
E
Ve log (y Pena (xx))))
yo
Vo log (- po.. (1:))]|
+ P(r 0/æz))
E
ymuto
Po (x:) (1 poaa (x;)))
E
logooooool [1]]
+ (I:)) (1 pe (x+))
E
V log log l= 12]
you
Po (o; (x:)) - po (r:)))
E
V log
E
V oo
y-u
of
(73)
Therefore, need solve the following problem:
min
S.L
= N.
(74)
We also employ the Lagrange multiplier method to solve this problem:
min
min
1
74 N),
(75)
m
11
where j is the Lagrange multiplier.
20

---

Published as a conference paper at ICLR 2026
Qwen2.5-Math-1.57
Qwon2.5-Math-1.5B
Gradient Norm
0.08
CurES-RPP
CurES-GRPO
RPP
GRPO
0.04
0.04
0.02
0.02
Qwen2.3-Math--7B
Qwen2.5-Math-77
Gradient Norm
CurES-RPP
CurES-GRPO
1.00
RPP
0.8
GRPO
0.75
0.6
0.50
0.4
0.2
0.00
0.0
Step
Step
Figure 7: Comparison of Average Gradient Norms. This figure compares the average gradient norms among
CurES-RPP, GRPO, and RPP. The CurES variants consistently exhibit higher combinations, suggesting that the
gradient norms in three out of the four dggorttn-anndmmodel-ee CurES effectively selects more informative prompts, thereby accelerating the training process.
By differentiating both sides with respect to n, and setting the derivative to 0, we obtain:
0J of + 1 =
n
(76)
On,
m²n
m
That is
n m mvp
(77)
According to the constraint:
0i
N =
v/
(78)
mvp
N
1=1
i=1
We get
11:
Σ N, o, a= TT (h (y, 2z 0ata))
(79)
Thus, we derive the rollout quantity allocation strategy for different prompts.
B EXTENDED EXPERIMENTAL RESULTS
We further analyze the evolution of average gradient norms across different model scales and op- timization algorithms (Figure 7 Overall, the CurES variants consistently exhibit stronger gradient signals compared their corresponding baselines. On Qwen2.5- 1.5B. both CurES-RPP and CurES-GRPO maintain substantially higher gradient norms throughout training. On @wwn2.55Matth 7B, CurES-RPP continues to yield larger gradients, while CurES-GRPO performs comparably to GRPO. Higher gradient norms indicate that the model receives more informative learning signals, suggesting that CurES effectively prioritizes prompts that accelerate parameter updates. Notably, for the larger 7B model, the optimizer tends to dampen gradient magnitudes more significantly, which partially reduces the advantage of CurES; nevertheless, the overall trend demonstrates its robustness and consistent benefit across scales.
Figure BuaneBBilluseee Billustrates the distribution of rollout allocations across prompts with different accuracy lev- els over successive training iterations. We observe that CurES adaptively concentrates rollouts
21

---

Published as a conference paper at ICLR 2026
Iteration
Iteration
Iteration
40
40
x:0.5
35
35
Rollout Quantity
10
10
30
13
25
25
20
20
20
15
15
15
10
10
X
0.0
0.2
0.4
0.5
0.8
1.0
0.0
6.2
0.4
0.6
0.5
1.0
0.0
0.2
0.6
0.5
1.0
Accuracy
Accuracy
Accuracy
Figure Distribution of rollout quantities with respect to aceuracy in CurES base on Qwen2.5-
Math-7B at different training iterations. CurES concentrates more rollouts on moderately difficult prompts.
Owen-Math- BiteenB--CCurcrcr CurES-GRPO
Qwen-Mathh Baon--Math-7--uueee 7B-CurES-RPP
Average Accuracy
0.82
0.82
0.78
0.78
0.76
0.72
0.70
0.70
Step
Wi:4, n::
Wi:d, m:16
n:34 m:322
N m::
W- m::
Figure 9: Performance convergence of Qwen2.5-Math-CurES- on MATH500 with different sam- pling configurations.
0.850
0.825
Faster
Average Accuracy
0:825
MM NW
0.600
0.755
0.755
0.455
CurES-querO.5e
200
400
600
20C
230
Step
Figure 10: Efficiency comparison of CurES against baselines on MATH500 with Swen2.5-Math 7B. Gray dashed lines indicate the steps required for CurES and the baseline to reach the highest average accuracy within the total training duration.
moderately difficult prompts, rather than uniformly sampling across the entire spectrum. This behav- ior aligns with the intuition that prompts with intermediate difficulty provide the most informative learning signal-beit neither trivially solved nor consistently incorrect. As training progresses, the distribution becomes increasingly peaked around this region, indicating that CurES dynamically refines its sampling strategy to focus computational effort on prompts that are most beneficial for improving policy performance.
We further present the performance convergence of CurES with different sampling configurations on Qwen2.5-Math-77B as shown in Figure We observe that CurES achieves stable improvements across all settings, with only minor differences in convergence speed and final accuracy among vary- ing rollout counts and prompt subsets. Notably, configurations with a moderate number of rollouts (c.g., N' = 4, 16) strike favorable balance, reaching higher accuracy with fewer steps com- pared to more extreme settings such as very large or very small rollout numbers. This demonstrates that CurES is robust sampling configurations and can effectively leverage diverse rollout budgets without significant degradation performance.
We also compare the training efficiency of CurES against the baselines MATH500 for 7B models. As shown in Figure [0] CurES reaches the highest accuracy achieved by the baselines 4.3x faster under GRPO and 3.7x faster under RPP. Importantly, CurES not only converges more rapidly but
22

---

Published as a conference paper at ICLR 2026
4.507x
3.860x
Average Accuracy
Averagge Lecurac
0.75
Accia.
0.63
Qwen2.5-ttath- www.aaat--------PE-----
Qwen2 waa.:---tttt----------
0:5
Quean2, 2. 5- -S-EPEE
Queen2, 2. 5-Hath-78-GBB
Qwen2 w-wrnl----tttt-----1- --------
Qween2 2 Math-hB--Crr Baa-----C-SSP--PPP
Qwen2.5 (aa.i/--tttthh-ae wtathhh118--VV--000
Qwen225 S. Nath-NtthhBBehe
2-----8-------- GVN-GRPO
10 Training Time (hours)
20
10
20
70
60
76
80
90
100
Training Time (hours)
Figure 11: Validation performance with training time of CurES against baselines on MATH500. The gray dashed lines mark the training time CurES and each baseline required reach the baselines' highest average accuracy within the total training duration.
Table 2: Ablation study on isolating the contributions of prompt sampling and rollout allocation based on the Qwen2.5-Mann 1ren2.5-Mathh115B model.
@ass@!
Averago 16
Methed
Avg.
MATHS00
GSM8K
GA023
MINERVA
OLYM
AIME24
AIME25
AMC23
Swen2.5-Math-1.55
40.20
43.90
25.19
1.67
1.67
20,00
+CurES GRPO
77.20
85.97
$1.43
31.62
37.33
13.33
10.42
52.19
44.94
+CurES-GRRPO w/o Rollout Allocation
76.20
49.61
9.38
*CurES GRPO w/o Prompt Sampling
77.00
84.00
50.25
29.04
36.59
13.33
11.46
50.94
44.08
Table Additional baseline comparison with DAPO (Yu al. based on the Qwen2.5-Math-1.55 model.
2025
and MoPPS (Qu et
2025al
Pass
Average @
Method
Avg.
MATH500
GSM8K
GA023
MINERVA
OLYM
AIME24
AIME25
AMC23
Owen2.5-Muth-1.58
40.20
43.90
25.19
11.40
21.04
1.67
1.67
14.84
20.00
+DAPO
71.40
86.28
49.61
27.94
35.26
12.50
7.71
51.72
42.80
+MoPPS
72.64
84.23
50.25
31.62
37.33
12.92
10.83
52.03
43.98
+CurES-GRPO
77.20
85.97
51.43
31.62
37.33
13.33
10.42
52.19
44.94
also exceeding the baselines throughout training. These findings confirm that the adaptive prompt selection mechanism of CurES significantly improves sample efficiency, enabling faster convergence and better final model quality.
Moreover, the results in Figure demonstrate that CurES significantly accelcrates convergence without compromising model performance. As indicated by the gray dashed which mark the time required to reach the baselines' peak accuracy-CurESS CurES achieves substantial speedups. Specif- ically, for the 1.5B model, CurES attains the target accuracy approximately 4.51x faster than the GRPO baseline (reducing the required time from 20.1 hours to just 4.5 hours). Similarly, for the 7B model, CurES delivers 3.86x speedup, reaching the performance threshold in 10.8 hours compared to the baseline's 's 41.8 hours.
Table presents the ablation results on the Qwen2.5-Matth 1.5B model, where we independently evaluate the contributions of the two key components in CurES: prompt sampling strategy and roll- out allocation mechanism. The empirical results demonstrate that both components are essential to the overall performance gains. In particular, removing either the difficulty aware prompt sampling strategy or the adaptive rollout allocation mechanism leads to a notable decrease in overall accuracy, thereby validating their complementary contributions to the effectiveness of CurES.
Table demonstrates the performance comparison of our method with DAPO (Yu al. 2025) and MoPPS (Wang al., 2025) the demonstrate notable improvements over the base model, CurES-GRPO achieves superior overall performance, indicating that gradient-base based framework provides more effective mechanism for accelerating training.
Math model. While both DAPO and MoPPS
Table reports the experimental results across different model architectures, where additionally evaluate our method on 1amma3.2-3--nsttte and Phi-4-mini Instrutntt Our findings demonstrate that CurES-GRPO consistently outperforms vanilla GRPO in both model families. This consistent
23

---

Published as a conference paper at ICLR 2026
Table 4: Performance comparison different methods on Llama3.2-3B 2-3--IIrrtt and Phi-4-mini- Instruct.
Pass
Average @ 16
Method
Avg.
MATH500
GSM8K
GAO23
MINERVA
OLYM
AIME24
AIME25
AMC23
-lamaa.2-3B-Instree
29.38
78.17
38.96
18.22
14.52
5.62
0.00
20.31
25.65
+GRPO
52.52
80.44
40.26
19.49
18.81
9.19
0.42
29.22
31.29
+CurES-GRPO
60.40
83.09
42.08
21.69
21.96
12.29
0.21
32.66
34.30
Phi-4-mini-Instruc
65.39
85.06
40.52
18.75
19.70
2.08
21.09
31.60
+GRPO
67.81
88.32
41.04
36.40
30.81
7.08
2.29
38.75
39.06
+CurES-GRPO
76.80
87.87
42.86
35.74
31.11
7.71
3.12
35.78
40.12
enhancement confirms the robustness and generalizability of the CurES framework across diverse model architectures.
Case Study
Problem Janet goes to the mall and spends $3.50 on ice cream, $7.50 each for movie tickets for herself and her younger sister, and $8.50 on a bracelet. her parents gave her $40, how many dollars does she have left?
Answer:
GVM Output First, let's calculate the total amount of money Janet spent. She spent $3.50 on ice cream, $7.50 movie ticket for herself, $7.50 movie ticket for her younger sister, and $8.50 on bracelet. So the total amount spent is: [3.50 + 7.50 + 7.50 + 8.50] Next, we need find out how much money she has left after spending this amount. Her parents gave her $40, so the remaining amount is: [40 (3.50 + 7.50 + 7.50 + 8.50)] Let's calculate this step by step using Python. "'yython
Amount spent on ice cream ice-cream = 3.50
Amount spent on movie tickets for herself and younger sister
movie-tickets 7.50 7.50
Amount spent on bracelet
bracelet 8.50
# Total amount spent
total-spent ice-cream movie-tickets bracelet
Amount given by parents
amount-given 40.00 # Amount left
amount-left amount-given total-spent print(amount-left)
"'output
14.0
After calculating, we find that Janet has 14 dollars left.
CurES Output Let's break down the problem step by step and then use Python to verify calculations.
1. First, need to calculate the total amount Janet spent:
$3.50 on ice cream
$7.50 each for movie tickets for herself and her younger sister (so $7.50 S7.50
$15.00)
$8.50 on a bracelet
So, the total amount spent is $3.50 $15.00 + $8.50 $27.00.
Janet's parents gave her $40. To find how much money she has left, we subtract the total amount spent from the $40 her parents gave her: [40 27 13]
The calculations confirm that Janet has 13 dollars left.
24

---

Published as a conference paper at ICLR 2026
Case Study
Problem The Sun's effective temperature, T,, is 5800 Kelvin, and its radius is x 10 em x 10) Compute the luminosity (power output) of Sun in erg/s. Treat the Sun blackbody radiator, and give your answer to one significant figure.
Answer: 4 1033
GVM Output The luminosity L of blackbody can be calculated using the
law:
L == AmR
where:
is the radius,
Te is the effective temperature,
is the Bocfan-Boltzman constant, approximately 5.67 10 SW 2. K K First, convert the radius from to meters:
R = x 10 cm = 7 x 10
The effective temperature given as:
T. 5800K
Now plug the values into the formula:
L = 4x x 10%) x 10 (5800))
Calculate R²:
R = (7 10 =9 49 x 10 4.9 10
Calculate T.l
T (5800)* 4 = 1.1316496 x 10
Substitute the intermediate results back into the luminosity equation:
L = (4.9 x 10"6) x 10- (1.1316999 x 10
Combine the constants and the powers of 10:
= 4 (4.9 x 5.67 x 1.1316496 10 14) = 4T (4.9 5.67 1.1316496 x 1023)
Calculate the product of the constants:
4.9 x 5.67 x 1.1316496 = 31.03
So,
L 2 4m (31.03 1023)
4m (3.103 x 102t)
Now compute the numerical value:
L 4 3.14159 x 3.103 x 10 z 39.4784 x 3.103 x 1021 122.5 x 10
L 1.225 1026
Rounded to significant figure:
Lx1x 10
Since IW 1J/s erg/s:
L 1 x 2626 x 10 erg/ss 1 30
The luminosity in erg/s is:
10
CurES Output The luminosity L of a blackbody is given by the Stefan-Boltzmann law:
= 1
where:
25

---

Published as a conference paper at ICLR 2026
R is the radius of the Sun,
* Te is the effective temperature of the Sun,
a is the Stefan-Boltzmann constant, a R 5.67 x 10 m m2 K-
Convert the solar radius from centimeters to meters:
R 10 em 7 10ºm
The effective temperature is:
Te = 5800K
L = (T 10 x (5.67 x 100 $) (5800)*
Calculate R²:
R (7 10y)² 49 10t6 4.9 10
Calculate :
T (5800)* 1.1316496 10"
Substitute the intermediate results:
= 4T x (4.9 x ) x (5.67 x 10 -0_8)0=--000-0))0 x (1.1316496 x 10²4)
Combine the constants and the powers of 10:
L 4. (4.9 5.67 1.1316496) 10
4T (31.37336) 102aw
L 4 x 3.14159 x 31.37336 x 102 394.444 -0 3.94444 1025
Rounded to one significant figure:
4x 10
Since 1W = 1J/s 107 erg///
L 4 2 x erg/s = 4 x 10 33
erg/s
The luminosity of the Sun is:
x 1033 erg/s
C USE OF LLMS
We use LLMs only to refine the language and grammar in our paper. We do use them for gener- ating research ideas or for finding related work. We provide our complete original text to OpenAl's GPT-4o with instructions make more professional, coherent, and native-sounding for research paper. We then carefully review all suggestions to guarantee that no factual content is altered and that all changes remain true to our original writing.
26


Reviews:
[{"review_id": "eK6mKPqUy3", "reviewer": "Reviewer_uxBd", "summary": "The paper introduces CurES that selects prompts and determines the number of rollouts associated with each prompt during post-training based on the difficulty of the prompt in order to improve training efficiency.", "questions": "In Figure 6, does one step for CurES-GRPO/RPP have the same computational cost as vanilla GRPO/RPP? Would the results have been different if the x-axis was computational cost (for instance measured in FLOPS) rather than steps?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method is based on a theoretical analysis of the relationship between training dynamics and difficulty of prompts.\n\nThe experiments demonstrate that CurES improves average accuracy in mathematical reasoning across a range of model sizes compared with alternative curriculum approaches, namely Speed-RL and GVM (the quantitative results in Table 1).", "weaknesses": "What the experiments related to the sampling behavior (in Section 5.2) are supposed to demonstrate is not clear. In Figure 4, “the distribution of estimated accuracy … becomes more concentrated around higher values,” but wouldn’t we expect this to be the case for any learning method?\n\nAlthough Speed-RL and GVM select the number of rollouts in a more straightforward manner, reporting their rollout numbers in Figure 3 would help highlight the difference of CurES.\n\nIn general, the sampling behavior analysis and the efficiency analysis in Sections 5.2 and 5.3 are only performed for CurES. Contrasting these results against Speed-RL and GVM can make these analyses more informative.\n\nHow do we arrive at the conclusion about Figure 5? Figure 5 doesn’t report the increase in computational cost with respect to N and N. How do we conclude they do not yield a proportional increase in performance? Assuming that’s true, how does this relate to the CurES’s efficiency?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YY0FikwNr4", "reviewer": "Reviewer_8JVY", "summary": "This work proposes CurES to enhance the training efficiency of Reasoning LLMs. The authors derive a theoretical link between gradient optimization efficiency and both the prompt sampling distribution and rollout quantity allocation. Based on this, CurES adaptively adjusts sampling probabilities and rollout allocations using Bayesian posterior estimation of prompt difficulty. Experiments on mathematical reasoning benchmarks show that CurES achieves faster convergence and higher accuracy than baselinr methods.", "questions": "- Line 350: \"an assignment of rollout quantities under a total sample budget of 8×1024\". Does it mean that the batch size for training CurES is set to 1024?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The work addresses a practically relevant problem—improving the training efficiency of reasoning LLMs.\n- CurES reallocates both prompt sampling probabilities and rollout quantities to enhance the training efficiency.\n- Many of the derived equations have closed-form solutions, which makes computation and optimization more tractable.\n- The presentation of several figures is excellent. For example, the distributions of rollout quantities and estimated accuracies in Figures 3 and 4 are clearly illustrated, providing intuitive insights into the CurES’s behavior throughout the training process.", "weaknesses": "- In Eq. (12), the $\\leq$ relation does not seem to strongly support the argument. Since the method aims to maximize the right-hand side (RHS), having a higher upper bound on the RHS does not necessarily imply a larger loss change in LHS. A $\\geq$ lower bound would better motivate and validate the theory and the method.\n- The training paradigm is restricted to iterative training across pre-divided dataset subsets, which limits generality. The way the dataset is partitioned may influence the algorithm’s performance. Moreover, each inner iteration requires rolling out every prompt to estimate the initial difficulty distribution, which introduces additional computational overhead.\n- The methodology presentation lacks clarity. While the paper provides detailed explanations of the gradient theory and the estimation of prompt difficulty, it does not clearly describe how sampling and budget allocation are actually conducted. Before seeing Algorithm 1 in the Appendix, it remains unclear how the algorithm operates in practice.\n- The paper could include more ablation studies to isolate the individual contributions of prompt sampling versus rollout allocation.\n- The notation introduced in Section 5.3 (Line 460) needs revision for clarity and consistency. \"pre-sampling scale (N)\": This term is ambiguous. Moreover, it conflicts with the earlier definition of N as the \"total rollout budget\" on Line 224. \"training-phase sample budgets coefficient (n)\": This term is imprecise, and the symbol n is not defined previously in the text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8FliZMsxg6", "reviewer": "Reviewer_Lzg2", "summary": "This paper discusses curriculum learning for training Large Language Models with Reinforcement Learning (RL) for reasoning tasks. From gradient analysis, the authors demonstrate the intrinsic relationship between optimization efficiency and prompt sampling distribution, as well as rollout quantities. Therefore, an curriculum learning algorithm called CurES is proposed to adjust the sampling probability and rollout quantities based on the prompt difficulty. Experiments show that CurES improves the efficiency and stability of GRPO across several math reasoning tasks.", "questions": "Please refer to the \"Weakness\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and well-motivated.\n2. The theoretical analysis is interesting and aligns with people's understanding on RLVR, Eq (11) shows that training with a prompt with higher variance is more helpful; while Equation (20) indicate that more rollout should be allocated to the prompts with higher variance of gradients. \n3. The assumption of Beta-distribution in Section 4.3 address a good trade-off between theory and practice, leading to a simple yet practical algorithm.\n4. From empirical results, CurES outperforms other baselines.\n5. Comprehensive experiments demonstrate the effectiveness of proposed algorithm compared to baselines", "weaknesses": "## Section 4.1: Mismatched derivation from single prompt to dataset-level behavior.\n\nFrom Eq (3) to Eq (11), the author analysis the impact of $|\\mathcal{L}(x; \\theta_{old}+d) - \\mathcal{L}(x;\\theta_{old})|$ to the variance of $r(x, y)$ based on Cramér–Rao inequality, which indicates that training on prompts with higher variance is more helpful. This is an interesting and meaningful observation. However, from Eq (11) to Eq (12), the derivation jumps from $|\\mathcal{L}(x; \\theta_{old}+d) - \\mathcal{L}(x;\\theta_{old})|$ to $|\\mathcal{L}(\\theta_{old}+d) - \\mathcal{L}(\\theta_{old})|$, which is actually **mismatched in the parameter update $d$**:\nin the first expression $d$ denotes the parameter update w.r.t. a **single prompt** $x$, while in the second it implicitly refers to the **dataset-level** update. Ideally, the second one should be replaced by\n$\\bar{d} = \\mathbb{E}_{x\\sim\\rho}[d]$. This implies that:\n\n1) $d \\neq \\bar{d}$, ; therefore, it is not mathematically correct to generalize directly from a single prompt to the dataset average.\n\n2) $\\bar{d}$ also depends on your sampling distribution $\\rho$, which is ignored. \n\nUnfortunately, all the subsequent derivations are built upon this mismatch.\n\n**Q1**: How can the authors justify this mismatch between $d$ and $\\bar{d}$? What's its influence to the analysis afterwards?\n\n**Q2**: Is it accurate to describe the variance term $\\sqrt{2δp_{θ_{old}} (x) (1 − p_{θ_{old}} (x))}$ by prompt difficulty? It's more related to the variance. For example, both too difficult and too simple prompt could have zero variance. \n\n## Section 4.2: Unexplained objective and beyond\n\nIn Section 4.2, the authors propose to optimize the objective Eq (15), which is $\\mathbb{E}[(\\mathcal{L}(\\hat{\\theta}) - \\mathcal{L}(\\theta_{old})  - \\Delta_{theo})^2 ]$ (according to Eq (58)). This objective is the **Mean-Square Error (MSE)** between *actual improvement* $\\mathcal{L}(\\hat{\\theta}) - \\mathcal{L}(\\theta_{old})$ and *theoretical improvement* $\\Delta_{theo}$. However, the paper does not provide sufficient intuition for why minimizing this MSE corresponds to a principled curriculum-learning objective under a rollout budget constraint.\n\n**Q3**: Why should Eq. (15) be viewed as a valid curriculum-learning objective? What is the intuition?\n\n**Q4**: Since Eq. (15) builds on Section 4.1, it inherits the mismatched $d–\\bar{d}$ issue. How does that affect the soundness of the subsequent results?\n\n**Q5**: Moreover, the advantage estimation $A_{\\theta_{old}}(x_i, y_i)$ also depends on the rollout number $n_i$ (and also the underlying policy gradient algorithm). This is ignored during the analysis.\n\nMoving forward, the authors derive an optimal allocation of rollouts in Eq (20), i.e., $n_i = \\frac{\\sigma_i}{\\sum_j \\sigma_j} N$, where $\\sigma_i$ is the variance of gradient of $x_i$ w.r.t $y$. Although the authors derive a symmetric computational form in Eq (21), it still seems computational intractable, especially for large langauge models. Concretely, the term $\\mathbb{E}{y\\sim\\pi_{\\theta_{\\text{old}}}}\\big[||\\nabla_\\theta \\log\\pi_\\theta(y|x_i)||^2\\big]$ requires computing the gradient norm w.r.t **each response $y_i$**, taking $N$ backward in total, while normal RL algorithm only needs to compute the average gradient once per batch. \n\n**Q6**: What's the **time** and **memory** overhead of CurES, especially on estimating $\\sigma_i$? Currently this part is missing in the Experiment Section.  \n\n## Other question\n\n**Q7**: What’s the performance of baseline algorithms with different sample budget? Figure 5 only shows the performance of CurES.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GWSOwRVusZ", "reviewer": "Reviewer_9BeW", "summary": "This paper studies curriculum learning for RL finetuning of reasoning LLMs under a verifiable-reward setting. The authors analyze how different prompts contribute to learning progress by examining the relationship between success probability and optimization potential. They find that prompts with intermediate accuracy are the most informative and propose CurES, a principled task sampling and rollout allocation framework. CurES estimates each prompt’s accuracy using a Beta posterior, samples prompts according to a concave transformation of their estimated uncertainty, and allocates rollout counts to minimize gradient variance. The method is implemented as a multi-iteration procedure that periodically refreshes estimates to adapt to distributional shifts. Experiments using Qwen2.5-Math models show that CurES achieves faster convergence and higher final accuracy than GRPO, GVM, Speed-RL, and RPP.", "questions": "see weaknesses", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper provides a principled derivation linking sampling and rollout allocation to expected loss change and gradient variance, and successfully turns the analysis into a practical algorithm.\n- The paper is well-structured.", "weaknesses": "- Narrow experimental scope. The training dataset is limited to Numina-Math, which limits evidence of general applicability to other task types or reward settings.\n- Limited ablation studies. The paper does not include results for partial variants that use only task sampling or only rollout allocation, making it unclear how much each component contributes individually.\n- Incomplete baseline comparison. Some relevant online data filtering or adaptive sampling methods, such as DAPO[1] and MoPPS[2], are mentioned but not compared experimentally. Moreover, the proposed Prompt Difficulty Assessment appears conceptually similar to MoPPS, and the paper should clearly articulate the differences. Additionally, utilizing optimization histories to guide efficient sampling shares similarities with Bayesian-based active sampling methods[3,4], which warrants further discussion in related work.\n- Lack of discussion on prior theoretical overlap. The theoretical finding that tasks with intermediate success probability are the most informative has been observed or hypothesized in several prior works[5,6,7,8]. The paper should acknowledge these connections and clarify how its derivation provides new insight beyond existing results.\n- Algorithmic clarity. Algorithm 1 is somewhat confusing, especially regarding the role and purpose of the pre-rollout stage, which requires additional explanation.\n- Limited model diversity. All experiments are conducted on Qwen2.5-Math models, leaving uncertainty about whether the method generalizes to other model families such as LLaMA or Mistral.\n\n[1] Dapo: An open-source llm reinforcement learning system at scale.\\\n[2] Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?\\\n[3] Model Predictive Task Sampling for Efficient and Robust Adaptation\\\n[4] Fast and Robust: Task Sampling with Posterior and Diversity Synergies for Adaptive Decision-Makers in Randomized Environments\\\n[5] Self-Evolving Curriculum for LLM Reasoning\\\n[6] Online Difficulty Filtering for Reasoning Oriented Reinforcement Learning\\\n[7] Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning\\\n[8] SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learning", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)