Prompt Curriculum Learning for Efficient LLM Post-Training
Authors: Zhaolin Gao
Joongwon Kim
Wen Sun
Thorsten Joachims
Sid Wang
Richard Yuanzhe Pang
Liang Tan
Venue: ICLR 2026 Poster (2026)
Keywords: reinforcement learning
large language models
post-training
curriculum learning
Primary area: reinforcement learning

Abstract: Reinforcement learning (RL) is widely used to post-train large language models for tasks such as mathematical reasoning and coding. However, the convergence of RL training remains sensitive to batching and prompt selection strategies. We investigate the factors that affect convergence, including batch size and prompt difficulty. Through large-scale experiments across multiple models and datasets, we show that there exists an optimal batch size that balances generation time and gradient quality, and that prompts of intermediate difficulty (where the model has roughly a 50\% chance of success) are the most sample-efficient for model convergence. Motivated by these findings, we propose Prompt Curriculum Learning (PCL), a lightweight algorithm that selects intermediate-difficulty prompts using a learned value model. PCL avoids costly rollouts and efficiently guides training by focusing on the most informative samples. Empirically, PCL either achieves the highest performance or requires significantly less training time to reach comparable performance across a suite of benchmarks. Compared to using rollouts to filter, PCL is $12.1\times$ and $16.9\times$ faster on identifying intermediate-difficulty prompts when training on MATH and DeepScaleR respectively.

Published as a conference paper at ICLR 2026
PROMPT CURRICULUM LEARNING FOR EFFICIENT LLM POST-TRAINING
Zhaolin ? Joongwon Kim 13 Wen Sun?, Thorsten Joachims?, Sid Wang', Richard Yuanzhe Pang', Liang Tan'
Meta Superintelligence Labs, 2 Cornell University, University of Washington
ABSTRACT
Reinforcement learning (RL) is widely used to post-train large language models for tasks such as mathematical reasoning and coding, However, the convergence of RL training remains sensitive to batching and prompt selection strategies. We investigate the factors that affect convergence, including batch size and prompt difficulty. Through large-scale experiments across multiple models and datasets, we show that there exists optimal batch size that balances generation time and gradient quality, and that prompts of intermediate difficulty (where the model has roughly 50% chance of success) are the most sample-efficia for model conver- gence. Motivated by these findings, we propose Prompt Curriculum Learning (PCL), lightweight algorithm that selects learned value model. PCL. avoids costly rollouts and efficiently guides training by focusing the most informative samples. Empirically, PCL either achieves the highest performance or requires significantly less training time to reach comparable performance across suite of benchmarks. Compared using rollouts to filter, PCL is 12. 1x and 16.9 faster on identifying cetemeddateddifffccul prompts when training on MATH and DeepScaleR respectively.
cerrmeddate-difficuul prompts using
INTRODUCTION
Recent large language models (LLMs), such OpenAI-ol (OpenAI, 2024b) and DeepSeek- R1 (DeepSeek-AA, 2025), have demonstrated strong performance by producing long chain-of-thought (CoT) solutions (Wei al., 2023; DeepSeek-AI, 2025; Zeng al., 2025). A key driver of these improvements is reinforcement learning (RL) with rule-hased rewards, using algorithms such as PPO (Schulman al., 2017) and GRPO (Shao al., 2024). By generating responses online from the current model, RL enables LLMs to self-explore and iteratively improve based on their own outputs.
Substantial effort has been devoted to improving both the performance and efficiency of RL training for LLMs (Brantley et al., 2025; Xu et al., 2025; An et al., 2025; Sun et al., 2025). A recurring insight across recent works (Yu et al., 2025; Zhang et al., 2025; Zheng al., 2025) is that training on prompts of intermediate difficulty (i.e., neither too easy nor too hard for the current policy) yields significantly better data efficiency. However, existing approaches on identifying intermediate prompts typically rely either rollouts from the current model dictionary that tracks average rewards from previous epochs. The former introduces substantial training overhead due to the high cost of online generation, while the latter suffers from off- off-policyness especially when the dataset is large. In addition, while these works primarily focus on prompt difficulty, many hyperparameters (e.g., batch size) can significantly affect convergence but remain underexplored in prior work.
In this paper, we systematically study how batch configuration and prompt selection jointly affect the convergence of RL training, and we these insights to design new, compute-efficient curriculum algorithm. We uncover two key findings. First, there exists an optimal batch size that achieves the best trade-off between faster generation time and smaller gradient noise. While larger batches reduce gradient noise and allow for higher learning rates, they also increase generation time, limiting update frequency. We identify a sweet spot at the transition point between sublinear
>g2920corne11 ededu

---

Published as a conference paper at ICLR 2026
Less
Higher Ranio
Rollouts
Frequent Updates
Gradiant Noise
Promot Diversity
600
TTC (x)
Time (saconds)
Faster
Largeri Batch Size
Number Of
Generations
Generation
Par
200
Time
Prompts
Prompt
PCL
12.38
26.9x
((x)
DeepScalelt
Figure 1:
We conduct systematic investigation of the trade-off's on generation time vs, batch size and number of prompts vs. generations per prompt. We identify an optimal batch size that achieves the best trade-off and discover that the prompts of intermediate difficulty are the most effective for learning. Building on these insights, we introduce Prompt Curriculum Learning (PCL), which trains a value model online for prompt filtering. Compared to the rollout-based filter method, PCL 12.1x and 16.9x faster during prompt filtering when training MATH and DeepScaleR respectively.
and linear generation time growth, where convergence speed is maximized. Second, prompts of intermediate difficulty are the most effective for learning. When prompt is too easy or hard, gradient signals tend vanish, leading to wasted compute. In contrast, prompts for which the model has a ~50% success rate require fewer samples to obtain informative updates. We validate this finding empirically across models, datasets, and batch configurations.
Building on these insights, we introduce Prompt Curriculum Learning (PCL), an efficient algorithm that dynamically selects prompts of intermediate difficulty using value model. At each step, PCL samples a large pool of candidate prompts, predicts their expected reward with single forward pass, and greedily selects those closest target threshold (e.g., 0.5). This approach avoids the overhead of rollout- baseet prompt filtering while also being much more on-policy than dictionary-bacn -ased methods. We benchmark PCL across a wide range of models and datasets, including Qwen3-Ban 4B, 8B) and Llama3.2 (3B) on MATH, Olympiad-Bene Bench, Minerva MATH, AMC, and AIME. Empirically, PCL either achieves the highest performance or requires substantially less training time to reach comparable performance.
(1.7B,
PROBLEM SETUP
Let æ denote a prompt (e.g., a math question), and let denote a sampled solution of length [y/ generated autoregressively from a policy æ, i.e., y ~ .-- æ:). We assume a binary reward function r(x,v) v) {0, 1}, where r(x,)) v)
if the final answer in is correct and O otherwise. Since the reward binary, we denote pr (x) = E [t=)(()) v)] as the probability of generating a correct answer from policy æ on prompt æ, and A(x,y) ((z,)( y) Pr (x) as the advantage. To optimize æ, we adopt the purely on-policy variant of GRPO (Shao et al., 2024; DeepSeek-AI, 2025), without KI. regularization to a fixed reference policy Tref (Yu al., 2025) and without standard deviation-base advantage regularization (Liu et al., 2025), by maximizing:
क. (
x. Met) A(x, v) xr Vet)
(1)
1=1 T.
(vn
where yr denotes the I-th token in the generated sequence y. We adopt this formulation to eliminate the off-policyness during updates, clipping heuristics, and additional hyperparameters, which would complicate our analysis in the following section. We note that this is clean RL objective that has the same gradient as policy gradient and can be directly derived from the original RI objective of maximizing expected reward: EE,D, (==( y. The derivation is provided Appendix A.
3 PRELIMINARY INVESTIGATIONS
In this section, present set of preliminary experiments that investigate the interplay between convergence, batch size, the number prompts per batch, and the number of generations per prompt. We first define them in detail.
2

---

Published as a conference paper at ICLR 2026
Batch size, denoted by b, refers to the total number of prompt-respoon pairs in a batch. In our purely on-policy setting, this number also corresponds to the number of pairs used in a single update. The batch size is given by the product of the number of prompts and generations per prompt. Batch size directly affects the generation time, as larger batches require longer to generate.
Number of prompts, denoted by m, refers to the number of unique prompts in batch. This quantity is closely related to the prompt diversity. Increasing the number of prompts improves the diversity of the batch, which in turn reduces gradient noise and stabilizes learning.
Generations per prompt, denoted by n, refers to the number of responses generated for each prompt. These responses used to estimate the expected reward, which is used to compute the advantage. The number of generations per prompt is related to the effective ratio, defined as the proportion of samples in the batch with non-zero advantages, i.c., the proportion of samples that contribute meaningful gradient signals. Increasing n improves the effective ratio. For example, for particularly challenging prompt, if 2. both responses may be incorrect, leading to zero advantage and zero gradient under the objective in Eq. I. In contrast, for n 16 or 32, it is much more likely that at least one response is correct, resulting in non-zero advantage and thus useful gradient updates. Therefore, increasing would result in a more accurate advantage estimation and higher effective ratio.
Convergence defined as the final training or validation reward achieved under a fixed compute and time budget (c.g., number of GPUs and wall-clock time). A method exhibits faster convergenee if, under the same computational resources, it reaches a higher reward. Convergence is influenced by generation time, prompt diversity, and effective ratio. Reducing generation time enables more frequent updates, while increasing prompt diversity and effective ratio reduces noise in the gradient and leads more stable and efficient training.
Overall, these quantities exhibit a natural trade-off. On the one hand, reducing generation time enables more frequent updates within fixed time budget, allowing the model train new rollouts from improved policies. On the other hand, increasing the number of prompts and generations per prompt reduces gradient noise with higher signal-to-noise ratio. In the following experiments, we perform comprehensive ablations with around 100K A100 GPU hours to identify the optimal balance between these competing factors.
3.1 OPTIMAL BATCH SIZE
Experiment Setup. We conduct experiments on both MATH (Hendrycks et al., 2021) and Deep- ScaleR (Luo et al., 2025) datasets. For MATH, we evaluate on the standard MATH500 split. For Deep- ScaleR, we include evaluations on MATH500, Minerva Math (Lewkowycz et al., 2022), Olympiad- Bench (He et al., 2024), as well as competition-level benchmarks including AMC 23, AIME 24, and Qwen3-8B-base
AIME 25. We report results across four models, Qwen3-1.7BB 1.7e-b Qwen3-4B-bass and Llama3.2-33 -iiii,, covering two model families and a range of sizes. All models are trained with a context length of 4,096 tokens. We use rule-based reward function based on math- wehheirrriee fy (Hug- ging Face, 2024), which assigns a reward of +1 for correct ones and 0 for incorrect ones or generations that exceed the context limit. All experiments synchronous training setup that alternates between generation and optimization phases. For each batch size, we ablate find the optimal learning rate with a total of 23 runs. Additional implementation and training details, including learning rate ablations, are provided in Appendix B.
implemented using the VERL (Sheng et al., 2025),
The results for Qwen3-48 1-------BBsss Base are presented in Fig. 2 and 3, including training reward a function of both training steps and wall-clock time (in hours), generation time per step using vLLM (K (Kwon et al., 2023), and test accuracy. For DeepScaleR runs, test accuracy is reported as the average across all six benchmarks. Full results are provided in Appendix C.
Larger batch sizes converge faster in terms of steps. As shown in Fig. 2 (Left), increasing the batch size consistently leads to faster convergence when measured in training steps. This is primarily because larger batches reduce gradient noise, allowing the use of higher learning rates without destabilizing training. The learning rates used in each configuration are listed in Tables 4 and 5.
Generation time grows sublinearly at first, then linearly. In Fig. 2 and 3, we plot generation time per step against batch size, alongside dashed reference line representing linear growth (intersecting the origin). We observe that generation time initially increases sublinearly with batch size, and transitions to linear growth as batch size continues to increase. This behavior is expected: When the
3

---

Published as a conference paper at ICLR 2026
0.9
0.9
(4096, 16)
Gon. Time: Step (bour)))
10
0.35
Praining Reward
Training Reward
(2065, 15)
0.830
MATH
0.8
0.8
0.825
Accuraccy
16)
0.820
0.7-
0.7-
0.815
0.6
0.6
(128, 16)
0.810
0.5
0.5
(64, (6)
0.805
15 Total Time (hours)
16)
(512, 16)
(4096,
Step
Batch Sizee (m,
0.6
0.6
(4096, 16)
Gea. Time Step (hwara))
10"
DeepScaleR
Training Reward
Truining Reward
(2048, 16)
Arg. Teat Accuracy
0.5
0.5
(1024, 16)
0.45
(512, 16)
0.44
(256, 16)
10"
(128, 16)
0.43
(64, 16)
0:42
100 200
10
15
(64. 16)
(512, 16)
(4096, 16)
Step
Total Time (hours)
Batch Sizee (mn, I)
Figure 2: (Left / Middle) Training reward as function of training steps and wall-clock time for Owen3-4B-Basd on MATH and DeepScaleR. The legend indicates the batch configuration in terms of (number of prompts m, generations per prompt n.). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale. For key takeaways, refer to the paragraph headers in Section 3.1.
MATH
DeepScaleR
10
0.550
6/434
-0.45
A0g Test Accracy
6A12
10.02
10"1
-081
10"
A.15
0.43
0.90
0.100
-0.4
5.031
(52.33)
(256, 32)
(2048.
(36, 64
(128. 44)
(1924, o0)
(32. 323
(356, 32)
(2014, 32)
(16.66) 64)
(128)
(1024, (4)
Ratch Size (m..) n)
Batch Sizee (m.
Batch Size (m.
Ratch Size (m,
Figure 3: Generation time per step and test accuracy across different batch size combinations (number of prompts m, generations per prompt n) for Qwen3-4B- on MATH and DeepScaleR.
batch size is small, the generation time is dominated by the longest response in the batch. As batch size increases, compute utilization becomes the bottleneck, and generation time scales more linearly.
Optimal batch size occurs at the transition point from sublinear to linear scaling, From Fig. (Middle / Right) and Fig. 3, there exists sweet spot in batch size that yields the best convergence speed. Extremely small large batch sizes lead to suboptimal performance. The optimal point for the fastest convergence tends to lie at the end of the sublinear regime and the beginning of the linear regime in generation time. Specifically, the optimal batch size our setting is around 8K. achieved with combinations (m, n) (512, 16), (256, 32), or (128, 64). In other words, the optimal batch size remains fixed, regardless of how is factorized into m and n. We hypothesize that this sweet spot achieves favorable balance: compared to smaller batch sizes, it can have linearly more generations with sublinear time growth; compared larger batch sizes, it allows more frequent updates in the same amount of time. To ensure robustness, validate this phenomenon across different model architectures and sizes, datasets, context lengths, hardware configurations, rollout engines (vLLM SGLang), and batch configurations. Full results are provided in Appendix C. Having established an optimal batch size, the natural question is: How should we determine the optimal decomposition into the number of prompts and generations per prompt?
4

---

Published as a conference paper at ICLR 2026
plx) - 0.00
000)=0255
pix) -.500
p(x)
p(x) 1.00
512,
1,0
0.47
Training Reward (hefore)
0.60
Training Reward (after)
0.55-
0..8-
Arg. Effective Ratio
0.8
0.05
Avg. Grad Norm
Avg. Tost Acerracy
0.50
01.6-
0.04
0.45
0.03
0.4
2.4
00.0
0.4
00.1
0.0
0.0
0.00
0.40
Steps
Steps
Thresheld (p(x)
Figure 4: (Left) Training reward before downsampling in terms of step with number of prompts m = 512 and generations per prompt n = 8. (Middle) Training reward after downsampling, (Right) Average effective ratio and gradient norm over training steps, and average test accuracy of six benchmarks across different thresholds. For key takeaways, refer to Section 3.2.
3.2 OPTIMAL NUMBER OF PROMPTS AND GENERATIONS PER PROMPT
We hypothesize that the optimal decomposition of the batch size is closely tied to the difficulty of the prompts. Specifically, for extremely easy difficult prompts, larger number of generations (n) may be necessary to achieve a high effective ratio. In contrast, for prompts of intermediate difficulty (p(x) 0.5), fewer generations may be sufficient.
Experiment Setup. We use DeepScaleR dataset and Qwen3-4B-Ban Base, and train under different decompositions. To control prompt difficulty, for each batch we first sample 4m prompts and generate responses for each prompt to estimate p(z), similar to Zhang et al. (2025). We then perform greedy downsampling select m prompts that are closest to specific difficulty threshold ((x) (0, 0.25, 0.5, 0.75, 1}, and sample n generations per selected prompt for training. We are not reusing the responses to train to avoid selection-induced bias, which keeps the ablation on comparable. We keep the total batch size fixed at m n 4096 and ablate n from 2 to 128. All other experimental configurations remain the same. Full results are shown Appendix
Downsampling successfully retains target-difficuly prompts. As shown in Fig. 4 (Left Middle), our downsampling procedure effectively retains prompts around the specified threshold. This validates the experimental design and ensures that training focuses prompts of controlled difficulty.
Higher improves effective ratio and p(x)= 0.5 has the highest effec- tive ratio. As shown in Fig. 4 (Right) and 5, increasing consistently im- proves the effective ratio, and prompts with p(z)= 0.5 achieve high effec- tive ratios even with relatively small
p00 1.00
ptt 00.5
pixl
p000 6.05
- LLG
Avg. Effective Ratio
0.8
Avg. Accurayy
0.46
0.6
0.64
0.44
n. For example, the effective ratio for n = 16 at p(x) = 0.5 already higher than any other thresholds even with = 128.
0.0
0.40
32
128
32
Number of Generations (n)
Figure 5: Average effective ratio over training steps and av- erage test accuracy of six benchmarks under different thresh- olds p(z) and generations per prompt n.
p(x) 0.5 has the highest gradient morm and test accuracy. As shown in Fig. (Right) and 5, training on prompts with p(z) = 0.5 yields the highest gradient norms and test accuracy. Interestingly, while increasing n benefits test accuracy for other difficulty levels, we find that for p(x:) 0.5, accuracy actually degrades beyond m), which increases gradient noise despite higher per-prompt sampling. Conversely, based on the previous section, since there exists optimal batch size, focusing p(z) = 0.5 allows us to use smaller n and a higher m which improves prompt diversity and also maintains a high effective ratio. could have the best of both worlds (effective ratio and prompt diversity) with
32. We suspect this is due to reduced prompt diversity (i.e., smaller
In other words, p(x) = 0.5. Full results, including ablations across all configurations, are provided in Appendix C, and theoretical connection of the gradient norm and p(x:) is provided in Appendix D.
5

---

Published as a conference paper at ICLR 2026
Algorithm 1 PCL
Require: Number of prompts m, generations per prompt n, threshold 7, sampling parameter
I: Initialize policy To. value network V
2: for prrt- 0 to T do
3: Sample a batch with km prompts: Dkm {x²) J=-1 c D.
4: Select a batch of m prompts using value model: Dm
arg min Exes VRL (x)
SED 18|-m
5: Generate for the batch: Dm = {(a', (1rd) (;=)) where yyd iid
TT(: x+))
6: Update Ne to T++1 using Dm.-
7: Update V to V with loss in Eq. 2.
8:
end for
PCL: PROMPT CURRICULUM LEARNING
The previous section demonstrates that prompts of intermediate difficulty (p(x:) 0.5) the most sample-efficient for RL training. However, estimating the difficulty of each prompt using actual generations from the policy can be computationally expensive, the generations for the filtered-out prompts are wasted. To address this issue, we propose a lightweight and efficient alternative: Prompt Curriculum Learning (PCL), which leverages a learned value model during online RL estimate prompt difficulty using single forward pass, significantly reducing computational overhead.
At training iteration t. we begin by sampling pool of km candidate prompts from the dataset where l is hyperparameter. For each prompt we use value model to predict its expected reward V Vx)), which approximates Pr (a:) (+=)) [r(,r, y)]. We then greedily select a subset m prompts target difficulty threshold T (defaulting to 0.5), ensuring that
whose predicted values are closest the batch is focused on prompts of intermediate difficulty. For each selected prompt, we generate n responses using the current policy and perform standard policy gradient updates. To update the value model, we only use the generated responses and minimize the prediction error between the estimated value Vz) and the empirical average reward across the n generations:
v v(x')
- 1 -
r( ((x),n"))
(2)
This allows us to improve the value model online, without requiring any additional rollouts. Since the value model only takes in the prompt input which is typically less than 1K tokens in length for math, we find that both training and inference of the value model incur negligible cost (see Appendix I for detailed breakdown). The full algorithm is summarized in Algorithm 1. Note that the value model V in our algorithm is one step behind the policy æ, which is acceptable since each update is small with so+1 Te. We further discuss the alternatives in Section 7.
EXPERIMENTS
Models & Datasets. We use the same sets of models and datasets for experiments as Section 3. We use the same-sized model as the policy for the value model when running PCL. All runs use a 2-day time budget, except for Qwen3-8B-Bas m = 512 and 16 as it is one of the best combinations we found in terms of convergence. Unless otherwise noted, PCL uses we evaluate the model after training on every 4K prompts (8 steps), and report the performance of the checkpoint that obtains the best average performance.
DeepScaleR, which trained for 3 days. We focus on
0.5 and k 4. Similar to Wang et al. (2025c) and Zheng et al. (2025),
We compare PCL against five baselines. We include original GRPO, which performs no prompt filtering and uniformly samples prompts from the dataset. This serves as a standard baseline to assess the impact of filtering strategies. Pre-filter is a heuristic approach that leverages fixed reference policy Reef to estimate prompt difficulty and filters out easy or hard prompts. Dynamic- sampling (DS) (Yu et al., 2025) uses n rollouts per prompt to estimate Pxx for km prompts and filters out prompts with P. = or 1. SPEED (Zhang et al., 2025) improves upon DS by first using Aiit rollouts to estimate where 2 Dinit- It then performs filtering and generates the remaining Minit.
Baselines.

---

Published as a conference paper at ICLR 2026
Table 1: Results on MATH and DeepScaleR. For each metric, the best- rest-performing method is
highlighted in bold, and the second-best is underlined. Time is the sum of training and generation time of the checkpoint that achieves the best average performance (excluding validation/checkponttin hours. For full DeepScaleR results across model sizes and families, refer to Appendix F.
in
Qwen3-8B- Bren3-8B-Basee
Qwen3-4B-Ban
Qwen3-1. 1ir---11B-Bsss MATH500
1--ma3.2-3B-it
MATH
MATH500
Time
MATH500
Time
Time
MATH500
Time
ieee
72.4
65.6
55.4
42.6
GRPO
86.4
28.3
83.0
29.2
73.6
22.0
56.2
5.80
Pre-filter
84,8
17.1
81.6
27.1
73.4
13.5
55.4
7.47
DS
87.8
37.8
82.6
37.1
73.8
27.6
56,8
19.3
SPEED
81.2
4.25
78.8
6.75
70.2
1.93
42.6
/
GRESO
87.2
29.1
83.0
33.1
73.4
17.6
56,6
7.37
PCL
88.2
37.2
83.4
14.0
73.8
24.8
57.8
14.3
Minerva Avg@4
AMC23
ATMF24
AIME25
DeepScaleR
MATH500
Olymp.
Avg@32
Avg@32
Avg@ 32
Avg.
Time
70.2
34.3
29.8
49.1
15.8
8.8
34.7
GRPO
87.2
57.9
45.3
70.1
25.3
22.7
51.4
43.0
Qwen3-Be 18--------
Pre-filter
86.4
54.6
44.2
69.8
26.9
22.6
50.7
67.4
DS
87.2
55.3
45.7
71.5
24.9
24.2
51.5
69.5
SPEED
82.4
46.4
40.3
66.6
21.1
15.7
45.5
19.3
PCL
56.2
46.8
71.2
25.2
23.9
52.0
rollouts. GRESO (Zheng al., 2025) keeps a dictionary of historical rewards based on generations from previous epochs and skips uninformative prompts using the dictionary. We tested GRESO on MATH but not on DeepScaleR, as DeepScaleR is large and limits the training to around epoch under the compute budget which prevents the use of dictionary-basee GRESO keep sampling and generating until there is full batch. Additional experiment details, including pseudo-codes and hyperparameters, are in Appendix E.
methods. DS, SPEED, and
5.1 CONVERGENCE COMPARISON
PCL either achieves the highest performance or requires significantly less training time to reach comparable performance. The main results summarized in Tables (for full DeepScaleR results, refer to Appendix F). Compared to prior baselines, PCL consistently achieves the highest performance across all four models the MATH dataset, and faster convergence at similar or better accuracy on DeepScaleR, When training Qwen3-8B -BBe on DeepScaleR, PCL converges 39.8% faster than DS, the second-best method in terms of average accuracy. DS requires significantly more time to converge, as it performs generation for all km prompts at each step with n generations per prompt. SPEED's efficient implementation pre-generates Minit rollouts at an earlier step with an old policy and uses them at the current step, treating them as if sampled from the current policy, While this approach reduces generation cost for estimating Pa. it introduces severe off-policyness. We observe that most of the SPEED runs crashed within few hours, leading to lower convergence time as it would crash afterward. On the other hand, GRESO also suffers from high degree of off policyness where the historical estimates based on outdated policies from the last epoch and may not reflect the current model's performance, especially when the dataset is large.
5.2 ANALYSIS & ABLATION
PCL consistently achieves either higher effective ratio or a lower generation time, while maintaining a focus on p(x) = 0.5 prompts. To better understand the training dynamics of each method, we visualize the effective ratio, generation time per step, and training reward after filtering in Fig. 6 when training Owen3-8B-Bard on DeepScaleR. PCL. consistently maintains a higher effective ratio compared to GRPO and Pre-filter. While DS and SPEED achieve an effective ratio of I due to resampling, they require significantly higher generation time, with relative increase of 105% and 81.8% for DS and SPEED respectively. The slightly higher generation time PCL compared to GRPO and Pre-filted is that harder prompts require longer generations, and, when the average accuracy of the model on the training set is higher than 0.5, PCL focuses on harder prompts than those methods. Interestingly, the effective ratio for Pre-filter starts higher than GRPO but quickly

---

Published as a conference paper at ICLR 2026
0.5
1.0
Gen. Time Stte (hours)
Training Reward (after)
0.4
0.40
Efflective Ratio
0..8
0.3
0.6
0.6
0.34
0.4
0.0
Pre-fttr
SPEED
PCL
Total Time (hours)
Methods
Total Time (hours)
Figure 6: Experiment on DeepScaleR with Qwen3-8B-Bass (Left) Effective ratio w.r.t. training time across five methods. Refer to the middle plot for legend. (Middle) Average generation time per step throughout the training. (Right) Training reward after downsampling. PCL either has a higher effective ratio lower generation time, and is consistently training p(x) 0.5 prompts.
1.0
MATH
1.0
DeepScaleR
MATH
Explained Variance
0.8
0.8
Explained Variance
0.6
0.6
10
0.4
0.4
0.4
0.3.3
0.2
0.2
0.2
T= 0.5
0.7
0.0
0.0
0.0
T- 0.9
Value
Filiz
0.2
-0.2
-0.2
25
50
75
100
25
50
75
100
25
50
75
100
Step
Step
Step
Figure Explained variance of PCL's value model using 16 gen- erations as the ground-truth difficulty (p(z)), and the explained variance using to 16 generations predict the difficulty (f(x)) on MATH and DeepScaleR with two Qwen3-1.7PP 1.n--- models as policy and value model. The accuracy of the value model similar to using around 3 generations estimate.
Figure 8: Explained variance on
MATH with Swen3-1.7BB Base for PCL's 's value model with fil- tering on different thresholds (7) and without filtering.
drops below. This behavior comes from how Pre-filter selects prompts: it excludes very difficult ones based on Tref- the policy improves during training, many previously difficult prompts transition into the intermediate-diffiee prompts were previously filtered out, they are never revisited, causing Pre-filter to keep training on easy prompts from the perspective of the current policy. In addition, shown in Fig. 6 (Right), PCL consistently focuses on eicemmediatedifffiuuee PCL after filtering stays closely to 0.5), whereas other methods gradually shift toward easier prompts as the policy improves which is suboptimal based on the findings in Section 3.
range (e.g.. p(x) 0.5) for the current model. However, because these
prompts throughout training (the training reward of
The accuracy of the value model is similar to using 3 generations to estimate. To investigate the prediction accuracy the value model, we compute the explained variance using the average reward of 16 generations as the ground-truth difficulty p(x). The explained variance is calculated as:
Var (fp(x*) ((p(xt)-Ve - (m
Var ([p()*)}
where Var denotes the variance. In addition, we also use the average reward of to 16 generations as the predicted difficulty f(x) and compute their explained variance. The explained variances are computed on prompts randomly sampled from the dataset before filtering. The results on MATH and DeepScaleR with two Qwen3- 1.7--Bae models as policy and value models are shown in Fig. Since the prediction head of the value model is randomly initialized, the initial explained variance very low. As training progresses, the value model improves steadily and achieves an explained variance comparable to using three rollouts per prompt for value estimation. Specifically, with km = 2048, generating n = rollouts per prompt takes 288 seconds on MATH and 396 seconds on DeepScaleR per step. In contrast, training and inference with the value model require only 23.9 and 23.5 seconds respectively, achieving 12.1x speedup on MATH and a 16.9x speedup on DeepScaleR.
Results visualized in Figure 1.
8

---

Published as a conference paper at ICLR 2026
The accuracy of the value model with filtering at T 0.5 matches that of training without filtering. One might expect the value model to suffer from filtering, as the training data is biased toward prompts with estimated difficulty near the threshold т, potentially limiting generalization. To investigate how the choice of threshold r affects the accuracy of the value model, we ablate over TE (0.1, 0.3, 0.5, 0.7, 0.7.0.9.. In addition, we train baseline value model without any prompt filtering (i.e., GRPO but with a value model trained alongside the policy) using Qwen3-1.7B-BB policy and value model on the MATH dataset. Results are presented in Fig. 8. We observe that the value model achieves the highest prediction accuracy when 7 = 0.5, with performance degrading as the threshold deviates further from 0.5 in either direction. Notably, the accuracy of the value model at = 0.5 is comparable to the no- no-filtering baseline, despite training on filtered subset of prompts. We hypothesize that filtering at T = 0.5 still captures a diverse set of reward outcomes, as the midpoint of the binary rewards. Moreover, if the average reward of the policy over the training data is not 0.5 (i.c., there is label imbalance), filtering around r = 0.5 may implicitly rebalance the data, thus improving generalization. In contrast, filtering with extreme values (e.g.. = 0.1 or 0.9) selects only very easy or very hard prompts, leading to severe label imbalance and reduced predictive accuracy. A deeper theoretical understanding of why 0.5 leads to such effective value model training is an interesting direction for future work.
for both the
PCL progressively focuses harder prompts during train- ing, despite a fixed threshold of r = 0.5. To better understand the training dynamics of PCL. we analyze how the difficulty of selected prompts evolves over time. Specifically, we use the initial reference policy Tre to generate 16 responses for each prompt in DeepScaleR and compute the average reward, which serves as a proxy for prompt difficulty (i.e., lower average rewards indicate harder prompts). During training on Qwen3- 8B-Base with PCL, we log the average Trer-based reward for the filtered prompts at each training step. The results shown in Fig. 9. For methods that do not perform prompt filtering (GRPO and Pre-filter), this average remains nearly constant, as these methods uniformly sample from the dataset. In contrast, for methods that apply filtering (DS, SPEED, and PCL., observe a consistent downward trend in the -based reward of selected prompts. This indicates that these methods focus on increasingly harder prompts as training progresses. Although PCL maintains fixed difficulty threshold of 7 = 0.5, as the policy improves, previously hard prompts would now appear intermediate (i.e., T 0.5), allowing PCI. to continually shift toward more challenging examples.
Training Reward (after)
GRee
0.5
Pre fthzz
ns
based on Tire
0.4
PCL
0.3
0.2
0.1
Total Time (hours)
Figure 9: Training reward of PCL after filtering based on Ree w.r.t. training time with DeepScaleR and Qwen3-8B
-B-see. Base. PCL progres-
sively focuses on harder prompts during training, despite fixed threshold of r 0.5.
RELATED WORK
LLM Post-training. Reinforcement learning (RL) has become a standard for post-training LLMs, including Reinforcement Learning from Human Feedback (RLHF) (Ouyang al., 2022; Christiano et al., 2023; OpenAI, 2024a; Team, 2025a), enabling the LLMs to generate faithful and harmless responses that closely follow the instruction, and Reinforcement Learning with Verifiable Rewards (RLVR) (OpenAl, 2024b; Yang et al., 2024; DeepSeek-Al, 2025; Qwen, 2025; Lambert et al., 2025; Team, 2025b), improving model reasoning capabilities using verifiable rewards. These methods typically use algorithms include PPO (Schulman et al., 2017), GRPO (Shao et al., 2024), DR- GRPO (Liu et al., 2025), OREO (Wang et al., 2024), DQO (Ji et al., 2024), and VinePPO (Kazemnejad et al., 2025).
Efficient RL for LLM Post-training. Given the huge parameter size for L.MMs, there a large body of work recently focusing on developing more efficient algorithms and data selection methods to enable more efficient RL training for LLMs. Algorithmically, DPO (Rafailov et al., 2024), RAFT (Dong et al., 2023), REBEL (Gao et al., 2024), REFUEL (Gao et al., 2025), PPO (Brantley et al., 2025), RAFT+ (Xiong et al., 2025), RLOO (Ahmadian et al., 2024), and REINFORCE++ (Hu et al., 2025) are all trying to construct new objective functions that either reduces the number of models used (c.g. value model, reference model, reward model) or reduces the number of generations required for online RI. Another line of works (Xia al., 2024; Muennighoff al., 2025; Ye et al.,

---

Published as a conference paper at ICLR 2026
2025a; Muldrew al., 2024; Das al., 2025; Wang et al., 2025e; Sun et al., 2025; Wang et al., 2025a; Lin et al., 2025) focuses on improving data selections by reducing the amount of training data to be more sample efficient. DAPO (Yu et al., 2025) and VAPO (Yue al., 2025) resample and keep generating until the effective ratio of the batch is 1 during each step of RI. training. However, the generations for a prompt that are either all correct incorrect are wasted. SPEED (Zhang et al., 2025) improves on top of these methods by using a smaller number of generations to estimate the effective ratio and only generate the rest of the generations if the existing ones are not all correct or incorrect. GRESO (Zheng et al., 2025) and MoPPS (Qu et al., 2025), on the other hand, avoids rollouts by using a dictionary or a Beta posterior and samples based on the historical rewards from the previous epoch. However, they would suffer from off- offlppolicnness especially when the dataset is large.
Our method is the combination of the best of both worlds where PCL. directly avoids costly rollouts and also is on-policy. Our method is closely related to classic class of machine learning techniques, Curriculum Learning (Bengio et al., 2009). Previous works have explored currieulum learning for LLM post-training (Lee et al., 2024; Wen et al., 2025; Shi et al., 2025) by either training on progressively harder prompts ordered before training or focusing on certain difficulty range on the fly during RL. Our work falls in this group by always focusing on intermediate difficulty prompts for the current policy.
Curriculum and Active Learning. More broadly, our approach connects to curriculum and active learning methods that adapt the training distribution over examples. In supervised learning, Min- dermann et al. (2022) prioritize examples based on loss on held-out set, while Ash et al. (2020) selects diverse and uncertain examples using gradient embeddings. Kawaguchi & Lu (2020) analyzes how sample ordering affects convergence under Ordered SGD. These methods assume access to per-example gradients or labels on a static dataset, whereas we operate in an online RLVR setting where each example is a long response that cannot be determined before generating. In RL. Parker- Holder et al. (2022) and Ye et al. (2025b) study environment curricula by evolving tasks through regret-based environment design or asymmetric self-play. Under their minimax regret formulations, the optimal teacher concentrates on the hardest solvable tasks, in contrast to our empirical finding that convergence is fastest when training on prompts of intermediate difficulty. PCL adapts and extends ideas from curriculum and active learning to the specific constraints of large-scale, on-policy RtyR-style LLM post-training.
7 DISCUSSIONS & CONCLUSION
PCL accelerates RL post- training by targeting two findings from our study: (1) there exists an optimal total batch size at the transition between sublinear and linear generation- time scaling, and (2) prompts of intermediate difficulty (p(z) a 0.5) yield the highest gradient signal and sample efficiency. trains a value model online identify such prompts, avoiding the wasted rollouts of generation-baset filtering (DS, SPEED) and the off-policyness of dictionary-based methods (GRESO). PCL either achieves the highest performance or requires significantly less training time to reach comparable performance. We include discussion on limitations in Appendix J.
While our experiments focus on binary correctness rewards, PCL naturally extends to non-binary scalar rewards. Since the value model (zz) estimates Ey-r(( [=((=, y) non-binary r(x,8) only changes its range and the meaning of the target threshold 7. In addition, we note that PCL alternates updates between the policy and the value model, meaning that V is always one step behind the current policy 1+ In practice, this lag does not hinder performance, as the per-step policy updates are small with n, a T Theoretically, Wang et al. (2025b) provably shows that the ranking of prompt difficulties is stable under small perturbations between N. and TC+1. We also experimented with using importance sampling to correct for this lag by reweighting based on Th- (3) x)/m((2) I)), but it does not improve the accuracy of the value model and computing T++1 (y) x:) is computationally expensive as y is thousands of tokens long.
We highlight that prompt filtering methods rely on an implicit assumption of prompt-level generaliza- tion: training
selected subset of prompts will improve performance the filtered-out ones. For example, PCL assumes that training on intermediate-difficuitl prompts leads to improvements on both easier and harder prompts, while DS and SPEED assume that gradually solving not-too-hard prompts enables the model to eventually handle harder ones. While this assumption holds in domains like math where problems often share structural similarities, may not generalize to other domains.
10

---

Published as a conference paper at ICLR 2026
REFERENCES
Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, and Sara Hooker. Back to basies: Revisiting reinforce style optimization for learning from human feedback in Ilms, 2024. URL https: /atpa:://aaxxiii http:///ggaaa////////a/a
47440
Chenxin An, Zhihui Xie, Xiaonan Li, Lei Li, Jun Zhang, Shansan Gong, Ming Z.hong, Jingjing Xu, Xipeng Qiu. Mingxuan Wang, and Lingpeng Kong. Polaris: A post-training recipe for scaling reinforcement learning on advanced reasoning models, 2025. URL attps:///kun... //hunlpp gi wwtpb/////////.// /
Jordan T. Ash, Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. Deep batch active learning by diverse, uncertain gradient lower bounds, 2020. URL https:////////..c /ata:///rre //axxiv.
org/abs / 9066 03671.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th Annual International Conference Machine Learning, ICML '09, pp. 41-48, New York, NY, USA, 2009, Association for Computing Machinery. ISBN 978 1781655581161. doi: 10.1145//553374415533338 URL https: /tw.i//iiiio/////////..d/ org/ 10. 1144/1155133311 .0013344.11155555 1553380.
Kianté Brantley, Mingyu Chen, Zhaolin Gao, Jason D. Lee, Wen Sun, Wenhao Zhan, and Xuezhou
Zhang. Accelerating rl for II reasoning with optimal advantage regression, 2025. URL https:
//taaxii.orgg/aaaeal..//a/ 250. 20686.
Zhipeng Chen, Yingqian Min, Beichen Zhang, Jie Chen, Jinhao Jiang, Daixuan Cheng, Wayne Xin Zhao, Zheng Liu, Xu Miao, Yang Lu, Lei Fang, Zhongyuan Wang. and Ji-Rong Wen. An empirical study on eliciting and improving rl-like reasoning models, 2025. URL https: abs/2503, 04 548.
org/
Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep reinforcement learning from human preferences, 2023. URL http:s///axxv..o///////o org/bbb..rr....... 1706. 03741.
Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rihf, 2025. htts: / arxiv. www.v.org/aaaieeee...aaaoweaaeee wwt./oogg/aaaaa..e..oooo 10000
10500.
DeepSeek-AI. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning,
2025. URL https:////// /ttess//aaraiee www.i.ooo./raaa...oo /2501.111.0 .2948.
Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, and Tong Zhang. Raft: Reward ranked finetuning for generative foundation model alignment, 2023. URL https: / /aaaiiiii.. org/ abs 2304. 06767.
Wei Fu, Jiaxuan Gao, Xujie Shen, Chen Zhu, Zhiyu Mei, Chuyi He, Shusheng Xu, Guo Wei, Jun Mei, Jiashu Wang, Tongkai Yang. Binhang Yuan, and Yi Wu. Areal: A large-scale asynchronous reinforcement learning system for language reasoning, 2025. URL abs/2505. 24298..
Zhaolin Gao, Jonathan Chang, Wenhao Zhan, Owen Oertell, Gokul Swamy, Kianté Brantley, Thorsten Jason D Lee, and Wen Sun. Rebel: Reinforcement learning via regressing
Joachims, Drew Bagnell, Jason relative rewards. Advances in Neural Information Processing Systems, 37:52354-52400, 2024.
Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, and Wen Sun. Regressing the relative future: Efficient policy optimization for multi-turn rlhf, 2025. 04612.
URL https:///aaat
org/
Chaoqun He, Renjie Luo, Yuzhuo Bai, Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang, Yuxiang Zhang, Jie Liu, Lei Qi. Zhiyuan Liu, and Maosong Sun. Olympiad- bench: A challenging benchmark for promoting agi with olympiad level bilingual multimodal scientific problems, 2024. https: wwwpvv.vvvv..////v//...////aaavv orgg worggabb/////ooooooro.ooooeeober 4222. 12000.........
Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset, 2021. URL https: a wwwaiivvaaoo www....w.vv............ /203 03874.

---

Published as a conference paper at ICLR 2026
Jian Hu, Jason Klein Liu, Haotian Xu, and Wei Shen. Reinforce++: An efficient rlhf algorithm with robustness to both prompt and reward models, 2025, URL URL.https://xxxxw.rrrg 25011. 03262.
Hugging Face. Math-verit herifverff. http:///w....tto..cooo////////aa
Kaixuan Ji, Guanlin Liu, Ning Dai, Qingping Yang, Renjie Zheng. Zheng Wu, Chen Dun, Quanquan Gu, and Lin Yan. Enhancing multi-step reasoning abilities of language models through direct q-function optimization. arXiv preprint arXiv:2410.09302 2024.
Kenji Kawaguchi and Haihao Lu. Ordered sgd: A new stochastic optimization framework for empirical risk minimization, 2020. URL htwps: /ttp/aaaxvvvvv..///v//eee/ea.avv wttp/org////////////o 04371.
Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. VinePPO: Refining credit assignment in rl training of Ilms, 2025. URL https: : wawaivvvvv..//.a org abs 2410.
Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get a baseline for free! In DeepRLStructPeede @CLR, 2019. URL BtLhttp::/ https////appssaaiiiasiiioooooo semanticscholas org/ CorpusID 19M.D::99441111
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang. Ying Sheng, Liammin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention, 2023. URL pettts://arxtee org/ abs 2309. 0 06180.
Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang. Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, and Hannaneh Hajishirzi. Tulu 3: Pushing frontiers in open language model post-training, 2025. URL https: http::://aaxivvooo abs/2415 15124.
Bruce W. Lee, Hyunsoo Cho, and Kang Min Yoo. Instruction tuning with human curriculum, 2024. 09518.
URL https::/aaaxii..ooo/
Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo. Yuhuai Wu, Behnam Neyshabur, Guy Gur-Ari, and Vedant Misra. Solving quantitative reasoning problems with language models, 2022. URL https: / arxiv. orgg abs 2206 14858.
Zhihang Lin, Mingbao Lin, Yuan Xie, and Rongrong Ji. CPPO: Accelerating the training of group relative policy optimization-bart based reasoning models, 2025. URL https: //ap:ii/////iiv///// org/ abs/ 2503. 22342.
Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl ri-zero-like training: A critical perspective, 2025. URL https: wtt.o/b//////00//// 20783.
Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Tang. Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol ol-preview with a 1.5b model by scaling rl, 2025. Notion Blog.
Sören Mindermann, Jan Brauner, Muhammed Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N. Gomez, Adrien Morisot, Sebastian Farquhar, and Yarin Gal. Prioritized training on points that are learnable, worth learning, and not yet learnt, 2022. URL httts:://axxt orgg alss 220 6 07137.
Niklas Muennighoff, Zitong Yang. Weijia Shi, Xiang Lisa Li. Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling, 2025. URL https: /tte://arxxite http///axvvv..////////////e www.ariwii...aaaaeaaaaaaa abs//////////// 2501. 19393.
William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models, 2024. URL https: pttttpss:///xiiiiiee http://wwwwwoooo/////////oo/ 08114.
12

---

Published as a conference paper at ICLR 2026
OpenAl. GPT-4 technical report, 2024a. URL
OpenAl. Learning reason with 1lms. OpenAl Blog Post, 2024b.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models follow instructions with human feedback, 2022. URL https::/ //axivv org/ abs 2203. 02155.
Jack Parker-Holder, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. Evolving curricula with regret-based environment design. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 174473-17498 PMLR, 17-23 Jul 2022. URL httpss: ptwpp/ooceeiioooooooooo www.m.essrrssss....mmmmmmmmeeeee v1 62 wwtpp//pp.pae....e wwtp://aa///aaeeeeeeeeeeeee
Yun Qu, Qi Wang, Yixiu Mao, Vincent Tao Hu, Bjöm Ommer, and Xiangyang Ji. Can prompt difficulty be online predicted for accelerating rl finetuning of reasoning models?, 2025. URL
https:// http:///axvvvvvo org/ abss 2507.04632.
Qwen. Qwen3 technical report, 2025. URL / 09388.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model, 2024. URL
https://arxiv.orre wttp:r/gg//////oo////ereeee////o 18290.
Lorenz Richter, Ayman Boustati, Nikolas Nüsken, Francisco Ruiz, and Omer Deniz Akyildiz. Vargrad: low-variance gradient estimator for variational inference. Advances in Neural Information Processing Systems, 18:13481-13499 2020.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy 2017.
optimization algorithms. arXiv preprint
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300 2024.
Guangming Sheng, Chi Zhang. Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rlhf framework. In Proceed- ings of the Twentieth European Conference Computer Systems, EuroSys '25, pp. 1279-1297. ACM, March 2025. doi: 10.114536690311336900 3689031. 3696075
URL http: /tt///xx./// doi. www...org/0000 1145/
Taiwei Shi, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. Efficient reinforcement finetuning via adaptive curriculum learning, 2025. URL https: /ttp:///aavvvvv/ae wwtp:///rrr//b//oeeooooo...oss 2504. 05520.
Yifan Sun, Jingyan Shen, Yibin Wang, Tianyu Chen, Zhendong Wang, Mingyuan Zhou, and Huan Zhang. Improving data efficiency for Iim reinforcement fine-tuning through difficulty-targeted on- line data selection and rollout replay, 2025. URL httph:tt///// /ttp:i//aaavvv//a http:///aa......raaaa////666... 05316
Kimi Team. Kimi k1.5: Scaling reinforcement learning with 1lms, 2025a. URL WRLhhtsss//axii
org/abs abs///b// 2202.511.1 12599.
Kimi Team. Kimi k2: Open agentic intelligence, 2025b. URL http:ttt///cc//..oo////////
1507.20533
Huaijie Wang, Shibo Hao, Hanze Dong. Shenao Zhang, Yilin Bao, Ziran Yang, and Yi Wu. Offline reinforcement learning for Ilm multi-step reasoning. arXiv preprint 11X::24112.16455 2024.
Liangyu Wang. Huanyi Xie, Xinhai Wang, Tianjin Huang, Mengdi Li, and Di Wang. Infinite sampling: Efficient and stable grouped rl training for large language models, 2025a. URL
22950.
13

---

Published as a conference paper at ICLR 2026
Qi Wang, Zehao Xiao, Yixiu Mao, Yun Qu, Jiayi Shen, Yiqin Lv, and Xiangyang Ji. Model predictive task sampling for efficient and robust adaptation, 2025b. URL https: httppss///axiv..//// org / abs/
2103.
Yiping Wang, Qing Yang. Zhiyuan Zeng. Liliang Ren, Liyuan Liu, Baolin Peng, Hao Cheng, Xuehai He, Kuan Wang, Jianfeng Gao, Weizhu Chen, Shuohang Wang, Simon Shaolei Du, and Yelong Shen. Reinforcement learning for reasoning in large language models with one training example, 2025c. URL https: www.aaavvvvo........ abss/200 20571.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi. Quoe Le, and Denny Zhou, Chain-of-thouggh prompting elicits reasoning large language models, 2023.
URL httph:ttp/// /ttp///axxx //aa/axivvoorrr abs /2001. 11903.
Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, and Xiangzheng Zhang. Light-r1: Curriculum sft, dpo and rl for long cot from scratch and beyond, 2025. URL https:
warxiv.orge abs/ 2503 10460.
Ronald Williams. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8(3-4):229-2566 may 1992. ISSN 0885-6125. doi: 10. (0.10077BF009992666 URL https::t///. /tt:/oo..oooooo org 110 1007 BB0000//F0000 /BT////09996666
Bo Wu, Sid Wang, Yunhao Tang, Jia Ding. Eryk Helenowski, Liang Tan, Tengyu Xu, Tushar Gowda, Zhengxing Chen, Chen Zhu, Xiaocheng Tang. Yundi Qian, Beibei Zhu, and Rui Hou. Llamarl: distributed asynchronous reinforcement learning framework for efficient large-scale Ilm training, 2025. URL https: https://arxivv www.ooogg/aao. 200pp//555...3.33
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning, 2024. URL https:/// http://aaxvvvoro/e/// abs/ /402. 04333.
Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang, Lei Wang. Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, and Hanze Dong. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce, 2025. URL https: / wwwaaiiiivv.... www.iiiw...........ooo...eoooeee abs /2044. 11343.
Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down- sampling rollouts in Ilm reinforcement learning, 2025. URL https: http://aavvvvvv/////ee//.a orgg /bbs/ 2504.13818
An Yang. Beichen Zhang, Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li, Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, Keming Lu, Mingfeng Xue, Runji Lin, Tianyu Liu, Xingzhang Ren, and Zhenru Zhang. Qwen2.5-math technical report: Toward mathematical expert model via self-improvement, 2024. URL htt.t://///// //arx//axvvvvv//// v. org/ abs/ / 2409. 12122.
Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern, Shijie Xia, and Pengfei Liu. Limo: Less more for reasoning, 2025a. URL https:/ /al:://aaaaaa httpaaxivvvvoo/.aa/aaaa hwt. abs 220.22522.222 03367.
Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, and Yuan Liu. Scalable reinforcement post-training beyond static human prompts: Evolving alignment via asymmetric self-play, 2025b. URL https: www.aaiivvvv..oo org/ abs 2411.0001000
Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, Haibin Lin, Zhiqi Lin, Bole Ma, Guangming Sheng, Yuxuan Tong, Chi Zhang. Mofan Zhang, Wang Zhang, Hang Zhu, Jinhua Zhu, Jiaze Chen, Jiangjie Chen, Chengyi Wang, Hongli Yu, Yuxuan Song, Xiangpeng Wei, Hao Zhou, Jingjing Liu, Wei- Ying Ma, Ya-Qin Zhang, Lin Yan, Mu Qiao, Yonghui Wu, and Mingxuan Wang. Dapo: An open-source Il reinforcement learning system at scale, 2025. URL https: / //aaiv.voggg////.. abs/2500 14476.
Yu Yue, Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang, TianTian Fan, Zhengyin Du, Xiangpeng Wei. Xiangyu Yu, Gaohong Liu, Juncai Liu, Lingjun Liu, Haibin Lin, Zhiqi Lin, Bole Ma. Chi Zhang, Mofan Zhang, Wang Zhang, Hang Zhu, Ru Zhang, Xin Liu, Mingxuan Wang, Yonghui Wu, and Lin Yan. Vapo: Efficient and reliable
14

---

Published as a conference paper at ICLR 2026
reinforcement learning for advanced reasoning tasks, 2025. URL URU/httsss/arrx/v..r///// 2504. 05118.
Weihao Zeng, Yuzhen Huang, Qian Liu, Wei Liu, Keqing He, Zejun Ma, and Junxian He. Simplerl- 200: Investigating and taming zero reinforcement learning for open base models in the wild, 2025. URL https::/aaaaaa http///axvvv.o/////eee/eee
18892.
Ruiqi Zhang, Daman Arora, Song Mei, and Andrea Zanette. Speed-rl: Faster training of reason- ing models via online curriculum learning, 2025. URL
abs 2506.
09016.
Haizhong Zheng, Yang Zhou, Brian R. Bartoldson, Bhavya Kailkhura, Fan Lai, Jiawei Zhao, and Beidi Chen. Act only when it pays: Efficient reinforcement learning for Ilm reasoning via selective rollouts, 2025. URL https: / arxiv org/ abs/2506. 02177.
Banghua Zhu, Michael Jordan, and Jiantao Jiao. Principled reinforcement learning with human feedback from pairwise or k-wise comparisons, In International Conference on Machine Learning, pp. 43037-43067. PMLR, 2023.
15

---

Published as a conference paper at ICLR 2026
Appendix
CONTENTS
A Problem Setup Details
17
Preliminary Investigation Details
18
B.1 Dataset Details
18
B.2
Model Details
18
B.3
Reward Details
18
B,4
Evaluation Details
18
B.5
Complete List of Experiments.
19
Preliminary Investigation Complete Results
22
C.1
Complete Results for Section 3.1
22
C.1.1 Results with varying m
22
C.1.2 Results with varying and
24
C.1.3 Results with different context length
26
C.1.4 Results with different hardware configuration
27
C.1.5 Results with different inference engine
28
C.2 Complete Results for Section 3.2
29
Connection between p(x) and Gradient Magnitude
32
Experiment Details
33
E.1 Baselines ..grithns.......................
33
E.2 Dataset, Model, Reward, Evaluation Details
................................
35
E,3 Hyperparameters
35
Complete Experiment Results
36
G Value Model Size Ablation
37
H Stochastic Batch Selection
38
Training Time Breakdown
38
J Limitations
39
16

---

Published as a conference paper at ICLR 2026
A PROBLEM SETUP DETAILS
Let x: denote a prompt (e.g., a math question), and let denote a sampled solution of length lyl generated autoregressively from a policy π, i.e., y ~ [- æ).. We assume a binary reward function r ((r,)) y) e {0, 1), where r(x, v) = if the final answer in y is correct and 0 otherwise. Our goal is to learn parameterized policy me that maximizes the expected reward over a dataset 'D of prompts:
((0) Ez-D, 1--oo
(4)
Following the standard REINFORCE derivation (Williams, 1992), the gradient of this objective can be written (0) E,,y/ [r(x, V log ( x)].
To reduce the variance of this estimator, it is common to subtract a baseline function that depends only on the prompt x, which does not change the optimum of the policy gradient (Kool al., 2019; Richter et al., 2020; Zhu et al., 2023; Shao et al., 2024). In this work, we use the expected reward under the current policy, (y (r,, y')] as the baseline, which standard in LLM post-training (Shao et al., 2024; DeepSeek-AI, 2025; Yu et al., 2025; Liu et al., 2025). Since the reward is binary, we define Pro (z) == (t))) [r y) as the probability of generating a correct answer, and A(x,)) (x.y) Pro (x) as the advantage. The policy gradient can be expressed as ( (()))) [A(x, Vo log (3 r)].
In practice. LLMs are trained with multiple updates on generations produced by some old policy Toas and the training is often stabilized using techniques such as PPO-style clipping (Schulman et al., 2017; Shao al., 2024; Xiong et al., 2025). However, we focus on purely on-policy setting, where each gradient step is followed by the collection of fresh rollouts. Specifically, at each iteration t, we perform a single gradient step to maximize:
J(0)= Eg-D, t- [A(r, v) log mo m)]
Note that the above objective has the same gradient as:
To ( x) J(0) = Reg- v-no, (12) [ y)
].
(6)
To, (y x))
since we are purely on-policy and me, the policy before the update and also serves as the sampling distribution.
Given the autoregressive nature of LLMs, we further decompose the objective into token-level form, treating each token as individual action:
| I i-1
(o I (et))) )
J(t) E,-D, (
A(z. y) log
lul y Tog x, (..aaa)
Eg-D, (+))
A(x, v)
(8)
where y denotes the I-th token in the generated sequence. Similarly, the above objective has the same gradient
Iul
To V. Met)
J(()= = Ey.D, D, yo (1w)
A(x,V) v)
Σ
(9)
To. (3/
2., yet1)
Normalize by the length of y, we arrive at
M Σ
To (y)
xr. [el
J(0)= = E DE y-To, (4w)
A(x,2) lyl
(10)
To. (311
x2
This objective corresponds to purely on-policy variant of GRPO (Shao et al., 2024; DeepSeek-AI, 2025), without KL regularization to fixed reference policy neef (Yu et al., 2025) and without standard deviation-bar based advantage regularization (Liu et al., 2025). We adopt this formulation to eliminate the off ooflpooicssess during updates, clipping heuristics, and additional hyperparameters. This results in clean experimental setup that is directly derived from the original RL. objective in Eq. 4.
17

---

Published as a conference paper at ICLR 2026
B PRELIMINARY INVESTIGATION DETAILS
B.1 DATASET DETAILS
Table 2: Dataset split, maximum prompt length, and maximum generation length
Dataset
Huggingface Dataset Card
Train Val
Prompt Length
Generation Length
MATH
7.5k - 5k
1,024
4,096
DeepScateR agentica-org/DDeepSaaleek Preview-Datan
40.3k -/
1,024
4,096
Table 3: Model prompt format
Model Family
Prompt Format
Qwen (Base)
[prompt) Let's think step by step and output the final answer within (boxed
Llama (Instruct)
< |sstarr _header id systtmmm /end lend_eeader header id) Cutting Knowledge Date: December
[x |start
2023 Today Date: 26 Jul 3024-</eot id [> start header id [>-ser < </end header id |>> [prompt) Tet's think step by step and output the final answer within (boxed [ < </eot <|statt header id
|>assistant <Jend_hewe headereiiddd id
B.2 MODEL DETAILS
We perform full parameter training 8 A100 GPUs using Qwen3-1.7B-Baal (model card: LitnoQQen3-177-Baaeee Qwen3-4B-Base (model card: Qwen/Qwen3-44-BBan Qwen3-8B-Bass (model card: Qwen/QQwen3-B-BBat and Llama3.2 2-3tBttt (model card: meta-llama/Llama-33 Instruct).
B.3 REWARD DETAILS
We use a rule-based reward function based on the correctness of the response with math-verify, assigning +1 for correct answers and for incorrect ones generations that exceed the context length. Recent studies (Chen et al., 2025) have proposed incorporating format-based rules into reward calculations to encourage models to follow specific output formats. However, in experiments, we observed no significant difference in performance with or without such format-base rewards. Therefore, for simplicity, exclude them from implementation.
B.4 EVALUATION DETAILS
Following prior work (Zeng et al., 2025), we evaluate model performance on a suite of stan- dard mathematical reasoning benchmarks, including MATH500 (Hendrycks al., 2021), Minerva Math (Lewkowycz et al., 2022), and OlympiadBench (He et al., 2024), as well as competition-level benchmarks such as AMC 2023, AIME 2024, and AIME 2025.
For smaller-scale datasets, report results using the average reward across multiple generations. Specifically, for Minerva Math, we report Avg04; 4; for AMC 2023, AIME 2024, and AIME 2025, we report Avg
For MATH experiments, we use decoding parameters top_k = 20, temperature 0.6, and top-p 0.95. For DeepScaleR experiments, top_k -1 (i.e., disabled), temperature 0.6, and top p p==099 0.95.
18

---

Published as a conference paper at ICLR 2026
B.5 COMPLETE LIST OF EXPERIMENTS
The learning rate for each batch size is tuned on a logarithmic scale using the Qwen3-8B-Base model. For all other models, adopt the corresponding optimal learning rafe found for Wwen3-8B-Basse The complete list of all the experiments is provided below with the chosen learning rate highlighted in bold.
Table 4: Complete List of Experiments for Math
#Prorpts
aCienerations (s)
Context Length
Workers
Engine
16
4096
VLLM
1024
LE 6/2E-6
128
16
4196
VLLM
2048
1E-4/2-6655 1ET
2E-6A4E-6/8EE
Qwen3-SB-kae base
512
16
4096
VLLM
8192
4E KE-6/8E-6// D/EE-///..E .SE.S
1024
16
4196
VLLM
16384
48-A/888-61 LAE-S
4096
16
1096
VLLM
65536
BE
64
16
41996
VILM
1024
2,00K-066
16
4/96
VLLM
2048
2,00E-066
16
VILM
4096
00000066
Qwen3-1B- B-baue
512
16
1096
VLLM
8192
8.00E--06
1024
16
4096
VLLM
16334
8.00E-06
32768
1.00-005
4096
1096 16
VLLM
65536
3.20E-05
16
4/99
VLIM
1024
2,006-066
16
4096
VILM
2048
2. 2,008-066
256
16
1096
VILM
4096
4.00EE-06
Dwen3-1177B.. base
512
16
4096
VILM
8192
8. 8.00E-06
VIEM
0.000006
2048
16
4/96
VILM
32768
1 1.00-005
4096
16
4096
VILM
65536
3 3,208-05
2.00E5-06
16
4196
VILM
2048
2. 2.004-06
256
16
1096
VILM
4096
1,005-066
Llama3.2/30
1024
16
4/96
VILM
16394
8. 8.00F-066
2048
16
4096
VILM
32768
(0088066
1.20E-05
32
41996
VILM
1024
2. 2.00k-06
256
32
1096
VLLM
8192
5.005-06
Qwen
32
65536
3.200-05
64
1096
VLLM
1024
2.00E-06
128
64
4096
VLLM
8192
8.00E-06
64
3.206-05
19

---

Published as a conference paper at ICLR 2026
Table 5: Complete List of Experiments for DeepScaleR
#Prompts (m6)
Length
Engine
Batch
64
16
4096
VLLM
1024
1E-6/2E-6
128
4096
VLLM
2048
S/DDS
4096
4096
66-6--
Qwend-8B--ee 8B--Bss
512
16
4096
VLLM
8192
26-6448-466E6
1024
16384
45-6/88-6/ 28- 666-5
2048
4096
VLLM
32768
SE-6/122-S/L 66--5
4096
4096 16
VLLM
65536
DEEA//22E-5/L..O 6E-5
4096
2.00E-066
128
4096
VLLM
2043
2,90E-06
256
16
4096
VLLM
4096
4,00 4.00E-06
Qwen3-4B
4.00E-06
1024
16
4096
VILLM
16384
8.00F-06
2048
16
4096
VLLM
52768
1,20E--55
1.20E-05
64
16
4096
VILM
1024
2.00E-06
128
4096
VILM
2048
2,00E-00
4096
4006
4.00F-066
Qwea3-L.7B B.-b----BB-bsee
512
16
4096
VILM
8192
4,00E-06
8
2048
4096
VLLM
32768
1.20E-05
4096
4096 16
VILMM
65536
1,200--05
128
16
4006
VILLM
2048
2,008-066
256
16
4096
VILM
4096
4.00E-06
Llama3.2-3B-it
4.00E-06
1024
16
4096
VILLM
163344
8.00F--06
2048
16
4096
VLLM
32768
8. 8,00E-06
1.20E-05
32
32
4096
VILM
1024
2,00E 1000-06
256
4096
VILM
8192
4.00E--00 -000-00
Qwen3-4B huse
2048
4096
65536
1,2006-05
16
64
4096
VILM
1024
2.00E-06
128
1024
4096
VLLM
65536
1.20E-05
64
16
8192
VILLM
1024
2.00F-066
128
256
16
8192
VILLM
4096
4.00%--06
Owen3-1B-bbase
512
8192
VILM
8192
4.00 4.00E-06
8.00EE-05
2048
16
8192
VILLM
32768
1,2088-05
4096
16
8192
VLLM
65536
1,20E-05
4096
256
1,00E-06
32
16
4096
VILM
512
1,00E-00
64
4096
VILM
1024
2.00E-00
Qwen3-4B-bel
128
4096
2048
2.008-066
256
16
4096
VILM
4096
4,00E
1024
4096
VLLM
16384
8.00E-00
16
16
4096
SGI
256
1.00F-066
16
4096
SGI ang
1024
2.008-066
128
4096
SGLang
2048
2. 2.00E-06
Qwen3-4B
SGLang
4.00E-06
512
16
4096
SGI ang
8192
4.00F--06
1024
4096
SGLang 16384
8.00 8.00E-06
2048
ang
32768
1,2085-05
4096
16
4096
SGLang
65536
1,20E-05
20

---

Published as a conference paper at ICLR 2026
Table 6: Complete List of Experiments for DeepScaleR (cont.)
Model
&Prompts
(Generations
Context Lengch
Num Workers
Engine
Batch Size
LR
32
128
4096
VLLM
4096
4,00E-06
64
64
4096
VLLM
4096
4,00E-06
128
4096
4,00E-06
256
16
4096
VILM
4096
4.00F-06
Qwen3-4B p(x)
4,00I-066
1024
4096
VLLM
4096
4,00E-06
2048
VLLM
4096
4.000-06
32
128
4096
VLLM
4096
4,00E 06
4.00E-466
128
32
4096
VLLM
4096
4, 4,00E-06
256
4096
VILMM
4096
4. 4.000066
Qwen3-4B. Been3-44-Buusee 4.B-Buse & p(ar) 0.25
512
4096
VILM
4096
4.00E.000 06
1024
4096
VLLM
4096
4. 00E000
2048
4096
VLLM
4096
4, 4.00E-06
32
128
4096
VLLM
4096
4, 0000-006
4096
128
32
4096
VILM
4096
4.00E-00 06
Qwen3-IB-Base p(r) 0.5
512
4096
VLLM
4096
4, 4,00E-06
1024
4096
VLLM
4096
4, 1.000-066
2048
4096
VILLM
4096
4. 4.00F-066
32
4096
VILLM
4096
4. 4.000-06
64
4096
VLLM
4096
4, 4.00E-06
128
32
4096
VLLM
4096
4, 0000-066
256
4096
1.000-006
Qwen3 & p(r)) 0.75
512
4096
VILLM
4096
4. 4.00F-06
2048
4096
VLLM
4096
4,00E-06
4096
64
64
4096
VLLM
4096
4,00E 06
128
4.00E-416
256
16
4096
VLLM
4096
4,00E-06
Qwen3-4B-BBee & p(x)
512
4096
VILMM
4096
4.00F-066
1024
4096
VLLM
4096
4.00E
2048
4096
VLLM
4096
4 4.00E-066
21

---

Published as a conference paper at ICLR 2026
C PRELIMINARY INVESTIGATION COMPLETE RESULTS
C.I COMPLETE RESULTS FOR SECTION 3.1
C.I.I RESULTS WITH VARYING
Qwen3-8B-Ban MATH
14096, 16)
Gen.. Step (hours)
Training Reward
(2048, 16)
0,88
Accuracy
0.8-
(1024,
-0.86
0.7-
0.7
(256, 16)
10"
01.04
(128, 16)
0.82
164, 16)
200
5
10
(65,
(512, 16)
(4096,
Total Time (hours)
Batch Size
Qwen3-44-Bart & MATH
0.9
(4096, 16)
Gen. Time Step (hours)
10a
0.835
Training Reward
Training Reward
(204K, 16)
0.830
Accuracy
0.8
0.8
(1024, 16)
(512, 16)
0,820
0.7
0.7
0.815
0.6
0.6
(128, 16)
0,810
(64, 16)
0.5
0.5
0.805
(512, 16)
16)
Step
Total Time (hours)
Batch Size (mn, n)
Qwen3-1.7B-BBB & MATH
14096,
Step (hours)
0.76
Training Reward
Training Reward
(2048,
00774
0.7
0.7
(1004, 16)
0.72
(512, 16)
10
0.70
(256, 16)
0.5
0.5
0.66
164,
1.00
200
500
10
15
(64,
(512,
(4096, 16)
Step
Total Time (hours)
Batch Size (m, n)
Llama3.2-388 & MATH
(4096, 16)
0.575
(hours)
Training Reward
0.8
Training Reward
(2048,
0.570
Accuracy
(1024,
0.7
0.7
0.565
(512,
1.6-
0.6
(256, 16)
0.5
0.5
(128, 16)
Gen.
0.555
0.4
0.4
(64, 16)
10
200
400
0.0
2.5
5.0
7.3
10.0
(64.16) 16)
(512, 16)
(4096, 16)
Step
(bours)
(m. n)
Figure 10: Results for all four models on MATH with n = 16. (Lefft / Middle) Training reward as function of training steps and wall-clock time. The legend indicates the batch configuration in terms (number prompts m, generations per prompt n). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are log scale.
22

---

Published as a conference paper at ICLR 2026
Qwen3-8B- B-B-BaseB & DeepScaleR
0.7
0.7
(4096,
Gen. Step (hours)
0.52
Training Reward
10'
Avg: Test Accuracy
(1024,
0.5
0.5
Training
(312,
(256, 16)
10
(128, 16)
0.3
0.3
(64, 16)
0.48
400
10
(64, 16)
(512, 16)
(4096, 16)
Total Time (hours)
Batch Size
Qwen3-4B- Owen3-4B-Base & DeepScaleR
(4096, 16)
Gen. Time Step (bours)
0.46
Training Reward
Training Reward
(2048, 16)
Arg: TeSt Accuracy
0.5
0.5
(1024, 16)
0.455
(512, 16)
0.4
0.4
0.44
10"
0.3
(128, 16)
(64, 16)
0.2
0.2
100
(64,
16)
(512, 16)
(4096, 16)
Step
Total Time (hours)
Batch Size (m, m)
Qwen3- 3-1.7B-Base & DeepScaleR
0.45
0.45
10
0.35
(409%, 16)
Gen. Step (hours)
0.40
0.40
Training Reward
Training Reward
(2048,
0.34
Test Accuracy
(1024,
0.33
10"1
0.25
0.25
(128,
0.20
0.20
(64, 16)
0.15
0.15
(64,
Step
Total Time (bours)
Batch Size (m, n)
Llama3.22 2--3BBt & DeepScafeR
0.45
0.45
0.40
0.40
(4096, 16)
Gen. Time Stup (1W0r3)
0.255
Training Reward
0.35
0.35
(2048, 16)
0.250
Avg. Tost Accuracy
(1024, 16)
0.30
0.30
0.245
10
0.25
0.240
0.20
0.235
(128,
0.15
0.230
(64, 16)
0.10
0.10
(64,
(4096,
Step
Total Time (hours)
Batch Size (m, n)
Figure 11: Results for all four models on DeepScaleR with n = 16. (Left / Middle) Training reward function of training steps and wall-clock time. The legend indicates the batch configuration in terms (number prompts m, generations per prompt n). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale.
23

---

Published as a conference paper at ICLR 2026
Table 7: Detailed Results for Fig. 11.
Minerva Avg@4
AMC23
AIME24
AIME25
Model
MATH500
Olymp.
Avg@32
Avg@32
Avg@32
Avg.
Wref
29.8
49.1
15.8
8.8
34.7
4096
52.2
70.9
21.5
49.1
2048
85.2
53.4
43.9
66.6
26.4
21.5
49.5
1024
85.6
54.9
45.9
70.5
22.8
19.3
49.8
Qwen3-8B-u Base
512
87.2
55.8
44.1
71.8
26.1
21.1
51.0
256
85.0
57.4
40.4
66.3
24.9
22.9
49.5
128
85.6
53.9
67.8
22.3
19.9
48.6
64
85.2
54.7
42.1
70.6
21.4
17.3
48.6
Tre
65.8
34.4
26.9
47.3
10.9
7.1
32.1
4096
45.8
59.8
16.4
43.0
2048
39.2
57.0
16.0
15.9
43.3
1024
81.6
46.1
40.2
59.3
18.1
16.1
43.6
Qwen3-4B-BP
512
84.0
49.7
38.8
62.9
17.1
17.9
45.1
256
82.8
48.1
40.3
66.3
17.8
18.1
45.6
128
83.8
46.0
42.6
59.5
18.2
15.6
44.3
83.2
48.2
64.1
17.6
16.8
44.9
57.0
23.9
21.8
29.0
3.8
1.1
22.8
4096
69.8
35.2
29.0
40.7
9.1
8.0
32.0
2048
42.0
12.2
32.7
1024
72.2
36.2
29.7
41.8
12.4
7.0
33.2
Qwen3-1.7B-Ban
512
71.8
37.1
30.1
44.2
12.7
6.1
33.7
256
72.6
35.6
31.5
46.9
10.1
7.2
34,0
128
68.4
35.2
30.0
43.3
10.9
6.7
32.4
64
70.2
36.5
40.8
11.2
32.7
42.8
12.3
13.8
19.7
4.6
0.4
15.6
4096
55.2
20.0
21.1
31.6
11.9
0.6
23.4
2048
55.8
19.3
21.2
30,9
13.8
0.9
23.6
1024
22.8
21.2
34.8
12.1
1.2
25.0
Llama3.2-3B-it
512
58.0
22.3
22.7
30.0
15.8
1.6
25.1
256
57.6
21.8
22.6
32.0
14.5
0.4
24.8
128
55.6
22.7
22.2
34.8
13.9
0.1
24.9
64
56.8
20.9
25.5
31.8
10.2
0.2
24.2
C.1.2 RESULTS WITH VARYING AND
Table 8: Detailed DeepScaleR Results for Fig. 12.
Minerva Avg @4
AMC23
AIME24
AIMH25
Model
MATH500
Olymp.
Avg.
Avg
Avg@32
Avg@32
65.8
34.4
26.9
47.3
10.9
7.1
32.1
32
32
80.4
47.5
37.4
57.4
17.5
14.4
42.4
256
32
83.2
49.1
38.6
63.4
17.3
16.0
44.6
Qwen3-4B-Base
2048
32
81.4
46.3
39.8
57.6
17.6
15.5
43.0
16
64
81.6
47.8
39,2
58.4
15.7
14.3
42.8
128
44.4
41.5
17.0
13.1
1024
48.7
39.2
16.4
24

---

Published as a conference paper at ICLR 2026
Qwen3-4B 4B & MATH
Gen. Time Step (hours)
0.83
(2048,
Training Reward
Training Rewurd
0.8
0.8
+0.82
Test Accuracy
(256, 32)
0.7
0.7
10"
0.6
(32, 32)
0.5
(256, 32)
(2048, 32)
Step
Total Time (hours)
Batch Size (i,, m))
10"
0,850
Gen. Time Step (hours)
(1024, 64)
Training Reward
Training Reward
0.8
0.8
Tos Accuris acy
0.820
(128, 64)
0.7-
0.815
10"
0.6
0,810
(16, 64)
0.5
0.5
(128,
Step
Total Time (hours)
Batch Size (m, n)
Qwen3-4B & DeepScaleR
0.6
Gea. Time Step (hours)
10a
(2048, 32)
0.45
Training Reward
Training Reward
Avg. Test Acuraacy
0.5
0.5
(256, 32)
0.45
0.3
0.3
132, 32)
0.42
100
200
(32. (256, 32)
(2048,32)
Step
Total Time (hours)
Batch (m, n)
0.6
0.6
Gen. Time/ Step (hmurs)
0.454
(1024, 64)
Training Reward
Training Reward
0.452
AVg Test ccuracy
0.5
0.5
0.430
(128, 64)
0.4
0.428
0.3
0.3
0.126
64)
0.424
100
200
10
15
20
(16. 54)
(128, 64)
(1024, 64)
Step
(hours)
(m. n)
Figure 12: Results for Qwen3-4B on MATH and DeepScaleR with n 32 and 64. (Left Middle) Training reward function of training steps and wall-clock time. The legend indicates the batch configuration in terms of (number of prompts per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale.
generations
prompt n)). (Right) Generation time
25

---

Published as a conference paper at ICLR 2026
C.1.3
RESULTS WITH DIFFERENT CONTEXT LENGTH
Qwen3-4B & DeepScaleR
Gon. Time Step (hours))
Training Reward
Training Reward
16)
-0.6
Test AasEAcc Acuuaac
(1024, 16)
0.4
(512, 16)
(256, 16)
0.42
0.3
0.3
(128, 16)
0.40
10"
(64, 16)
0.2
0.2
100
10
(64. 16)
(512, 16)
(4096, 16)
Stup
Total Time (hours)
Batch Size
Figure 13: Results for Qwen3-4B on DeepScaleR with context length 8192 (other results are using 4096 context length). (Left Middle) Training reward as function of training steps and wall-clock time. The legend indicates the batch configuration in terms of (number of prompts m, generations per prompt n). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale.
Table 9: Detailed DeepScaleR Results for Fig. 13.
Minerva Avg@4
AMC23
AIME24
Model
Context Len.
MATH500
Olymp.
Avg@32 32
Avg@32
Avg 32
Avg.
Tiat
65.8
34.4
26.9
47.3
8K
80.4
47.9
39.2
60.7
15.9
15.1
43,2
128
8K
83.0
50.6
38.4
62.3
14.2
16.4
44.1
256
8K
82,8
50.4
42.2
62.0
19.3
18.0
45.8
Qwen3-4B-Ban
512
8K
81.2
49.6
41.0
65.3
17.0
12.5
44.4
80.8
46.6
39.7
63.7
18.9
44.4
37.5
$4.5
4096
8K
79.8
45.3
38.9
58.4
14.7
13.8
41.8
26

---

Published as a conference paper at ICLR 2026
C.1.4
RESULTS WITH DIFFERENT HARDWARE CONFIGURATION
Qwen3-4B & DeepScaleR
0.6
0.6
(1024, 16)
Gen. Time Step (hours)
-0.6
Training Reward
Training Reward
(512, 16)
10"
-0.5
Arg Test Aceur Accuracy
(256. 16)
0.4
0.4
0.43
(64, 16)
(32, 16)
10"
0.41
(16, 16)
0.2
0.2
200 400
0
1/1
20
(16, 16)
(12H. 16)
(1024, (6)
Total Time (hours)
Batch Size
Figure 14: Results for Qwen3-4B DeepScaleR with only I rollout worker with 8 GPUs (other results are using rollout workers, per GPU), (Left Middle) Training reward as a function of training steps and wall-clock time. The legend indicates the batch configuration in terms of (number of prompts m, generations per prompt ni). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale.
Table 10: Detailed DeepScaleR Results for Fig. 14.
Minerva Avg@4
AMC23
AIME24
AIME25
Model
Num. Worker
MATH500
Olymp.
Avg@32
Avg@32
Avg@32
Avg.
Tiue
34.4
78.4
47.2
39.2
58.0
15.3
12.4
41.7
32
1
81.8
47.0
39.0
61.3
17.1
14.9
43,5
64
1
83.6
49.4
40.6
58.7
15.9
16.4
44.1
Qwen3-4B-Bas
82.4
48.7
39.2
62.7
15.7
44.3
256
82.8
62.6
17.5
512
81.6
47.6
40.7
62.1
15.5
44,2
1024
80.4
47.8
40.4
60.1
18.9
16.9
44.1
27

---

Published as a conference paper at ICLR 2026
C.1.5
RESULTS WITH DIFFERENT INFERENCE ENGINE
Qwen334B & DeepScaleR
10'
0.455
Training Reward
0.6
Training Reward
0.5
(1024, 16)
(3048, 16)
Gen. Time Step (hours)
+0450
Avg. Accuracy
(512, 16)
0:445
00"
(256. 16)
0.4-
0.4
16)
(64. 16)
0.435
(32, 16)
10
0.430
(16, L6)
0.2
01.2
1000
(125, (1024,
Step
Total Time (hours)
Batch Size (m, n)
Figure Results for Qwen3-4B on DeepScaleR with SGL.ang (other results are using VLLM). (Left Middle) Training reward as function of training steps and wall-clock time. The legend indicates the batch configuration terms of (number prompts m, generations per prompt ri). (Right) Generation time per step and test accuracy across different batch sizes. The dashed line represents the linear increase that intercepts the origin and the generation time for the largest batch size. Both axes are in log scale.
Table 11: Detailed DeepScaleR Results for Fig. 15.
Minerva Av
AMC23
AIME24
AIME25
Model
Inference Eng.
MATH500
Olymp.
Avg@32
Avg@32
Avg@32
Avg.
Tref
65.8
34.4
26.9
47.3
10.9
7.1
32.1
SGLang
81.8
37.3
17.1
17.4
43.4
32
SGLang
81.2
40.0
17.0
14.9
44.0
64
SGI ang
81.6
50.9
39.8
60.2
15.3
15.4
43.9
Qwen3-4B-Bas
128
SGLang
84.0
48.8
39.7
62.9
15.2
17.1
44.6
256
SGLang
81.8
49.4
40.0
60.4
16.6
17.7
44.3
512
SGLang
82.2
40.9
17.0
15.2
44.8
1024
SGLang
81.2
51.3
41.0
62.9
16.8
16.1
44.9
2048
SGLang
81.2
46.0
40.6
61.0
17.3
14.4
43.4
4096
SGI ang
82.0
47.8
39.8
61.9
16.2
13.8
43.6
28

---

Published as a conference paper at ICLR 2026
C.2
COMPLETE RESULTS FOR SECTION 3.2
plx - 0.00
p(x) (X))-025 -025
pix) 0.50
p(x) 0.75
p(x) 1.00
Training Reward (before) Training Reward (befrre) Training Roward (before) Training Roward (before) Training Reward (befors) Training Roward (before) Training Reward (before)
Training Reward (after)
1.0
0.06
Ang. Effeetive Ratto
Text Acnurao comurayy
Grad Norm
-3.48
0.4-
0.4
0.4
00.00
25
75
25
75
0.0
0.25
0.5
0.75
1.0
Trainnn Reward (alber)
1.0
18:
1024, i
0.00
0.6
0.8
Augg Efffediiee Ratio
0.8
Arg. Grad Norm
0.46
Avp. Accurac
0.5
3..4
0.2
0.2
75
25
75
0.75
1.0
Training Reward (dfter)
Avg. Edfective Ratio
Grad Norm
-0445
Avg. Tost Arcuracy
00.00
0.6
0.6
0.4-
0.4
0.4
00.00
0.3
0.0
0.0
0.00
0.40
0.55
Training Reward (alter)
1.0
2.8
256,
Ang Effeedivv Rato
0.8
1.0
0.00
Avg. Gradd Norm
+0.45
Avg. Text Acruracy
0.5
3.44
0.4
0.4
0.4
0.00
0.2
50
100
50
100
0.0
0.25
0.5
0.75
1.0
Tritning Reward (wBer)
128,
32
0.6
Ang. Erflectrve Ratio
Avg: Grad Norma
0.45
Ang. Teat Accuracy
0.8
0.6
0.6
00.00
0.5
0.0
0.0
0.00
0.440
Training Reward (after)
1.0
0.06
Aug. Effective Ratio
Grad Norm
Avg. Test Assureyy
0.5
00.04
+3.44
0,4-
0.4
0.4
00,00
50
100
100
0.0
0.25
0.5
0.75
1.0
Training Reward (aller)
m= 32, a 128
0.6
0.3
Avgg. E.ffective Ratoo
0.3
Avg. Grad Norm
0.46
Avg. Accuracy
0.5
0.6
00
0.2
0.0
0.00
0..00
50
100
100
0.5
0.25
1.0
Steps
Steps
Threshold (p(x)
Figure 16: Results for Qwen3-4B on DeepScaleR with different p(x) under different decompositions, grouped by number of prompts and generations per prompt n. (Left) Training reward before downsampling in terms of step. (Middle) Training reward after downsampling. (Right) Average effective ratio, gradient norm, and accuracy across different thresholds.
29

---

Published as a conference paper at ICLR 2026
(2048,
(1024, 4)
(512,
(256, 16)
32)
(64, 64)
(32, 128)
0.00
1.0
1.0
Training Reward (before)
0.6
Truining Reward (after
(after)
Avg. Effective Ratie
0.46
0.5
0.6
0.6-
0.4
0.4
0.4
0.
0.2
0.0
0.00
0.40
20
40
20
2
p(x)
00.25
1.0
1.0
Training Reward (before)
0.6
Training Reward (after)
0.8
Avg. Effeetive Ratio
0.46
0.5
0.6
0.6
Toxt
0.4
0.4-
0.42
02-
0.0
0.0
0.40
p(x) 0.50
1.0
1.0
Training Reward (before)
0.6
Training Reward (ufter)
0.8
Arg. Effective Ratio
0.8-
0.3
0.44
Text
0.4
0.4
0.42
0.2-
0.0
0.0
0.40
p(x) - 0.75
Training Reward (before)
0.6
Training Reward (ufter)
0.8
Avg. Effective Ratio
0.8
0.5
0.6
0.44
Test
0.4
0.4-
0.4
0.42
0.2
0.2
0.3
20
40
50
80
20
40
60
80
p(x)
1.00
1.0
1.0
Training Reward (hefore)
0.6
Training Reward (after)
0.98
Arg. Effective Ratio
0.88
0.46
0.6
0.6
0.64
Text
0.4
0.42
0.2
0.2-
0.3
50
75
1.00
25
50
75
100
Steps
Steps
Number of Generations (1)
Figure 17: Results for Qwen3-4B on DeepScaleR with different p(x)) under different decompositions (number prompts m, generations per prompt n)), grouped by p(x)). (Left) Training reward before downsampling in terms of step. (Middle) Training reward after downsampling. (Right) Average effective ratio, gradient norm, and test accuracy across different thresholds.
30

---

Published as a conference paper at ICLR 2026
Table 12: Detailed DeepScaleR Results for Fig. 16 and 17.
Minerva Avg@4
AMC23 Avg@32
AIME24
AIME25
x(x)
MATH500
Olymp.
Avg.
Av
Avg @ 32
Wref
65.8
34.4
26.9
47.3
10.9
7.1
32.1
2048
2
82.8
44.8
39.4
54.3
15.3
13.4
41.7
1024
4
81.8
44.7
38.4
54.8
17.1
14.5
41.9
512
8
82.6
48.1
40.0
57.5
14.7
14.4
42,9
0
256
16
83.4
46.6
39.4
58.0
16.7
14.9
43.2
128
32
79.8
45.1
38.1
59.4
18.2
16.9
42.9
64
64
83.4
47.3
40.7
59.3
15.4
13.6
43.3
32
128
81.0
49.1
38.5
54.8
17.7
16.2
42.9
2048
2
80.6
46.9
39.4
58.3
15.5
11.1
42.0
1024
4
84.2
48.2
39.9
61.6
16.6
16.5
44.5
512
8
81.8
49.3
40.1
61.2
20.7
15.5
44.8
0.25
256
16
82.8
47.0
38.5
63.2
19.6
19.3
45.1
128
81.0
50.1
63.0
21.6
14.4
45.0
64
64
83.4
49.1
41.6
58.8
19.1
17.8
45,0
32
128
84.0
49.1
40.0
61.9
19.5
19.0
45.6
2048
2
83.2
50.7
40.0
60.9
19.2
19.7
45.6
1024
81.4
53.7
63.7
17.2
45.6
512
8
84.2
50.1
40.3
64.6
21.5
16.1
46.1
0.5
256
16
82.8
49.7
41.5
62.4
19.5
19.3
45.9
128
32
84.4
51.8
39.8
64.4
20.1
15.7
46.0
64
64
83.6
48.8
39.4
61.9
19.9
16.8
45.1
32
128
81.6
45.7
40.8
62.0
20.1
16.9
44.5
2048
2
81.0
48.7
39.0
56.6
17.5
14.6
42.9
1024
4
82.0
49.7
38.6
58.3
16.5
14.5
43.3
512
8
81.6
47.6
40.6
60.9
16.6
18.2
44.3
0.75
256
16
81.4
46.1
39.1
64.0
17.2
14.2
43.7
128
32
81.6
50.0
40.3
63.6
15.0
15,9
44.4
64
64
81.0
50.0
41.1
61.9
15.1
17.6
44.4
32
128
82.0
49.3
40.6
62.0
17.7
17.1
44.8
2048
2
78.6
46.6
41.6
55.8
14.3
10.6
41.3
1024
4
80.8
48.8
38.3
60.2
16.7
14.0
43.1
512
82.4
45.7
39.9
59.0
18.8
43.7
1
256
16
82.0
46.7
40.6
58,8
18.4
15.5
43.7
128
32
81.4
48.2
39.3
57.6
17.0
15.2
43.1
64
64
81.2
46.9
40.7
56.8
18.8
14.1
43.1
32
128
82.4
45.5
39.7
62.9
16.6
18.6
44.3
31

---

Published as a conference paper at ICLR 2026
D CONNECTION BETWEEN p(x) AND GRADIENT MAGNITUDE
We now analyze the squared norm of the gradient in Eq. (6):
[[VaJa || (40 Va log ()]x))]]
(11)
Using Jensen inequality, have:
(1) v) Toll s s Byrr (1x) (I) log (o())>)
(12)
= [t (1)) v) VI log To (v]=))]
(13)
Apply Cauchy-Schwars
(1g= V| log ( V (() botl:///.ii..////m/.////////o///
(14)
Now let's derive for (A(x,
(x(=,))))(((111))) (r(22)) Pro (x))) Pro (P. (z) - Pro (x)) + (1 Pno (z)) (n.
(15)
(16)
Pro Pr, (1
(17)
which is maximized at Pr. (x) 1. Therefore, intermediate-level prompts yield the largest expected magnitude of advantage and upper bound on gradient updates. These observations motivate prompt curriculum strategy that prioritizes prompts with intermediate difficulty (Pra (x) = 2) to enhance training efficiency.
32

---

Published as a conference paper at ICLR 2026
E EXPERIMENT DETAILS
E. BASELINES ALGORITHMS
We list the pseudo-code for GRPO, Pre-filter, DS, and SPEED, For the pseudo-code of GRESO, please refer to Zheng et al. (2025).
Algorithm 2 GRPO
Require: Number of prompts m, generations per prompt
Initialize policy To
2: for L= 0 to T do
3: Sample a batch with m prompts: Dm (-') CD.
4: Generate n responses: Dm {<*,, (yr0) ) 15-1] ) where yid iid
TT(1
5: Update to #++1 with GRPO objective using Dm
6: end for
Algorithm 3 Pre-filter
Require: Number of prompts mi, generations per prompt n, pre-filter generations Opres thresholds
Plow and Phigh Initialize policy To
2: Generate ppre responses for each prompt in the dataset: {y' Jj-1 TT(-l2)) for each D
3: Filter to keep prompts with accuracy between Plow and Phighi
Ex
Plow
Σ
(x,,0) < <high)
Plrre 5=1
4: for = 0 to T do
5: Sample a batch with m prompts: Dm (-) CDD
6: Generate n responses: Dm {a (1 ( } where yhd iid
TT(: (+)+
7: Update to T41 with GRPO objective using Dm
8: end for
Algorithm 4 Dynamic-sampling (DS)
Require: Number of prompts m, generations per prompt n, sampling parameter I-
1: Initialize policy To. Dbuller
2: for &= 0 to T do
3: while 'Drufter| < m do
4: Sample batch with km prompts: Dem fmm
Jel CD. km
5: Generate responses: Dam {(x*, (1r5) 3 :-1
where yfd me(: xt)
6: Select prompts with mean reward between 0 and 1:
n
Doutter + Doufter U {xx, (v') E Dam
0 <
Σ r(z,)") <1}
n j-1
7:
end while
8: Sample batch with prompts: Dm (xt, (ytd)n ) } cD D
9: Druffer Ø
10: Update to m. with GRPO objective using D...
11:
end for
33

---

Published as a conference paper at ICLR 2026
Algorithm 5 SPEED
Require: Number of prompts generations per prompt n. sampling parameter k, screening number
of responses Ainit (n 2 Tunit)
1: Initialize policy To. Druffer + Ø, Daccopted Ø
2: for t= 0 to do
3:
while (Douterl m do
4: Sample batch with km prompts: Dem (xt) km D.
Ji-l
5:
Generate Aimit times for Dem and n Plinit times for Daccopted:
Dkm {(x", f 1==1 (1
Panic f. where yo lid Te(- x²)
D acceepted D aeceepted U {(x",
)) ) [Pasosal where yid lid (=( x²).
J==1 Add D Paccepted to D Dufferi Douffer Druffe D accopptd
6:
7:
Select prompts with mean reward between and and add to Daccepted
Pinit
D Daccopted + {(x, fur
f (iint) Dkm 10<
Σ
r(x,8') < 1}
Pinit
1-1
8:
end while
9: Sample a batch with m prompts: Dm {at,, (1 c Dufferr
10: Drutter Drulter D
Update to T with GRPO objective using Dm
12: end for
34

---

Published as a conference paper at ICLR 2026
E.2 DATASET, MODEL, REWARD, EVALUATION DETAILS
We adopt the exact same setting in our preliminary investigation. Refer to Appendix B for details on datasets, models, rewards, and evaluations.
E.3 HYPERPARAMETERS
We list the hyperparameters for each method below. We tune the sample batch size / sampling parameter for DS, SPEED, GRESO, and PCL with the bolded one as the best one. For PCL, we always use the same-sized model as the value model for the main results. The learning rate of the value model in PCL is tuned within {le 6, 3e 6, le - 5}. All learning rates make the value model converge to the same performance but larger learning rates one. Therefore, we pick le-6 as the learning rate for the value model. We also tune the sampling parameter & from {2, 8) and observe that it has minor effect the accuracy of the value model. Larger values of & lead to higher effective ratios, as the filtering becomes more aggressive. However, increasing beyond yields only marginal improvements in the effective ratio. Therefore, we = for all experiments on PCL. An ablation on the value model size is included in Appendix G.
unstable than the smaller
Dataset
Method
Parameters
MATH
GRPO
512 Ir Se 6
16
DeepScaleR
GRPO
m = 512
16
Ir 4e 6
MATH
Pre-filter
m 512
Ir = Se Plow 0
Ppr = 16
Phigh
DeepScaleR
Pre-filter
512
= de
Ppre = 16
Plow
Phigh
MATH
DS
512
16 1/2/4
DeepScaleR
DS
512 Ir 1e
16 k= 1/2/1
MATH
SPEED
m = 512
== 16 k= 1/2/4
Ir 8c 6 Pinit 4/8
DeepScaleR
SPEED
m = 512
n 16 k= 1/2/4
Ir = 4e 6 4/8
MATH
GRESO
m = 512
= 16 Edefault 768/1021
Ir 8e 6
Peasy 0.5
Phard 0.5
Oearg 0.083
Qhard 0.167
Ap 0.01
DeepScaleR
GRESO
512
Ir= = 4e
gdefaul: 768/1024
Peasy 0.5
Phard 0.5
Ocasy 0,083
Qhard 0.167
Ap 0.01
MATH
PCI.
m = 512
= 16
Ir = 8e - 6
Iraitic 1e-6/3c-6/1o-5
r 0.5
k 2/4/8
DeepScaleR
PCL
m 512
Ir= 4e 1e-00 = 0.5
(raii 1e-6/3e-6/le-5
k= 2/4/8
35

---

Published as a conference paper at ICLR 2026
F COMPLETE EXPERIMENT RESULTS
Table 13 shows the full results on DeepScaleR. PCL consistently achieves either the highest final accuracy substantially reduced wall-clock time comparable accuracy. We summarizes the results on DeepScaleR below, comparing PCL with the best or the second best method for each model in terms of test accuracy:
Qwen3-8B-Bast PCL is top-performing method and converges in 41.8 hours, compared to 69.5 hours for DS, which also achieves lower average accuracy. PCL. converges 39.8 % faster.
Qwen3-4B-Bat PCL achieves 45.7 test accuracy and converges 32.8 hours, compared to 45.8 accuracy and 40.1 hours for DS. PCL therefore converges 18.2% % faster while maintaining similar performance.
Qwen3-1.7B-Base PCL is again the best-performing method and converges in 23.3 hours, whereas Pre-filter requires 44.2 hours and yields lower average accuracy. PCL converges 47.3% faster.
Llama3.2-3B-it. PCL reaches 26.5 accuracy in 28.7 hours, compared to 26.7 accuracy in 40.6 hours for DS, achieving similar accuracy with 29.3% % faster convergence.
The consistent 1.3 to 2x reductions in convergence time are practically impactful in real RL pipelines where rollouts the dominant cost.
Table 13: Full Results on DeepScaleR. For each metric, the best- best-performing method is highlighted in bold, and the second-best underlined. Time is the of training and generation time of the checkpoint that achieves the best average performance (excluding validation checkpointing) in hours.
Minerva Avg@4
AMC23
AIME24
AIME25
DeepScalcR
Method
MATH500
Olymp.
Avg@32
Avg@32
Avg@32
Avg.
Time
Fref
70.2
34.3
29.8
49.1
15.8
8.8
34.7
GRPO
87.2
57.9
45.3
70.1
25.3
22.7
51.4
43,0
Qwen3-8B-Base
Pre-filter
86.4
54.6
44.2
69.8
26.9
22.6
50.7
67.4
DS
87.2
55.3
45.7
71.5
24.9
24.2
51.5
69.5
SPEED
82.4
46.4
40.3
66.6
21.1
15.7
45.5
19.3
PCL
88.4
56.2
46.8
71.2
25.2
23.9
52.0
41.8
Fye
65.8
34.4
26.9
47.3
10.9
7.1
32.1
GRPO
83.4
51.0
40.1
60.7
16.1
20.7
45.3
45.5
Pre-filter
47.8
40.0
60.2
16.2
44.4
Qwen3-4B-Base
DS
51.6
62.4
18.5
18.0
45.8
40.1
SPEED
79.4
45.4
38.3
60.3
15.7
14.5
42.3
10.7
PCL
83.0
50.6
40.9
60.8
19.4
19.4
45.7
32.8
Fed
57.0
23.9
21.8
29.0
3.8
22.8
GRPO
72.4
37.7
31.2
44,9
11.2
6.7
34.0
46.2
Pre-filter
74,0
36.5
32.6
45.6
11.7
7.8
34.7
44,2
Qwen3-1.7B-BB
DS
73.2
36.9
31.9
42.7
10.8
7.7
33.9
41.7
SPEED
73.0
34.4
30.2
37.2
9.2
7.1
31.8
22.7
PCL
35.6
46.3
12.5
34.9
23.3
Frdf
42.8
12,3
13.8
19.7
4.6
0.4
15.6
GRPO
55.2
23.1
40.0
13.3
25.7
47.5
Pre-filter
24.5
35.5
26.2
Llama3.2 it
DS
57.2
23.3
37.1
26.7
SPEED
51.4
20.2
20.1
32.0
10.6
0.8
22.5
3.86
PCL
58.8
23.9
24.0
35.2
15.0
2.1
26.5
28.7
36

---

Published as a conference paper at ICLR 2026
G VALUE MODEL SIZE ABLATION
We ablate the value model used in PCL across three different model sizes, with results on explained variance of the value prediction shown in Figure 18. Similar Section 5, the average reward of 16 generations for each prompt as the ground-truth p(x). Overall, larger value models exhibit faster convergence compared to smaller ones. On the MATH dataset, all three value models eventually converge to similar performance levels. However, on the larger DeepScaleR dataset, we observe substantial gap after 100 training steps: the smaller value model significantly underperforms relative to its larger counterparts.
We hypothesize that the smaller value model may require more training steps to reach comparable accuracy and that, given sufficient time, all models could eventually converge to similar point. Nonetheless, this result highlights the benefit larger value models in the early stages of training, especially on large-scale datasets. In addition, 4B value model performs similarly to 8B model. This suggests that the size the value model does not need to scale proportionally with the policy. While we have not tested larger policies due to resource constraints, expect diminishing returns from larger value models.
Qwen3-1.7B-Ban
IT: Qwen3-4B-Bay
Qwen3-8B SB-Baac
1.0
1.0
1.0
Explained Variance -MTTH
0.8
0.8
0.6
0.4
0.4
0.4
V: Qwen3-1.7D Base
0.0
Vi Qwen5-ll--Bel
0.0
V: Qwen3 Base
0.2
0.2
0.2
25
s0 Step
75
100
25
50
75
100
25
50
75
100
Step
Step
Owen2-1.7B-Bat
Qwen3-4B-Bar
Qwen3-8B-Baa
Explained Variance DeepScaleR
1.0
1.0
1.0
0.6
0.6
0.6
0.4
0.4
0.4
0.2
0.2
0.0
0.0
0.0
50
75
100
25
50
75
100
50
100
Step
Step
Step
Figure 18: Explained variance MATH and DeepScaleR with different combinations of Qwen3 base models (1.7B/ 4B / 8B) for policy (T) and the value model (V).
37

---

Published as a conference paper at ICLR 2026
H STOCHASTIC BATCH SELECTION
In this section, we provide an ablation on using stochastic batch selection instead of greedy selection around threhold Since the reward is binary, we sample from Beta distribution whose support naturally lies in [0, J. We set a 3 for the Beta distribution and ablate a and 3 from 2 to 10, higher values indicate more focus on 0.5 difficulty prompts. From our experiments. greedy sampling and sampling based on Beta distribution perform similarly when a and & are large. However, using smaller values of a and 3, which spread the probability mass more broadly, actually degrades convergence, likely because it focuses less on 0.5 difficulty prompts. The results are detailed below for 1lamma32--3B-1l on MATH dataset.
Table 14: Results on MATH with Llama3.2 2-BB it under different (a.3) settings. Time is the sum of training and generation time the checkpoint that achieves the best average performance (excluding validation/checkpoinnng) in hours.
Q
MATH500
Time
56.4
10.8
5
5
56.2
15.9
57.2
15.6
greedy
57.8
14.3
We therefore adopt greedy top-m selection based threshold 7, which is simple without additional hyperparameters, and already delivers strong performance.
I TRAINING TIME BREAKDOWN
Table 15:
Breakdown of per-step policy and value training/inference time in seconds, averaged over the full training run, when training Qwen3-8B-Ban on MATH with three different sized value models (Qwen3-8B- 8B--ase, Qwen3-4BB 4a------B.... Qwen3-1.7B-Bass
Value Model
Generation
Value Computation
Policy Update
Value Update
Qwen3-8B-BB
573
2.99
769
20.5
Qwen3-4B-BB -ase
567
2.11
763
12.4
Qwen3-1.7B-Bal Base
582
1.16
760
8.03
For reasoning tasks, the prompt is typically much shorter than the response, which significantly reduces value model compute and memory compared to policy updates. We include a detailed breakdown of the time per step when training Qwen3-8B Base on MATH with three different sized value models (Qwen3-8B- -B--ass Table 15. Value computation time takes less than 1% of the generation time, and value model update time takes less than 3% of the time to update the policy.
Syve3-44--Base Qwen3-1.7B-Bass
The results are shown in
38

---

Published as a conference paper at ICLR 2026
J LIMITATIONS
While PCL demonstrates strong empirical performance across a range of models and datasets, our study has several limitations that open avenues for future work.
Purely on-policy setting. Our experiments are conducted entirely in purely on-policy RL setting, where new generations are sampled after each policy update. While this simplifies the analysis and avoids additional hyperparameters (e.g., clipping), it may reduce the generalization to more complex training pipelines that leverage off-policy data or replay buffers.
Focus on synchronous setting. Our preliminary investigation and PCL are evaluated in syn- chronous training setup where data generation and policy updates are alternated step-by-step. How- ever, many large-scale RL pipelines for LLMs adopt asynchronous architectures for better through- put (Wu et al., 2025; Fu al., 2025). Extending our analysis and PCL to asynchronous settings may require more sophisticated value model training and prompt selection strategies to handle stale or partially updated policies.
Relatively short context lengths. We limit our experiments to a maximum context length of 4,096 tokens due to compute constraints. While this setting is sufficient for the datasets used (c.g., MATH, DeepScaleR), real-world LLM deployments often involve much longer contexts. From our analysis, for longer context length, the batch size that transitions from sub-linear to linear generation time is larger. Future work could explore the interplay between prompt difficulty, batch decomposition, and context length in long-context regimes.
Limited training horizon. Our experiments constrained relatively short training runs (e.g.. 2-3 days), which may not fully capture long-term convergence behavior, especially for larger models and datasets. Although we observe strong carly-stage performance, it remains an open question whether our analysis in Section 3 would generalize to much longer training runs.
Limited task diversity. Our current evaluation is restricted to mathematical reasoning benchmarks, which provide clean and controlled environment for studying curriculum mechanisms but do not capture the full breadth of real-world LLM applications. such as code generation would require compiling and executing test cases during reward computation, substantially increasing runtime and compute cost. Due to limited computational resources, we were unable to include such experiments. We view applying PCL to broader set of tasks, including coding, multi-step tool use, and open-ended dialogue, as an important direction for future work that would further validate and potentially refine the proposed curriculum framework.
particular, extending PCL to domains
39


Reviews:
[{"review_id": "21yBWzguV7", "reviewer": "Reviewer_tnns", "summary": "This paper re-uses classical active batch selection method (phrased as curriculum learning) in the post-training of large language models. \n- Method: At each gradient update step, the method first samples a large pool of prompts, then chooses a subset from the pool as the training batch. The choice is based on signals from the value function. \n- Experiments: The authors conduct the experiments using MATH and DeepScaleR, with models ranging from 1.7B to 8B. As in Table 1, the performance gain is relatively neutral.", "questions": "I encourage the authors to consider more fundamental algorithmic improvements, instead of simply re-using old ideas as low-hanging fruit, which can be misleading. For example, can we do better than top-m subset selection? One related line of research is stochastic batch acquisition [[Kirsch et al., 2024](https://openreview.net/pdf/abd7cf88617c72ff4ec92264fb8a7919ef9ffee3.pdf)]. It would be helpful if the authors could provide a thorough discussion on the limitations of current work and include new experiments with better algorithms that bring something new to the academia. I am happy to raise my score if these concerns are sufficiently addressed.  \n\nReferences:     \n[1] Kirsch, Andreas, Sebastian Farquhar, Parmida Atighehchian, Andrew Jesson, Frederic Branchaud-Charron, and Yarin Gal. \"Stochastic batch acquisition: A simple baseline for deep active learning.\" arXiv preprint arXiv:2106.12059 (2021).", "limitations": "", "rating": 2, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- Presentation: The write-up is clear, and the descriptions of the experiments are detailed and easy-to-understand.\n- Experiments: The attempt in understanding \"optimal\" batch size is very interesting (although there should be many artifacts in experimental settings that affect the optimality other than the factors proposed by the authors. It would be nice to further scale. One such example is [[Goyal et al., 2018](https://arxiv.org/pdf/1706.02677)]).\n\nReferences:  \n[1] Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. \"Accurate, large minibatch sgd: Training imagenet in 1 hour.\" arXiv preprint arXiv:1706.02677 (2017).", "weaknesses": "- Lack of novelty: While the manuscript is well-written as a technical report, it offers limited methodological novelty or new insights to the field. The findings are well-known, and numerous prior works have explored similar directions. There appears to be little distinction between the proposed method and existing approaches, aside from the specific experimental setting (models/datasets). Consequently, this work seems better suited as a report or blog post, rather than a distinct contribution to academic research.\n- Missing citations and comparisons: Related works should be put in the main body of the paper, instead of in the appendix. Also, please properly cite prior works on active batch selection and curriculum rl, e.g., [[Mindermann et al., 2020](https://arxiv.org/pdf/2206.07137)], [[Ash et al., 2019](https://arxiv.org/pdf/1906.03671)], [[Parker-Holder et al, 2022](https://arxiv.org/abs/2203.01302)], which all to some extent show it is better to learn from examples of medium difficulty. The proposed method appears to be a special case of many existing works, like [[Ye et al., 2024](https://arxiv.org/pdf/2411.00062)] but using greedy selection, or [[Muldrew et al., 2024](https://arxiv.org/pdf/2402.08114)] but using reinforce instead of dpo, or [[Kawaguchi et al, 2020](https://arxiv.org/pdf/1907.04371)] but using value as the selection criteria. A proper comparison with these works is necessary to accurately situate the paper within the fundamental literature and clarify the specific methodological contribution. \n- Empirical performance: The empirical gains are marginal compared to existing baselines, in terms of both accuracy and computational efficiency. Further algorithmic exploration is needed to demonstrate significant performance improvements.  \n\nReferences:  \n[1] Kawaguchi, Kenji, and Haihao Lu. \"Ordered sgd: A new stochastic optimization framework for empirical risk minimization.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2020.  \n[2] Ye, Ziyu, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, and Yuan Liu. \"Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play.\" ICML, 2024.  \n[3] Mindermann, Sören, Jan M. Brauner, Muhammed T. Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen et al. \"Prioritized training on points that are learnable, worth learning, and not yet learnt.\" In International Conference on Machine Learning, pp. 15630-15649. PMLR, 2022.  \n[4] Ash, Jordan T., Chicheng Zhang, Akshay Krishnamurthy, John Langford, and Alekh Agarwal. \"Deep batch active learning by diverse, uncertain gradient lower bounds.\" arXiv preprint arXiv:1906.03671 (2019).    \n[5] Parker-Holder, Jack, Minqi Jiang, Michael Dennis, Mikayel Samvelyan, Jakob Foerster, Edward Grefenstette, and Tim Rocktäschel. \"Evolving curricula with regret-based environment design.\" In International Conference on Machine Learning, pp. 17473-17498. PMLR, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiGYuePvMG", "reviewer": "Reviewer_MXSJ", "summary": "This paper systematically studies how batch size and prompt difficulty affect LLM post-training dynamics and produces two key findings. First, the optimal batch size occurs at the transition point between sub-linear and linear generation time scaling. Second, they empirically confirm that prompts of intermediate difficulty (where the model has a ~50% success rate) provide the highest gradient norms and lead to the best test accuracy, validating insights from prior work.\n\nBuilding on these findings, the paper proposes Prompt Curriculum Learning (PCL). PCL trains a value model alongside the main policy to estimate a prompt's difficulty (expected reward) with a single forward pass. At each step, it greedily samples prompts whose predicted value is closest to 0.5. This value model serves as a lightweight and on-policy substitute for computationally expensive methods like rollout-based filtering and avoids the off-policy staleness of dictionary-based approaches, proving to be ~12-17 times faster at identifying informative prompts.\n\nEmpirical results across various math reasoning benchmarks show that PCL either achieves state-of-the-art performance or reaches comparable results in significantly less training time.", "questions": "Please see weaknesses", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper contains systematic and rigorous experimentation to empirically validate all its core claims on optimal batch size, prompt difficulty and the effectiveness of PCL. \n - PCL is a relatively lightweight algorithm that addresses the high computational cost of rollout-based filtering and the off-policy staleness of dictionary-based methods.\n - The value model trained has reasonable accuracy (same as 3 samples per prompt) and provides a clear enough signal on prompt difficulty and significantly reduces training time during the filtering stage. \n - The paper is well presented and easy to follow with a clear narrative.", "weaknesses": "- A core assumption in the paper is that rewards are binary. There are many tasks like creative writing where partial rewards are the norm. It is not clear how intermediate difficulty can be mathematically determined in those cases. Similarly, it is unclear whether the value model would succeed in accurately estimating the value of a prompt in such such tasks. \n - For practical reasons, oftentimes rollouts and policy updates happen simultaneously. In such a setup the value model would be even further behind the policy and it is not clear if the accuracy of value prediction would be affected in any way. \n - Filtering exclusively for intermediate difficult could lead to the model regressing on easy tasks and never improving on the hard ones. An interesting experiment could be to do weighted sampling of the prompts during training instead of greedy sampling to allow some easy and hard prompts to also be included in training. \n - While more efficient than rollouts, PCL requires training and maintaining a second, often same-sized, value model, which doubles the memory footprint compared to single-model approaches.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QtzzJoCDiZ", "reviewer": "Reviewer_ZNyN", "summary": "This paper investigates strategies for efficient post-training of large language models (LLMs) using reinforcement learning (RL). The authors first conduct a systematic study on the effects of batching and prompt difficulty on RL convergence. They identify two key findings: (1) an optimal batch size exists at the transition point from sublinear to linear growth in generation time, and (2) prompts of intermediate difficulty, where the model has approximately a 50% success rate, are the most sample-efficient for training. Based on these insights, the paper proposes Prompt Curriculum Learning (PCL), a lightweight algorithm that uses a value model to efficiently select intermediate-difficulty prompts. This avoids the high computational cost of rollout-based filtering. Experiments on mathematical reasoning benchmarks (MATH and DeepScaleR) show that PCL achieves competitive performance with significantly less training time compared to several baselines.", "questions": "1. The idea of using a value model for estimating sample difficulty is somewhat connected to LLM-as-judge methods; can you add some discussions in related work?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. To my knowledge, this is the first work to systematically analyze the optimal batch size and the optimal decomposition into the number of prompts and generations per prompt.\n2. Using an additional value model for difficulty estimation before rollout is an interesting idea.\n3. The paper is well-written and easy to understand.\n4. The authors have conducted comprehensive empirical evaluation to support the findings on Math datasets.", "weaknesses": "1. The main weakness I see is that in many cases, the performance gaps between PCL and GRPO is less than 1%, thus the significance of conducting active curriculum learning is somewhat limited in these settings. And the experimental evaluation is too narrow, as it relies solely on mathematical datasets. I suggest trying to highlight the effectiveness and test the generalization of PCL in other reasoning tasks, such as coding.\n2. Some related baselines are missing for discussion and comparison (e.g. MoPPS [1]). I mean this can be added to show iteration numbers as an efficiency metrics as the ablation to show connections (Unnecessary to implement for all experiments considering the time constraint in rebuttal). \n3. This paper lacks systematic theoretical analysis regarding why an additional value model is sufficient to predict prompt difficulty. (In Line 300, it says: \"Note that the value model $V$ in our algorithm is one step behind the policy $\\pi$, which is acceptable since each update is small with $\\pi_{t+1}\\approx\\pi_{t}$\".) However, I think this can be resolved by relating to some existing works or adding some existing theoretical findings. For example, this method belongs to a model predictive sampling strategy [2], which utilizes the optimization history to build up a risk-ware module. Note that using iteration $\\pi_{t}$'s generalization to approximate $\\pi_{t+1}$'s task difficulty evaluation outcome first occurs in MPTS [2], and this provides a rigorous predictive foundation. Adding reference and theoretical discussions on MPTS as this work's predictive foundation can well support the validation of the prediction results.\n4. \"we find that both training and inference of the value model incur negligible cost and can be\n  completed under 30 seconds for each step\", I think providing a detailed breakdown of time consumption for the whole training process (e.g., rollout, log_prob, policy model update, value model update, ...) would better support this claim.\n\n[1] Can prompt difficulty be online predicted for accelerating RL finetuning of reasoning models?\n\n[2] Model Predictive Task Sampling for Efficient and Robust Adaptation.\n\nOverall, I am quite positive on this work in terms of novelty and contributions. I'll update the score if the above weaknesses are addressed in the revised version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tfj3w9Trwi", "reviewer": "Reviewer_SyDP", "summary": "This paper investigates how prompt difficulty and batch configuration affect the efficiency of RL post-training for LLMs. The authors show empirically that there exists an optimal batch size that balances generation time and gradient quality, and that prompts of intermediate difficulty (where the model has roughly a 50% chance of success) are the most sample-efficient for model convergence. They propose Prompt Curriculum Learning (PCL), which trains a value model V(x) to estimate expected reward and select prompts closest to a target threshold $\\tau = 0.5$, enabling efficient on-policy filtering without expensive rollouts. PCL is shown to either exceed the performance of baseline methods or require less time to train for a similar performance.", "questions": "1. What is the empirical distribution of prompt difficulty $\\pi(x)$ across the full dataset for the initial policy and throughout training? Without this, it is unclear whether intermediate-difficulty prompts are abundant, rare, or emergent as a function of policy improvement.\n2. If intermediate prompts are scarce early on, is learning bottlenecked?\n3. Did you evaluate selecting a band of difficulties (e.g., $\\tau \\pm \\epsilon$) such as 0.2–0.8 or 0.4–0.6, instead of a point target? I think this may alleviate the scarcity issue by increasing sample diversity while still maintaining meaningful gradient feedback.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Advances the efficiency of LLM post-training based on interesting empirical findings about the batch size and the intermediate difficulty of prompts, which are supported by broad ablations across models, datasets, and a significant spend of resources.\n* The value network to select prompts is lightweight and easy to adopt.\n* The experimental section is well presented and results show the method reaches highest accuracy, or reaches baseline accuracy significantly faster.", "weaknesses": "* The method assumes that the dataset always contains a sufficient number of prompts with approximately 50% success probability. However, during early training, there may be very few intermediate-difficulty prompts, which could hinder learning and cause training to stagnate, as the model would not encounter harder examples. Similarly, toward the end of training, if the model improves rapidly, only a small subset of prompts may continue to be selected, potentially biasing the training distribution. The paper does not appear to analyze or address these edge cases, which could affect the robustness of the proposed curriculum.\n* Filtering may introduce distributional bias by over-representing specific mathematical subtypes while under-exposing the model to both foundational easy patterns and extremely difficult reasoning structures. This could lead to topic drift or a collapse toward narrower reasoning styles, as the curriculum disproportionately focuses on intermediate-difficulty prompts. The paper does not provide an analysis of how filtering affects topic coverage or reasoning diversity over training.\n* The lack of variance reporting weakens the paper's claims of robustness, as the absence of standard deviation metrics makes it difficult to assess the statistical significance of the reported improvements. Moreover, the paper does not present results across multiple random seeds for the final evaluations, leaving open the possibility that the observed gains may be sensitive to initialization or sampling noise.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rM7o4evm3E", "reviewer": "Reviewer_BGex", "summary": "The paper investigates how batching and prompt selection affect reinforcement learning post-training of large language models, identifying two key empirical findings: an optimal total batch size where rollout time scaling transitions from sublinear to linear, and the highest learning efficiency when prompts have intermediate difficulty (success probability ≈ 0.5). Building on these insights, the authors propose Prompt Curriculum Learning (PCL), which trains a lightweight value model to estimate per-prompt reward and selects prompts near a target difficulty without costly on-policy rollouts. Across several math reasoning benchmarks (MATH, Olympiad-Bench, AIME, etc.) and models (Qwen3, Llama3.2), PCL matches or surpasses other post-training baselines.", "questions": "Please refer to the weaknesses section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper focuses on a practically important yet underexplored question, how batching and prompt difficulty affect the efficiency of RL post-training for large language models, and grounds its approach in two clear empirical observations.\n\n2. Through systematic experiments, the study identifies an optimal total batch size and shows that prompts of intermediate difficulty yield the most effective learning signal, offering useful insights for designing efficient RL fine-tuning pipelines.\n\n3. The method is validated across several models and math reasoning benchmarks with comprehensive ablations, demonstrating consistent performance trends and providing a transparent analysis of efficiency, stability, and cost trade-offs.", "weaknesses": "1. The effectiveness of PCL may heavily rely on the accuracy of the value model. In more complex settings, a small or undertrained value model may fail to provide reliable estimates, leading to suboptimal prompt selection and degraded performance.\n\n2. Concentrating too much on p(x)≈0.5 (especially with large n, small m) lowers prompt diversity, which can reduce robustness and even degrade accuracy beyond some point.\n\n3. The experiments focus almost entirely on math reasoning tasks, leaving it unclear whether the findings on optimal batch size and prompt difficulty generalize to more diverse domains such as code generation or instruction following.\n\n4. If the average reward over the dataset is far from 0.5 (label imbalance), filtering near 0.5 may implicitly rebalance, which can be good or bad depending on goals; it’s another place PCL’s assumptions can break.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)