Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
Authors: Yifan Sun
Jingyan Shen
Yibin Wang
Tianyu Chen
Zhendong Wang
Mingyuan Zhou
Huan Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Data Efficiency
Online Data Selection
Experience Replay
Reinforcement Learning
Large Language Model
Primary area: deep_learning
TL;DR: We propose two techniques to improve the data efficiency of LLM RL fine-tuning: difficulty-targeted online data selection and rollout replay.

Abstract: Reinforcement learning (RL) has become an effective approach for fine-tuning large language models (LLMs), particularly to enhance their reasoning capabilities. However, RL fine-tuning remains highly resource-intensive, and existing work has largely overlooked the problem of data efficiency. In this paper, we propose two techniques to improve data efficiency in LLM RL fine-tuning: difficulty-targeted online data selection and rollout replay. We introduce the notion of adaptive difficulty to guide online data selection, prioritizing questions of moderate difficulty that are more likely to yield informative learning signals. To estimate adaptive difficulty efficiently, we develop an attention-based framework that requires rollouts for only a small reference set of questions. The adaptive difficulty of the remaining questions is then estimated based on their similarity to this set. To further reduce rollout cost, we introduce a rollout replay mechanism inspired by experience replay in traditional RL. This technique reuses recent rollouts, lowering per-step computation while maintaining stable updates. Experiments across 6 LLM-dataset combinations show that our method reduces RL fine-tuning time by 23% to 62% while reaching the same level of performance as the original GRPO algorithm. 
Our code repository is available at https://github.com/ASTRAL-Group/data-efficient-llm-rl/.

Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-tageeted Online Data Selection and Rollout Replay
Yifan Sun UIUC
Jingyan Shen* New York University
Yibin Wang" UIUC
Tianyu Chen University of Texas at Austin
Zhendong Wang Microsoft
Mingyuan Zhou University of Texas at Austin
Huan Zhang
UICC
Abstract
Reinforcement learning (RL) has become an effective approach for fine-tuning large language models (LLMs),, particularly enhance their reasoning capabilities. However, RL. fine-tuning remains highly resource-intensive,, and existing work has largely overlooked the problem of data efficiency. In this paper, propose two techniques to improve data efficiency in LLM RL fine-tuning: difficulty- targeted online data selection and rollout replay. We introduce the notion of adaptive difficulty to guide online data selection, prioritizing questions of mod- erate difficulty that are more likely yield informative learning signals. To estimate adaptive difficulty efficiently. we develop an attention-bare framework that requires rollouts for only a small reference set of questions. The adaptive difficulty of the remaining questions is then estimated based on their similarity this To further reduce rollout cost, we introduce a rollout replay mecha- nism inspired by experience replay in traditional RI. This technique reuses recent rollouts, lowering per-step computation while maintaining stable updates. Ex- periments across 6 LLM-dataset combinations show that method reduces RL fine-tuning time by 23% to 62% while reaching the same level of perfor- mance as the original GRPO algorithm. Our code repository is available at htttps://gitaei com/ Aww.AASTRRALAAA GataAAAATA-----raaaaaaaaaaa-a-
11m-r1/
Introduction
Reinforcement learning (RL. has emerged promising and increasingly adopted paradigm for fine-tuning large language models (L.LMs) toward stronger reasoning capabilities [10
30 15 533
Despite a steady stream of algorithmic improvements [51 28 50), relatively little attention has been paid to improving the data efficiency of LLM RL fine-tuning. This gap is particularly concerning given that RL fine-tuning for LLMs is notoriously computationally expensive
'Equal contribution.
'Correspondence to Yifan Sun <yifan5001 11 www.iinon...immooiiii edu> and Huan Zhang
<huan@huan-Zhh com>.
For example, Luo et al. (30] report that training a relatively small 1.5B -arameeet model on just 40K samples required over 3,800 A GPU hours-equivalent to approximately $4,500 in compute cost-even before scaling to larger models or longer training horizons.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Next RL Fine-tuning Iteration
Dificulty-targette
Rollout Replay
Online Data Selection
Reference Qquestions
Adaptie Dfficuity
Retrieue
Training
FIFO Replay Buffer
Recent Rolioutts
Policy Update
Model
Questions
ffiuuityy Predictor
Uniobaled Questions
Adoptii beficuity
Rollouts
LLM
Writetn
Current Roliouts
Rollout
Questions with Difficulty Closest 0.5
Figure I: Overview of our framework combining Difficulty-targetee Online Data Selection and Rollout Replay. At each training step, the online data selection module selects training questions with adaptive difficulty near 0.5, requiring rollouts only on a small reference set ($4.1 $4.2). The rollout replay module combines current rollouts with retrieved recent rollouts from FIFO buffer, and the current rollouts are stored into the buffer for future use ($4.3).
In this paper, we present two simple yet effective techniques to improve the data efficiency for LLM
RL fine-tuning: Difficulty-taggetel Online Data Selection and Rollout Replay. Our goal is to reduce both (I) the number of training steps required to match the performance of the original GRPO algorithm, and (2) the per-step computational cost.
Total RL fine-tuning time
Number of training steps
X
Time per step
Rolueed Difficuly-argttal Online Selection
Reluced
Rollour Reploy
Difficult--targeted Online Data Selection (DOTS) In RL, tasks that are too easy too difficult often provide limited learning signal [7] Moreover, since the policy evolves during training, is
crucial to adopt an online and adaptive mechanism for selecting informative data [34 377 To this
end, we introduce the notion of adaptive difficulty, which measures how likely the current policy is to fail on a given question. At each training step, we prioritize questions of moderate adaptive difficulty, as these are most likely to yield meaningful learning signals.
However, computing adaptive difficulty exactly requires executing multiple rollouts per question, which is computationally expensive. To address this, we propose an attention-based adaptive difficulty prediction framework that efficiently estimates difficulty without generating full rollouts for all questions. each training step, we generate rollouts only for small reference set and compute their ground-truth adaptive difficulty. The difficulty of the remaining questions is estimated attention.
by comparing them to the reference set using similarity- Rollout Replay (RR) To further reduce the cost of rollout generation, replay mechanism, motivated by experience replay in standard RI. (7]. At each training step, we generate fewer new rollouts and reuse past rollouts from recent steps. A bounded First-In-First-Ou (FIFO) buffer is maintained to store recent rollouts, from which we retrieve samples to complete each training batch. Although this makes GRPO slightly off-policy, our modified GRPO loss ensures stability, and thus RR effectively reduces per-step training time without degrading model performance.
introduce simple rollout
Our key contributions are summarized as follows:
We propose a novel attention- adaptive difficulty prediction framework that efficiently esti- mates how likely question will be answered incorrectly by the current policy, without requiring full rollouts for all questions.
Guided by this difficulty prediction framework, we introduce an adaptive difficulty-targetedd online data selection mechanism (DOTS) for RL fine-tuning, supported by theoretical justifications. DOTS prioritizes questions of moderate difficulty relative the current policy, accelerating convergence.
We develop rollout replay (RR) mechanism that reuses recently generated rollouts. With modified GRPO training loss, RR remains stable and effectively reduces per-step rollout cost.
Extensive experiments on six L.L.-dataset combinations show that our method reduces RI. fine- tuning time by 23% to 62% while achieving the same performance the original GRPO algorithm.

---

2
Related Work
Online Data Selection Data selection seeks accelerate training by focusing computation on the
most informative examples [2]. A key limitation of static data selection methods is their assumption that the importance of samples remains fixed throughout training. Online methods instead periodically
reselect data during training to reflect the model's evolving state [46 52 29 [6 20). Such adaptability
particularly important in RL, where non-stationary policy updates and environment dynamics
necessitate continuous re-evaluation of data utility 34 47 36].
Experience Replay On-policy algorithms such Proximal Policy Optimization (PPO) 41 and
Group Relative Policy Optimization (GRPO) [42 have become standard choices for online RL fine- tuning in LLM reasoning tasks (10). However, their reliance on freshly collected rollouts for each policy update leads to substantial data inefficiency and computational overhead [27] Experience replay mitigates this by maintaining fixed-size buffer of recent transitions collected by the policy, Instead of discarding data after a single use, the buffer enables multiple passes over past rollouts,
thereby improving sample efficiency and stabilizing training 54 39
3 Problem Setup
GRPO We focus on the GRPO algorithm 42 with verifiable rewards. For each question q. group of G' individual responses {= are sampled from the old policy Tola- The advantage of the i-th response calculated by normalizing the group-level rewards {r. {ri) where r; E (0,1):
G
Å, = r; mcan (ri))=))
Compared the original formulation proposed by 42 we remove the standard deviation normaliza-
tion, as it has been shown to introduce bias into the optimization process 28 Based on this, the GRPO objective can be formulated as:
JGRPO( = E By tor3 (ot (:19)
G
lodl
min (O),())Ai, clip(ra.(0),11 - c +)).
3KL (TT Tree)
The first term represents clipped policy update, where the ratio term (0) s (o (10, represents the probability ratio between the current and old policies. A KL penalty (RI || Tref) is applied with respect to a fixed reference policy Tref. weighted by a scalar coefficient 3.
entorelee clg,on
Online Data Selection Let D = { denote the full dataset of N questions. In standard GRPO, each policy update uses batch of questions uniformly sampled from D. However, all questions contribute equally to learning progress. In particular, questions that are either too easy or too hard relative to the current policy's capability may yield weak gradient signals, slowing convergence.
To address this, we consider an online data selection setting [52] At each step t, a batch B, c D of fixed size B is selected based on the current policy ne. Unlike static data pruning, this selection is repeated throughout training and adapts to the evolving policy. While more frequent selection allows better adaptation, it also increases computational overhead. In practice, when policy updates are relatively stable, it is often more efficient to perform data selection every (a (e.g., 2,4,8...) steps, selecting sequence of f batches be used in the subsequent updates 29 44).
4 Method
Our proposed method is two-fold: (1) Difficulty-targeted Online Data Selection, which reduces the number of training steps needed to achieve the same performance as the original GRPO algorithm by prioritizing questions of moderate difficulty, and (2) Rollout Replay, which reduces the per-step computational cost by reusing recent rollouts. Full pseudocode is provided in Algorithm [
We propose using adaptive difficulty to guide online data selection. The adaptive difficulty of question is defined with respect to the current policy and reflects how challenging the question is for
3

---

Attention Score
Ground Truth
Reference Question
Diffiullty
Solve (aresin(x) aresin(2x) 3]
Q:41
X
Predicted Difficulty
Unlabeled Question
What the volue of (2x+ when x 3?
0.C1
X
D
Solve [rrssin)si(ss)) *]
0.927
hyperbolo x 6e dy" -By 27.
Find distance between the the
0.04
X
0:375
Mf and ((2)))2)))) for find (=644)
0.08
X
0.875
Figure 2: Illustration of our attention-based adaptive difficulty prediction framework. For each unlabeled question, we compute its embedding and attend to reference questions to obtain similarity scores. The predicted difficulty of the unlabeled question is obtained by computing an attention-weiiee weighted average, where similarities to reference questions serve as attention scores over their associated difficulties. functions. The model assigns high attention reference question that tests a closely related concept and has difficulty of 1.0. As a result, the predicted difficulty also close to 1.0. All difficulty values shown correspond to adaptive difficulty scores computed at the same step.
this example, the unlabeled question involves inverse trigonometric
the policy at the current stage of training. Formally, at step f, for each question g, we sample group of G responses {o () ) from the current policy and obtain their corresponding rewards {r² R==1 The adaptive difficulty at step / then computed as:
G
-1- (
d q
This value represents the average failure under the current policy, with higher values indicating greater difficulty. Unlike static difficulty measures, adaptive difficulty evolves with the policy and provides a dynamic signal for selecting informative training samples.
Challenge: How to estimate adaptive difficulty efficiently? key challenge in using adaptive difficulty is that computing it requires executing multiple rollouts, which is one of the most expensive components in LLM RL fine-tuning[] This raises the question: can we estimate adaptive difficulty efficiently without generating rollouts for all questions? To address this, propose a lightweight attention-base based adaptive difficulty prediction framework that generates rollouts for only small reference subset of questions. The adaptive difficulty of the remaining questions is then estimated by comparing them to reference questions with known difficulty values using similarity-basee attention, thereby avoiding full rollouts. See Fig. 2 for illustration.
4.1 Attention-based Adaptive Difficulty Prediction Framework
At each step t, given the full training dataset D, we first sample a small subset of K questions (e.g.. 128 or 256) uniformly at random to form the reference set Diet- For each question in the reference set.
we execute rollouts and compute its adaptive difficulty at step t, denoted by {d! 1
J using Eq. 2
For the remaining N - K questions, we aim to estimate their adaptive difficulty without performing rollouts. To this end, employ a lightweight embedding model Eg to encode questions and capture similarity. We first compute the embeddings {2, Eg (g:)) for all reference questions. Denote h the embedding dimension. Then, for each unlabeled question 9. we compute its embedding 2q Eg(g) and use similarity-eeighted averaging to estimate its adaptive difficulty:
exp(= 2/ 2/VhT)
K
(0t)
0 Σ 1= exp 27 2//hh)
Calibration To improve the prediction performance, we apply Platt scaling 335 that utilizes the information of mean and standard deviation of the reference set difficulties. Specifically, let
For instance, generating rollouts for batch of 512 samples with maximum sequence length 3072 takes (09.83 secouds on 8 L40S GPUs, nearly half of the total step time.

---

1 = = d.6 and o / R K ( (d 22) denote the mean and standard deviation of the reference difficulties step t. These two statistics passed through a lightweight MLP to produce scale and bias parameters (w((), b(O)) = MLP((u!0) o(1)). We then apply calibrated transformation to the predicted difficulty:
1 g,cal
(witt) (log d
log(1 -)/)) + b(1).
where (-) denotes the sigmoid function. The MLP is optimized using binary cross-entropy loss. Full training details can be found in and Appendix
C.I
4.2
Adaptive Difficulty- Online Data Selection
At each training step, with the adaptive difficulty prediction framework, we now efficiently obtain the adaptive difficulty for all questions the training set. Inspired by prior work on goal curriculum in RL 8 48)
we prioritize questions whose predicted difficulty is closest to 0.5.
This selection strategy selects questions that neither too easy too hard for the current policy, as these are intuitively the most informative for learning. Moreover, GRPO, when all sampled rewards for question either 0 or the moup-normalizer advantage becomes identically zero, resulting in no gradient signal. By focusing on questions with predicted difficulty near 0.5, we avoid such degenerate cases and ensure each update contributes meaningfully policy gradients, thereby accelerating optimization convergence. We formalize this intuition in Theorem which shows that the expected gradient magnitude is maximized when the reward success rate 0.5 (i.e., the adaptive difficulty is also 0.5). A complete proof is provided in Appendix B
Theorem (Maximal Gradient Signal at 50% Success Rate). Consider a single question q, where G responses {or)e are sampled independently from the current policy Te(- g). Each response re- ceives a binary reward ri E {0, 1}, sampled i..dd from a Bernoulli(p) distribution, where p represents the reward success rate. Define the group-relaive advantage Ås as in Eg. clipped policy gradient estimator for this question without KL penalty = Σ A, log (o q). Under mild assumptions on the reward and the likelihood gradients Vo log To (o, 9) (detailed in Appendix By the expected squared norm of the gradient satisfies:
1 We consider the un-
Elggg & p(1 - p) (1 11-)
and is maximized when p = 0.5.
Discussion:
Our dynamic selection mechanism implicitly promotes diversity. As questions near the target difficulty are repeatedly selected and trained on, their predicted difficulty gradually deviates from 0.5. They are then less likely to be sampled again, allowing other explored questions to enter the selection pool. This dynamic prevents overfitting to small subset of questions and encourages broader coverage over time.
4.3 Rollout Replay
To further improve data efficiency, we aim to reduce the time cost of each training step. Since rollout generation one of the most expensive components, inspired by experience replay in traditional RL. Specifically, at each training step, we generate new rollouts for only fraction 8. of the batch, where ô (0, 1). and fill the remaining (1 (1-6) samples using recent rollouts sampled from a FIFO replay buffer Proplay with capacity C.
adopt a rollout replay mechanism,
However, naively reusing past rollouts introduces bias into the policy gradient estimation, as the data is no longer drawn from the current policy. This mismatch can lead to unstable training and performance degradation [3I. Inspired by off off-policy variants of PPO [38, propose modified GRPO loss using importance sampling with respect to the behavior policy Th under which the rollouts stored in the buffer were originally collected:
JeRPO.RR (0) = EyzD, (o:31 ~ Ta (-1g)
G
lod (min (O), clip(f -c +,1+ (Too |l meee))
G
0
5

---

Algorithm GRPO with DOTS and RR
Require:
Initial policy model Ta, reward model Tg. training dataset D, target difficulty a, batch size
B, total steps T. reference set size K, sampling temperature т, adaptive difficulty prediction framework DIFFPRED ($4.1) 1). fresh rollout fraction E (0, 1], buffer capacity C
Initialize replay buffer R
2: Set ae f- m
3: for L...,, do
// Adaptive difficulty prediction
4: Sample reference set Dret= DD uniformly at random, where (Deell K
5: for each q E Diet do
6:
Generate G outputs {o 1i=1 Te.. ( (q)
7: Compute rewards r = Typ (o]) for i [G] and difficulty score dp 2 (1-
8: end for
9: Predict adaptive difficulty dy = DDFFPRED(Pret. {dg E Deet,, g') for all q' E D Diet
Sample rollout batch Brotout of size 5B from D according to:
exp (-Idg -ldg- -al/T)
P(g)
Ed'ED exp
(-ldg- al/T)
11:
for each q E Byollou do
Generate G outputs {o]) Tesal Compute rewards r = r (o.) for i E [G] and group average reward Fg = G Ee
13:
Obtain advantages Ay and policy probabilities Te.. (o q) for i [G] end for
15:
// Rollout Replay and update
16:
Sample (1 - 5) B samples from buffer R to complete batch B if R > (1 B) Update policy Tg using modified GRPO objective JaRPo.RR on batch B // Store informative rollouts in buffer
18:
Add (g. { A7 To (o) ()) )&m) to R for Baollot Fg $ (0.
19:
Remove the oldest samples from R until [R| < C
Set T8 + TT
21:
end for
where (0) (o. (.: By appropriately controlling the buffer size C, we empirically demonstrate that rollout replay improves sample efficiency while maintaining training stability. For each newly generated rollout, if the group average reward is neither 0 nor (i.e., the sample yields non-zero gradient signal), we store the question, its sampled rollouts, computed advantages, and policy probabilities into the buffer. When the buffer is full, the oldest samples are discarded.
sl.00119.00.ca)
5 Experiments
5.1
Experimental Setup
LLMs and RL training datasets We perform GRPO training on three model scales: Qwen2.5- Math-1.5B, Qwen2.5-3B, and Qwen2.5-Math-78 [4)] We adopt four open-source datasets for training: MATH [33. DeepScaleR-400 [30], Open-Reasonee Reasomer-Zero-577 (ORZ) [15] and DeepMath- 103K (12). For MATH, include all level 3-5 questions. For the other three datasets, sample 8K to 10K subsets to construct the training pools. These datasets span diverse mathematical domains and difficulty levels. In total, experiment with six LLM- LLM-training dataset combinations to assess the effectiveness of our framework.
practice, we observe that off-the-shelf pretrained embedding models struggle to capture fine-grained similarity between math questions. To address this, we freeze the Qwen2.5-Math- 3-layer MLP adapter with calibration head, using binary cross-entropy loss. We fix the reference
Implementation details for adaptive difficulty prediction framework
backbone [49] and train

---

Owen2.55Mathv3 MATH
Qwen225-Math-1500 DeepScaleR
5wen25-Mach-1 ORZ
53
54
54
52
52
52
51
Aug Accurscy
50
50
Ang Accuracy
50
49
26 25% Faster
23, 23.30%
85% Paster
48
Faster
47
45
44
2 Training Time (Hours)
0
Training Time (Hours))
Training Time (Hours)
Qwen2.55338 DeepScaleR
Qwen2.5-38 DeopMath
Qwen2.5-Math. DeepScaleR
51
60
50
49
Aug Accuracy (88)
48
58
Aug Accuracy
Accuracy
56
48
47
Paster
47
Faster
54
18.03%%
46
46
45
45
44
0
2
2
8
10
Training Time (Hours)
Training Time (Hours)
Training Time (Hours)
DOTS+RR (Ours)
Original GRPO
Figure 3: Average accuracy curves of our method and original GRPO under various LLM-dataset combinations. The curves show average performance aggregated over four bench- marks with exponential smoothing for visualization. The error bars represent 95% confidence intervals across independent runs. Although both methods are trained for the same number of steps (60), our curve is shorter in duration because RR reduces the wall-clock time per step. Our method consistently outperforms the original GRPO throughout training and reduces the time required to match the original GRPO's ' final accuracy after 60 training steps by an average of 40.7%.
set size to 256. Additional implementation details are provided in Appendix C.I and ablation results on key components are presented in Appendix E.I
Implementation details for RL training We employ the verl framework 43 to perform GRPO training. We use batch size of 512 and mini-batch size of 64 in verl's configuration, resulting in 8 gradient steps per training step. Across all experiments, we train for 60 training steps, yielding total of 480 gradient steps. For each prompt, we generate 8 rollouts. The maximum rollout length is set to 3072 tokens for the Owen2.5-Matt series models (due to max position embedding limits) and 4096 tokens for Qwen2.5-3B. For reward computation, we use a simple rule-based function based solely on answer correctness, without incorporating any format-related signals. The 1.5B and 3B models are trained on L40S GPUs, while the 7B model trained on 8 A100 GPUs. For DOTS, data selection is performed every steps. For RR, we choose the fresh rollout ratio 5 as 0.5 and buffer capacity C' E (256, 512]. All RL training hyperparameters are detailed Appendix D.2
Evaluation We adopt the official Qwen2.5-Math evaluation implementation 49 setting the maximum generation length to 3072 tokens for Qwen2.5-Me series models and 4096 tokens for
Qwen2.5-3B. Following [10 30 28 we evaluate RL model performance on standard mathematical reasoning benchmarks, including GSMSK (3], MATH500 (25), Minerva Math [22] and Olympiad- Bench (II. Accuracy is measured using sampling temperature of 0.6, top-p of 0.95, and the standard prompt template, consistent with [TO] We exclude benchmarks with very few questions, such as AIME 24 (30 questions) and AMC 23 (40 questions), their small size results in high evaluation variance and unreliable performance comparisons on smaller models (14). We report the final performance as the average accuracy across the four benchmarks mitigate mmnchmark-speciffi variance. As the baseline, we use the original GRPO algorithm with uniform batch selection.
5.2
Main Results
The total training costs can be decomposed into two components: the number of steps required to reach target performance and the average wall-clock time per step. Each training step involves
7

---

Table 1: Percentage of training steps saved, per-step time saved, and total training time saved.
Results are averaged over four mathematical reasoning benchmarks and reported relative to the original GRPO baseline. All timing measurements are conducted on the same computational devices.
Model
Dataset
Steps Saved (%)
Time Saved/Step (%)
Total Time Saved (%)
MATH
16.67
11.71
26.25
Qwen2.5-Math-ie
DeepScaleR
43.33
11.69
49.85
ORZ
13.33
11.66
23.30
DeepScaleR
26.67
11.52
35.10
Qwen2.5-3B
DeepMuth
56.67
11.35
61.65
Qwen2.5-Math-7B
DeepScaleR
40.00
13.39
48.03
processing fixed-size batch, consisting primarily of rollout generation and policy update, To ensure a fair comparison, each set of experiments is run on the same type and number of GPU devices.
Our method reaches the same performance as the original GRPO with fewer steps. Tab. reports the number of training steps required by DDTS+RR to match the final performance of the original GRPO at 60 steps. Across all LLM-dataset combinations, our method consistently reaches the same performance with substantially fewer steps, achieving reductions ranging from 13.33% to 56.67%. These results demonstrate that DOTS significantly accelerates convergence by prioritizing informative training samples.
Our method reduces per-step cost. our experiments, rollout generation accounts for approxi- mately 47%, 46%, and 54% of the total per-step time for the 1.5B, 3B, and 7B models, respectively, with the remaining time primarily spent on policy updates By reducing the number of fresh rollouts per step, our RR strategy leads to 11 11%13% reduction in per-step training time, as shown in Tab. [
Our method significantly reduces total training cost. As shown in Fig. 3 DOTS+RR (orange)
consistently outperforms the original GRPO (blue) throughout training, maintaining higher accuracy at almost every step. Across all six settings, DOTS+RR reduces total training time by an average of 40.7%, with the largest improvement observed on Qwen2.5-3B trained on DeepMath (61.65%).
5.3 Effectiveness of Adaptive Difficulty Prediction Framework
To better understand why method acceler- ates training effectively, we examine whether the attention-based prediction framework can accurately estimate adaptive difficulty and con- sistently prioritize informative training signals throughout learning.
Table 2: Average Pearson correlation (p) be- tween predicted and ground-truth adaptive dif- ficulties. Reported as mean + standard deviation over 60 training steps.
Model
Datuset
MATH
0,7843 0.0243
The adaptive difficulty prediction aligns with evolving training dynamics. To assess the fit- of online predictions, we collect ground- truth adaptive difficulty labels from training batches and compute the Pearson correlation between these labels and the predicted difficulty scores. As shown in Tab. 2 our framework con-
Qwen2.5-Math-1555
DeepScaleR
0.7244
0.0318
ORZ.
0.7153 0.0257
Qwen2,5.3B
DeepScaleR
0.7789 +: 0.0191
DeepMarh
0.7029 0.0082
Owen2.5-Math-77
DeepScaleR
0.7076 = 0.0195
sistently achieves strong Pearson correlation (p >> 0.7) across settings, demonstrating its ability to effectively track policy behavior throughout training. Additional qualitative examples are provided in Appendix C.2 to offer further insight into our attention- prediction mechanism.
Our prediction framework effectively filters out uninformative samples. As discussed in $4.2 questions with adaptive difficulty values of 0 or 1 correspond cases where all rollouts receive identical reward. In such cases, the group-normalized advantage becomes zero, yielding no gradient signal. We define effective questions those with adaptive difficulty strietly between O and 1. As shown in Fig. on average across all LLMM-datase combinations, DOTS selects 25.4% more effective
"In practice, for longer generation lengths, such as 8K and 16K, rollout time increases substantially, making it the dominant computational bottleneck. In such settings, our rollout replay mechanism can yield even greater wall-clock savings.
8

---

Qwen2.5-Math-j.5e 1.0--1155--AAT
Swven2S-Math--1.. 1.58-oeegScaalllee oR
Effective Question Raloo
2.0
0.8
0.6
0.4
0.2
Qwen2.5 3-3 3B-DeepMath
QwenZ.5 Math D8+DeepSrallRR
Effectiv Question Rabio
1.0
0.8
0.66
2222
0.4
0.2
10
20
30
40
50
60
10
20
30
40
50
60
10
20
30
40
50
60
Training Steps
Training Steps
Training Staps
DOTS (Ours)
Original GRPO
Figure 4: Ratio of effective questions (i.e., questions with adaptive difficulties strictly between and 1) during training across various LLM-training dataset combinations. Annotated percent- ages indicate the per-step increase in effective question ratio achieved by DOTS compared to original GRPO, averaged across the training process. Our adaptive prediction framework consistently selects more informative samples throughout training.
questions than the original GRPO, demonstrating clear advantage in selecting more informative questions throughout training, thereby accelerating convergence.
Our prediction framework incurs minimal computational overhead and scales efficiently to large datasets. By caching question embeddings and using a lightweight encoder, our prediction framework remains highly efficient- processing 10K samples in just 1.71 seconds at deployment.
5.4 Analysis and Discussion
We further investigate three important questions: (Q1) What are the individual contributions of DOTS and RR to training efficiency? (Q2) How does DOTS compare to online data selection method based on external difficulty labels? (Q3) Do DOTS and RR remain effective in non-mattematical domains?
DOTS accelerates convergence, while RR reduces per-step cost. As shown in Fig. 5(a), training guided by DOTS alone yields steeper learning curve compared to original GRPO. Fig. ((b) shows that incorporating RR further reduces training time by approximately 20% without sacrificing performance. These results show that DOTS and RR improve RL training efficiency in complementary ways.
DOTS outperforms online data selection method based on external difficulty labels. We com- pare DOTS with an online data selection baseline that relies on external difficulty annotations (e.g.. annotated by GPT-4o-mini), where training questions are selected at different stages based on static difficulty labels, gradually shifting from easier harder questions over time.
Specifically, we use the DeepScaleR dataset and label each question with GPT-4o-mini, following the difficulty annotation prompt introduced in 30) Each question is annotated 32 times, and the average score is used as its final difficulty. We then follow a staged curriculum: in the first third of training steps, batches are sampled from the easiest third of the dataset; in the middle third, from the medium- difficulty third; and in the final third, from the hardest third. To ensure a fair comparison of online data selection strategies, we compare this baseline with DOTS (without RR). As shown in Fig. [a), our DOTS method consistently outperforms this baseline on both Qwen2.5-Math 1.5B and Qwen2.5-3B. Moreover, such methods require expensive external labeling and offer limited adaptability, as they typically follow hand-crafted curricula that demand extensive manual design and tuning. In contrast, by leveraging adaptive difficulty, DOTS automatically adjusts to the model's learning progress without relying on external supervision, enabling more scalable and efficient training.
DOTS and RR improve RL data efficiency beyond mathematics. To further examine the generality of our approach beyond the math domain, we apply the full training and evaluation pipeline to the science domain using the curated SCP-25K dataset (26), which mostly contains advanced physies, chemistry, and biology questions. We adopt the Owen2.5-3B model and train new adaptive difficulty

---

Qwen2.5-Maan 1reen25-MMth-11.. Math-1.58-MMA1
Qwen2.5 Math-1-1---eeeeeee- DBDDepeSa
Qwen2.5 Sween..-Math-222 M--Math---------
54
52
Accuragy
50
(a)
48
44
Qwen2.5-Math-1.5BMMe MATH
Qwen2.5-Mtth-155B Delesee
Qwen22.-Meth-1.30 OR
54
52
Acouracy (N)
50
(b)
48
Similar performance. 20% time!
Similar performance. less time!
Similar performance, but -20% less time!
44
20
40
60.0
20
40
600
20
60
Training Stops
Training Steps
Training Steps
Original
Figure 5: Average accuracy curves of (a) DOTS vs. Original GRPO, and (b) DOTS+RR vs. DOTS on the Qwen2.5-Math-1..B model. The curves show average performance aggregated over four benchmarks with exponential smoothing. Note that the x-axis is the number of steps (rather than time). (a) DOTS consistently outperforms the original GRPO and leads to faster convergence. (b) Incorporating RR reduces training time by 20% while preserving the performance of DOTS.
Qwen225-Mathi- 5-Math-1.56 DeepScaleR
Qwen22.5338 DeepScaleR
Qwen2.5-3B SCP
51
50
56
MMLU Accuracy (66)
Aag Acuraoy
50
Aug Accuracy
48
48
47
52
46
50
44
44
0
20
Training Steps
40
60
20
Training Steps
40
Training Time
4
DOTS
External Difficulty based Curriculum Baseline
Original GRPO
DOTS RR (Durs)
Original GRPO
(a) Comparison between DOTS (ours) and external difficulty- based curriculum baseline. The curves show average performance
(b) Results on the science subsets of MMLU using Qwen2.5-3B trained on the SCP-25K dataset. Our method sig- nificantly outperforms original GRPO in this non-math domain.
aggregated over four benchmarks with exponential smoothing for visu- alization. Note that the x-axis is the number of steps (rather than time).
Our method consistently outperforms the baseline.
Figure 6: Comparison of DOTS with external difficulty-based curriculum baseline (Ieft) and generalization to non-math domain (right).
predictor, while keeping all other RL settings unchanged. We evaluate performance on the science subsets of MMLU, including questions in the fields of physics, chemistry, and biology. As reported in Fig. b)., our method continues to significantly improve RL data efficiency in this non-math domain, demonstrating its broader applicability.
6 Conclusion
In this paper, we propose two techniques to improve the data efficiency of LLM RL fine-tuning: Difficulty-targeted Online Data Selection and Rollout Replay. We hope these effective techniques will encourage future work to explore data-centric approaches to improving LLM RL fine-tuning.
10

---

Acknowledgment
We would like to express our heartfelt thanks Rayne Amami for helpful discussions and inputs.
References
[1] Pranjal Aggarwal and Sean Welleck, LI: Controlling how long a reasoning model thinks with reinforcement learning. arXiv preprini arXiv:2503.0400 2025.
[2] Alon Alhalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang. Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, et al. A survey on data selection for language models, 2024. URI https:/aarxiv. org/abs/2402.. 16827, 2022.
[3] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reicchro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:21 10. 2021.
[4] Nicholas E Corrado and Josiah P Hanna. On-policy policy gradient reinforeement learning without on-policy sampling. arXiv preprint (rXiv:2311.08222 1.08290, 2023.
[5] Ganqu Cui, Lifan Yuan, Zefan Wang. Hanbin Wang. Wendi Li. Bingxiang He, Yuchen Fan, Tianyu Yu, Qixin Xu, Weize Chen, et Process reinforcement through implicit rewards. arXiv preprint arXiv:2502.0 2025.
[6] Junwei Deng, Yuzheng Hu, Pingbang Hu, Ting-wei Li, Shixuan Liu, Jiachen T. Wang. Dan Ley, Qirun Dai, Benhao Huang. Jin Huang, Cathy Jiao, Hoang Anh Just, Yijun Pan, Jingyan Shen, Yiwen Tu, Weiyi Wang. Xinhe Wang. Shichang Zhang. Shiyuan Zhang, Ruoxi Jia, Himabindu Lakkaraju, Hao Peng. Weijing Tang. Chenyan Xiong, Jieyu Zhao, Hanghang Tong, Han Zhao, and Jiaqi W. Ma. A survey of data attribution: Methods, applications, and evaluation in the era of generative ai. 2025. doi: (0.2139/ssm.5551005
Available SSRN: attps://esrrt ctm/////c.cc/oc/e///c/a//cc/ abstrratttt555154444
William Fedus, Prajit Ramachandran, Rishabh Agarwal, Yoshua Bengio, Hugo Larochelle, Mark Rowland, and Will Dabney, Revisiting fundamentals of experience replay. In International conference on machine learning, pages 3061-3071, PMLR, 2020.
[8] Carlos Florensa, David Held, Xinyang Geng, and Pieter Abbeel. Automatic goal generation for rein- forcement learning agents. In International conference on machine learning, pages 1515-1528, PMLR, 2018.
[9] Bofci Gao, Feifan Song, Zhe Yang, Zefan Cai, Yibo Miao, Qingxiu Dong, Lei Li, Chenghao Ma. Liang Chen, Runxin Xu, et al. Omni-math: A universal olympiad level mathematic henchmark for large language models. arXiv preprint 10X00:2410.07985, 2024.
[10] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, al. Deepseck-rl: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint 250i 11.500..1949., 12948, 2025.
[11] Chaoqun He. Renjie Luo, Yuzhuo Bai. Shengding Hu, Zhen Thai, Junhao Shen, Jinyi Hu, Xu Han. Yujic Huang, Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad-level bilingual multimodal scientific problems. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages
2024.
[12] Zhiwei He, Tian Liang, Jiahao Xu, Qiuzhi Liu, Xingyu Chen, Yue Wang, Linfeng Song, Dian Yu, Zhenwen Liang, Wenxuan Wang, et al. Deepmath-1033 103k: A large-scale, challenging, decontaminated, and verifiable mathematical dataset for advancing reasoning. arXiv preprint arXiv:2504. 1456, 2025.
[13] Dan Hendryeks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Erie Tang, Dawn Song, and Jacob Steinhardt. Mcasuring mathematical problem solving with the math dataset. Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[14] Andreas Hochlehnert, Hardik Bhatnagar, Vishaal Udandarao. Samuel Albanie, Ameya Prabhu, and Matthias Bethge. A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility. arXiv preprint arXivv2504.00006 2025.
[15] Jingcheng Hu, Yinmin Zhang, Qi Han, Daxin Jiang, Xiangyu Zhang, and Heung- Yeung Shum. Open- reasoner-zero: An open source approach to scaling up reinforcement learning on the base model. arXiv preprint arXiv:2503.24244 2025.

---

[16] Angela H Jiang. Daniel L-K Wong, Giulio Zhou, David G Andersen, Jeffrey Dean, Gregory R Ganger. Gauri Joshi, Michael Kaminksy, Michael Kozuch, Zachary Lipton, et al. Accelerating deep learning by focusing on the biggest losers. arXiv preprint arXiv:1910.0000 1910.00762. 2019.
[17] Niels Justesen, Ruben Rodriguez. Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, and Sebastian Risi. Illuminating generalization in deep reinforcement learning through procedural level generation. arXiv preprint 8806 10729, 2018.
[18] Feiyang Kang, Hoang Anh Just, Yifan Sun, Himanshu Jahagirdar, Yuanzhi Zhang, Rongxing Du, Anit Ku- mar Sahu, and Ruoxi Jia. Get more for less: Principled data selection for warming up fine-tuning in Ilms. arXiv preprint arXiv:2405.02774 2024.
[19] Feiyang Kang, Yifan Sun, Binghing Wen, Si Chen, Dawn Song. Rafid Mahmood, and Ruoxi Jia. Autoscale: Scale-aware data mixing for pre trainini 1lms. arXiv preprint arXiv:2407.2017. 2024.
[20] Angelos Katharopoulos and François Fleuret. Not all samples are created equal: Deep learning with importance sampling. In International conference on machine learning, pages 2525-2534, PMLR, 2018.
[21] Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, al. TQ" " ulu 3: Pushing frontiers in open language model post-training. arXiv preprint arXiv:2411 2024.
[22] Aitor Lewkowyez. Anders Johan Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Venkatesh Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. In Advances in Neural Information Processing Systems, 2022.
[23] Jia Li, Edward Beeching, Lewis Tunstall, Ben Lipkin, Roman Soletskyi, Shengyi Costa Huang, Kashif Rasul, Longhui Yu, Albert Jiang, Ziju Shen, Zihan Qin, Bin Dong, Zhou, Yann Fleureau, Guillaume Lample, and Stanislas Polu. Numinamath, htttps://huge: https/////ggggggceegeeem co AI inaaaa CoT 2024. Report available at hhttps://git //gttub.. com /ww.///neetaniiaaaaaaaaaaaaaaaaa numina/ aimo prrogress-prizo wttbbbblbbb/a/aa/eee httprttnnminaaaattattt .df
[24] Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2502.1 1886, 2025.
Limr: Less is more for rl scaling. arXiv preprint
[25] Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2023.
[26] Mingjie Liu, Shizhe Diao, Ximing Lu, Jian Hu, Xin Dong, Yejin Choi, Jan Kautz, and Yi Dong. Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models. arXiv preprint arXiv:2505.2 24864, 2025.
[27] Qisai Liu, Zhanhong Jiang, Hsin-Jung Yang, Mahsa Khosravi, Joshua Russell Waite, and Soumik Sarkar. HP3o: Hybrid-policy proximal policy optimization with best trajectory, 2025. URL https /openreview www.eee/foooo
YmJJ
[28] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl-zero-like training: A critical perspective, 2025. URL https:/aarsii. org/abs/2503.20 20783.
[29] Tlya Loshchilov and Frank Hutter. Online batch selection for faster training of neural networks. arXiv preprint 15 1.106333,, 2015.
[30] Michael Luo, Sijun Tan, Justin Wong, Xiaoxiang Shi, William Y. Tang, Manan Roongta, Colin Cai, Jeffrey Luo, Tianjun Zhang, Li Erran Li, Raluca Ada Popa, and Ion Stoica. Deepscaler: Surpassing ol -rrrrvieww with 1.5b model by scaling rl, 2025. Notion Blog.
[31] Wenjia Meng. Qian Zheng, Gang Pan, and Yilong Yin. Off-policy proximal policy optimization. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pages 9162-9170, 2023.
[32] Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, et al. Imitate, explore, and self-improve: A reproduction report on slow-thinking reasoning systems. arXiv preprint arXiv: 0244111,.941,, 2024.
[33] Youssef Mroueh. Reinforcement learning with verifiable rewards: Grpo's effective loss, dynamics, and success amplification. arXiv preprint XXiv:2503.06633, 2025.
12

---

[34] Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. Journal Machine Learning Research, 21(181):1-50, 2020.
[35] John Platt al. Probabilistic outputs for support vector machines and comparisons to regularized likelihond methods, Advances large margin classifiers, 10(3):11-7, 1999.
[36] Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre-Y Oudeyer. Teacher algorithms for curriculum lcarning of deep rl in continuously parameterized environments. In Conference on Robot Learning, pages 835- 853, PMLR, 2020.
[37] Rémy Portelas, Cédric Colas, Lilian Weng, Katja Hofmann, and Pierre- Oudeyer. curriculum learning for deep rl: A short survey. arXiv preprint arXiv:2003.04664, 2020.
Automatic
[38] James Queeney, Yannis Paschalidis, and Christos G Cassandras. Generalized proximal policy optimization with sample reuse. Advances in Neural Information Processing Systems, 34:11909 11919, 2021.
[39] David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy I Lillicrap, and Gregory Wayne. Experience replay for continual learning. In H. Wallach, H. Larochelle, A. Beygelzimer, d'Alché-Bue, E. Fox, and R. Garnett, editors, Advances Neural Information Processing Systems. volume 32. Curran As- sociates, Inc., 2019. URL
aarrafaddssafffffoooaaaa///fff.d Paper .pf
[40] Tom Schaul, John Quan, Ioannis Antonoglou, and David Silver. Prioritized experience replay, 2016. URL
https://arxiv org/ aaaaabaab//111 05952
[41] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint 17X7::1777.06477 2017.
[42] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, arXiv preprint arXiv:2402.003300 2024.
[43] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint (xXiv:2409.19255 2024.
[44] Hwanjun Song, Minscok Kim, Sundong Kim, and Jac-Gil Lec. Carpe diem, seize the samples uncertain" at the moment" for adaptive batch selection. In Proceedings of the 29th ACM International Conference on Information Knowledge Management, pages 1385-1394, 2020.
[45] Milan Vojnovic and Se- Young Yun. What is the alignment objective of grpo? arXiv preprint arXiv:2502. 18548, 2025.
[46] Jiachen Tianhao Wang. Tong Wu, Dawn Song. Prateek Mittal, and Ruoxi Jia. Greats: Online selection of high-quality data for Ilm training in every iteration. Advances in Neural Information Procexsing Systems, 37:131197-131 2024.
[47] Rui Wang, Joel Lehman, Jeff Clune, and Kenneth O Stanley. Paired open-ended trailblazer (poct): Endlessly generating increasingly complex and diverse learning environments and their solutions. arXiv preprint arXiv: / 190733 2019.
[48] Xin Wang, Yudong Chen, and Wenwu Zhu. A survey on curriculum learning. IEEE transactions on pattern analysis and machine intelligence, 44(9):4555 4576, 2021.
[49] An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. arXiv preprint 2512 2024.
[50] Edward Yeo, Yuxuan Tong, Morry Niu, Graham Neubig, and Xiang Yue. Demystifying long chain-of- thought reasoning llms. arXiv preprint arXiv:2502.03377. 2025.
[51] Qiying Yu, Zheng Zhang, Ruofci Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source Ilm reinforcement learning system at scale, arXiv preprint 25xvv:2033.1 14476, 2025.
[52] Zichun Yu, Spandan Das, and Chenyan Xiong. Mates: Model-aware data selection for efficient pretraining with data influence models. Advances in Neural Information Processing Systems, 30:108 118713511-77999,, 2024.
13

---

[53] Weihao Zeng. Yuzhen Huang, Qian Liu, Wei Liu, Keqing He. Zejun Ma, and Junxian He. Simplerl-zoo: Investigating and laming reinforcement learning for open base models in the wild. arXiv preprint wwXi:2253.18882 2025.
[54] Shangtong Zhang and Richard Sutton. A deeper look experience replay. arXiv preprint mxXi:1712.01275.20 2017.
14

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly include the claims made in the paper.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of the work in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifiennt specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Assumptions and proofs are provided in Appendix C.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
15

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification:
Reproducibility information is provided in Appendix A.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material?
the data and code, with sufficient instructions
Answer:
[Yes]
Justification:
We release our code repository in Appendix A and use only publicly-available datasets.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines http::://niss.ccpppbbppp
quides/CodeSubmn seiobboicc for more details.
While we encourage the release of code and data, we understand that this might not be possihle,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nniss//.///////////e:::eeee cc/public/ qu..des//ooeeSubbllsssooool..... for more details.
16

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 5.1 and Appendix detail our experimental sctup.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification: The main results in Section 5.2 are accompanied by error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification: Section 5.1 and Section 5.2 report the GPU type and execution time.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
17

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics http:///eeerippeppppiiieeeeeeeee wwtpc//wwwwp/tciceeeeeeeeeeeeeee cet.eliieeeehhcccceclllllleeeee
Answer: [Yes]
Justification:
We reviewed the Code and found no conflicts.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: There is no socictal impact of the work performed.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g.. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should deseribe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
18

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: All third party models and datasets are properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new datasets or models.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
19

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
recognize that procedures this may vary significantly between institutions and Iocations, and we expect authors adhere to the NeurIPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this rescarch? Note that if the LLM is used only for writing, editing, or formatting purposes and does impact the core methodology, seientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are used only writing and formatting purposes.
Guidelines:
answer NA means that core method development this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https http::::e/eeiipp...co//feeerrree http::/s......ooeeeeeee///////// for what should or should not be described.
20

---

A Discussions
A.1 Limitations and Future Work
Our adaptive difficulty prediction framework currently relies randomly sampling a reference of K questions at each selection step. While effective, the quality of the reference set can influence prediction performance. In principle, one could improve prediction performance by selecting a more diverse reference set that better covers the training set. Building on this idea, natural extension is to fix shared set of K reference questions (with sufficient coverage) across training, re-evaluating their adaptive difficulty each selection step.
Moreover, while demonstrate the effectiveness of experience replay in the GRPO setting, our current strategy relatively straightforward: we randomly replay rollouts associated with questions whose average reward across all rollouts is neither O nor 1. A promising direction for further improving efficiency is to incorporate more principled replay strategies, such those inspired prioritized experience replay [40] 54)
Another potential extension of our method lies in the construction of input embeddings for difficulty prediction. Specifically, instead of relying solely on the question text, one could incorporate reference solutions to enrich the representation. Preliminary experiments suggest that including reference solutions slightly improve the accuracy adaptive difficulty prediction. However, this approach may have limited applicability practice, as reference solutions are not available for all datasets (c.g., DeepScaler and ORZ).
Finally, we note that generating rollouts for the reference set can introduce nontrivial computational overhead, especially when the reference size is large. To mitigate this, we reuse rollouts from reference questions whose predicted difficulty is near 0.5, effectively incorporating them into training. This strategy reduces rollout generation cost by 4-12% per step while maintaining final performance.
A.2 Extended Related Work
RL. fine-tuning for LLMs (with verifiable rewards) has recently attracted significant attention, driven part by success DeepSeek-RI [0]. Compared to the original GRPO algorithm [42]. recent work has proposed several algorithmiic improvements: DAPO [51] introduces techniques such as clip-higher dynamic sampling. token-level policy gradient loss. and overlong reward shaping, while Dr. GRPO (28) removes the length and standard deviation normalization terms to improve stability. Beyond these algorithmic enhancements, [5 provide thcoretical insights into GRPO, while [53 50 conduct large-scale empirical studies across models,
identifying key design choices that enable effective RL fine-tuning.
In contrast, relatively little attention has been paid to data-centric approaches, despite their demonstrated potential in other areas of LLM training [8] 19 []. LIMR (24) explores a static data selection strategy for RL fine-tuning by prioritizing samples based on their alignment with the policy's 's learning trajectory. However, it requires a full training run over the entire dataset beforehand, limiting its practicality. Our online data selection method DOTS more efficient and applicable in realistic settings. addition. prior work not explored the use of rollout replay in LLM RL fine- theettunng. which we show can further reduce training costs.
B Proofs
Proof of Theorem 1. We restate Theorem and provide a complete proof below.
Theorem 1 (Maximal Gradient Signal at 50% Success Rate). Consider single question q, where G responses {or) are sampled independently from the current policy To(- q). Each response receives binary reward rs E {0, 1}, sampled ii.dd from a Bernoullíp) distribution, where p represents the reward success rate. Define the group-relative advantage A as in Eq. We consider the unclipped policy gradient estimator for this question without KL penalty:
G g g= Σ A.Volog (o( g))
Under mild assumptions on the reward and the likelihood gradients Vo log (o(s q)) the expected squared norn of the grudient satisfies:
Elllg| x p(1 p) -1-1111)),
and is maximized when p = 0.5.
Proof. Let r. f0. } be the binary reward for response o,, sampled i.i.d. from Bernoullipp) distribution. Define the gllup-relative advantage as:
A = r,
21

---

We aim to analyze the expected squared norm of the gradient estimator
V. lo I g)-
Assume that the gradients V log (o, q) have bounded second moment:
E Vo log (o ()]l C < co.
We compute the full second moment of the gradient estimator, where the expectation is taken with respect to me:
[[|b||
ArA,]
lo 9) Vo log (o a)
G
Cov (AiAj, Va log nolo: 4)) T Vo xo 19))
We introduce a weak-dependence assumption that the correction term 12 is negligible compared to the leading term T
T2
<
Tj
Therefore, it suffices focus our analysis on the leading term Tı. By assumption, the log-liko likelihood gradients are zero-mean, independent, identically distributed across
V, i= j i * j.
V (o 2) Vo log 1
So.
B[|gg| V. Σ R((
We compute [[[A]]. Let 1 F Tjs then:
E E.A.] B[(r;: +) Var(r: - -F) Var(ri) Var(F) Cov(ri.
Since Bernoullipp) and rj i.idd.
Var(r,) =p(1- - PP) Var(F) p(1 - p)
Cov F) p(1 - p)
G
G
Substitute in:
p(1 p)
2. p(1 - p)
p(1 p) (1 (1-
B[A?] p -p)
G
G
Therefore,
(1 (1-1
E =
p(1 p)
which is maximized when p 0.5.
"To support this assumption empirically, compute the ratio on LLM-dataset combinations. Specifically, we randomly sample 512 questions each dataset, generate rollouts per question, and evaluate the ratio. The results (mean + standard deviation) are: Qwen2.5- -athh- 1.5B + MATH: 0.081 0.0065, and Qwen2.5-3B DeepMath: 0.081 = 0.0051. These consistently low ratios empirically validate the weak- dependence assumption.
22

---

Remark: Extension to Multi-component Rewards. Theorem focuses on binary rewards for simplicity. following standard practice in recent RLVR literature. Its core derivation computing the second central moment of group-normalized rewards [ extends naturally to more complex reward formulations.
derivation-commpuing
For example, consider a reward composed of two independent components: a correctness term G ~ Ber((a)) and a format term fi (8m( where the total reward is T; G. + ft. Then,
F[ (a(1 a) + 3(1 B)) 7
which is maximized when both o = 0.5 and / = 0.5. This demonstrates that our insight applies naturally to multi-component rewards and highlights the generality of the result.
Details of Adaptive Difficulty Prediction Framework
C.1
Design and Implementation Details
The core of our adaptive difficulty prediction framework lies in obtaining proper embeddings to enable attention- based weighted prediction, as descrihed in Section To achieve this efficienlly, freeze the Qwen2.5-Math, 1.58-Instruct model as the backbone and augment it with a lightweight adapter and a calibration head.
The adapter a -aativvaee MI.P with three hidden layers, each containing 896 units and a dropout rate of 0.1. A LayerNorm is applied to the projection output to stabilize training. The calibration head a two-layer MLP that takes the mean and standard deviation of reference set difficulties as input. The first output passes through a Softplus activation to yield the scale parameter (e()), while second transformed by a Tanh activation produce bounded bias term 6())
We collect training data from of LLMs that are disjoint from our policy models. These include Qwen2.5- Instruct and Qwen2.5 1...5-MMat--nnttrue series (49), Eurus-2-7BB 2-7B-PRIME [3). Mathstral-7B- 1 DeepSeek R1- Distill- Siilll-wen-ILa 1.5B [10]. DeepSSaleR-LL5B Preview [30. and Qwen2.5-7 @wwen..57B--Sing Si-SillleRL-Zooo [53] For each model, we sample query questions and reference questions from math datasets and compute their adaptive difficulty as supervision labels. Specifically, each training instance consists of a query question g. reference with known difficulty scores {q., K and ground-truth difficulty label dy. Repeating this procedure across models yields the training dataset D prod-train-
We train the adapter and calibration head using standard binary cross-entropy loss:
LaCK
du log dg.cal + (1 - d,) log(1 dy,cal]
[Dpred
where da.eal is the calibrated predicted difficulty for the query question.
C.2
Qualitative Examples
Tab. presents a qualitative example from the DeepScaler dataset using Qwen2.5-3B the policy model, showing one unlabeled question along with the reference questions receiving the highest and lowest attention scores. The example demonstrates that our difficulty prediction framework assigns higher attention to reference questions that share key mathematical topics and structures (e.g., rhombus, incircle), while fown-weightie unrelated questions.
D Implementation Details
D.1 Training Datasets and Models
Our experiments involve three model sizes: Qwen2.5 1wen2.5-Mathe 1.5B, Qwen2.5-3B, and S0wen..5Matth 49 We adopt four open-soure source mathematical reasoning datasets for RL fine- tine-tuning:
MATH [3
This dataset contains 12,500 competition-Ievel problems from sources such as AMC and
AIME, spanning seven mathematical subjects and five difficulty levels. Following 24 53) we merge the train and test splits and retain only Level 3-5 questions. These are guaranteed to have overlap with the MATH500 benchmark to prevent data contamination.
DDepScaleR-400k [30: A collection of approximately 40,000 curated mathematical problems from AMC (0r3, AIME 111944-2020) Omni-MATH 99, and Still 32 Deduplication is performed
23

---

Table 3: Qualitative example illustrating the similarity-basee attention mechanism in adaptive difficulty prediction. The table shows one unlabeled question along with its top- and bottom-ranked reference questions by attention score. High-attention references (red) typically share similar concepts and difficulty with the target question (e.g. rhombus and incircle geometry), while low-attention references (blue) diverge in topic and substantially easier.
Data Source: DeepScaleR
Unlabeled Question
[ground truth adaptive difficulty = 1.000, predicted difficulty = 0.907]
the rhombus ABCD, point Q divides side BC in the ratio 3 starting from vertex B, and point is midpoint of side AB, is known that the median CF of triangle CEQ equal to 2v/2, and EQ v2. Find the radius of the cirele inscribed in rhombus ABCD.
#
Attention Score
Adaptive Difficulty
Reference Question
0.487
1.000
Rhombus ABCD has LBAD < 90" There a point P on the incirele the rhombus such that the distances from P to the lines DA, AB, and BC are 9, 5, and 16, respectively. Find the perimeter of ABCD.
2
0.093
1.000
Circle wh with radius is inscribed in strip S having border lines and b. Circle w2 within S with radius is tangent externally to cirele wa and is also tangent to line a. Circle wa within S is tangent externally to both circles w1 and w2, and is also tangent to line b, Compute the radius of circle was
255
0.000
0.125
package milk with a volume of liter cost 60 rubles. Recently, the purpose of economy, the manufacturer reduced the package volume to 0.9 liters and increased its price to 81 rubles. By what percentage did the manufacturer's revenue increase?
256
0.000
0.125
Given lan (a - 2, find the value of sin (2ax f)-
using embedding based retrieval, and ungradable problems are filtered to ensure high-quality reward signals. We randomly sample 10,240 problems for training.
Open-Reasoner-Zero-Se (ORZ) [15): This dataset includes 57,000 high qualituaity reasoning problems sourced from AIME (up to 2023), AMC, MATH, Numina-MATH (23). and Tulu3 MATH (21. Extensive cleaning via rule-based and LLM-based filters ensures evaluability and difficulty balance. We sample 8, 192 problems for training.
Deep.Math-103K [12: A large-scale dataset focused on high-difficulty mathematical problems. constructed with rigorous data decontamination procedures support reliable benchmark evaluation. We sample 8,192 problems for training.
D.2 RL Fine-tuning Details
Tab. summarizes the hyperparameters used in our GRPO training. We adopt the same configuration across all experiments. Following 1515 remove the KL regularization terms. For reward computation, simple rule-based function based solely on answer correctness, without incorporating any format-related signals. Specifically, reward of is assigned for exact matches with the reference answer, and otherwise. Answer matching is implemented using the Math-Verify library K We adopt a standard chain-of-thought (CoT) prompt template, provided in Tab. 5
D.3 Implementation Details of DOTS and RR
We present the detailed hyperparameter settings of Algorithm in Tab, 6 For DOTS, data selection is performed every two steps during RL fine-tuning.

---

Table 4: Detailed RL fine-tuning recipes.
Optimizer
A
Total Batch Size
512
Mini Batch Size
64
Learning Rate
le-6
LR Schedule
Constant
Weight Decay
0
Warm-up Ratio
Number of Training Steps
60
Number of Gradient Steps
480
Max Prompt Length
1024
Max Rollout Length
3072/4096
Number of Rollouts Per Prompt
Rollout Sampling Temperature
0.6
Rollout Sampling Top-p
0.95
GPU Hardware
8x NVIDIA L40S/8x NVIDIA A100
Table 5: Prompt template used for RL fine-tuning and evaluation. The placeholder <question>> is replaced with the actual mathematical question during fine-tuning and evaluation. Special tokens "<lim starti>" and "<lim endl>" are omitted for clarity.
system
Let's think step by step and output the final answer within \boxed(). user <question>
assistant
D.4 Evaluation Details
Consistent RL fine-tuning, we use a sampling temperature 0.6, top-p 0.95, the same prompt template. We evaluate model performance on four commonly-used mathematical reasoning benchmarks and report the average accuracy to mitigate benchmark-specilff variance.
GSM8K 3
A set of 1,319 grade school math word problems from the GSM8K dataset, requiring
multi-step arithmetic reasoning.
MATH500 25 A widely used subset of the MATH test split [3] These problems excluded
from our MATH training data.
Minerva Math 22 A set of 272 undergraduste-level science and math questions from MIT Open-
CourseWare.
OlympiadBench A benchmark of 675 problems from international math olympiads and physics
contests.
We exclude benchmarks with very few questions, such as AIME (30 questions) and AMC (40 questions), as their limited size leads to high evaluation variance and unreliable performance comparisons for smaller models [14]. We further justify this exclusion by evaluating the original GRPO on AIME 24 across various LLM-datase, combinations. Specifically, each of the 30 AIME 24 questions is evaluated 8 times, and the average accuracy (avg@8) 8) is computed regular intervals during training. As shown Table the accuracy fluctuates considerably across training steps without a clear upward trend. This high variance across steps underscores the difficulty of obtaining reliable evaluation signals on such small-scale datasets, especially for smaller models with limited reasoning capacity.
E
Additional Experimental Results
E.1
Ablation Study the Adaptive Difficulty Prediction Framework
Off-the-shelf embeddings fail capture difficulty structure. We evaluate baseline that directly uses frozen embeddings from the Qwen2.5-Mait 1---Mahh-..555
model without any further training or calibration. In contrast, our framework incorporates trained adapter layers and a calibration head. As shown in Tab. g our framework consistently achieves significantly higher Pearson correlation with the ground-trun cound-truth adaptive difficulty
25

---

Table Hyperparameters of DOTS and RR.
Target Difficulty a
0.5
Reference Set Size K
256
Data Sampling Temperature Fresh Rollout Fraction 5
1c-3
0.5
Bufler Capacity C
256/512
Table Accuracy of original GRPO on AIME 24 across training steps. Each of the 30 questions is evaluated 8 times, and avg@8 accuracy is reported every 10 training steps. The results show high variance without clear trends, which limits evaluation reliability especially for smaller models.
Steps
Qwen2.5-Math-LL5B DeepScaler Qwen2.5-3B + DeepScaler
Qwen2.5-Math-7B: DeepScaler
10
10.42
7.50
20.42
20
15.83
6.25
24.17
30
15.83
8.75
24.17
40
11.67
9.17
23.33
50
16.67
7.50
25.42
60
13.75
5.42
20.00
across all settings. The poor performance of the off-the-shelf baseline highlights the necessity of further adapter layers and calibration for accurately predicting question difficulty.
Table 8: Ablation study on training with adapter and calibration. Comparison of average Pearson correlation (p) between predicted scores and ground-truth adaptive difficulties, reported as mean + standard deviation over 60 training steps. Results show that training with adapter layers and calibration significantly improves prediction performance.
Off-th-shelf
Our Method
Model
Dataset
Embedding
(With Adapter Layers Calibration)
MATH
0.2682 0.0207
0.7843 0.0243
Owen2.5-Math-1.5.
DeepScaleR
0.2064
0.0518
0.7244 + 0.0318
ORZ
0.1598
0.0266
0.7153 0.0257
Qwen2.5-3B
DeepScaleR
0.2688 0.0369
0.7789 0.0191
DeepMath
0.0671 1 0.0168
0.7029 1: 0.0082
Qwen2.5
DeepScaleR
0.1983 0.0254
0.7076 0.0195
DOTS is robust to the size of reference set.
We further investigate the impact of the reference set size K in RL fine-tuning, Fig. F compares the performance of the original GRPO and DOTS under reference set sizes of 128 and 256, using Swen2.5-Math-11.. reference size of 128 yields performance comparable that 256. This indicates that DOTS robust to smaller reference sets, enabling more efficient rollout collection without sacrificing RL. fine-tuning quality.
and Qwen2.5-3B on the DeepScaleR dataset. The results show that
E.2 Additional Experiment with Extended Training Horizon
To further verify the stability of our findings, extend training 100 training steps (600 gradient steps) under two settings: Siven..5-Math-1l 1.5B + DeepScaleR and Qwen2.5-3B + DeepMath. Notably, our method continues to outperform the original GRPO baseline.
E.3
Additional Results under Different Evaluation Views
the main text, Fig. B presents performance over wall-clock time, while Fig. usess training steps. For completeness, we provide alternate versions: Fig. shows the step-based view corresponding Fig. and Fig. 10 shows the time-based view for Fig.
Across both views-training steps and wall-clock Lime-DOTS-RR and DOTS consistently demonstrate strong performance, confirming the robustness of our improvements regardless of presentation format. Interestingly. as shown in Fig. [0] although DOTS (without RR) incurs small overhead from reference rollouts and difficulty prediction, requires substantially fewer training steps reach the same final accuracy as the original GRPO. As a result, the overall training time is often reduced despite the per-step overhead.
26

---

Qwen2.5-Maiie D.Se-Dee
51
50
52
Aug (60)
Aug Accurssy (%)
50
4.8
47
48
46
45
44
Training Steps
TrapningSteps
DOTS (Ref Size=256)
DOTS (Ref Size=126)
Original GRPO
Figure Average accuracy curves of DOTS (Ref Size 256), DOTS (Ref Size 128), and Original GRPO on Qwen2.5-Math-
and Qwen2.5-3B. The curves show average performance aggregated over four benchmarks with exponential smoothing for visualization. Note that the x-axis the number of steps (rather than time). The results show that reference set size of 128 achieves performance comparable that of 256, indicating the robustness of method to smaller reference sets.
Qwen2.5-Math-1.58 DeepScaleR
Qwen2.5-38 DeepMath
54
52
(96)
fug Accuraty (96)
48
50
48
46
46
44
Training Sleps
Training Sleps
DOTS RR (Ours)
Original GRPO
Figure Extended training to 100 training steps for two settings: @wwen..5MathhLL.3B DeepScaleR and Qwen2.5-3B + DeepMath. Our method consistently outperforms the original GRPO baseline.
27

---

Qwen2.5
Qwen2.5
53
54
acturas /
51
Accurssy
Aug Actur (67
50
49
16.67% Fewer Steps
13.33% Fewer Steps
Feweer Steps
48
47
45
Training Stops
Training Stops
Training Stops
Qwen2.5 36 DeepScaleR
Qwen2.5 2.3.2.338B. 38 DespMath
Qwen2.5 Math 7B DeepSraleR
51
49
60
Avg (6g)
49
Accurocy 18
ArgAccurrcyy (60
58
56
48
26.67% Steps
56.57% Feoer Staps
54
Fewer Staps
46
46
52
45
50
44
20
40
60
20
40
60
20
40
60
Steps
Training
Training Steps
(Ours)
Original GRPO
Figure 9: Performance curves with training steps as x-axis. Replot of Fig. 3 Busing training steps.
Qwen2.5-Meth-1.5B MATH
Qwen2.5-Meth-1.. DeepScaleR
Qwen2.5-Mathn Swenn.5-Math-1155- ORZ
Nvg Accurasy 196)
52
Aug Accuracy (96)
52
52
50
50
48
48
Nug
46
46
44
2
Traiiing Time (Hours)
Training Time (Hours)
Training Time (Hours)
Owen2.55Math-11.. MATH
DeepScaleR
QwenZ.5 Swen225-Math-11.5 Math-1158- + ORZ
54
52
log Accurecy (98)
Avg Accuraccy (R)
52
50
50
50
48
48
Aug
46
46
Training Time (Hours)
Training Time (Hours)
Training Time (Hours)
DOTS+RR
DOTS
Original GRPO
Figure 10: Performance curves with wall-clock time x-axis. Replot of Fig, using wall-clock time.
28


Reviews:
[{"review_id": "eYiFLBsp4A", "reviewer": "Reviewer_9x3o", "summary": "This paper proposes a method to improve data efficiency in reinforcement learning (RL) fine-tuning of large language models (LLMs), primarily through two techniques: Difficulty-targeted Online Data Selection (DOTS) and Rollout Replay (RR) . The authors validate the effectiveness of their approach both theoretically and experimentally, conducting extensive evaluations across multiple LLM-dataset combinations. The results show that the proposed method reduces training time by 25% to 65% while maintaining performance.\n\nOverall, the paper is well-structured, clearly written, and presents a thorough experimental design with strong practical value. Especially given the high computational cost of current RL training methods, this work offers an efficient and reproducible solution.", "questions": "--Enhance Novelty Explanation:\nClarify the core differences between DOTS and automatic curriculum learning methods, emphasizing the innovation of using an attention mechanism to achieve zero-shot difficulty transfer.\n--Supplement Empirical Validation of the Independence Assumption:\nConduct empirical tests of the independence assumption in Theorem 1, such as binning statistics of gradient norm distributions across different success rate intervals p .\n--Refine Method Details:\nClearly specify the architecture of the MLP prediction framework, and discuss how well the assumptions in Theorem 1 hold in practical policy gradient settings.\n--Error Analysis by Question Type and Dynamic Reference Set Expansion:\nBreak down prediction errors by question type, and explore mechanisms for dynamically expanding the reference set. Include additional experiments—such as evaluating on non-mathematical tasks (e.g., AlpacaEval)—to validate generalization.", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\n--Clear problem definition:\nThe paper precisely identifies the core issue of low data efficiency in LLM reinforcement fine-tuning and quantifies the severity of the problem using computational cost analysis (e.g., 3,800 A100 GPU hours / $4,500). The motivation is clear and highly relevant to real-world applications.\n--Comprehensive experimental design:\nThe experiments cover six LLM-dataset combinations (ranging from 1.5B to 7B parameters) and are evaluated on widely used mathematical reasoning datasets such as MATH and DeepMath. Multiple evaluation metrics are employed, providing solid empirical support for the claims.\n--Strong theoretical foundation:\nAppendix provides a complete proof of Theorem 1 , which states that gradient magnitude is maximized when the reward success rate is 50%. This theorem underpins the difficulty selection strategy and is supported by rigorous mathematical derivation.\n\nWeaknesses:\n--Incremental innovation:\nWhile DOTS and RR are effective, they represent incremental improvements over existing methods. Difficulty-based curriculum learning has been studied extensively in RL, and the paper does not sufficiently clarify the fundamental differences between DOTS and classical curriculum learning approaches. Similarly, RR is essentially a variant of experience replay, adapted for GRPO, but its technical contribution remains a straightforward extension rather than a novel breakthrough.\n--Assumption mismatch in Theorem 1:\nTheoretical result Theorem 1 assumes independent and identically distributed (i.i.d.) Bernoulli rewards and that the policy gradient is independent of the reward distribution. However, in LLM generation tasks, token predictions are inherently sequential and interdependent—early errors can propagate and affect later tokens and final rewards. Thus, the i.i.d. assumption may not hold, potentially undermining the validity of the theorem in practical settings.\n--Missing implementation details of the prediction framework:\nThe lightweight MLP adapter used for adaptive difficulty prediction lacks detailed architectural description (e.g., number of layers, activation functions). Additionally, it is unclear how the training data for the adapter is constructed and whether it is independent of the RL training process.\n--Potential failure cases in difficulty prediction:\nThe prediction framework estimates adaptive difficulty via embedding similarity interpolation. However, in domains like mathematics, surface-level similarity does not necessarily imply similar difficulty levels. For example, \"taking derivatives\" and \"computing integrals\" may have similar embeddings but vastly different difficulties. Furthermore, the reference set (only 256 samples) may lack sufficient coverage, leading to unreliable predictions for unseen question types.", "was_revised": true, "final_justification": "The author's response has addressed my concerns, and I keep my initial rating of Borderline Accept.", "rebuttal": "We sincerely thank Reviewer 9x3o for the thoughtful and constructive feedback. In response to the suggestions, we have conducted **additional experiments during the rebuttal period** on (1) providing **empirical evidence** supporting our theoretical analysis assumption and (2) **increasing the reference set size** from 256 to 512 to demonstrate that our current choice is sufficient.\n\nBelow, we address the questions and concerns point by point.\n\n---\n\n> **W1 & Q1:Clarification on innovation**\n\n**Re.:** We would like to clarify the novelty of our contributions. Prior work on difficulty-based curriculum learning either relies on *external difficulty labels* [1,2], which we show in §5.4 to be less effective than our method, or computes *adaptive difficulty* but requires full rollouts for every question [3], making them computationally impractical. In contrast, we propose a **novel attention-based adaptive difficulty prediction framework** that **efficiently and accurately estimates adaptive difficulty without full rollouts**. We also provide Proposition 1 to theoretically justify the benefit of selecting questions with difficulty near 0.5. Together, these innovations enable scalable and principled online data selection for LLM RL fine-tuning, which prior methods cannot achieve due to either limited effectiveness or prohibitive cost.\n\nRegarding RR, while experience replay is common in off-policy RL, it has not, to our knowledge, been applied to LLM RL fine-tuning; we **bridge this gap** by adapting it with a modified GRPO loss to enable stable and efficient rollout reuse.\n\nFinally, our method improves data efficiency of LLM RL fine-tuning from two **complementary angles**: DOTS reduces the number of steps needed, while RR lowers the cost per step. This unified approach to optimizing both axes has not been explored in prior work.\n\n> **W2 & Q2: Independence assumptions in theoretical analysis**\n\n**Re.:** Thank you for your question. It involves two parts: (1) the i.i.d. assumption on Bernoulli rewards and (2) the assumption that the policy gradient is independent of the reward. We address both below.\n\n(1) **i.i.d. reward assumption is reasonable and standard**\n\nThis assumption holds in our setting because **given the same prompt**, each response is **randomly sampled** from the **same policy**, and the reward is then computed by a deterministic rule-based verifier. As a result, the only source of randomness lies in the policy’s stochastic generation process, and the rewards are i.i.d. from the same Bernoulli (p) distribution. This assumption is commonly used in prior work [4,5].\n\n(2) **Independence between gradient and reward can be relaxed without affecting the main conclusion & empirical evidence to support this relaxed assumption**\n\nWe agree that assuming full independence between policy gradients and rewards may be overly strong in practice. In the original proof, we adopted this assumption for simplicity. However, we now provide a more **general and relaxed** formulation **without affecting our main conclusion**.\n\nWe now expand the original derivation (L65, Appendix C) to include the covariance correction term:\n\n$\\mathbb{E}[||g||^2] =\\underbrace{\\sum\\_{i,j=1}^G\\mathbb{E}[\\hat{A}_i \\hat{A}_j] \\cdot \\mathbb{E}[\\nabla\\_{\\theta}\\log \\pi\\_\\theta(o_i \\mid q)^\\top\\nabla\\_\\theta \\log \\pi\\_\\theta(o_j \\mid q)]}\\_{T_1}$ + $\\underbrace{\\sum\\_{i,j=1}^G \\mathrm{Cov} (\\hat{A}_i\\hat{A}_j,\\nabla\\_\\theta\\log \\pi\\_\\theta(o_i \\mid q)^\\top \\nabla\\_\\theta \\log \\pi\\_\\theta(o_j \\mid q))}\\_{T_2}$.\n\nWe denote the first term as the **independence term** and the second as the **covariance correction term**. To relax the independence assumption, we introduce a weak-dependence assumption:\n\n$ \\frac{\\left|T_2\\right|}{T_1}  \\ll 1$.\n\nUnder this assumption, our original subsequent derivation and final conclusion remains unchanged, as the **independence term continues to dominate**. **This confirms that our previous theoretical conclusion was not incorrect, but rather based on a simplified case that can now be justified under a broader condition.**\n\n+ **Empirical evidence:** To support this relaxed assumption **empirically**, we compute the covariance-to-main ratio described above. We randomly sample 512 questions for each of two LLM-dataset combinations, generate 8 rollouts per question, and evaluate the ratio. The results are:\n\n– Qwen2.5-Math-1.5B + MATH: 0.081 ± 0.0065\n\n– Qwen2.5-3B + DeepMath: 0.081 ± 0.0051\n\nThese low ratios empirically validate the weak-dependence assumption.\n\nWe thank the reviewer again for raising this point. We will revise the theoretical section in the final version to explicitly incorporate the relaxed assumption and include the empirical results that support it.\n\n> **W3 Part 1 & Q3: Implementation details of adaptive difficulty framework**\n\n**Re.:** The implementation details of the adaptive difficulty framework are provided in L81-95 in Appendix D.1. Specifically, the adapter is a GELU-activated MLP with three hidden layers, each containing 896 units and a dropout rate of 0.1. A LayerNorm is applied to the projection output to stabilize training. \n\n> **W3 Part 2: The construction of training data for the adapter**\n\nThe construction process is detailed in Appendix D.1 (L87–93). Each training instance consists of a query question, a reference set with known difficulty scores, and a ground-truth adaptive difficulty label. To build this dataset, we sample questions from the math datasets and use a fixed set of **external LLMs (disjoint from the policy model)** to generate responses and compute adaptive difficulty scores, which serve as supervision signals for training the adapter.\n\nThe adapter training **is independent of the RL process**, as the policy model trained during RL is excluded at this stage, and the adapter is trained before RL fine-tuning begins. This ensures that **no information from the policy model is leaked**.\n\n\n> **W4 & Q4 Part 1: Potential failure case of adaptive prediction framework & Coverage of Reference Set**\n\n**Re.:** We agree that in mathematics, surface-level similarity does not always imply similar difficulty. This is precisely **why we do not rely on off-the-shelf embeddings**. Instead, we train an adapter to refine the embedding space to capture difficulty signals rather than surface form. As shown in Table 5 of the ablation study (L148–154, Appendix F.1), this leads to a substantial improvement in prediction alignment with ground-truth adaptive difficulty compared to off-the-shelf alternatives.\n\nTo **demonstrate that a reference set size of 256 is sufficient**, we conduct **additional experiments** increasing the size to 512. As shown in the table below, the improvement in adaptive difficulty prediction is **marginal** (typically 1–2%). This suggests that increasing the reference set beyond 256 offers diminishing returns in prediction performance while incurring higher computational cost.\n\nFinally, we agree that dynamically adjusting the reference set size based on question types is an interesting direction. We leave this exploration to future work.\n\n**Table: Average Pearson correlation ($\\rho$) between predicted and ground-truth adaptive difficulties under different reference set sizes.** \n\n|Model|Dataset|Ref Size = 256|Ref Size = 512|\n|-|-|-|-|\n|Qwen2.5-Math-1.5B|DeepScaleR|0.7244 $\\pm$ 0.0318 |0.7434 $\\pm$ 0.0365|\n|Qwen2.5-3B|DeepScaleR |0.7789 $\\pm$ 0.0191| 0.7916 $\\pm$ 0.0193|\n|Qwen2.5-Math-7B|DeepScaleR|0.7076 $\\pm$ 0.0195| 0.7140 $\\pm$ 0.0431|\n\n> **Q4 Part 2: Generalization to non-math domains**\n\n**Re.:** We would greatly appreciate clarification on which aspect of generalization you are referring to:\n\n(1) If the concern is about the adaptive difficulty prediction framework, we note that cross-domain generalization is not required in our setting. Instead, for each domain, we can train a separate MLP adapter, which is both **justified and practical**.\n\n- **Justification:** This adapter is trained **specifically for the data domain used in the subsequent LLM RL fine-tuning**, which is **known in advance**. **If data from a particular domain will be used during RL fine-tuning, it is naturally included in the adapter training data.** Therefore, there is no need to generalize to out-of-domain data. This setting is fundamentally different from standard supervised learning, where models are expected to perform well on unseen distributions.\n\n- **Practicality:** The training cost is low, taking about 11.5 minutes on a single NVIDIA A100 GPU. This makes per-domain retraining entirely manageable in practice.\n\n(2) If the question instead concerns the generalization of our full data-efficient RL fine-tuning pipeline beyond math, we have **conducted an additional experiment in the science domain** using the SCP-25K dataset. Evaluation results on the science subsets of MMLU confirm that our method continues to **significantly improve data efficiency**, supporting its **broader applicability**. Please refer to our response to Q2 & L1 of reviewer NvgW for detailed results due to space limits here.\n\n---\n\n[1] Lee, Bruce W., Hyunsoo Cho, and Kang Min Yoo. \"Instruction Tuning with Human Curriculum.\" In Findings of the Association for Computational Linguistics: NAACL 2024, pp. 1281-1309. 2024. \n\n[2] Team, Kimi, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li et al. \"Kimi k1. 5: Scaling reinforcement learning with llms.\" arXiv preprint arXiv:2501.12599 (2025). \n\n[3] Bae, Sanghwan, Jiwoo Hong, Min Young Lee, Hanbyul Kim, JeongYeon Nam, and Donghyun Kwak. \"Online difficulty filtering for reasoning oriented reinforcement learning.\" arXiv preprint arXiv:2504.03380 (2025).\n\n[4] Vojnovic, Milan, and Se-Young Yun. \"What is the Alignment Objective of GRPO?.\" arXiv preprint arXiv:2502.18548 (2025).\n\n[5] Shi, Taiwei, Yiyang Wu, Linxin Song, Tianyi Zhou, and Jieyu Zhao. \"Efficient reinforcement finetuning via adaptive curriculum learning.\" arXiv preprint arXiv:2504.05520 (2025)."}, {"review_id": "tWlWfg6FYL", "reviewer": "Reviewer_1Wxg", "summary": "This paper presents a new method for selecting LLM RL training problems via their difficulty level for the current policy. This is done by rolling out the policy on a small core-set of reference problems and estimating problem difficulty for the remainder of the train set using an attention based scoring. Selected rollouts are placed in a replay buffer and used to train the current policy (by additionally modifying the standard GRPO loss accomodate off-policy data). Problems are sampled by the replay buffere reweighted by their likelihood under the policy by which they were generated. Experiments show the method improves sample efficiency by 25%-65%.", "questions": "- Why not at least also include reference solutions in the embedding adaptive diffuclty embeddings to better calibrate solution difficulty?\n- In a similar direction, why not include an oracle training baseline estimating problem difficulty by rolling out the model for every problem? While this may be costly, it seems possible to maintain slightly off-policy difficulty estimates by reusing training rollouts.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths:**\n\n- The presentation and layout of the paper is clear and easy to follow.\n- The evaluation is done across multiple model scales (1.5B, 3B, 7B) and multiple datasets\n- The paper addresses a key bottleneck in the LLM RL training pipeline: rollout generation inefficiency.\n\n**Weaknesses:**\n\n- I'm a a bit skeptical of the adaptive difficulty assessment approach: many questions may be semantically similar in statement but may actually be quite different in difficultly (e.g. consider the simplicity of Goldbach's conjecture). The correlation between estimated difficulty and ground-truth also does not seem to be very strong.\n- In the theorem, it is assumed rewards are independent from the gradient ($\\nabla_\\theta log (o_i | q_i)$). Can the authors motivate why these assumptions are reasonable?", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank Reviewer 1Wxg for the thoughtful and constructive feedback. In response to the suggestions, we have conducted **additional experiments during the rebuttal period** on (1) providing **empirical evidence** supporting our theoretical analysis assumptions, (2) **including reference solutions** when training the adaptive difficulty prediction framework, and (3) **adding an oracle training baseline** using full rollouts for every question.\n\nBelow, we address the questions and concerns point by point.\n\n---\n\n> **W1: Effectiveness of the adaptive prediction framework & correlation between estimated difficulty and ground-truth**\n\n**Re.:** Thank you for the thoughtful feedback. We agree that in mathematics, surface-level similarity does not always imply similar difficulty. This is precisely **why we do not rely on off-the-shelf embeddings**. Instead, we train an adapter to refine the embedding space to capture difficulty signals rather than surface form. As shown in Table 5 of the ablation study (L148–154, Appendix F.1), this leads to a substantial improvement in prediction alignment with ground-truth adaptive difficulty compared to off-the-shelf alternatives.\n\nWhile the overall Pearson correlation ranges from 0.7 to 0.8, we emphasize that our goal is not to precisely predict the difficulty of **every question**. Rather, at each training step, we only need to **identify B questions** that are highly likely to be informative (those near 0.5 adaptive difficulty). Our method achieves this effectively: it significantly increases the ratio of effective questions (as shown in Figure 4 on Page 8), and selects about **30–40% more** moderate-difficulty questions compared to random sampling, resulting in substantial data efficiency gains.\n\n> **Q1: Incorporate reference solutions in the embeddings**\n\n**Re.:** Thank you for the suggestion! We conduct an **additional experiment** on the MATH dataset by including reference solutions when constructing the input embeddings. This results in a **slight improvement** in Pearson correlation, indicating that solutions can provide helpful additional signals beyond the problem statement. However, since reference solutions are **not available for all datasets** (e.g., DeepScaleR and ORZ), this approach may have limited applicability. We will include this discussion in the revised version.\n\n**Table: Comparison of average Pearson correlation ($\\rho$) between predicted and ground-truth adaptive difficulties.**  Reported as mean ± standard deviation over 60 training steps.\n\n|Model+Dataset|w.o. Solutions | w. Solutions |\n|-|-|-|\n|Qwen2.5-Math-1.5B+MATH| 0.7843 $\\pm$ 0.0243 | 0.7940 $\\pm$ 0.0229 |\n\n\n> **W2: Independence assumptions in theoretical analysis**\n\n**Re.:** Thank you for your question. We agree that assuming full independence between policy gradients and rewards may be overly strong in practice. In the original proof, we adopted this assumption for simplicity. However, we now provide a more **general and relaxed** formulation **without affecting our main conclusion**.\n\nWe now expand the original derivation (L65, Appendix C) to include the covariance correction term $T_2$:\n\n$\\mathbb{E}[||g||^2] =\\underbrace{\\sum\\_{i,j=1}^G\\mathbb{E}[\\hat{A}_i \\hat{A}_j] \\cdot \\mathbb{E}[\\nabla\\_{\\theta}\\log \\pi\\_\\theta(o_i \\mid q)^\\top\\nabla\\_\\theta \\log \\pi\\_\\theta(o_j \\mid q)]}\\_{T_1}$ + $\\underbrace{\\sum\\_{i,j=1}^G \\mathrm{Cov} (\\hat{A}_i\\hat{A}_j,\\nabla\\_\\theta\\log \\pi\\_\\theta(o_i \\mid q)^\\top \\nabla\\_\\theta \\log \\pi\\_\\theta(o_j \\mid q))}\\_{T_2}$.\n\nWe denote the first term as the **independence term** and the second as the **covariance correction term**. To relax the independence assumption, we introduce a weak-dependence assumption:\n\n$ \\frac{\\left|T_2\\right|}{T_1}  \\ll 1$.\n\nUnder this assumption, our original subsequent derivation and final conclusion remains unchanged, as the **independence term continues to dominate**. **This confirms that our previous theoretical conclusion was not incorrect, but rather based on a simplified case that can now be justified under a broader condition.**\n\n* **Empirical evidence:** To support this relaxed assumption **empirically**, we compute the covariance-to-main ratio described above. We randomly sample 512 questions for each of two LLM-dataset combinations, generate 8 rollouts per question, and evaluate the ratio ($ \\frac{\\left|T_2\\right|}{T_1}$). The results are:\n\n– Qwen2.5-Math-1.5B + MATH: 0.081 ± 0.0065\n\n– Qwen2.5-3B + DeepMath: 0.081 ± 0.0051\n\nThese low ratios empirically validate the weak-dependence assumption.\n\nWe thank the reviewer again for raising this point. We will revise the theoretical section in the final version to explicitly incorporate the relaxed assumption and include the empirical results that support it.\n\n> **Q2: Oracle training baseline with full-rollout difficulty estimates**\n\n**Re.:** Thank you for the insightful suggestion. We implemented such an oracle baseline by performing full rollouts on all questions every $\\mu$ steps and selecting $\\mu \\times B$ questions, prioritizing those with adaptive difficulty near 0.5, for the next $\\mu$ training steps. We set $\\mu = 6$ to allow the oracle baseline to use approximately **twice the computational budget of DOTS**. To make the comparison fair, we also removed rollout replay from both methods to isolate the effect of difficulty-based selection. \n\nAs shown in the table below, this oracle baseline **underperforms our DOTS method**. This is likely because these difficulty scores become increasingly off-policy as training progresses. Increasing the frequency of full rollouts could mitigate this issue, but would incur **even greater computational cost**—recall that running full rollouts every 6 steps already doubles the training cost compared to DOTS.\n\nIn contrast, our adaptive difficulty prediction framework strikes a strong balance between **on-policy difficulty estimation** and **computational efficiency**: it **generates on-policy difficulty predictions** with **minimal overhead**, enabling it to better capture evolving training dynamics and deliver stronger performance.\n\n\n**Table: Average accuracy of different methods on Qwen2.5-Math-1.5B+DeepScaleR.**\n| Steps | DOTS (Ours) | Oracle | Original GRPO |\n|-|-|-|-|\n|10| **48.61**| 47.68  | 47.71|\n|20| **50.54**| 49.69  | 49.09|\n|30| **51.74**| 50.84  | 50.16|\n|40| **52.47**| 51.65  | 50.88|\n|50| **53.12**| 52.36  | 51.50|\n|60| **53.60**| 52.69  | 51.93|"}, {"review_id": "3jKN9HwDv0", "reviewer": "Reviewer_puy2", "summary": "This paper aims to improve the data efficiency of reinforcement learning for LLM post-training. The authors recognized problems that not all training samples are used effectively to provide useful learning signals, and data from past iterations are simply discarded. Therefore, the authors proposed to techniques, difficulty-targeted online data selection (DOTS) and rollout replay (RR) to reduce the number of training steps and the computational costs per step. Specifically, DOTS introduces an “adaptive difficulty” metric that selects data of moderate difficulty to maximize the learning signal for each training step, while RR lowers the cost by reusing rollouts from recent training steps. With these two techniques combined, the paper demonstrates that the method reduces the RL training time by 25% to 65% percent without sacrificing performance.", "questions": "1. It is unclear from the paper whether the reference set’s rollouts are used in the training process other than the adaptive difficulty prediction.\n2. Relatedly, when the reference set is 256 and the training batch size is 512, substantial time is spent on the rollout of the reference set. Have you considered using those samples for training (especially when their difficulty is around 0.5), for replay, or are they just discarded?\n3. While there are high-level computational time comparisons, it is unclear how much time was taken for the additional difficulty prediction step. Can the authors present a breakdown of time for the overall process? Also, will the efficiency advantage (wall-clock time wise) still be present when there is only POTS (no RR)?\n4. The experimental results lack credibility and reproducibility. There is only one run reported, without error bars. The experimental run is also terminated early - the convergence behavior is not seen in the plots.\n5. Another questionable presentation of the result is that the ablation results in Figure 5 are shown on training step, while Figure 3 is shown on GPU time. I would like to see both as the x axis, for both overall result and ablation to assess reliably.\n6. Though AIME results are excluded, they could have been reported separately because it could be possible that the designed online data selection method does not help with learning extremely hard questions like AIME.", "limitations": "Please discuss the limitations in the main paper instead of the appendix B.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. Recognizing the data efficiency in RL as a major problem to address, this is also a timely contribution.\n2. The results do show the great performance of the proposed method in terms of efficiency gain.\n3. The proposed method is shown to greatly improve the ratio of effective questions in the training, as shown in Figure 4.\n\nWeaknesses:\n\n1. The empirical results could lack credibility due to the limited number of runs, and some additional issues described in the Questions below.", "was_revised": true, "final_justification": "The authors provided additional results during the short rebuttal period. My questions are answered. Though the improvements are marginal in some experiments, I feel the paper is above the threshold for acceptance. I will raise my score.", "rebuttal": "We sincerely thank Reviewer puy2 for the thoughtful and constructive feedback. In response to the suggestions, we have conducted **additional experiments during the rebuttal period** on (1) adding **standard errors** for 3 LLM-dataset combinations to support statistical significance, (2) **extending the training horizon** to observe convergence behavior, and (3) evaluating on the **AIME benchmark**.\n\nBelow, we address the questions and concerns point by point.\n\n> **W1 & Q4 Part 1: No error bars**\n\n**Re.:** Thank you for the suggestion. We have added error bars for 3 LLM-dataset combinations by independently repeating both DOTS+RR and original GRPO three times. The results confirm that the improvements of our method over original GRPO are **statistically significant**, with **nearly all p-values below 0.05**. \n\nDue to space limitations, detailed results are provided in our response to Reviewer NvgW, W1.\n\n> **Q4 Part 2: Extended training to observe convergence behavior**\n\n**Re.:** Due to resource constraints, we initially limited all runs to 60 steps. Following your suggestion, we extended training to **100 steps** for two settings: Qwen2.5-Math-1.5B+DeepScaleR and Qwen2.5-3B+DeepMath. The updated results are shown in the table below and remain consistent with our original conclusions. \n\n**Table A: Average accuracy (%) of DOTS+RR (Ours) and original GRPO over 100 training steps on Qwen2.5-Math-1.5B with DeepScaleR and Qwen2.5-3B with DeepMath datasets.**\n\n|Steps|Qwen2.5-Math-1.5B+DeepScaleR||Qwen2.5-3B+DeepMath||\n|-|-|-|-|-|\n||DOTS+RR|Original GRPO|DOTS+RR|Original GRPO|\n|10|**48.59** |47.67|**48.10**|47.74|\n|20|**50.02** |49.13|**48.39**|48.37|\n|30|**51.19** |50.36|**48.88**|48.26|\n|40|**52.09** |51.13|**49.06**|48.37|\n|50|**52.70** |51.69|**49.24**|48.45|\n|60|**53.14** |52.19|**49.38**|48.57|\n|70|**53.49** |52.62|**49.51**|48.62|\n|80|**53.84** |52.91|**49.63**|48.79|\n|90|**54.07** |53.15|**49.64**|48.85|\n|100|**54.31**|53.29|**49.66**|48.87| \n\n> **Q1 & Q2: Reuse of reference set rollouts**\n\n**Re.:** This is an insightful point. In our current experiments, rollouts from the reference set are **not reused** for training. \n\nInspired by your suggestion, we conducted **additional experiments** to assess the benefit of reusing reference rollouts for training when their predicted difficulty is near 0.5. Approximately 10% of the reference questions per step fall into this moderate-difficulty range (see Table B below). Reusing the rollouts associated with these questions in RL training can **reduce overall rollout generation time by 4–12%** per step, while maintaining the same final policy performance. **We find this strategy effective and will adopt it in the revised version.**\n\nRelatedly, to **clarify the overhead**, in our experiments, data selection is conducted **every 2 steps** (stated in L185), so after amortization, **only 128 (instead of 256)** reference questions are needed to do rollouts per step. \n\n**Table B: Proportion of questions with adaptive difficulty=0.5 in reference sets at different training steps.**\n|Model+Dataset|Step 0|Step 20|Step 40|Step 60|\n|-|-|-|-|-|\n|Qwen2.5-Math-1.5B+DeepScaleR|0.11| 0.08|0.07|0.04|\n|Qwen2.5-Math-1.5B+ORZ|0.11|0.13|0.08|0.12|\n|Qwen2.5-3B+DeepMath|0.12|0.06|0.04|0.04|\n\n> **Q3 Part 1: Detailed time breakdown of the overall process**\n\n**Re.:** Thank you for your question. We provide a detailed breakdown of the time cost of DOTS+RR (Ours) on the Qwen2.5-Math-1.5B and Qwen2.5-3B models trained on DeepScaleR, averaged over 60 steps. The experiments are conducted on 8x NVIDIA L40S GPUs. The overall process consists of four components: (1) **Reference rollout time**, (2) **Adaptive difficulty prediction time**, (3) **Selected question rollout time**, and (4) **Actor update time**.\n\nBelow is the breakdown of the average time (in seconds) amortized per step. We highlight three key observations:\n\n(1) The cost of adaptive difficulty prediction is negligible.\n\n(2) The selected question rollout time is relatively short due to the use of replay in DOTS+RR.\n\n(3) As mentioned in our response to Q1&Q2, reference rollouts whose predicted difficulty is close to 0.5 can be reused for training, thereby further reducing the cost of selected question rollouts.\n\n**Table C: Breakdown of the average time (in seconds) amortized per step of DOTS+RR.** We note that data selection is performed once every two steps (as stated in L185), so reference rollouts are only generated at those steps, and the reported cost is amortized accordingly.\n\n|Component|Qwen2.5-Math-1.5B| |Qwen2.5-3B| |\n|-|-|-|-|-|\n| |Avg Time per Step (s)|Percentage (%)|Avg Time per Step (s)| Percentage (%) |\n|Reference Rollout|37.76|16.78%|61.73|17.36%|\n|Adaptive Difficulty Prediction|1.17|0.46%|1.21|0.34%|\n|Selected Question Rollout|79.27|35.23%|126.94|35.70%|\n|Actor Update|106.95|47.53% |165.69|46.60%|\n\n\n> **Q3 Part 2: Wall-clock efficiency of DOTS (without RR) vs. original GRPO**\n\n**Re.:** Yes, even when using DOTS without RR, **the efficiency advantage in wall-clock time is still present**.\n\nAlthough DOTS (without RR) introduces a small additional overhead due to reference rollouts and difficulty prediction, it requires **much fewer steps to reach the same final accuracy** as the original GRPO. As a result, the **overall training time is reduced**. \n\nFor example, on the Qwen2.5-Math-1.5B + DeepScaleR setting, DOTS reaches the final accuracy of original GRPO **31.23%** faster in terms of wall-clock time. More results can be found in the response to Q5 below.\n\n> **Q5: Add training steps as x-axis in Figure 3 and GPU time in Figure 5**\n\n**Re.:** Thank you for the suggestion. Since figures are not permitted during the rebuttal, we provide the corresponding results in tabular form to address both of your concerns:\n\n(1) **Figure 3 → Now with training steps as x-axis (Table D)**:\n\n(2) **Figure 5 → Now with GPU time as x-axis (Table E)**.\n\nAcross both views (training steps and wall-clock time), DOTS & RR and DOTS consistently demonstrate strong performance, confirming the robustness of our advantage **regardless of the presentation format**. We will include the updated figures in the revised appendix.\n\n**Table D: Average accuracy of DOTS+RR (Ours) and original GRPO across various LLM-dataset combinations.**\n| Steps | Qwen2.5-Math-1.5B + MATH | | Qwen2.5-Math-1.5B + DeepScaleR | | Qwen2.5-Math-1.5B + ORZ | | Qwen2.5-3B + DeepScaleR | | Qwen2.5-3B + DeepMath | | Qwen2.5-Math-7B + DeepScaleR | |\n|-|-|-|-|-|-|-|-|-|-|-|-|-|\n||DOTS+RR|Original GRPO|DOTS+RR|Original GRPO|DOTS+RR|Original GRPO|DOTS+RR|Original GRPO|DOTS+RR| Original GRPO|DOTS+RR|Original GRPO|\n|10|49.03|48.19|48.39|47.99|48.42|48.17|48.93|47.82|47.86|47.66|63.44|61.99|\n|20|50.61|49.82|50.19|49.06|50.72|50.17|49.77|49.09|48.43|48.25|65.07|64.55|\n|30|51.69|51.11|51.52|49.89|52.28|51.46|49.95|49.65|48.86|48.08|66.38|65.76|\n|40|52.42|51.78|52.27|50.81|53.00|52.35|50.25|50.01|49.01|48.29|67.09|66.70|\n|50|52.84|52.27|53.01|51.44|53.75|53.00|50.55|50.31|49.13|48.43|67.54 |67.13|\n|60|53.12|52.66|53.52|51.95|54.15|53.52|50.65|50.47|49.13|48.58|67.91|67.58|\n\n**Table E: Accuracy comparison of DOTS+RR, DOTS (without RR), and original GRPO on Qwen2.5-Math-1.5B in the ablation study. The “Final” row refers to the last evaluation point for each run and the “Time to Achieve original GRPO Final Performance” row refers to the required time to match original GRPO final performance. DOTS can still reduce training time compared to original GRPO. DOTS+RR consistently achieves comparable performance with DOTS while saving the most training time.**\n\n|Time(h)|MATH|||DeepScaleR|||ORZ|||\n|-|-|-|-|-|-|-|-|-|-|\n||DOTS+RR|DOTS|Original GRPO|DOTS+RR|DOTS|Original GRPO|DOTS+RR|DOTS|Original GRPO|\n|0.50|48.49|47.76|47.75|47.95|47.45|47.65|47.85|47.26|47.36|\n|1.00|50.01|49.03|48.99|49.72|48.86|48.24|50.00|48.94|49.06|\n|1.50|51.03|50.10|50.22|50.90|50.23|49.14|51.42|50.32|50.51|\n|2.00|51.92|51.20|51.11|51.60|51.04|49.71|52.50|51.38|51.34|\n|2.50|52.42|51.71|51.49|52.27|51.60|50.46|53.00|52.13|52.02|\n|3.00|52.84|52.20|52.06|52.86|52.21|50.95|53.75|52.62|52.62|\n|3.50|53.07|52.53|52.27|53.43|52.65|51.44|54.09|53.16|53.00|\n|4.00|–|52.85|52.55|–|53.00|51.76|–|53.56|53.42|\n|Final|53.12|53.28|52.66|53.52|53.52|51.95|54.15|54.12|53.52|\n|Time to Achieve original GRPO Final Performance (h)|2.81|3.64|4.14|2.25|2.91|4.24|2.94|3.81|4.15|\n|Time Saved|**32.1%**|**12.1%**|–|**46.9%**|**31.2%**|–|**29.2%**|**8.3%**|–|\n\n> **Q6: AIME results exclusion**\n\n**Re.:** We initially excluded AIME results because AIME is known to cause instability in evaluation for smaller models due to high variance and limited data size (only 30 questions) [1]. \n\nTo support this point empirically, we provide **additional experimental results** using various LLM-dataset combinations with the original GRPO. Specifically, we evaluated each of the AIME 24 questions 8 times and computed the average accuracy (avg@8) across training steps.  \n\nAs shown in the following table, accuracy fluctuates heavily across steps, with no clear trend (results shown every 10 steps). This **high variance across checkpoints** makes it difficult to obtain **reliable evaluation signals** on AIME.\n\n**Table F: Accuracy of original GRPO on AIME 24.**\n|Steps| Qwen2.5-Math-1.5B+DeepScaler|Qwen2.5-3B+DeepScaler|Qwen2.5-Math-7B+DeepScaler|\n|-|-|-|-|\n|10|10.42|7.50|20.42|\n|20|15.83|6.25|24.17|\n|30|15.83|8.75|24.17|\n|40|11.67|9.17|23.33|\n|50|16.67|7.50|25.42|\n|60|13.75|5.42|20.00|\n\nAs suggested, we will include AIME evaluation results for all models in the appendix **for reference**, and provide a detailed discussion on why we exclude AIME from the main analysis.\n\n---\n\n[1] Hochlehnert, Andreas, Hardik Bhatnagar, Vishaal Udandarao, Samuel Albanie, Ameya Prabhu, and Matthias Bethge. \"A sober look at progress in language model reasoning: Pitfalls and paths to reproducibility.\" arXiv preprint arXiv:2504.07086 (2025)."}, {"review_id": "r6M44YHgWE", "reviewer": "Reviewer_NvgW", "summary": "In this paper, the authors address the computational inefficiency of reinforcement learning fine-tuning for large language models by introducing two techniques: Difficulty-targeted Online Data Selection (DOTS) and Rollout Replay (RR). DOTS employs an attention-based adaptive difficulty prediction framework that efficiently selects training questions with moderate difficulty, providing the most informative learning signals by generating rollouts for only a small reference set. Simultaneously, RR reduces computational costs by reusing recent rollouts and generating new rollouts for only a fraction of each training batch, using a modified loss function to maintain training stability. Experimentally validated across six LLM-dataset combinations, their approach demonstrates remarkable performance, reducing RL fine-tuning time while preserving the original GRPO algorithm's performance.", "questions": "1. The theorem is based only on binary rewards. How to generalize it to more complex reward scenarios?\n2. Why did you choose a mathematical reasoning dataset?  Have you tried to verify it on non-mathematical fields?", "limitations": "1. Only validates mathematical reasoning datasets, generalization is unknown.\n2. Difficulty selection may inadvertently reinforce training data bias.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 4, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1. Rigorous experimental methodology with comprehensive testing across multiple LLM models and datasets\n2. Theoretically grounded approach, including a formal theorem (Theorem 1) explaining the rationale behind difficulty-based sample selection.\n3. Demonstrates substantial time savings (25-65% reduction) across different model sizes and datasets.\n\nWeaknesses:\n1. No statistical significance tests due to high computational costs (acknowledged in the paper)\n2. Pre-trained embedding models have difficulty capturing fine-grained similarities in mathematical problems. The paper alleviates this problem by fine-tuning an MLP adapter, but the adapter may overfit and its generalization ability to problems in different fields is questionable.", "was_revised": true, "final_justification": "The author's response to this paper addressed most of my concerns. Considering the contribution and application potential of this paper, I decided to improve my score.", "rebuttal": "We sincerely thank Reviewer NvgW for the thoughtful and constructive feedback. In response to the suggestions, we have conducted **additional experiments during the rebuttal period** on (1) adding **standard errors** for 3 LLM-dataset combinations to support statistical significance, and (2) evaluating our data-efficient RL training pipeline on a new **non-math** domain.\n\nBelow, we address the questions and concerns point by point.\n\n---\n\n> **W1: No statistical significance tests due to high computational costs**\n\n**Re.:** Thank you for the suggestion. We have added error bars for 3 LLM-dataset combinations by independently repeating both DOTS+RR and original GRPO three times. The results are shown in the table below and confirm that the improvements of our method over original GRPO are **statistically significant**, with **nearly all p-values below 0.05**.\n\nSpecifically, we report the average accuracy (mean ± standard error) based on three independent runs. We also perform statistical testing using one-sided Welch’s t-tests. Due to the rebuttal policy, we cannot include plots, so we report each metric at 0.5-hour intervals in the tables below. Since DOTS+RR completes training earlier than original GRPO, we report results only up to the DOTS+RR completion point for comparison purposes.\n\nThese additional runs represent our best effort within the limited rebuttal period and available compute resources. In the final version, we will extend this analysis to the remaining 3 combinations and increase the number of independent runs.\n\n\n**Table A: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-Math-1.5B with DeepScaleR dataset.**\n\n| Time(h)    | DOTS+RR (Ours)    | Original GRPO    | p-value   |\n|--|--|--|--|\n| 0.50       | 48.11(±0.094)     | 47.18(±0.277)    | **0.033** |\n| 1.00       | 49.97(±0.128)     | 48.30(±0.057)    | **0.001** |\n| 1.50       | 51.12(±0.112)     | 49.29(±0.084)    | **0.000** |\n| 2.00       | 51.89(±0.155)     | 49.95(±0.126)    | **0.000**     |\n| 2.50       | 52.47(±0.132)     | 50.62(±0.097)    | **0.000**     |\n| 3.00       | 53.01(±0.096)     | 50.98(±0.027)    | **0.001**     |\n| 3.50       | 53.45(±0.034)     | 51.50(±0.064)    | **0.000**     |\n| 3.75       | 53.60(±0.058)     | 51.66(±0.053)    | **0.000**    |\n\n\n\n**Table B: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-Math-1.5B with ORZ dataset.**\n\n| Time(h)    | DOTS+RR (Ours)    | Original GRPO    | p-value   |\n|--|--|--|--|\n| 0.50       | 47.68(±0.159)     | 47.79(±0.245)    | 0.636     |\n| 1.00       | 49.91(±0.045)     | 49.29(±0.131)    | **0.015**     |\n| 1.50       | 51.27(±0.080)     | 50.66(±0.130)    | **0.011**     |\n| 2.00       | 52.24(±0.133)     | 51.37(±0.080)    | **0.004**     |\n| 2.50       | 52.87(±0.092)     | 52.16(±0.166)    | **0.016**     |\n| 3.00       | 53.62(±0.086)     | 52.78(±0.148)    | **0.007**     |\n| 3.50       | 54.00(±0.046)     | 53.15(±0.141)    | **0.009**     |\n| 3.67       | 54.07(±0.067)     | 53.41(±0.139)    | **0.013**     |\n\n\n**Table C: Average accuracy (mean ± standard error) of our method and original GRPO on Qwen2.5-3B with DeepMath dataset.**\n\n| Time(h)    | DOTS+RR (Ours)    | Original GRPO    | p-value   |\n|--|--|--|--|\n|0.50| 46.88(±0.102)| 45.99(±0.141)    | **0.004**     |\n| 1.00| 47.91(±0.104)| 47.53(±0.228)    | 0.116     |\n| 1.50| 48.17(±0.155)| 48.06(±0.205)    | 0.346     |\n| 2.00| 48.27(±0.151)| 48.08(±0.133)    | 0.199     |\n| 2.50| 48.51(±0.189)| 48.05(±0.139)    | 0.064     |\n| 3.00| 48.71(±0.171) | 48.15(±0.110)    | **0.030**     |\n| 3.50| 48.82(±0.154)     | 48.12(±0.087)    | **0.013**     |\n| 4.00| 48.89(±0.161)     | 48.27(±0.135)    | **0.022**     |\n| 4.50| 48.98(±0.158)     | 48.30(±0.161)    | **0.020**     |\n| 5.00| 49.07(±0.132)     | 48.34(±0.148)    | **0.011**     |\n| 5.50| 49.05(±0.138)     | 48.42(±0.156)    | **0.020**     |\n| 6.00| 49.13(±0.167)     | 48.44(±0.141)    | **0.018**     |\n| 6.04| 49.13(±0.167)     | 48.46(±0.143)    | **0.020**     |\n\n\n> **Q2 & L1: Choice of mathematical reasoning dataset and generalization to non-math domains**\n\n**Re.:** We focused on mathematical reasoning datasets because math is a representative and important use case for LLM reasoning, and prior work on both algorithmic [1,2] and data-centric [3] improvements in LLM RL fine-tuning commonly adopts math datasets for training and evaluation.\n\nThat said, inspired by your suggestion, we conduct an **additional experiment** in a non-math domain. Specifically, we apply our full pipeline, including both RL training and evaluation, to the **science domain** using the curated SCP-25K dataset [4], which contains advanced physics, chemistry, and biology questions. We use the Qwen2.5-3B model and train a new adaptive difficulty predictor. All other RL settings are kept unchanged.\n\nWe evaluate performance on the science subsets of MMLU, and the results (reported in the table below) show that our method continues to **significantly improve RL data efficiency in this non-math domain**, supporting its **broader applicability**.\n\n**Table D: Average accuracy of DOTS+RR (ours) and original GRPO on Qwen2.5-3B with science subsets of MMLU.**\n|Step|DOTS+RR (Ours)|Original GRPO|\n|-|--|-|\n|10|**53.35**|53.28|\n|20|**55.36**|54.38|\n|30|**56.41**|54.44|\n|40|**56.48**|54.18|\n|50|**56.98**|53.84|\n|60|**57.31**| 53.80|\n\n> **W2: Generalization ability of the MLP adapter**\n\n**Re.:** We do not require the MLP adapter to transfer across domains. Instead, for each domain, we can train a separate adapter, which is both **justified and practical**.\n\n- **Justification:** This adapter is trained **specifically for the data domain used in the subsequent LLM RL fine-tuning**, which is **known in advance**. **If data from a particular domain will be used during RL fine-tuning, it is naturally included in the adapter training data.** Therefore, there is no need to generalize to out-of-domain data. This setting is fundamentally different from standard supervised learning, where models are expected to perform well on unseen distributions.\n\n- **Practicality:** The training cost is low, taking about 11.5 minutes on a single NVIDIA A100 GPU. This makes per-domain retraining entirely manageable in practice.\n\n\n> **Q1: Generalization of Theorem 1 to complex rewards**\n\n**Re.:** Theorem 1 uses binary rewards for simplicity, a standard choice in recent literature on RLVR (Reinforcement Learning with Verifiable Rewards). Its key step involves computing the second central moment of the group-normalized rewards $\\mathbb{E}[\\hat{A}_i^2]$, which can be easily recomputed under more complex reward formulations.  \n\nFor example, if the reward consists of two independent components such as a correctness reward $c_i \\sim \\text{Bern}(\\alpha)$ and a format reward $f_i \\sim \\text{Bern}(\\beta)$, with $r_i = c_i + f_i$, then we have $\\mathbb{E}[\\hat{A}_i^2] = (\\alpha(1-\\alpha) + \\beta(1-\\beta))(1 - 1/G)$, which is maximized when both $\\alpha = 0.5$ and $\\beta = 0.5$. This shows that our insight generalizes naturally to multi-component reward settings.\n\n> **L2: Potential bias due to difficulty-based selection**\n\n**Re.:** If the concern refers to the possibility that our difficulty prediction framework consistently favors a particular type of question (e.g., from certain domains or with certain formats), we believe this is **not a significant issue** in our setup. \n\nSince our method selects questions whose predicted adaptive difficulty under the *current* policy is close to 0.5, those that have been repeatedly selected and trained on will become easier and thus gradually fall outside the 0.5 region. As a result, they are less likely to be selected again, allowing other question types to enter the selection pool over time. This dynamic naturally promotes broader coverage during training.\n\nWe will include a relevant discussion in the revised version.\n\n---\n\n[1] Yu, Qiying, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Weinan Dai et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n[2] Lin, Zhihang, Mingbao Lin, Yuan Xie, and Rongrong Ji. \"Cppo: Accelerating the training of group relative policy optimization-based reasoning models.\" arXiv preprint arXiv:2503.22342 (2025). \n\n[3] Li, Xuefeng, Haoyang Zou, and Pengfei Liu. \"Limr: Less is more for rl scaling.\" arXiv preprint arXiv:2502.11886 (2025).\n\n[4] Liu, Mingjie, et al. \"Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.\" arXiv preprint arXiv:2505.24864 (2025).\n\n[5] Zeng, Thomas, Shuibai Zhang, Shutong Wu, Christian Classen, Daewon Chae, Ethan Ewer, Minjae Lee et al. \"VersaPRM: Multi-Domain Process Reward Model via Synthetic Reasoning Data.\" In Forty-second International Conference on Machine Learning. \n\n[6] Namgoong, Hyuk, Jeesu Jung, Sangkeun Jung, and Yoonhyung Roh. \"Exploring domain robust lightweight reward models based on router mechanism.\" arXiv preprint arXiv:2407.17546 (2024)."}]

Author rebuttal:

Meta-review: This paper proposes a method to improve data efficiency in reinforcement learning (RL) fine-tuning of LLMs. All reviewers and I agree that the paper is well-written and well-motivated. The experiments are finally comprehensive, while some reviewers raised concerns about experiments in the initial phase. 

During the rebuttal phase, the authors made great efforts and supplemented additional experiments, so most reviewers were satisfied with the response and raised their scores to BORDERLINE ACCEPT. 

After reading the paper, I'm also inclined to accept this paper. The authors should revise the paper in the camera-ready version according to the suggestions given by reviewers
Decision: Accept (poster)