Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Authors: Haizhong Zheng
Yang Zhou
Brian R. Bartoldson
Bhavya Kailkhura
Fan Lai
Jiawei Zhao
Beidi Chen
Venue: NeurIPS 2025 poster ()
Keywords: LLM
GRPO
Reinforcement Learning
Primary area: deep_learning

Abstract: Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance, but at the cost of significant computational overhead. In this paper, we first show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, like Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, and Qwen2.5-32B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation. We make our code publicly available at https://github.com/Infini-AI-Lab/GRESO/.

Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Haizhong Zheng
Yang Zhout
Brian R. Bartoldson'
Bhavya Kailkhura²
Fan Lai
Jiawei Zhao
Beidi Chen
Carnegie Mellon University
Lawrence Livermore National Laboratory
SUniversity of Illinois Urbana Chaaaa--a
*Meta AI (FAIR)
Abstract
Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance, but at the cost of significant computational overhead. In this paper, we first show that substantial portion this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO broad range of math reasoning benchmarks and models, like -wen2.5-Math- 1.5B, DeepSeek-RR R1 1 ------itiill--e Swwilll-wen--.... Qwen2.5-14B, and Qwen2.5-328
-wen2.55Mathh show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup total training time without accuracy degradation. We make code publicly available at GitHut
Owen2.5-Math- Matth-1.55
Qwen2 5-Math-78
56
4.3M fewer rollouts
rollouts
Accuracy (96)
Accuracy (56)
80-
50
36
Baseline
Baseline
Ours
Ours
10
12
Rollout (M)
Rollout (M)
on the DAPO + MATH dataset and evaluate them on five math reasoning benchmarks: MATH500, AMC,, Gaokao, Minerva, and Olympiad Bench. Compared to the baseline method (Dynamic Sampling), our approach (GRESO) reduces rollout overhead by up to 2x while achieving comparable training performance, improving the efficiency of rollout scaling.
Figure 1: We train S.wen..5-Mat----.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
34 40,, such as OpenAl's ol and DeepSeek's RI, leverage Chain- of-Thought as a form of test-time scaling to significantly enhance the reasoning capabilities of large language models (LLMs). Reinforcement Learning (RL) techniques, including PPO (35] and GRPO [9]. have emerged as key drivers of this progress. By generating data online during each training iteration (i.e., rollout), reinforcement learning enables models to iteratively refine their reasoning strategies through self-exploration, often achieving or even surpassing human-level performance [34 38 39]. Notably, scaling computational resources sample responses for more prompts at this rollout stage can further enhance training. which allows models to selectively utilize higher-quality data and thus train models with better converged performance [46] 49] However,
Recent reasoning models 2
scaling up rollouts introduces significant computational overhead, as rollout remains a major bottle-
neck in RL training 33 37 42 58) For instance, as shown in Figure filtering out uninformative
example and resampling to fill the batch with effective data (also known as Dynamic Sampling in [49] improve model performance, but it comes at the cost of significantly increased rollout overhead. Motivated by this challenge, we aim to investigate the following research question:
How can we perform more selective allouts-focusing on sampling more valuable prompts to make this rollout scaling more efficient?
Existing methods face several limitations in ad- dressing this question. First, some approaches [24] 43 attempt to improve data efficiency by prun- ing datasets before training. These methods typ- ically rely on training a model to identify valuable data points; however, there is conclusive ev- idence that such strategies improve the overall efficiency of RL training well. Second, these static pruning methods overlook the fact that the value of data point can vary across models and training stages, limiting their ability to support adaptive data selection. Finally, online selection approaches such as Dynamic Sampling [49] per- form oversampling and filter out uninformative data only after rollout, Icading to substantial addi- tional rollout cost. Estimating data quality accu- rately and efficiently before rollout remains a challenging and underexplored problem.
82-
17.5
Auerage
15.2
7.3
32
2.5
Mamdaon
AMC
Giodao
Opyasied
alwurd
0:1
Traliig
Figure 2: Left:: GRPO training with more effec- tive data through Dynamic Sampling (DS) leads to improved final model performance. Right: However, DS requires additional rollouts to maintain the same training batch size.
Consequently, ideal selective rollout algorithm for efficient LLM RL should have the following properties: 1) Online data selection. Instead of relying on an auxiliary model trained offline to pre-prune the dataset, ideal method should perform data selection online during training. This avoids the additional overhead of training a separate model and enables decisions to be made based on the current training states. 2) Model-based data value estimation. Data values evolve throughout training and vary across different models, requiring a selective rollout strategy to adapt dynamically to different models and training stages. 3) Low computational overhead. To ensure scalability, the selective rollout strategy should introduce minimal additional cost during training.
In this paper, aim to design efficient selective rollout strategy for LLM RL to make rollout scaling more efficient. We begin by analyzing the training dynamics of prompts across epochs and observe strong temporal consistency across different training epochs (Section B). In particular, prompts that yield zero advantage for all sampled responses in one epoch are more likely to do so in future epochs well. This temporal correlation suggests that historical reward dynamics can be leveraged to predict and preemptively skip those uninformative prompts before rollout. Building on these observations, we propose GRESO (GRPO with Efficient Selective Rollout) in Section 4 depicted in Figure an online efficient pre-rollout filtering algorithm that reduces rollout cost by selectively skipping prompts predicted to be uninformative. Instead of performing filtering after rollout, GRESO estimates a skipping probability for each prompt based on its reward dynamics
In GRPO, many examples yield identical rewards across all responses, resulting in zero ad advantage and thus contributing no learning signal during training.
2

---

during training prior to the rollout stage, significantly reducing prompt selection overhead and making the rollout scaling more efficient.
In Section 5t
we empirically verify the efficiency of GRESO on six math reasoning benchmarks
and five models: Qwen2.5-Matt-11558 477, SeepSeek-R---isiil----eeeeee
(9] Qwen2.5-Math
7B 147. Qwen2.5-14B [47], and Qwen2.5-32B 447). Our evaluation results show that GRESO achieves up to 2.4x speedup in rollout and 2.0 speedup in total training time while maintaining comparable accuracy (Section 3.2). We also conduct more detailed study on how GRESO reduces training overhead by performing selective rollout and ablation study on different components of
GRESO in Section 5.3
2 Related Work
RL for LLM Reasoning. Reinforcement learning (RL) was initially used to align model outputs with human preferences [7]. Since then, RL has become a commonly used technique for fine-tuning LLMs, enabling them generate more helpful, harmless, and honest responses by incorporating reward signals from human feedback [3] 66.. Recent advances 19, 12 40 49] in LLM reasoning show that Reinforcement Learning with Verifiable Reward (RLVR), which relies verifiable reward signals instead of model-generated scoress, can effectively improve model reasoning ability. These gains achieved using various policy optimization methods such PPO Encouraged by the success of RLVR, a growing body of work [15] 19 27
45 35 and GRPO [36).
52 has emerged further improve reinforcement learning methods for LLM reasoning. For instance, methods such as VinePPO [1)]. VC-PPO (51), and VAPO (50] aim to enhance LLM reasoning by optimizing the value function Meanwhile, DAPO 49 introduces several techniques to improve GRPO, including Dynamic Sampling, which filters out zero-variance prompts and refills the training batch with effective training data through resampling.
30
Data Selection for LLM. In addition to improving training algorithms, another line of work [7 31 44 48 seeks to enhance the efficiency and effectiveness of LLM training through data selection strategies. Several approaches [5 16 44 focus on pruning data used for supervised fine-tuning, For example, S1
31 reduces large set of 59k examples to just 1k high-quality samples. In parallel, another thread of research [8] 11
24 28, 32 43 targets improving data efficiency in reinforcement learning for L.MMs. For instance, recent research [24 43 shows that only small subset of the original training dataset is necessary for GRPO to improve the model's reasoning ability. However, those methods rely on training models with the full dataset first to identify important samples and do not offer clear improvements in end-to-end RL training efficiency.
3 Observation
In this section, we study the impact of uninformative prompts--pecfficalyy zero-variance prompts-on GRPO training. We empirically show that a high zero-variance ratio can hurt the training performance (Section 3.22). Our analysis reveals a strong temporal consistency in prompt value: prompts that are uninformative one training epoch tend to remain uninformative future epochs, which inspires the design of GRESO (Section 3.3)
3.1 Background: Group Relative Policy Optimization (GRPO)
Group Relative Policy Optimization (GRPO) is a variant of Proximal Policy Optimiza- tion (PPO) [35] tailored for language model fine-tuning. Instead of computing advantages using value function, GRPO normalizes reward scores within groups of responses sampled for the same prompt, which largely improves the training efficiency. GRPO has shown superior performance in recent advances 19 24 43 49 in RL for LLMs, especially for reasoning tasks. GRPO aims to maximize the following objective:
(o) = =Elq ~ P(Q), (or) Te (0]g)]
To(
A,, clip To(oi 4)
Teaa (o:lg) c, (0) 4, - -DDxL (molltress)
min
G
To (o.1g)
1=1
(1)
3

---

where A, is the advantage, computed using group of rewards {ri, ........ r.) corresponding to the outputs within each group:
(...,
Ar, mean (R)
(td(
The advantage of each response is computed as normalized reward within a group of repeated rollouts. When all responses in group receive the same reward, regardless of whether they are all correct or all incorrect, the resulting reward variance zero, and the computed advantages for those responses are all zero. As result, these examples provide no learning signal during training. In this paper, we refer to such prompts as tero-variane prompts, while prompts that yield non-identical rewards across responses termed effective prompts.
3.2 Reduction of Effective Prompts in GRPO Training
The existence of zero-advantage prompts can largely reduce the effective prompt ratio training batch. As shown Figure 3 during GRPO training on Qwen2.5-Math- effective prompts keeps decreasing as the training proceeds: at
GRPO
[47]. the ratio of
Effectbe Rall
0.8
0.6
the late stage of training, this ratio can be around only 20%. A varying ratio of effective prompts can potentially hurt training stability and final model performance 44
0.4
A potential way to address this instability issue is oversample and select batch only containing effective prompts, which is also known as Dynamic sampling (DS) 49). As shown in Fig- ure Left, GRPÓ with DS consistently outperforms the vanilla GRPO, particularly on datasets such as AMC and AIME24, also with a higher average accuracy. This performance gain stems from DS's ability filter out zero-variance prompts, thereby stabilizing training. While DS leads to better performance, it incurs significantly higher computational cost due to its need to oversample more data to maintain the training batch size of effective prompts (as shown in Figure Right). For instance, the zero-variance prompt ratio 80%, DS needs to perform around five times rollouts to maintain the training batch size. However, substantial amount of rollout computation is wasted on prompts that ultimately result in zero- variana prompts. Identifying such prompts prior to rollout can significantly reduce computational overhead.
0.0
Training Stop
Figure 3: Dynamics of effec- tive prompts ratio in each step in GRPO training. The ratio keeps decreasing as the training pro- ceeds.
3.3 Temporal Correlation of Prompts across Epochs
Training data typically exhibits strong temporal correlations across epochs (24, 41 55 We hypothesize that zero-variance prompts in GRPO training similarly have such strong correlations in their training dynamics, enabling opportunities for more efficient identification of these prompts prior to the rollout stage. To test this hypothesis, we conduct a study on the temporal correla- tion of variane prompts in GRPO training. Specifically, we train Qwen2.5-Math, GRPO and measure two probabilities to study the temporal correlation of zero-vane varrivar prompts: 1) P (Previous [Current): The probability that prompt identified zero-variance in the current epoch was also zero-variance in any previous epoch. 2) (Current wwwreen//eeeiii (Previou)) The probability that prompt identified as zero-variance in any previous epoch remains zero-variance in the current epoch.
with
The results shown in Figure-4alinde indicate that zero-variance prompts exhibit strong temporal correlations throughout training. We have two key observations: 1) Prompts previously identified zero- variance are likely to remain zero-variance PPrrvvoous Current) curve shows that the majority of vero-variance prompts in a given epoch (e.g., over 90%) were also identified as zero-variance in earlier epochs. 2) Some zero-variance prompts can become effective again in future epochs. CCurrent/PPeeiionn ous) curve shows that approximately 20% of prompts previously labeled as zero- variance become effective prompts that contribute to training again. This suggests that, rather than statically pruning zero-variance prompts, beneficial to retain some degree of exploration helps retain potentially valuable prompts.

---

Dynamic Sampling
GRESO
Training Dynamics
Responses
Epoch
Responses
Prompt
D00-00
Prompt
D00-00
Pobabiity
Prompt
DD0.-00
Prompt
0.5
Prompt
Rallutt
DD0-000
Prompt
CRLSC
Rollout
Prompt
000..000
Prompt
.00..00
6.-
Accamnn: Previnett Afrestout] Cammitt
Prompt
DD0.-00
Prompt
DDD--00
Prompt
-
000-00
Prompt
65
10
Traning Epoch
-tectivn Travinn
Wasted Generation
Htective
Zenowarance
Explontion
(a)
(b)
Figure 4: (a) Temporal correlation of examples across epochs. Prompts previously identified as zero-variance are likely to remain zero-variance. (b) Pipeline comparison between Dynamic Sampling and our GRESO method. Unlike Dynamic Sampling, which filters out zero- varivneee prompts after rollout, GRESO efficiently predicts and filters them based on training dynamics before rollout, which improves rollout efficiency. The probabilistic filtering also allows zero-variance prompts to still be occasionally sampled, enabling the model revisit potentially valuable prompts.
Methodology: GRESO
In this section, building the two observations discussed in Section we design GRESO (GRPO with Efficient Selective Rollout), a novel, online, efficient selective rollout algorithm that predicts and skips pero-variance prompts using reward training dynamics before the rollout stage. The overall algorithm is illustrated in Algorithn
4.1 Detection and Filtering with Reward Training Dynamics
SOTA method 449 selects effective training data by first oversampling and then filtering out zero- variance prompts after rollout, which incurs expensive rollout overhead. Building on our observation in Section 3.3 that zero- vero-nariance prompts tend remain uninformative in future epochs, propose to leverage reward training dynamics to detect and filter these prompts before rollout to save rollout computation (as shown in Figure
More specifically, we formalize the problem zero-variance prompt detection as follows. During training, each prompt x; is associated with training dynamics trace:
T: (ei,1, Ri,), .... (ei,ay Ri,n);
where et, denotes the epoch number of the j-th sampling for example 2,, and R., (r lk=1
represents the set of response rewards obtained in that epoch. The goal of our algorithm is predict whether v; is zero-variance prompt-i.e., one that yields identical rewards for all responses based on its reward dynamics T; prior to rollout.
4.2 Probabilistic Pre-rollout Prompt Filtering
Probabilistic Filtering. To utilize the reward training dynamics, we propose a probabilistic filtering strategy: each prompt is calculated with filtering probability based on its current training dynamics. As observed in Section some zero-variance prompts can become effective again in later epochs. A key advantage of this crobabilistic-basee approach is that it naturally balances exploitation and exploration, allowing zero- variinariance prompts to still be occasionally sampled, rather than being deter- ministically discarded too early. This enables the model to revisit potentially valuable prompts. More specifically, given prompt z; whose training dynamics trace is T, = (Ci,1, R,.))...,,ee Ri,)) calculated the filtering probability by:
(x))) 1- p.:
(3)
5

---

Algorithm 1: Training Iteration in GRESO
Input: Dataset D; Default rollout batch size Bdefault, Training batch size Bú; Probability step size Ap;
Base exploration probability: Peasy, Phardi Targeted zero-variance percentage: Cleasy Otarrd-
B +- 0; B, t- Brefanit, Heasys Rhard, Retntal f- 0, 0, O;
1 Rollout Stage.
2 repeat
{rij + Sample prompts from D and filter with Eq. 3 until batch size B;:
{. B,x Rollout generation on { /=1
[=l BxX
(Is; ri)
i=1 &- filter out zero-var prompt in {z;; r= B.x.
Mency
Measy + filtered easy zero-var prompt count;
Whard
Thard + filtered hard zero-var prompt count;
Thotal
Mhoal B₁
< B U r Bxx
1a Adaptive rollout batch size.
10
B, < min(
(Bdefault, min(RRefauut Adaptive rollout batch size calculated by Eq. 6
11 until [B] 2 B;;
Adjust Base Exploration Probability.
12 if Teacy/ Maay///ouat 2 Oeasy then Peasy f Peasy Ap
else Peasy s Peasy + Ap;
14 if Thard/ Teonal 2 Chard then Phard &- Phard Ap;
15 else Phard 4- Phard" + Ap:
1e GRPO Training.
16 + select B, examples from B:
Update actor model with GRPO on B;
2; = max
E n0.
Is
(4)
1, if all rewards in Rag are identical,
15 00 otherwise,
(5)
where P. is the base exploration probability controlling how likely a prompt is selected for rollout. 2; represents the number of most recent consecutive rollouts for prompt that were zero-variance.
Self-adjustable Base Exploration Probability. One challenge of the above probabilistic filtering algorithm lies in determining the base exploration probability, which can vary across models, datasets, and even different training stages. In addition, different base probabilities may be appropriate for easy and hard zero-variance prompts. Manually selecting the probabilities for all scenarios is impractical.
To address this challenge, GRESO employs an adaptive algorithm that automatically adjusts the base exploration probability at each training iteration (Lines 14-18 in Algorithm [ Rather than requiring users to manually select the base probability, which can vary across different settings, GRESO only requires a target zero-variance percentage. then automatically increases or decreases the exploration rate by step size Ap based whether the observed zero-variance percentage is above or below the target. We set A to 1% in all our evaluations. Additionally, instead of using a single base exploration probability, GRESO maintains two separate values: for easy nero-variance prompts and another for hard ones. When computing the filtering probability py (zt). GRESO first determines whether æ; is an easy or hard zero-variance prompt and then applies the corresponding exploration probability
A daptive Sampling Batch Size. In the current design of Dynamic Sampling [49]. if the number of valid examples insufficient to meet the training batch size requirement, the training performs
"e set the target zero-we variance ratio to 25% for experiments and allocate between easy and hard prompts in an 1 2 ratio (i.e., 8.3% for easy and 16.7% for hard zero-variance prompts), based on the intuition that, as models become more capable during training, more exploration on hard examples can be more beneficial. However, more optimal allocation scheme may exist, which we leave for our future study.

---

rollout using fixed batch size. However, this may result in wasted computation when only a small number of additional examples are needed complete the training batch. To further improve rollout efficiency, GRESO adopts an adaptive rollout batch size:
_BA a1
B, min Briefault
(6)
where Belefault is the default rollout batch size, B& is the number of examples needed to fill the training batch, a is the current vero-variance example ratio in this iteration (as some rollouts have already occurred in this iteration), and [8 is safety factor, which is fixed at 1.25 across all our evaluations, to
ensure sufficient valid examples are collected. We provide an ablation study in Section 5.3 to evaluate the contribution of this adaptive batching mechanism to GRESO's s overall performance.
5 Experiment
In this section, we evaluate GRESO on multiple benchmarks using three different models. The evaluation results show that GRESO achieves comparable performance Dynamic Sampling while significantly reducing rollout and training costs:
In Section 5.2. we show that GRESO reduces up to 8M rollouts and achieves up to 2.4x speedup in rollout and 2.0x speedup in total training time compared to Dynamic Sampling without accuracy degradation.
In Section 5.3 we conduct detailed study on how GRESO reduces training cost with selective rollout, and we also conduct an ablation study on the contribution of GRESO components.
5.1
Experimental Settings
Models & Datasets. We run our experiments on Qwen2.5-Math-I.5B 47), DeepSeek-RI-Disee
Qwen-1.5B (]. and Qwen2.5-Math-77 (7). For --ttn..5-MMt--15B//// models, we use 4096 as the context length, as it is the maximum context length for those two models. For DeepSeek-R2 -1- Distill-Qwen-L5BB we set the context length 8196. For Owee2.55-14B/32B models. we use 16k as the context length. For training datasets, we evaluate our methods on two datasets: 1) DAPO+MATH (DM): We combine the DAPO dataset [9), which contains only integer solutions, with the MATH solutions. We find that training on DAPO alone
dataset [14, which also contains can degrade performance on LaTeX- benchmarks, so we augment with MATH to preserve formatting diversity and improve generalization. 2) OPEN-RI 30k subset (RI): A 30,000-example
subset of the OPEN-RI math dataset [10]
Training& & Evaluation. Our method is implemented based on verl 37 pipeline and uses vLLM 22 for rollout. We use 4xH100 for Qwen2.5-Math-1155 training and 8xH100 for Qwen2.5-Math- and DeepScek-RR -1 - Sei-D-iill--we----ee 1.5B. For benchmark datasets, we use six widely used complex mathematical reasoning benchmarks to evaluate the performance of trained models: Math500 [4
26
AIME24 [1. AMC 22. Minerva Math 23 Gaokao (33. Olympiad Bench [13] Similar to
43 we
evaluate models on those benchmarks every steps and report the performance of the checkpoint that obtains the best average performance on six benchmarks. We also include more detailed experimental
settings in Appendix
5.2 End-to-end Efficiency Comparison
No performance drop with up to 3.35x fewer rollouts. To verify the effectiveness of GRESO, we present a comprehensive evaluation of GRESO and Dynamic Sampling (DS), which filters out zero-variance examples and resamples to fill the batch with effective data, across six math reasoning benchmarks, using three different model settings in Table T The models are trained either the DAPO + MATH dataset (DM) or the Open R1 subset (ORI). We report both the performance and the number of rollouts from the checkpoint that achieves the best average performance across six benchmarks. Across all training settings, GRESO achieves comparable accuracy as DS, while significantly reducing the number of rollout samples achievinn up to 3.35 fewer rollouts. For example, on 1ree2.5--Mth--7 trained on the DM dataset, GRESO achieves a comparable average accuracy DS (57.5% vs. 57.8%), while reducing the number of rollouts from 13.1M to 6.3M. These results demonstrate that GRESO maintains performance while substantially lowering the cost on rollouts. Similar improvements are observed across other evaluation settings.
7

---

Table Performance (%) comparison across six math reasoning benchmarks. We train three models on DAPO + MATH (DM) and the Open RI subset (OR1). Compared to Dynamic Sampling (DS), GRESO achieves similar accuracy while significantly reducing the number of rollouts.
Dataset Method
Math500
AIME24
AMC
Gaokao
Miner.
Olymp.
Avg.
Rollout
Qwen2.5-MMr 1.1.5--Math-1555
DS
77.3
16.7
61.7
64.2
31.8
38.7
48.4
7.6M
GRESO
76.6
15.0
61.4
66.2
33.3
38.5
48.5
3.3M
DS
77.1
16.7
50.3
65.5
30.9
39.7
46.7
3.8M
ORI
GRESO
76.1
20.0
50.6
65.1
30.0
39.2
46.8
1.6M
DeepSeek- RI-Distill Seelll-ween----. I.5B
DS
87.9
36.7
71.7
78.7
35.3
55.9
61.0
2.4M
DM
GRESO
87.7
36.7
71.1
78.4
33,9
55.1
60.5
1.6M
DS GRESO
84.8
25.0
68.4
74.0
34.1
54.2
56.7
2.4M
ORI
85.9
26.7
66.9
75.2
33.6
55.5
57.3
1.5M
@wen2.5-Math-7B
DS GRESO
82.9
34.2
79.2
71.7
35,4
43.6
57.8
13.1M
DM
82.2
32.5
80.7
70.2
35.4
44.1
57.5
6.3M
DS
82.9
34.2
63.1
67.3
34.9
46.3
54.8
11.4M
ORI
GRESO
82.3
35.0
64.5
36.5
45.7
55.1
3.4M
Up to 2.4x wall-clock time speed-up in rollout and 2.0x speed-up in training. To better understand the efficiency of our pro- posed methods, we report the detailed end- to-end training time (1000 steps) breakdown for different stages: rollout generation, ac- tor model update, and other overheads (e.g.. reference model and advantage calculation). -wen2.5-Math-i 1.5B is trained 4xH100 GPUs, while the other two models are trained 8xH100 GPUs. Table compares the training time breakdown between GRESO and Dynamic Sampling for models trained on the DAPO MATH dataset. For all three models, GRESO significantly reduces rollout time time-achieving up to 2.4x speedup in roll- and 2.0x speedup in total training time compared to DS. For instance, Math-7B, GRESO reduces rollout time from 155.9 hours to 65.5 hours, cutting overall training time from 178.0 to 88.3 hours.
Table 2: Training time (hours) breakdown and com- parison for models trained on DAPO + MATH dataset. GRESO consistently lowers rollout cost and achieves up to 2.4x speedup in rollout and 2.0 speedup in total training cost over Dynamic Sampling.
Method Training
Other
Rollout
Total
Qwen2.5 Fren2.5-Maalle 5-Math-1.BB
DS GRESO
8.1
3.6
41.0 (1.x)
(1.00)
3.9
25.2 (1.6x)
37.9 (1.4x)
DeepSeek-RL Baiekk-R---iiiee RRI-Disill-Qwen--ll O.5nl--Qeen-L.B
DS GRESO
6.1
3.3
92.4 (1.0x)
(1.0)
4.0
62.0 (1.5x)
72.7 (1.4x)
Qwen2.5-Math-7
DS GRESO
16.1
6.1
(1.00
Qwen2.5-
16.6
6.3
65.5 (2.4x)
88.3 (2.0x)
Scaling to 14B and 32B models. Besides the models presented in Table [ we further eval- uate GRESO on larger models to better verify its performance at scale. The reported accuracy represents the average performance across the six benchmarks listed in Table I along with AIME25. Both AIME24 and AIME25 are evalu- ated using @vg@ 16 accuracy. Figure 3 compares the accuracy (%) curves of GRESO and DS on Qwen2.5-1444 14B and Qwen2.5-32B, Both GRESO and DS exhibit similar convergence speed with respect to training steps, indicating that GRESO does not compromise model accuracy and con- vergence spped. However, GRESO requires
Qwen-2.5-288
Qwen-2.5-328
Accuracy (96)
35
25
GRESO
GRESO
Steps
Steas
Figure 5: Accuracy (%) curves comparison be- tween GRESO and DS Qwen2.5-14B and Qwen2.5-32B.Bol Both methods exhibit similar con- vergence speed with respect training steps.

---

Effective Prontpt Rabio
Roliur Time per Step
2.30
1.70
1000
1.50
Rollout pat Stop
02-
EReciiv
1.0
0.70
05
0.50
4000
GRESO
GAES0
<2a
400
190
23
2000
Training Step 230
435
Training Step
Thaining Step
(a)
(b)
(c)
(d)
Figure 6: Training dynamics analysis of Qwen-Math-1..5 1.5B trained on the DAPO MATH dataset: (a) Effective prompt ratio in each step. GRESO maintains a consistently higher effective prompt ratio during training. (b) To obtain the same number of effective prompts per batch, GRESO requires less rollout time. (c) GRESO achieves more effective rollouts for training under the same rollout time budget compared Dynamic Sampling. (d) Ablation study on adaptive batch size (ABS) for sampling: Both ABS and GRESO effectively reduce the number of rollouts per training step.
fewer rollouts per training step. For instance, over total of 800 training steps, Dynamic Sam- pling consumes 3.45M rollouts, whereas GRESO requires only 1.74M rollouts.
Performance on coding tasks. Besides math tasks, also verify the effectiivneess of GRESO cod- ing tasks. Table 3 presents a performance compar- ison the lodde_conessta [25] dataset using the DeepSeet-RI--istill--Weee 1.5B model, evaluated
Table 3: Performance (%) comparison on coding tasks. We train Seepseek-RR--iistiee Qwen-1.5B on the code_contest dataset and evaluate on LiveCodeBench.
on LiveCodeBench [8]] GRESO achieves the high-
est accuracy, reaching 22.8%, slightly outperform- ing DS (22.1%) and clearly outperforming GRPO (19.9%). Besides, GRESO accomplishes this with a much lower rollout cost: 1.07M compared 1.9M for DS, indicating that GRESO delivers better performance while using significantly fewer rollouts. These results demonstrate that GRESO significantly improve the training effectiveness without hurt model performance on code generation tasks.
Method
GRPO
DS
GRESO
Acc (avg @4)
19.9
22.1
22.8
# Rollouts
0.5M
1.9M
1.07M
Apply GRESO RLOO. Besides GRPO, GRESO also enhances the efficiency of methods that suffer from the cero-variance rollout issue, like RLOO 200 Table ree- ports the performance comparison by ap- plying GRESO to the RLOO algorithm us- ing the Qwen2.5-Me 5..55--Mhh---- model. Similar to applying GRESO on GRPO, when inte- grated with RLOO, GRESO also achieves comparable accuracy to Dynamic Sam- pling (57.0% v.s. 56.8%), but need much fewer rollouts. The reported accuracy represents the average performance across the six benchmarks used in Table
Table 4: Performance (%) comparison by apply- ing GRESO on RLOO (Qwen2.5-Math-7B) Besides GRPO, GRESO also improve the efficieny of methods suffering from zero- rollout issue.
Method
RLOO
RLOO+GRESO RLOO+DS
Acc (avg@4)
55.1
57.0
56.8
# Rollouts
0.81
1.2M
2.56M
5.3 Analysis and Ablation Study
In this section, we use Qwen-Math-1.51 trained on the DAPO MATH dataset to analyze in detail how GRESO reduces training overhead by enhancing rollout quality, and we also conduct an ablation study on the contribution of each component in GRESO.
GRESO improves effective prompt ratio and rollout efficiency. As shown in Figure 6al compared Dynamic Sampling, where effective prompt ratio steadily decreases during training, since GRESO filter out many zero- variance prompts before rollout, GRESO consistently maintains a significantly higher effective prompt ratio. For instance, as effective prompt ratio drop to around 20% in the late stage of training, GRESO maintains the effective prompt ratio larger than 70%. This higher ratio directly translates into reduced rollout time per training step, as fewer zero-variance prompts are sampled. Figure 6b shows that GRESO has significantly less rollout time per step compared to dynamic sampling. Figure (ec compares the total number of effective rollouts used during training under the same rollout time budget for GRESO and Dynamic Sampling. GRESO consistently

---

generates more effective rollouts over time. For instance, GRESO reaches 2 million effective rollouts around 25 hours, while Dynamic Sampling requires over 40 hours to achieve the same, which demonstrate the efficiency of GRESO.
Dynamics of self-adjustable base exploration
probabilities. A key parameter in GRESO is the base exploration probability P. defined in Equation
Probani Probality
Dey the
S As discussed in Section 4.2 this probability can vary depending on the model, dataset, and training stage. Instead of manu- ally tuning Ps. GRESO employs an adaptive mechanism to automatically adjust it during training. Specifically, GRESO maintains sepa- rate exploration probabilities for hard and easy zero-variance prompts, denoted as Pe,hard and Pe,cay, respectively. In Figure 7al we plot the dynamics of both herdard and pe,casy, along with the ratio of easy and hard zero-variance prompts over time. We observe that after the first training epoch, both exploration probabilities initially decline. However, as the model ability improves, Pe hard begins to increase, enabling more exploration of hard examples during later stages of training. Fig- ure 7b shows the dynamics of easy and hard zero-ver aaro-variance ratios. Unlike Dynamic Sampling, GRESO effectively maintains both ratios close to their target values during training, which demonstrates the effectiveness of its self-adjusting mechanism.
Bove Camorra
1363 21
2/n a0
Twining Step
Thaining Step
(a)
(b)
Figure 7: (a) Dynamies of base exploration prob-
abilities. (b) Dynamics of easy and hard zero-
variance prompt ratio.
Prompt
Selection Dynamics. In Figure 8 we present case study illustrating how GRESO selects or skips prompts over train- ing epochs. Each row stands for a prompt, and each column stands for an epoch. We observe that very easy prompts tend to remain easy throughout training; although frequently skipped, GRESO still occasionally selects them to ensure a minimal level of exploration. For prompts of moderate difficulty, the model becomes stronger over time, these prompts gradually become easier and are increasingly skipped. In contrast, some hard prompts become solvable (i.e., effective prompts) in later epochs even easy prompts. However, certain hard prompts remain unsolved throughout training.
Epoch
Ablation study on adaptive batch size (ABS) for sampling. In addition to the pre-rollout prompt selection algorithm based on training dynamics, another key component of GRESO is the adaptive batch size (ABS) for sampling. When only a small number of effective prompts
Select
5kip (Easy)
Skib (Hardd)
Selection Dynamics of different prompts in GRESO. Each row is a prompt, and each column is an epoch.
Figure 8:
needed to fill the training batch,
ABS enables rollout on a smaller batch instead of using the default large sampling batch size, thereby reducing unnecessary computation. Figure 6d compares the number of rollouts per training step across three methods: Adaptive Batch Size (DS + ABS), and GRESO. DS maintains fixed sampling batch size, leading to consistently high sampling overhead. DS + ABS dynamically adjusts the batch size, reducing the number of samples in earlier steps, but still shows increasing sampling as training progresses and the effective prompt ratio decreases. In contrast, GRESO consistently maintains a much lower number of samples per step due its more selective rollout strategy combined with ABS. resulting in significantly reduced rollout overhead.
Dynamic Sampling (DS), DS with
Conclusion
In this paper, we present GRESO, selective rollout algorithm for LLM RL. GRESO aims to improve RL training efficiency by selecting effective prompts before rollout to save unnecessary overhead on sampling uninformative prompts. GRESO leverages reward dynamics efficiently filter out zero-variance prompts before rollout and significantly improve the RL training efficiency. Our empirical evaluation demonstrates that GRESO significantly improves end-to-end training efficiency, achieving up to 2.4x rollout speedup and 2.0x overall training speedup. We believe that the method and findings in our work can inspire more research on designing more efficient selective rollout algorithms to accelerate RL for LLM.
10

---

References
[1] Art Problem Solving. Aime problems and solutions. httt:://arrroffprrbbmmooo....... index. index.ppp/
ant_aanndddddiinnanaaaa Accessed: 2025-04-20.
[2] Art of Problem Solving, Ame problems and solutions. index. php:tiite=-MC__robblaee
Solutinn
Accessed: 2025-04-20.
[3] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint XXix:2204.00866 2022.
[4] Irina Bejan, Artem Sokolov, and Katja Filippova. Make every example count: On the stability and utility of olffinfluence for learning from noisy nlp datasets, arXiv preprint arXiv:2302.1395 2023.
[5] Lichang Chen, Shiyang Li, Jun Yan, Hai Wang, Kalpa Gunaratna, Vikas Yadav, Zheng Tang, Vijay Srimivasan, Tianyi Zhou, Heng Huang, al. Alpagasus: Training a better alpaca with fewer data. In The Twelfth International Conference on Learning Representations, 2024.
[6] Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. Deep rein- forcement learning from human preferences. Advances in neural information processing systems, 30, 2017.
[7] Josef Dai. Xuehai Pan, Ruiyang Sun. Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, and Yaodong Yang. Safe rlhf: Safe reinforcement learning from human feedback. arXiv preprint arXiv:2310.12773 2023.
[8] Nirjhar Das. Souradip Chakraborty, Aldo Pacchiano, and Sayak Ray Chowdhury. Active preference optimization for sample efficient rihf. arXiv preprint arXiv:2402.10500 2024.
[9] DeepSeek-AL, Daya Guo. Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. Wu, Zhibin Gou, Zhihong Shao, Zhuashu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li. Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu. Haocheng Wang. Honghui Ding. Huajian Xin. Huazuo Gao, Hui Qu, Hui Li. Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang. Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang, Kuai Yu, Lean Wang. Lecong Zhang, Liang Zhao, Litong Wang, Liyue Zhang, Lei Xu, Leyi Xia, Mingehuan Zhang, Minghua Zhang, Minghui Tang, Meng Li. Miaojun Wang, Mingming Li. Ning Tian, Pampan Huang. Peng Zhang. Qiancheng Wang. Qinyu Chen. Qiushi Du, Ruiqi Ge, Ruisong Zhang. Ruizhe Pan, Runji Wang. R. J. Chen, R. L. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang. Shuiping Yu, Shunfeng Zhou, Shuting Pan, S. S. Li, Shuang Zhou, Shaoqing Wu, Shengfeng Ye, Tao Yun, Tian Pei. Tianyu Sun, T. Wang, Wangding Zeng. Wanjia Zhao, Wen Liu, Wenfeng Liang, Wenjun L. Xiao. Wei An, Xiaodong Liu. Xiaohan Wang. Xiaokang Chen,
Gao. Wengin Yu. Wentao Zhang, Xiaotao Nie. Xin Cheng. Xin Liu, Xin Xie, Xingchao Liu, Xinyu Yang. Xinyuan Li, Xuecheng Su, Xuheng Lin, X. Q. Li. Xiangyue Jin. Xiaojin Shen. Xiaosha Chen. Xiaowen Sun, Xiaoxiang Wang. Xinnan Song. Xinyi Zhou, Xianzu Wang, Xinxia Shan, Y. K. Li, Y. Q. Wang. Y. X. Wei, Yang Zhang, Yanhong Xu, Yao Li, Yao Zhao, Yaofeng Sun, Yaohui Wang. Yi Yu, Yichao Zhang, Yifan Shi, Yiliang Xiong, Ying He, Yishi Piao, Yisong Wang, Yixuan Tan, Yiyang Ma, Yiyuan Liu, Yongqiang Guo, Yuan Ou, Yuduan Wang, Yue Gong. Yuheng Zou, Yujia He, Yunfan Xiong. Yuxiang Luo, Yuxiang You, Yuxuan Liu, Yuyang Zhou, Y. X. Zhu, Yanhong Xu, Yanping Huang, Yaohui Li, Yi Zheng, Yuchen Zhu, Yunxian Ma, Ying Tang, Yukun Zha, Yuting Yan, Z. Z. Ren, Zehui Ren, Zhangli Sha. Zhe Fu, Zhean Xu, Zhenda Xie. Zhengyan Zhang, Zhewen Hao, Zhicheng Ma, Zhigang Yan, Zhiyu Wu, Zihui Gu, Zijia Zhu, Zijun Liu, Zilin Li, Ziwei Xie, Ziyang Song, Zizheng Pan, Zhen Huang, Zhipeng Xu, Zhongyu Zhang, and Zhen Zhang. Deepseek rl: Incentivizing reasoning capability in ilms via reinforcement learning. arXiv preprint arXiv:2501. 12948, 2025.
[10] Hugging Face. Open rl: A fully open reproduction of deepseck-rl, January 2025.
[11] Mehdi Fatemi, Banafshch Rafice, Mingjie Tang, and Kartik Talamadupula. Concise reasoning via reinforcement learning. arXiv preprint arXiv:2504.05146 2025.
[12] Jiaxuan Gao, Shusheng Xu, Wenjie Ye, Weilin I Liu, Chuyi He, Wei Fu, Zhiyu Mei, Guangju Wang, and Yi Wu. On designing effective rl reward at training time for Ilm reasoning. arXiv preprint 251.::2410.15115. 2024.
[13] Chaoqun He, Renjie Luo, Yuzhuo Bai. Shengding Hu, Zhen Leng Thai, Junhao Shen, Jinyi Hu, Xu Han, Yujie Huang. Yuxiang Zhang, et al. Olympiadbench: A challenging benchmark for promoting agi with olympiad- -evel level bilingual multimodal scientific problems. arXiv preprint 2xx::24022.1008 2024.
11

---

[14] Dan Hendrycks. Collin Burns, Saurav Kadavath, Akul Arora. Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. NeurlPS, 2021.
[15] Jian Hu. Reinforce++: A simple and efficient approach for aligning large language models. arXiv preprint arXiv:2501.0322 032.22,,...0.,..... 2025.
[16] Hamish Ivison, Noah A Smith, Hannanch Hajishirzi, and Pradeep Dasigi. Data-efficient finetuning using cross-task nearest neighbors. In Findings of the Association for Computational Linguistics: ACL2023, pages 9036-9061, 2023.
[17] Hamish Ivison, Muru Zhang, Faeze Brahman, Pang Wei Koh, and Pradeep Dasigi. Large-scale data selection for instruction tuning. arXiv preprini vrXiv:2503.01807 2025.
[18] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang. Sida Wang, Armando Solar- Lezama, Koushik Sen, and lon Stoica. Livecodebench: Holistic and contamination free evaluation of large language models for code. arXiv preprint arXiv:2403.07972 2024.
[19] Amirhossein Kazemnejad, Milad Aghajohari, Eva Portelance, Alessandro Sordoni, Siva Reddy, Aaron Courville, and Nicolas Le Roux. Vineppo: Unlocking potential for Ilm reasoning through refined credit assignment. arXiv preprint arXiv:2410.01679, 2024.
[20] Wouter Kool, Herke van Hoof, and Max Welling. Buy 4 reinforce samples, get baseline for free! 2019.
[21] Byung-Jae Kwak, Nah-Oak Song. and Leonard E. Miller. Performance analysis of exponential backoff. IEFF/ACM transactions on networking, 13(2):343-355, 2005.
[22] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez. Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, pages 611-626, 2023.
[23] Aitor Lewkowyez, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Systems, 35:3843-3857, 2022.
[24] Xuefeng Li, Haoyang Zou, and Pengfei Liu. arXiv:2502. 18886. 2025.
Limr: Less is more for rl scaling. arXiv preprint
[25] Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwesser, Rémi Leblond, Tom Ec- cles, James Keeling. Felix Gimeno, Agustin Dal Lago, Thomas Hubert, Peter Choy, Cyprien de Mas- son d' Autume, Igor Babuschkin, Xinyun Chen, Po-Sen Huang, Johannes Welbl, Sven Gowal, Alexey Cherepanov, James Molloy, Daniel Mankowitz. Esme Sutherland Robson. Pushmeet Kohli, Nando de Freitas, Koray Kavukcuoglu, and Oriol Vinyals. Competition-level code generation with alphacode. Science, 37866624)10022-1997 2022.
[26] Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. arXiv preprint arXiv:2305.2003: 2023.
[27] Zichen Liu, Changyu Chen, Wenjun Li, Penghui Qi, Tianyu Pang, Chao Du, Wee Sun Lee, and Min Lin. Understanding rl -zzro-likke training: A critical perspective. arXiv preprint arXiv:2503. 207783,, 2025.
[28] Zijun Liu, Boqun Kou, Peng Li, Ming Yan, Zhang, Fei Huang, and Yang Liu. Enabling weak 1lms to judge response reliability via meta ranking. arXiv preprint arXiv: 1214,,1116, 2024.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations, 2019.
[30] Michael Luo, Sijun Tan, Roy Huang, Xiaoxiang Shi, Rachel Xin, Colin Cai, Ameen Patel, Alpay Ariyak, Qingyang Wu, Ce Zhang, et al. Deepcoder: A fully open-source 14b coder at o3-mini level, 2025.
[31] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Fei-Fei, Hannanch Hajishirzi, Luke Zetllemoeer Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. s1: Simple test-time scaling. arXiv preprint 293.93300..1393 2025. 2025.
[32] William Muldrew, Peter Hayes, Mingtian Zhang, and David Barber. Active preference learning for large language models. In Proceedings of the 41st International Conference on Machine Learning, pages 16577-36594 2024.
12

---

[33] Michael Noukhoviteh, Shengyi Huang, Sophie Xhonneux, Arian Hosseini, Rishabh Agarwal, and Aaron Courville. Asynchronous rlhf: Faster and more efficient off-policy rl for language models. arXiv preprint urXiv:2410.18255 18252, 2024.
[34] OpenAI, :, Aaron Jaech, Adam Kalai, Adam Lerer, Adam Richardson. Ahmed EI- El-Kishky, Aiden Low, Alec Helyar, Aleksander Madry, Alex Beutel. Alex Carney, Alex Iftimie. Alex Karpenko, Alex Tachard Passos, Alexander Neitz, Alexander Prokofiev, Alexander Wei, Allison Tam, Ally Bennett, Ananya Kumar, Andre Saraiva, Andrea Vallone, Andrew Duberstein, Andrew Kondrich, Andrey Mishchenko, Andy Applebaum, Angela Jiang, Ashvin Nair, Barret Zoph, Behrooz Ghorbani, Ben Rossen, Benjamin Sokolowsky, Boaz Barak, Bob McGrew, Borys Minaiev, Botao Hao, Bowen Baker, Brandon Houghton, Brandon McKinzie, Brydon Eastman, Camillo Lugaresi, Cary Bassin, Cary Hudson, Chak Ming Li, Charles de Bourcy. Chelsea Voss, Chen Shen, Chong Zhang. Chris Koch, Chris Orsinger, Christopher Hesse, Claudia Fischer, Clive Chan, Dan Roberts, Daniel Kappler, Daniel Levy, Daniel Selsam, David Dohan, David Farhi, David Mely, David Robinson, Dimitris Tsipras, Doug Li, Dragos Oprica. Eben Freeman. Eddie Zhang, Edmund Wong, Elizabeth Prochl. Enoch Cheung. Eric Mitchell. Eric Wallace. Erik Ritter, Evan Mays, Fan Wang. Felipe Petroski Such, Filippo Raso, Florencia Leoni, Foivos Tsimpourlas. Francis Song. Fred von Lohmann, Freddie Sulit, Gcoff Salmon, Giambattista Parascandolo, Gildas Chabot, Grace Zhao, Greg Brockman, Guillaume Leclere, Hadi Salman, Haiming Bao, Hao Sheng. Hart Andrin, Hcssam Bagherinezhad, Hongyu Ren, Hunter Lightman, Hyung Won Chung. Ian Kivlichan, Ian O'Connell, Ian Oshand, Ignasi Clavera Gilaberte, Ilge Akkaya, Ilya Kostrikov, Ilya Sutskever. Irina Kofman, Jakub Pachocki, James Lennon, Jason Wei, Jean Harb, Jerry Twore, Jiacheng Feng, Jiahui Yu, Jiayi Weng, Jie Tang. Jieqi Yu, Joaquin Quiñoonero Candela, Joe Palermo, Joel Parish, Johannes Heidecke, John Hallman, John Rizzo, Jonathan Gordon, Jonathan Uesato, Jonathan Ward, Joost Huizinga, Julie Wang, Kai Chen, Kai Xiao, Karan Singhal, Karina Nguyen, Karl Cobbe, Katy Shi, Kayla Wood, Kendra Rimbach, Keren Gu-Lemberg, Kevin Liu, Kevin Lu, Kevin Stone, Kevin Yu, Lama Ahmad, Lauren Yang. Leo Liu, Leon Maksin, Leyton Ho, Liam Fedus, Lilian Weng. Linden Li, Lindsay McCallum, Lindsey Held, Lorenz Kuhn, Lukas Kondraciuk, Lukasz Kaiser, Luke Metz, Madelaine Boyd, Maja Trebacz, Manas Joglekar, Mark Chen, Marko Tintor, Mason Meyer, Matt Jones, Matt Kaufer, Max Schwarzer, Meghan Shah, Mehmet Yatbaz, Melody Y. Guan, Mengyuan Xu, Mengyuan Yan, Mia Glaese. Mianna Chen, Michael Lampe. Michael Malek. Michele Wang. Michelle Fradin, Mike McClay, Mikhail Pavlov, Miles Wang, Mingxuan Wang, Mira Murati, Mo Bavarian, Mostafa Rohaninejad, Nat McAleese, Neil Chowdhury, Neil Chowdhury, Nick Ryder, Nikolas Tezak, Noam Brown, Ofr Nachum, Oleg Boiko, Oleg Murk, Olivia Watkins, Patrick Chao, Paul Ashbourne, Pavel Izmailov, Peter Zhokhov, Rachel Dias, Rahul Arora, Randall Lin, Rapha Gontijo Lopes, Raz Gaon, Reah Miyara, Reimar Leike, Remny Hwang, Rhythm Garg, Robin Brown, Roshan James, Rui Shu, Ryan Cheu, Ryan Greene, Saachi Jain, Sam Altman, Sam Toizer, Sam Toyer. Samuel Miserendino, Sandhini Agarwal, Santiago Hernandez, Sasha Baker, Scott McKinney, Scottie Yan, Shengjia Zhao, Shengli Hu, Shibani Santurkar, Shraman Ray Chaudhuri. Shuyuan Zhang, Siyuan Fu. Spencer Papay. Steph Lin. Suchir Balaji, Suvansh Sanjeev, Szymon Sidor, Tal Broda, Aidan Clark, Tao Wang. Taylor Gordon, Ted Sanders, Tejal Patwardhan, Thibault Sottiaux, Thomas Degry, Thomas Dimson, Tianhao Zheng, Timur Garipov, Tom Stasi, Trapit Bansal, Trevor Creech, Troy Peterson, Tyna Eloundou. Valerie Qi. Vineet Kosaraju, Vinnie Monaco, Vitchyr Pong, Vlad Fomenko, Weiyi Zheng, Wenda Zhou, Wes McCabe, Wojciech Zaremba, Yann Dubois, Yinghai Lu, Yining Chen, Young Cha, Yu Bai, Yuchen He, Yuchen Zhang, Yunyun Wang. Zheng Shao, and Zhuohan Li. Openai 01 system card, 2024.
[35] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[36] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song. Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402. 03300000000000 2024.
[37] Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, and Chuan Wu. Hybridflow: A flexible and efficient rihf framework. arXiv preprint arXiv: 19256. 2024.
[38] Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. Salmon: Self- alignmettt with instructable reward models. In International Conference Learning Representations, 2024.
[39] Zhiqing Sun, Yikang Shen, Qinhong Zhou, Hongxin Zhang, Zhenfang Chen, David Cox, Yiming Yang. and Chuang Gan. Principle-dris drivel self aligmmenttt of language models from scratch with minimal human supervision. Advances in Neural Information Processing Systems, 36:2511-2566 2023.
[40] Kimi Team, Angang Du, Bofei Gao, Bowei Xing, Changjiu Jiang, Cheng Chen, Cheng Li, Chenjun Xiao, Chenzhuang Du Chonghua Liao, et al. Kimi kl. 5: Scaling reinforcement learning with Ilms. arXiv preprint arXiv:2501.12599 2025.
13

---

[41] Mariya Toneva, Alessandro Sordoni, Remi Tachet des Combes. Adam Trischler, Yoshua Bengio, and Geoffrey J Gordon. An empirical study of example forgetting during deep neural network learning. In International Conference on Learning Representations. 2018.
[42] Leandro von Werra, Younes Belkada, Lewis Tunstall. Edward Beeching, Tristan Thrush. Nathan Lambert, Shengyi Huang. Kashif Rasul, and Quentin Gallouédec. Trl: Transformer reinforcement learning. https
//github. //github..om//uuggiine www.woogg//onneeeeeoeeeeooooeeee ace/trl 2020.
[43] Yiping Wang. Qing Yang. Zhiyuan Zeng. Liliang Ren, Lucas Liu, Baolin Peng. Hao Cheng, Xuchai He. Kuan Wang. Jianfeng Gao, et al. Reinforcement learning for reasoning in large language models with one training example. arXiv preprint arXiv:2504.2 20571, 2025.
[44] Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: selecting influential data for targeted instruction tuning. In Proceedings of the 41st International Conference on Machine Learning, pages 54 5410--54132, 2024.
[45] Wei Xiong, Jiarui Yao, Yuhui Xu, Bo Pang. Lei Wang, Doyen Sahoo, Junnan Li, Nan Jiang, Tong Zhang, Caiming Xiong, et al. A minimalist approach to Ilm reasoning: from rejection sampling to reinforce. arXiv preprint 2504 / 1343, 2025.
[46] Yixuan Even Xu, Yash Savani, Fei Fang, and Zico Kolter. Not all rollouts are useful: Down-sampling rollouts in IIm reinforcement learning. arXiv preprint arXiv:2504. 13818, 2025.
[47] An Yang, Beichen Zhang. Binyuan Hui, Bofei Gao, Bowen Yu, Chengpeng Li. Dayiheng Liu, Jianhong Tu, Jingren Zhou, Junyang Lin, et al. Qwen2. 5-math technical report: Toward mathematical expert model via self-improvement. arXiv preprint arXiv:2409. 222./22200.22122, 2024.
[48] Yixin Ye, Zhen Huang, Yang Xiao, Ethan Chern. Shijie Xia, and Pengfei Liu. Limo: Less is more for reasoning. arXiv preprint arXiv:2502.03387, 2025.
[49] Qiying Yu, Zheng Zhang, Ruofei Zhu, Yufeng Yuan, Xiaochen Zuo, Yu Yue, Tiantian Fan, Gaohong Liu, Lingjun Liu, Xin Liu, et al. Dapo: An open-source IIm reinforcement learning system at scale, arXiv preprint or Xiv:2503. 14476, 2025.
[50] Yufeng Yuan, Qiying Yu, Xiaochen Zuo, Ruofei Zhu, Wenyuan Xu, Jiaze Chen, Chengyi Wang. TianTian Fan, Zhengyin Du, Xiangpeng Wei, al. Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks. arXiv preprint arXiv:2504.051/88 2025.
[51] Yufeng Yuan, Yu Yue, Ruofei Zhu, Tiantian Fan, and Lin Yan. What's behind ppo's collapse in long-cot? value optimization holds the secret. arXiv preprint arXiv:2503.00 014491 2025.
[52] Xiaojiang Zhang, Jinghui Wang, Zifci Cheng, Wenhao Zhuang, Zheng Lin, Minglei Zhang, Shaojie Wang. Yinghan Cui, Chao Wang. Junyi Peng, Shimiao Jiang, Shiqi Kuang, Shouyu Yin, Chaohang Wen, Haotian Zhang. Bin Chen, and Bing Yu. Srpo: A cross dromi- implementation of large-scale reinforcement learning on Ilm, 2025.
[53] Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying. Liang He, and Xipeng Qiu. Evaluating the performance of large language models gaokao benchmark. arXiv preprint 30X.v:2305.1247, 2023.
[54] Yanli Zhao, Andrew Gu. Rohan Varma, Liang Luo, Chien-Chin Huang. Min Xu, Less Wright, Hamid Shojanazeri, Myle Ott, Sam Shleifer, et al. Pytorch Isdp: experiences on scaling fully sharded data parallel. arXiv preprint arXiv:2304.1 112:22304.11277. 2023.
[55] Haizhong Zheng. Bridging Data and Hardware for Efficient Machine Learning Model Scaling. PhD thesis, 2024.
[56] Haizhong Zheng, Rui Liu, Fan Lai, and Atul Prakash. Coverage-centric coreset selection for high pruning rates. In 11th International Conference on Learning Representations, ICLR 2023, 2023.
[57] Haizhong Zheng, Elisa Tsai, Yifu Lu, Jiachen Sun, Brian R Bartoldson, Bhavya Kailkhura, and Atul Prakash. Elfs: Label-free coreset selection with proxy training dynamics. The Thirteenth International Conference on Learning Representations, 2025.
[58] Yinmin Zhong, Zili Zhang, Bingyang Wu, Shengyu Liu, Yukun Chen, Changyi Wan, Hanpeng Hu, Lei Xia, Ranchen Ming, Yibo Zhu, et al. Optimizing (RLHF) training for large language models with stage fusion. In 22nd USENIX Symposium on Networked Systems Design and Implementation (NSDI 25), pages 489-503, 2025.
[59] Tianyi Zhou, Shengjie Wang, and Jeff Bilmes. Curricuuum learning by optimizing learning dynamics. In International Conference on Artificial Intelligence and Statistics, pages 433-441, PMLR, 2021.
14

---

Acknowledgment
LLNL-afiiliate authors were supported under Contract DE-AC52-07NA27344 and supported by the LLNL-LDRD Program under Project Nos. 24-ERD-010 and 24-ERD-058. This manuscript has been authored by Lawrence Livermore National Security, LLC, under Contract No. DE-AC52-07NA27344 with the U.S. Department of Energy. The United States Govermment retains, and the publisher, by accepting the article for publication, acknowledges that the United States Government retains non non-exclusive, paid-up up, irrevocable, world-wide license to publish or reproduce the published form of this manuseript, or allow others do so, for United States Government purposes.
A Overview
We begin in Section B by discussing the limitations of our method. Section CisionCCio highlighhs the broader societal and practical impact of improving rollout efficiency for LLM training. Section [ details our experimental setup, and Section G presents additional empirical experiments and analysis.
B Limitations
While GRESO effectively filters out the most obvious zero-variance training prompts- those that contribute no learning signal to the model, it does not estimate or rank the value of the remaining prompts, which can also contain uninformative prompts that provide limited contribution to training. A potential future work for GRESO is extend its filtering mechanism beyond binary decisions by incorporating a finer-graines scoring or ranking system to prioritize prompts based on their estimated training utility. Despite that, view GRESO an important first step toward such advanced data selection algorithm for efficient rollout and believe it provides a solid foundation for more adaptive and efficient reinforcement learning in LLM training.
C Broader Impact
This work enhances the efficiency and scalability of RL-based fine-tuning for language models by introducing lightweight, selective rollout mechanism that filters out uninformative prompts. By significantly reducing redundant computation, our method lowers overall training costs. This makes easier for institutions with limited computational budgets to train strong models, helping democratize access to advanced AI. Furthermore, our approach promotes more sustainable and resource-efficient practices, encouraging future research toward greener and more inclusive large-scale training.
D Discussion
While similar temporal correlation principles have been explored in areas such as curriculum learning, how model and leverage temporal signals to detect and filter zero-variance prompts prior to rollout remains underexplored and non-trivial. For instance, unlike prior works 59 that operate on static datasets, RLVR involves online data generation, where the training data can vary significantly across iterations, making it non-trivial to apply existing approaches directly. To the best of our knowledge, our paper is the first propose a practical and effective solution filter out zero-variance prompts in RLVR, by tackling the following key challenges: 1) How to detect and filter uninformative prompts before performing rollouts (i.e., without access the full training data)? 2) How to perform such detection and filtering with minimal overhead? 3) How to balance exploration and efficiency, ensuring that potentially useful prompts can be revisited in the future training stage? Each these challenges is non-trivial in the RLVR setting. Our proposed method, GRESO, offers lightweight mechanism that effectively addresses all three and demonstrates strong empirical performance across multiple benchmarks and RI. algorithms.
E Reproductivity
introduced our detailed experimental setting Section and we also include our code in
https:/ htte::://tttttt http://gggghbbb.om///iiinn/ AI AIE--A/GE-LAE///////
15

---

F Detailed Experimental Setting
Models & Datasets. We run our experiments Qwen2.5-Math-1.BB 471.
DeepSeek-RI-Diss
Qwen- 5B [9,, and Qwen2.5- 5-Mthh- [47]. For Qwen2.5-Math- -at---tht-----BBBBB models, we use 4096 as the context length, it is the maximum context length for those models. For DeepSeek- R1--tstilll Qwen-1.5B, we set the context length to 8196. For training datasets, we train our methods on two datasets in two settings: 1) DAPO+MATIE (DM); We combine the DAPO dataset [49], which contains only integer solutions, with the MATH dataset [4], which also contains formatted solutions. We find that training on DAPO alone can degrade performance
benchmarks, we
augment it with MATH to preserve formatting diversity and improve generalization. 2) OPEN-RI 30k subset (RI): A 30,000-example subset of the OPEN-RI math dataset [T0]
Training. Our method is implemented based on verl [37 pipeline and uses vLLM 22 for rollout. We use 4xH100 for Owen2.5-Math-15. -Distill-Qwen--.. 1.5B. We set the rollout temperature to 1 for vLLM [22]. The training batch size is set to 256, and the mini-batch size to 512. We sample 8 responses per prompt. We set the default rollout sampling batch size as 384. For DeepSeek-Re ---is---------i--------------- 1.5., we set the context length to 8196. For SWven2.5-44/322 and DeepSeek- -Ineee-----B-e ---Distill-Qweat -wel----eenn555 for coding tasks, context length to 16k. The training batch size is set to 128, and the mini-batch size to 512. We also sample responses per prompt. We set the default rollout sampling batch size as 192. We train all models for 1000 steps, and we optimize the actor model using the AdamW W [29] optimizer with constant learning rate of le-6. We use 31 0.9, 32 0.999, and apply a weight decay of 0.01. We use the following question template to prompt the LLM. For reward assignment, we give score of 0.1 for successfully extracting an answer and a score of 1.0 if the extracted answer correct. Similar to 49 we remove the KL-divergence term. The optimization is performed on the parameters of the actor module wrapped with Fully Sharded Data Parallel (FSDP) [54] for efficient distributed training. We use 4 H100 for Swen2.5-Math- .5B training and 8 H100 for Qwen2.5-Mathh -1- DeDisill-Qwen--... Qwen- (as has a longer context length.) We set the targeted zero-variance percentage to 25% for all experiments and allocate it between easy and hard prompts in an 1 : 2 ratio (i.e., 8.3% for easy and 16.7% for hard zero-variance prompts), based on the intuition that, as models become more capable during training, more exploration on hard examples can be more beneficial. However, a more optimal allocation scheme may exist, which we leave for future study, We set the initial exploration probability to 50% and base exploration probability adjustment step size Ap for base exploration probability to 1 %. We also set a minimal base exploration probability to 5% to ensure minimal level of exploration
training and 8xH100 for Qwen2.5-Math-75 and DeepSeek-
set the
and
vero-variance prompts throughout training.
GRESO with Fixed Parameters Across All Experiments. Although GRESO introduces a few hyperparameters, we argue that hyperparameter tuning is not major concern in practice. We designed GRESO (e.g., self-adjustable base exploration probability) to be robust under default settings and conducted all experiments using single fixed set of hyperparameters across models and tasks. The consistent performance observed across different models and tasks demonstrates that GRESO does not rely extensive hyperparameter tuning, making it both practical and easy to integrate into existing RL fine-tuning pipelines.
Evaluation. For benchmark datasets, we use six widely used complex mathematical reasoning benchmarks evaluate the performance of trained models: Math500 14 26). AIME24 M. AMC (2). Minerva Math (23 Gaokao [53] Olympiad Bench [13. Same as the training setting, For Qwen2.5- Math-1.5B/7 models, we use 4096 as the context length. For DeepSeek-Ri- R1-tttilll Sirill--Qen--Il 1.5B, we set the context length to 8196. Similar to 43), we evaluate models on those benchmarks every 50 steps and report the performance of the checkpoint that obtains the best average performance on six benchmarks. We evaluate all models with temperature 1 and repeat the test set 4 times for evaluation stability, i.e., pass@I (avg@44, for all benchmarks.
Question Template
Please solve the following math problem: | Question Description J. The assistant first thinks about the reasoning process step by step and then provides the user with the answer. Return the final answer in \boxed{) } tags, for example \boxed(1) 1). Let's solve this step by step.
16

---

G Additional Experiments
G.1 Impact of Targeted Zero-variance Percentage
In this section, we study how varying the targeted varoinarincce percentage impacts training and rollout efficiency. In addition to the default setting of 25% used throughout our experiments, we also evaluate alternative values of 0, 50%, 100% (i.e., always allow exploration). As shown Table different zero-variance targets give nearly identical accuracy. We also present the number of rollouts per step in Figure 9 When we reduce the
7909
Rolleut per Step
6009
25%
100%
5303
targeted zero- variance ratio to O, observe that the number of rollouts per step remains similar to that the 25% setting. be attributed to two factors. First, we
1232
2002
This lack of difference enforce a minimum exploration rate of 5%, which ensures that some exploration still occurs. As result, the actual vero-variance percentage never truly reaches O. Second, we always oversample some data in the first batch of rollouts in each iteration provide some redundancy to avoid the second batch of rollouts. With this setting, as long the first batch generates enough effective training data to fill the training batch, regardless of whether the target is or 25%, the total number of rollouts remains approximately the same. In addition, as the targeted zero-variance percentage increases, more zero-variance prompts are allowed during rollout, leading to a higher number of rollouts per step. When the targeted percentage becomes sufficiently large, GRESO gradually approaches the behavior of dynamic sampling with adaptive rollout batch size.
Training Stop
Figure 9: Comparison of the num- ber of rollouts across different tar- get zero- zero- variance ratios.
Table 5: Average accuracy across six math reasoning benchmarks under different targeted zero- variance percentages.
Target (%)
25
50
100
Acc. (%)
48.1
48.5
48.5
48.4
G.2 Alternative Design: Linear Backoff
In addition to the probabilistic filtering approach introduced in Section 4.2 of the main paper, we also explored an alternative solution for filtering zero- veroinariance prompts during the early stages of this project. One such method is the backoff algorithm (21 (e.g., linear backoff). Specifically, if prompt is identified as zero-variance in the most recent rollouts, is skipped for the
Linear Backoff
Zero-Vrriance Ratio
0.4
0.3
next k training epochs. However, there are several limitations to this approach. As discussed in Section of the paper, the degree
of cxploration should adapt to the model, dataset, and training stage. The linear backoff algorithm schedules the next rollout for zero-vaze variinne prompt & epochs into the future. As result, we wish to adjust the exploration intensity dynamically based on new observations evolving training dynamies, the backoff algorithm cannot directly affect prompts that have already been deferred future epochs. For instance, shown in Figure [00 unlike probabilistic filtering, filtering based on linear backoff' can cause periodic fluctuations in zero- variance prompt ratio, which differs from the smoother dynamics enabled by probabilistic filtering This lack of flexibility limits its ability to adapt exploration strategies in fine-grained or responsive manner, which motivated the design of our current GRESO algorithm based on probabilistic filtering.
100
200
250
300
Training Stop
Figure 10: Zero-variance prompt ratio dynamic for linear backoff.
G.3
Case study of Filtered Examples
To better understand the behavioral patterns of our selective filtering algorithm, we present case study of prompts that were frequently skipped selected during training from the MATH [14 dataset. We
17

---

categorize the examples into three groups: Frequently Skipped Prompts (Easy), Frequently Skipped Prompts (Hard), Frequently Selected Prompts. We observe that frequently skipped easy prompts often involve straightforward calculations routine applications of formulas, making them more likely to solved across all sampled responses. Frequently selected prompts tend to exhibit moderate difficulty, contributing more consistently to model improvement. As for frequently skipped hard prompts, these problems are too challenging for the model solve, even across multiple rollouts, resulting in zero variance among the rewards and ultimately failing to contribute to training.
Frequently Skipped Prompts (Easy)
1. Question: Johnny has 7 different colored marbles in his bag. In how many ways can he choose three different marbles from his bag to play a game? Solution: 35.
2. Question: The number n is prime number between 20 and 30. If you divide by 8, the remainder What value of n?? Solution: 29.
3. Question: Evaluate: 10
10-2.50
Solution: 10.
4. Question: The Ponde family's Powerjet pumps 420 gallons of water per hour. At this rate,
how many gallons of water will it pump in 45 minutes? Solution: 315.
5. Question: Suppose that n, n n + 3, + are five consecutive integers. Determine
a simplified expression for the sum of these five consecutive integers. Solution: 5n + 10.
Frequently Skipped Prompts (Hard)
1. Question: A parabola and an ellipse share focus, and the directrix of the parabola is the line containing the minor axis of the ellipse. The parabola and ellipse intersect two points. Given that the equation of the ellipse is x² + yf = 1, find the distance between those two g
25
points. Solution: 4v14
3
2. Question: In triangle ABC, AB AC==A= 100, and BC = 56. Circle P has radius 16 and is tangent to AC and BC. Circle Q is externally tangent to P and is tangent to AB and BC. No point of circle Q lies outside of AABC. The radius of circle Q can be expressed in the form m nvk, where m, n, and are positive integers and is the product of distinct primes. Find + nk. Solution: 254.
Question: Let EFGH, EFDC, and EHBC be three adjacent square faces of cube, for which EC = 8, and let A be the eighth vertex of the cube. Let L, J, and K, be the points on EF, EH, and EC, respectively, so that EI = EJ = EK = 2. A solid S is obtained by drilling a tunnel through the cube. The sides of the tunnel are planes parallel to AE, and containing the edges TJ, JK, and KT. The surface area of S, including the walls of the tunnel, is m + n /p, where m, n, and are positive integers and p is not divisible by the square of any prime. Find m + n + p. Solution: 417.
4. Question: Let a and 6 be nonnegative real numbers such that
sin(ar + b) = sin 29r
for all integers x. Find the smallest possible value of a. Solution: 10x 29.
5. Question: Four people sit around circular table, and each person will roll standard six-sided die. What is the probability that no two people sitting next each other will roll the same number after they each roll the die once? Express your answer as a common fraction. Solution: 72*
18

---

Frequently Selected Prompts
Question: Let x, M, and = three positive real numbers whose sum is no one of these numbers is more than twice any other, then find the minimum value of the product .y2. Solution: 32
2. Question: The number
T5i/60 te 17xi/60 + ee 27x1/60 + E 37mi/60
+
47mi/60
9F
expressed in the form reth where 0 2m. Find 0. Solution:
20
Question: For what values of x is
0 10x² + 25.r
8 x
nonnegative? Answer interval. Solution: 20.
Question: Determine all real numbers a such that the inequality | + 2ax + 3a S has exactly one solution in Solution: 1,
Question: By starting with a million and alternatively dividing by and multiplying by Anisha created a sequence of integers that starts 1000000, 500000, 2500000, 1250000, and so on. What is the last integer in her sequence? Express your answer in the form a", where a and b are positive integers and is small as possible. Solution: 512.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract and introduction clearly state the contributions of this work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We discuss the limitations of the work in the Appendix. Here summary: GRESO only filters out the most obvious zero-variance training prompts that contribute nothing training, but does not discuss how estimate the value of other prompts. However, we believe GRESO is an important first step for improving the efficiency of rollout scaling in RL for LLMs.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
20

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
No theorem or proof is included in the paper.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes].
Justification:
The paper has disclosed all the information in the method and experiment part.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open- open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
21

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: (Yes]:
Justification: We include our code in the supplementary material and plan to publicize the code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nips. wwtpcc/pccppcpcceeeooooooop httpic/gggudeee/ddSSuuubbiiiii seriissssoooooiii
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes].
Justification: The experimental setting is clearly described in the experiment section and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:
[Yes]
Justification: All evaluation repeated four times to calculate the average accuracy. We also evaluate method various settings to verify our conclusion.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes].
Justification: The compute resource information included in the experimental setting section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes].
Justification: The research conducted in the paper fully conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes].
Justification: We discuss broader impacts our work in the Appendix. Here summary: This paper advances the efficiency and scalability of reinforcement learning for language model fine-tuning by proposing a lightweight, selective rollout mechanism. By reducing redundant computation uninformative prompts, approach helps lower
23

---

energy consumption and resource barriers for training bigh-performing models, particularly for institutions with limited computational resources.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), fairness considerations (c.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA].
Justification: Our paper does not introduce any assets that have a high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use usel-use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes].
Justification: We have explicitly mentioned the citations for the datasets and have ensured that all conditions are fully respected.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
24

---

The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g.. website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code is anonymized and provided the supplementary materials. Docu- mentation includes basic usage and setup instructions.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA].
Justification:
We do not include any experiments with human subjects or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This work does not involve human subjects or crowdworkers.
Guidelines:
25

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
We only use LLM to polish paper writing.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy hutts:://earrpps.cc//ooeeeeeeee for what should or should not be described.
26


Reviews:
[{"review_id": "kHriqzDM1O", "reviewer": "Reviewer_3M88", "summary": "This paper introduces GRESO, a lightweight rollout filtering strategy for efficient reinforcement learning (RL) on large language models (LLMs). GRESO dynamically adjusts each prompt’s sampling probability to limit the frequency of zero-variance rollouts and thereby conserve consumption, all with few hyperparameters to tune. On mathematical reasoning tasks, it achieves substantial reductions in rollout cost without performance loss. Its key innovation lies in an automatic control of sampling probability to maintain a preset zero-variance target by monitoring the sampling history, ensuring both exploration and efficiency.", "questions": "Please see weaknesses.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths：\n\n1. The paper addresses an important challenge of rollout efficiency in RL on LLMs with a simple yet effective approach. \n2. The design of the dynamic control of exploration probability is insightful and well motivated. \n3. The manuscript is well writen and easy to follow, and the purpose of each component is convincingly justified. \n\nWeaknesses：\n\n1. The experiments on DeepSeek-R1-Distill-Qwen-1.5B seems problematic: an 8192 context length is clearly too short and likely incurs a high truncation rate during training. The authors should either include a more robust setting in the main result or explain their choice. \n2. Although the zero-variance target is chosen intuitively and leave the discussion to future work, a thorough ablation on this parameter would be highly informative and would significantly strengthen the paper. \n3. The ablation study of adaptive batch size is not entirely convincing. If the comparison is based on rollouts per step, then the optimal batch size would be 1 per GPU, which is obviously unreasonable. A direct comparison of actual rollout time savings with ABS should be provided. \n4. Several issues in the empirical results need further clarification: \n4.1 In Figure 6, the zero-variance ratio of easy problems fails to converge to the target, and the corresponding base exploration probability is bounded excessively high. This may result from too large a Δp; if so, the method should be revised to handle this case. \n4.2 In Figure 5(d), the rollout per step for GRESO shows large spikes after step 600, which seems abnormal given the continuous adaptation of batch size.\n4.3 The DS/GRESO rollout ratio of DM-Qwen2.5-1.5B is smaller than in Figure 5 (a). Does this indicate a performance drop when training steps are fixed? Including a step-to-accuracy curve would help address this concern.", "was_revised": false, "final_justification": "", "rebuttal": "## Response to Reviewer 1 (3M88)\n\nThank you very much for your thoughtful review and constructive suggestions. We are glad that the reviewer found our work addresses an important challenge in RL for LLMs, and the dynamic control of exploration probability is insightful. We appreciate the opportunity to address the points you have raised. \n\n> **Q1:**  The context of Deepseek model is 8k, which can introduce a high truncation rate.\n\n**A1:**  Thank you for pointing this out. We are currently running the 32k training and will make sure to put the results in the final version. We also note that, despite using truncation, we use the same setting for baselines and our method to guarantee the **fairness of the comparison** in the evaluation.\n\nIn the meantime, we would like to respectfully argue that such a truncation setting is also **commonly used in the previous works [1,2,3,4]** to improve training efficiency, as longer context lengths significantly increase memory usage and wall-clock time. \n\nImportantly, we view the context length constraints can be treated as a special reward constraint to encourage models to reason in a limited context length (e.g., 8k). During training, we observed that the initial truncation rate was around 50%, but the model quickly adapted—dropping to below 3% after approximately 200 iterations—while continuing to improve in accuracy.\n\nWe appreciate the reviewer’s concern and will include a more detailed discussion in the final version. We will also include the 32k results in the final version.\n\n> **Q2:**  An ablation study on zero-variance target can strengthen the paper.\n\n**A2:** Thank you for the suggestion. We have conducted an ablation study on the targeted zero-variance percentage, as presented in Appendix F.1, showing consistent improvement across a wide range of zero-variance targets. Our results also show that when the percentage is sufficiently high (e.g., 25% used in our experiments), GRESO matches the final accuracy of dynamic sampling. Importantly, **all our RL training experiments adopt the same target value**, indicating that this parameter is non-sensitive and easy to select in practice. \n\n> **Q3:**  Ablation study of adaptive batch size should also consider actual rollout time savings.\n\n**A3:**  We appreciate the reviewer’s insightful observation. It is indeed true that using a batch size of 1 would minimize the number of rollouts per step, but not the overall rollout time. This is because inference latency for LLMs scales nonlinearly with batch size—small batch sizes are not time-efficient due to under-utilized resources, while large batch sizes can significantly reduce per-sample cost.\n\nTo illustrate this, we conducted an experiment using Qwen2.5-Math-1.5B on 4 H100 to measure the actual rollout time at different batch sizes. The results are shown below:\n\n| batch size | 384   | 256  | 192  | 128  | 64   | 32  | 16  |\n|---|---|---|---|---|---|---|---|\n| time   | 135.2| 94.4 | 70   | 57.48| 43   | 36  | 35  |\n\nWe observe that when the batch size is small (e.g., ≤64), increasing the batch size yields per-sample rollout little time saving. However, beyond a certain point, the rollout time increases almost linearly with batch size. Therefore, by using ABS to reduce the batch size in this regime, we can significantly reduce overall rollout time without sacrificing efficiency.\n\nIn our training setup, we set the default rollout batch size to 384. When ABS reduces the batch size adaptively (e.g., from 384 to 128), it leads to a substantial reduction in actual rollout time. This is the core motivation behind ABS, and we will clarify this point in the final version of the paper.\n\n> **Q4.1:**  Figure 6: the zero-variance ratio of easy problems fails to converge to the target, and the corresponding base exploration probability is bounded excessively high. This may result from too large a Δp; if so, the method should be revised to handle this case. \n\n**A4.1:**  Thank you for pointing this out. The reason the base exploration probability appears excessively high is that we enforce a minimum exploration rate (5%) to ensure a non-zero level of exploration. This lower bound helps prevent the model from entirely ignoring potentially informative examples. While this constraint may prevent the zero-variance ratio from fully converging to the target, it guarantees a certain level of exploration throughout training. Importantly, even without fully reaching the target ratio, our method still eliminates a significant amount of unnecessary rollouts and achieves substantial speedup. We will revise the manuscript to clarify this design choice and avoid potential confusion.\n\n> **Q4.2:** In Figure 5(d), the rollout per step for GRESO shows large spikes after step 600, which seems abnormal given the continuous adaptation of batch size. \n\n**A4.2:** In the rollout stage, we enforce a minimum rollout batch size of 64, as small-batch LLM inference does not yield significant differences in wall-clock time (as discussed in Q3). The spikes observed after step 600 in Figure 5(d) are caused by GRESO triggering a second round of rollouts, during which an additional 64 × 8 responses are sampled. We will clarify this behavior in the revised version to avoid confusion.\n\n> **Q4.3:** The DS/GRESO rollout ratio of DM-Qwen2.5-1.5B is smaller than in Figure 5 (a). Does this indicate a performance drop when training steps are fixed? Including a step-to-accuracy curve would help address this concern.\n\n**A4.3:** The rollout ratio in Figure 5(a) reflects the per-step ratio, while the cumulative ratio is much closer to the values reported in Table 1. In Table 5, we report the best-performing checkpoints during training, evaluated every 50 iterations as described in Appendix D. We observed that the final iteration is not always the best one. However, after convergence, the accuracy tends to fluctuate slightly rather than showing a clear performance drop. We will include a step-to-accuracy curve in the revised version of our paper.\n\n---\n\nThanks again for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version.\nWe hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions!\n\n---\n\n[1] Wang, Yiping, et al. \"Reinforcement learning for reasoning in large language models with one training example.\" arXiv preprint arXiv:2504.20571 (2025).\n\n[2] Liu, Mingjie, et al. \"Prorl: Prolonged reinforcement learning expands reasoning boundaries in large language models.\" arXiv preprint arXiv:2505.24864 (2025).\n\n[3] Cui, Ganqu, et al. \"The entropy mechanism of reinforcement learning for reasoning language models.\" arXiv preprint arXiv:2505.22617 (2025).\n\n[4] Cheng, Daixuan, et al. \"Reasoning with exploration: An entropy perspective.\" arXiv preprint arXiv:2506.14758 (2025)."}, {"review_id": "0BCYYJzXlo", "reviewer": "Reviewer_KuoT", "summary": "This paper introduces GRESO, at selective rollout method for reinforcement learning in LLM reasoning tasks. The key observation is that prompts yielding zero reward variance (uninformative) in one training epoch are likely to remain uninformative in subsequent epochs due to temporal consistency. GRESO leverages this by predicting and skipping such prompts before rollout using reward dynamics, reducing computational overhead. The method employs probabilistic filtering with adaptive exploration rates and batch sizing. Evaluations on math reasoning tasks (using 1.5B/7B models) report up to 2.4× rollout speedup and 2.0× total training speedup compared to Dynamic Sampling, without accuracy degradation.", "questions": "- How does GRESO fundamentally differ from prior work on data pruning via training dynamics?\n\n- Can GRESO scale to models >30B parameters or non-mathematical tasks (e.g., dialogue, code generation)?", "limitations": "I would encourage the authors to discuss the potential negative societal impact of their work. For example, what if the prediction is not perfect? How to solve the out-of-distribution problem?", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n- The proposed method is clearly explained and the authors attempted to address the high computational cost issue of rollouts for LLMs.\n\n- The authors provide a practical solution with detailed implementation (adaptive batch sizing, dual exploration rates).\n\nWeaknesses:\n\n- The observation that zero-variance samples are not very helpful for the LLM finetuning is not novel. For example, [1] also utilized similar mechanism to filter the dataset.\n\n- Temporal correlation of sample utility is well-known in curriculum learning and NLP tasks [2,3]. GRESO applies this concept to RL rollouts but without significant algorithmic innovation.\n\n- This paper has a limited scope to math reasoning and small models (≤7B). Modern LLM RL (e.g., 70B+ models) that may face distinct challenges (e.g., prompt complexity, reward sparsity) are not addressed here. Speedup gains may not generalize.\nThere is no formal mathematical analysis of how skipping prompts affects convergence, reward shaping, or generalization beyond test accuracy.\n\nReferences:\n\n[1] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025).\n\n[2] Zhou, Tianyi, Shengjie Wang, and Jeff Bilmes. \"Curriculum learning by optimizing learning dynamics.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[3] Bejan, Irina, Artem Sokolov, and Katja Filippova. \"Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.", "was_revised": true, "final_justification": "I appreciate the authors for doing additional experiments on larger models and clarify their contribution on identifying zero-variance samples. After the discussion, I still have the following concerns:\n\n1. The evaluation might be problematic. Note that in the rebuttal, the authors showcase the way they report the final results, which is picking up the highest performance during a time period. Here is an example given by the authors:\n\nThe test accuracy trace for the GRPO training run:\n\nIteration\t300\t350\t400\t450\t500\t550\t600\t650\nAcc(avg@4) with (std)\t17.2\t18.3\t18.0\t19.8\t19.9\t19.7\t19.5\t19.6\n\nApparently, the best model is from 500 iterations (0.5M rollouts I think). The authors report the result of 19.9 with 0.55M rollouts. So I assume that the reported # rollouts corresponds to the time period, which could be longer than the time when the best model occurs. In that sense, I highly encourage the authors to limit the same number of rollouts for fair comparison. Simply picking up the best model and reporting the performance has a great uncertainty and randomness. The training curves of LLMs can differ a lot with different initial seeds and the time when the best model occurs can also differ a lot.\n\n2. I notice that in the original version of the paper, the authors did a comprehensive evaluation over 1.5B and 7B models. I appreciate the experiments over 14B and 32B models during the rebuttal and discussion phase. But I still encourage the authors to do a comprehensive evaluuation for 14B and 32B models and put the results into Table 1 (i.e., accross six tasks).", "rebuttal": "## Response to Reviewer 2 (KuoT)\n\nThank you very much for your thoughtful review and constructive suggestions. We are glad that you found our presentation clear and our solution practical, with detailed implementation. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.\n\n---\n\n> **Q1:** The observation that zero-variance samples are not very helpful for the LLM finetuning is not novel. For example, [1] also utilized similar mechanism to filter the dataset.\n\n**A1:**  Thanks for bringing this for discussion. As acknowledged in the paper, this observation has been discussed in prior work[1,4], and we do not claim it as a contribution in our paper. Our key contribution lies in **how to identify zero-variance samples prior to performing rollouts**—a nontrivial challenge that, to the best of our knowledge, has not been addressed in prior work, and our work is the first method to provide a practical and efficient solution to this problem.\n\nThank you for highlighting this point—we will further make this distinction clearer in the revised version.\n\n> **Q2:** Temporal correlation of sample utility is well-known in curriculum learning and NLP tasks [2,3]. GRESO applies this concept to RL rollouts but without significant algorithmic innovation.\n\n**A2:**  While we agree with the reviewer that leveraging temporal correlation in sample utility is a well-established principle in curriculum learning and active learning, we respectfully argue that **our paper tackles a fundamentally different and underexplored problem** for RLVR for LLMs, where zero-variance prompts emerge as a unique challenge, which was discussed by recent work [1,4]. \n\nWhile similar temporal correlation principles have been explored in areas such as curriculum learning, how to model and leverage temporal signals to detect and filter zero-variance prompts prior to rollout remains underexplored and non-trivial. For instance, unlike prior works [2,3] that operate on static datasets, RLVR involves online data generation, where the training data can vary significantly across iterations, making it non-trivial to apply existing approaches directly.\n\nTo the best of our knowledge, our paper is the **first** to propose a practical and effective solution to filter out zero-variance prompts in RLVR, by tackling the following key challenges:\n\n1. How to detect and filter uninformative prompts before performing rollouts (i.e., without access to the full training data)?\n\n2. How to perform such detection and filtering with minimal overhead?\n\n3. How to balance exploration and efficiency, ensuring that potentially useful prompts can be revisited in the future training stage?\n\nEach of these challenges is non-trivial in the RLVR setting. Our proposed method, GRESO, offers a lightweight mechanism that effectively addresses all three and demonstrates strong empirical performance across multiple benchmarks and RL algorithms.\n\nHowever, we agree with the reviewer that a discussion of these works would strengthen the paper. We will conduct a comprehensive discussion of the related prior works, including [2,3], on curriculum learning and active learning in the revised version to better clarify the connections and differences.\n\n\n> **Q3:** This paper has a limited scope to math reasoning and small models (≤7B). Modern LLM RL (e.g., 70B+ models) that may face distinct challenges (e.g., prompt complexity, reward sparsity) are not addressed here. Speedup gains may not generalize. There is no formal mathematical analysis of how skipping prompts affects convergence, reward shaping, or generalization beyond test accuracy.\n\n**A3:** \n\n**Training on Code Tasks**: To further evaluate the generalizability of GRESO, we applied it to the code generation domain. Specifically, we trained DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset and evaluated the models on the LiveCodeBench benchmark. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):\n\n|Method|GRPO|GRESO|DS|\n|----|---|-----|--|\n|Acc(avg@4)|19.9|22.8|21.6|\n|#Rollouts|0.55M|1.07M|1.8M|\n\nWe trained all the models until convergence. **GRESO achieves comparable accuracy to DS with fewer rollouts**, and GRPO fails to match the final accuracy of GRESO and DS.\n\nThese results suggest that GRESO is also **effective in other reasoning domains** such as code generation.\n\n**Scale to 32B model.** Thank you for the suggestion. We have already initiated experiments on 14B and 32B-scale models to evaluate the scalability of GRESO. Due to the high computational cost, these runs are still in progress, but we report interim results below (average accuracy across the 6 benchmarks used in the paper):\n\n|14B (150 iterations) |GRESO|DS|\n|------|----|----------|\n|Acc(avg@4)|55.4|56.3|\n|#Rollouts|353k|455k|\n\n|32B (100 iterations) |GRESO|DS|\n|------|----|----------|\n|Acc(avg@4)|60.4|59.2|\n|#Rollouts|254k|303k|\n\nThese preliminary results indicate that GRESO achieves comparable accuracy to Dynamic Sampling while using fewer rollouts. We are continuing these experiments and will update the results during the discussion phase. We will also ensure that the complete results, especially for the 32B model, are included in the final version of the paper.\n\nBesides, we agree that a formal theoretical understanding of the mechanism of our method is an important and open-ended research question. The primary goal of this work is to develop a practical and effective method for improving rollout efficiency. GRESO demonstrates consistent speed-up across multiple models and domains, which we believe provides strong evidence of its effectiveness in practice. We hope that our method and findings can serve as a foundation for future theoretical investigations into the dynamics of selective rollout in RL-based LLM training.\n\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version.\nWe hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.\n\n---\n\n[1] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025).\n\n[2] Zhou, Tianyi, Shengjie Wang, and Jeff Bilmes. \"Curriculum learning by optimizing learning dynamics.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[3] Bejan, Irina, Artem Sokolov, and Katja Filippova. \"Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.\n\n[4] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025)."}, {"review_id": "CNuj8IQ9tA", "reviewer": "Reviewer_vbkh", "summary": "The paper proposes a new online data selection strategy in RL for LLM Reasoning. Based on the findings of the strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs, the paper propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By applying GRESO on models like Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, the experimental results show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.", "questions": "The proposed method seems to have many hyper parameters to tune, such as $\\beta$, $\\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios?\nHow is the performance and training efficiency affected by different hyperparameters?\nIs the effectiveness of the algorithm robust to different hyper parameters?", "limitations": "1. The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not hold when the base model or initial policy is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "## Strengths\n1. The method is evaluated on multiple initial policy including Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, and shows that  GRESO consistently brings significant wall-clock time speedup in rollout and speedup in total training time without accuracy degradation.\n2. The analysis of training dynamics of different queries is thorough, especially the insights about the temporal consistency between uninformative queries are interesting. The statistical analysis such as Figures 5/6/7 also helps the reviewer better understands the influence of the proposed algorithm.\n\n## Weaknesses\n1. The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not holds when the base model is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.\n2. The proposed method seems to have many hyper parameters to tune, such as $\\beta$, $\\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios, and how is the performance so as the training efficiency affected by different hyper parameters?", "was_revised": true, "final_justification": "The author's response partially resolved my concerns.\nIn fact, the common practice is to use an SFT/RFT model as the initial policy instead of the base model, so usually queries are not used   multiple times to avoid overfitting.\n\nAccording to other review comments and their corresponding responses, I tend to keep my score.", "rebuttal": "## Response to Reviewer 3 (vbkh)\n\nThank you very much for your thoughtful review and constructive suggestions. We are glad that you found our evaluation extensively shows GRESO achieves performance gain and the analysis is thorough. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.\n\n---\n\n> **Q1:**  The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not holds when the base model is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.\n\n**A1:** Multi-epoch RL training is a common practice in existing RLHF and RLVR pipelines [1,2,3,4], even for strong base models such as 32B-scale LLMs [1]. In our experiments, we also observe that training over multiple epochs consistently leads to performance improvements. Therefore, we believe this is a valid and practical setting that can benefit from selective rollout strategies like GRESO.\n\nHowever, we acknowledge this as a valid limitation. One potential way to address it is to use the base model to profile the dataset before training, enabling effective data filtering even within the first epoch. We thank the reviewer for this insightful suggestion and will work on it as a direction for future work.\n\n> **Q2:**  The proposed method seems to have many hyper parameters to tune, such as $\\beta$, $\\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios, and how is the performance so as the training efficiency affected by different hyper parameters?\n\n**A2:**  While our method introduces several hyperparameters, we argue that **these are not manually tuned for each task and do not pose a significant burden in practice**:\n\n1. The exploration bounds $p_{\\text{easy}}$ and $p_{\\text{hard}}$ are automatically adjusted during training and do not require manual specification.\n\n\n2. The target zero-variance percentage $\\alpha$ is discussed in Appendix F.1. We show that GRESO is robust across a wide range of values, and as long as $\\alpha$ is sufficiently high, GRESO shows pretty good performance.\n\n3. The exploration adjustment rate $\\beta$ is a parameter inherited from GRPO; we follow prior work and simply set $\\beta = 0$, which works well in our setting.\n\nMore importantly, we use **the same default configuration across all experiments and consistently observe efficiency gains**, indicating that GRESO works robustly without the need for extensive hyperparameter tuning.\n\n---\n\nThanks again for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version. We hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.\n\n---\n\n[1] Yu, Qiying, et al. \"Dapo: An open-source llm reinforcement learning system at scale.\" arXiv preprint arXiv:2503.14476 (2025).\n\n[2] Yue, Yu, et al. \"Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.\" arXiv preprint arXiv:2504.05118 (2025).\n\n[3] Fu, Wei, et al. \"AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning.\" arXiv preprint arXiv:2505.24298 (2025).\n\n[4] Wang, Shenzhi, et al. \"Beyond the 80/20 rule: High-entropy minority tokens drive effective reinforcement learning for llm reasoning.\" arXiv preprint arXiv:2506.01939 (2025)."}, {"review_id": "EA89XHC2EI", "reviewer": "Reviewer_sAP6", "summary": "This paper introduces GRESO (GRPO with Efficient Selective Rollout), a novel method to enhance the efficiency of reinforcement learning (RL) for large language model (LLM) reasoning by selectively skipping uninformative prompts before rollout. The authors identify that prompts yielding zero advantage (zero-variance prompts) in one training epoch are likely to remain uninformative in subsequent epochs, leading to unnecessary computational overhead. GRESO leverages this temporal consistency in reward dynamics to predict and filter out such prompts proactively, using an online, lightweight algorithm with probabilistic filtering and adaptive batch sizing. Evaluated across multiple math reasoning benchmarks and models, GRESO achieves up to 2.4× speedup in rollout time and 2.0× reduction in total training time while maintaining model accuracy. The method addresses key limitations of prior approaches like Dynamic Sampling, which filters prompts after rollout, by reducing wasted computation.", "questions": "Besides the weaknesses above, further questions are as follows:\n\n- Can the authors provide some insights on the exploration mechanism in GRESO?\n- Since original DAPO is implemented using a 32B model, do the authors test GRESO on larger models, such as 32B?\n- Can GRESO be extended to other domains beyond math reasoning, such as coding?\n- How do the authors design the prompt using in training and test sets? I have tested several setups (including (1) DAPO with original prompt, (2) DAPO with DeepScaleR's prompt, (3) Skywork-OR1 with DeepScaleR's prompt and (4) Skywork-OR1 without prompt) and found that the prompt can influence on the model performance and entropy. For example, (1) has significant lower model entropy (0.62) compared with (2) (0.93) at the begining. Also, the performance of (1) drops largely after training 750 steps while (2) performs well. I would like to know the reason behind the prompt.", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "#### Strengths\n\n- The idea is novel and interesting.\n- The paper is easy to follow.\n- The code is provided.\n\n#### Weaknesses\n\n- The experiments lack baselines. Although the authors compare GRESO with DS of DAPO, the comparison is limited to a single method. A more comprehensive comparison with other methods would strengthen the paper, such as comparing with GRPO [1]. Also, I have run GRPO with DAPO+MATH dataset using DeepSeek-R1-Distill-Qwen-1.5B on 8x H800. After training 1170 steps, the model achieves 36.771 on AIME24 (avg@32) and training these steps only require 51 hours and 1.2M rollouts. Although some hyperparameters (e.g., response length) and package versions (e.g., vllm) may be different and this direct comparison is not fair, most important hyperparameters are the same by checking the provided training scripts and I suggest the authors compare GRESO with GRPO.\n- The experiments lack base methods. Efficient Selective Rollout can be used with other RL algorithms such as VC-PPO [2] and VAPO [3]. Experiments with these methods would provide a more comprehensive evaluation of GRESO's effectiveness.\n\n[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.\n\n[2] What’s behind ppo’s collapse in long-cot? value optimization holds the secret.\n\n[3] Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.", "was_revised": true, "final_justification": "All my concerns are addressed by the author's rebuttal.", "rebuttal": "## Response to Reviewer 4 (sAP6)\n\nThank you very much for your thoughtful review and constructive suggestions. We are glad that you found our idea both interesting and novel, and the presentation clear. We appreciate the opportunity to address the points you raised and hope you will consider raising your score in light of our response.\n\n---\n\n> **Q1:**  GRPO can be treated as a baseline.\n\n**A1:**  Thanks for your suggestion to compare GRESO with GRPO. We would like to address your concerns from the following two perspectives:\n\n1. We have already included a GRPO as a baseline in Figure 1 in the main paper: GRPO without DS can achieve a worse final accuracy in some settings.\n\n2. GRESO is complementary to GRPO and can be applied to accelerate vanilla GRPO.\n\n3. Discussion of GRPO performance on DeepSeek-R1-Distill-Qwen-1.5B.\n\n**1. GRPO without DS can achieve a worse final accuracy in some settings.**\nWe included GRPO as a baseline for training the Qwen2.5-Math-7B model in Figure 1. We show that, although GRPO training is faster, but achieves a worse final accuracy.  This phenomenon appears in several settings, like the 32B model results reported in the DAPO paper [1]. Similarly, in our additional RLOO (Q2) and Code (Q5) evaluations in the rebuttal, we also notice that vanilla GRPO achieves an inferior performance. For this reason, many recent methods adopt DS as a default component to stabilize training [1,2,3]. Therefore, in our main comparison table, we focus our comparison on DS rather than GRPO.\n\n**2.GRESO is complementary Vanilla GRPO**:  More importantly, GRESO is not incompatible with GRPO—in fact, the **GRESO and GRPO are complementary**. While our paper focuses on applying GRESO to accelerate DS, the same idea can be used to improve the efficiency of GRPO as well. Given a batch of prompts in GRPO, GRESO can be used to predict which prompts are likely to result in zero-variance rollouts. These prompts can then be skipped during rollout generation, reducing redundant computation. To verify this claim, we conduct an evaluation by applying GRESO to GRPO for training DeepSeek-R1-Distill-Qwen-1.5B on the DAPO+MATH dataset. The results are as follows:\n\nBest Accuray for AIME24 (This result exceeds the paper-reported number because Table 1 reports checkpoints selected for the best average accuracy across benchmarks, whereas here we report the best accuracy on this individual benchmark.):\n\nBest Accuray for AIME24:\n\n|AIME24|GRPO|GRPO+GRESO|\n|------|----|----------|\n|Acc(avg@4)|39.5|40.6|\n|#Rollouts|561k|322k|\n\nBest Accuray for AIME25:\n\n|AIME25|GRPO|GRPO+GRESO|\n|------|----|----------|\n|Acc(avg@4)|33.3|32.3|\n|#Rollouts|461k|377k|\n\nTotal #rollouts for 1000 iterations:\n\n|1000 Iterations|GRPO|GRPO+GRESO|\n|------|----|----------|\n|#Rollouts|1.02M|650k|\n\nFor both AIME24 and AIME25, GRESO achieves comparable performance while significantly reducing the number of rollouts. For all 1000 training iterations, GRPO has accumulated 1.02 million rollouts, whereas GRPO + GRESO only requires 650k—achieving over 35% reduction in rollout cost for the same number of updates. These results confirm that **GRESO not only accelerates DS training but also boosts the efficiency of vanilla GRPO by reducing redundant rollouts.**\n\n\n**3. Discussion of GRPO performance on DeepSeek-R1-Distill-Qwen-1.5B for Math.**\nThank you for sharing the evaluation results on DeepSeek-R1-Distill-Qwen-1.5B.  We also note that the effectiveness of DS varies depending on the training dataset difficulty for the base model. For instance, when training on DeepSeek-R1-Distill-Qwen-1.5B, the observed zero-variance rate is relatively low (around 50%), allowing GRPO to maintain training quality. In contrast, models such as Qwen2.5-Math-7B exhibit much higher zero-variance rates (up to 80%) under the same setup, which significantly harms performance. Furthermore, as we will present in Q5, when training DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset, vanilla GRPO yields inferior performance.\n\n\nWe thank the reviewer again for the insightful suggestion. We will include this discussion and the additional experimental results in the revised version of the paper to better illustrate the relationship between GRESO and GRPO\n\n\n> **Q2:** Selective rollout can also be applied with other baselines like VC-PPO and VAPO.\n\n**A2:** We agree that GRESO can be applied to other RL methods. However, VC-PPO and VAPO are value-based approaches that do not suffer from the zero-variance issue in RLVR, which GRESO is specifically designed to address. To demonstrate the generality of GRESO, we applied it to RLOO [4] on the MATH + DAPO benchmark using Qwen2.5-Math-7B and report average accuracy across 6 benchmarks used in the paper. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):\n\n\n|Method|RLOO|RLOO+GRESO|RLOO+DS|\n|---|---|---|----|\n|Acc(avg@4)|55.1|57|56.8|\n|#Rollouts|0.81M|1.2M|2.56M|\n\nThese results show that GRESO improves accuracy over vanilla RLOO while requiring **less than half** the number of rollouts compared to RLOO+DS. This demonstrates that GRESO is both effective and efficient in other RLVR methods.\nWhile GRESO was not applied to value-based methods in this work, we agree that selective rollout for value-based RL is an important direction. We will include a discussion of this extension in the main paper, and we consider it a promising direction for our future research.\n\n> **Q3:**  Can the authors provide some insights on the exploration mechanism in GRESO?\n\n**A3:**  As shown by the green curve in Figure 4(a) of the main paper, approximately 20% of prompts previously labeled as zero-variance become effective again in subsequent training. This highlights the importance of maintaining a certain level of exploration—even for prompts that were previously considered uninformative.\nGRESO addresses this by computing an exploration probability for each prompt based on its recent zero-variance history. Specifically, the more frequently a prompt has been identified as zero-variance in consecutive rollouts, the less likely it is to be informative, and thus it is assigned a lower exploration probability. This adaptive mechanism allows GRESO to selectively revisit potentially useful prompts while minimizing unnecessary rollouts.\n\n> **Q4:**  Evaluation on 32B models.\n\n**A4:**  Thank you for the suggestion. We have already initiated experiments on 14B and 32B-scale models to evaluate the scalability of GRESO. Due to the high computational cost, these runs are still in progress, but we report interim results below (average accuracy across the 6 benchmarks used in the paper):\n\n|14B (150 iterations) |GRESO|DS|\n|------|----|----------|\n|Acc(avg@4)|55.4|56.3|\n|#Rollouts|353k|455k|\n\n|32B (100 iterations) |GRESO|DS|\n|------|----|----------|\n|Acc(avg@4)|60.4|59.2|\n|#Rollouts|254k|303k|\n\nThese preliminary results indicate that GRESO achieves comparable accuracy to Dynamic Sampling while using fewer rollouts. We are continuing these experiments and will update the results during the discussion phase. We will also ensure that the complete results, especially for the 32B model, are included in the final version of the paper.\n\n> **Q5:**  Can GRESO be extended to other domains beyond math reasoning, such as coding?\n\n**A5:**  Yes, GRESO can be extended beyond math reasoning. To further evaluate the generalizability of GRESO, we applied it to the code generation domain. Specifically, we trained DeepSeek-R1-Distill-Qwen-1.5B on the code_contests dataset and evaluated the models on the LiveCodeBench benchmark. The results are as follows (we report the best accuracy and the corresponding number of rollouts at that iteration):\n\n|Method|GRPO|GRESO|DS|\n|----|---|-----|--|\n|Acc(avg@4)|19.9|22.8|21.6|\n|#Rollouts|0.55M|1.07M|1.8M|\n\nWe trained all the models until convergence. **GRESO achieves comparable accuracy to DS with fewer rollouts**, and GRPO fails to match the final accuracy of GRESO and DS.\nThese results suggest that GRESO is also **effective in other reasoning-intensive domains** such as code generation. Thank you for the suggestion, we will include the results in the final version of our paper.\n\n> **Q6:**  How are prompts designed for the paper?\n\n**A6:**  We empirically selected our prompt by adding a reasoning instruction to the default Verl prompt used for Math tasks (our final prompt is provided in Appendix E). We also evaluated the DAPO prompt during our experiments, but found that it led to inferior performance. We hypothesize that, one possible reason for the performance drop is the mismatch in answer formatting — the DeepSeek distilled model is trained to place the final answer in a box, whereas the DAPO template does not follow it.\n\n---\n\nThanks for the attentive reading of the manuscript and constructive feedback. We will incorporate these changes into our final version.\nWe hope our response addresses all the concerns and that the reviewer will consider raising the rating accordingly. We are more than glad to answer any further questions.\n\n---\n\n[1] DAPO: An Open-Source LLM Reinforcement Learning System at Scale\n\n[2] ProRL: Prolonged Reinforcement Learning Expands Reasoning Boundaries in Large Language Models.\n\n[3] Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning\n\n[4] BUY 4 REINFORCE SAMPLES, GET A BASELINE FOR FREE!"}]

Author rebuttal:

Meta-review: This paper proposes GRESO, a lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts in reinforcement learning for LLM reasoning, achieving up to 2.4x rollout speedup without accuracy degradation.

The paper addresses an important computational bottleneck in LLM RL training with a practical solution. The authors demonstrated strong responsiveness to reviewer feedback, providing extensive additional experiments across model scales (up to 32B) and domains (math and coding). The adaptive filtering mechanism represents a solid technical contribution. The method's demonstrated effectiveness across diverse settings and substantial computational savings make it a valuable contribution to the field.

Given the efforts the authors made during the rebuttal process, most reviewers provided positive reviews and found the approach insightful and well-motivated with good experimental design. Reviewer KuoT strongly suggested running comprehensive evaluations of large models and including training curves in the paper. Given the limited resources available for academic research and the extensive experiments already conducted on 1.5B/7B models with up to 32k context length, this remaining issue does not constitute a solid reason to reject this paper. However, I encourage the authors to follow the reviewer's suggestions and update the manuscript in the camera-ready version, which will definitely improve the solidness of the original draft.

**Overall, I recommend accepting this paper.**
Decision: Accept (poster)