Title: Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rollouts
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

GRPO

Reinforcement Learning
Abstract: Reinforcement learning, such as PPO and GRPO, has powered recent breakthroughs in LLM reasoning. Scaling rollout to sample more prompts enables models to selectively use higher-quality data for training, which can stabilize RL training and improve model performance, but at the cost of significant computational overhead. In this paper, we first show that a substantial portion of this overhead can be avoided by skipping uninformative prompts before rollout. Our analysis of reward dynamics reveals a strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs. Based on these insights, we propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By evaluating GRESO on a broad range of math reasoning benchmarks and models, like Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, Qwen2.5-Math-7B, Qwen2.5-14B, and Qwen2.5-32B, we show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation. We make our code publicly available at https://github.com/Infini-AI-Lab/GRESO/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRESO (GRPO with Efficient Selective Rollout), a novel method to enhance the efficiency of reinforcement learning (RL) for large language model (LLM) reasoning by selectively skipping uninformative prompts before rollout. The authors identify that prompts yielding zero advantage (zero-variance prompts) in one training epoch are likely to remain uninformative in subsequent epochs, leading to unnecessary computational overhead. GRESO leverages this temporal consistency in reward dynamics to predict and filter out such prompts proactively, using an online, lightweight algorithm with probabilistic filtering and adaptive batch sizing. Evaluated across multiple math reasoning benchmarks and models, GRESO achieves up to 2.4× speedup in rollout time and 2.0× reduction in total training time while maintaining model accuracy. The method addresses key limitations of prior approaches like Dynamic Sampling, which filters prompts after rollout, by reducing wasted computation.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n- The idea is novel and interesting.\n- The paper is easy to follow.\n- The code is provided.\n\n#### Weaknesses\n\n- The experiments lack baselines. Although the authors compare GRESO with DS of DAPO, the comparison is limited to a single method. A more comprehensive comparison with other methods would strengthen the paper, such as comparing with GRPO [1]. Also, I have run GRPO with DAPO+MATH dataset using DeepSeek-R1-Distill-Qwen-1.5B on 8x H800. After training 1170 steps, the model achieves 36.771 on AIME24 (avg@32) and training these steps only require 51 hours and 1.2M rollouts. Although some hyperparameters (e.g., response length) and package versions (e.g., vllm) may be different and this direct comparison is not fair, most important hyperparameters are the same by checking the provided training scripts and I suggest the authors compare GRESO with GRPO.\n- The experiments lack base methods. Efficient Selective Rollout can be used with other RL algorithms such as VC-PPO [2] and VAPO [3]. Experiments with these methods would provide a more comprehensive evaluation of GRESO's effectiveness.\n\n[1] DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models.\n\n[2] What’s behind ppo’s collapse in long-cot? value optimization holds the secret.\n\n[3] Vapo: Efficient and reliable reinforcement learning for advanced reasoning tasks.\n\nQUESTIONS:\nBesides the weaknesses above, further questions are as follows:\n\n- Can the authors provide some insights on the exploration mechanism in GRESO?\n- Since original DAPO is implemented using a 32B model, do the authors test GRESO on larger models, such as 32B?\n- Can GRESO be extended to other domains beyond math reasoning, such as coding?\n- How do the authors design the prompt using in training and test sets? I have tested several setups (including (1) DAPO with original prompt, (2) DAPO with DeepScaleR's prompt, (3) Skywork-OR1 with DeepScaleR's prompt and (4) Skywork-OR1 without prompt) and found that the prompt can influence on the model performance and entropy. For example, (1) has significant lower model entropy (0.62) compared with (2) (0.93) at the begining. Also, the performance of (1) drops largely after training 750 steps while (2) performs well. I would like to know the reason behind the prompt.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new online data selection strategy in RL for LLM Reasoning. Based on the findings of the strong temporal consistency in prompt value: prompts that are uninformative in one epoch of training are likely to remain uninformative in near future epochs, the paper propose GRESO (GRPO with Efficient Selective Rollout), an online, lightweight pre-rollout filtering algorithm that predicts and skips uninformative prompts using reward training dynamics. By applying GRESO on models like Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, the experimental results show that GRESO achieves up to 2.4x wall-clock time speedup in rollout and up to 2.0x speedup in total training time without accuracy degradation.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1. The method is evaluated on multiple initial policy including Qwen2.5-Math-1.5B, DeepSeek-R1-Distill-Qwen-1.5B, and Qwen2.5-Math-7B, and shows that  GRESO consistently brings significant wall-clock time speedup in rollout and speedup in total training time without accuracy degradation.\n2. The analysis of training dynamics of different queries is thorough, especially the insights about the temporal consistency between uninformative queries are interesting. The statistical analysis such as Figures 5/6/7 also helps the reviewer better understands the influence of the proposed algorithm.\n\n## Weaknesses\n1. The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not holds when the base model is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.\n2. The proposed method seems to have many hyper parameters to tune, such as $\\beta$, $\\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios, and how is the performance so as the training efficiency affected by different hyper parameters?\n\nQUESTIONS:\nThe proposed method seems to have many hyper parameters to tune, such as $\\beta$, $\\alpha$, $p_{easy}$ and $p_{hard}$, is there any principles or guidelines to tune these parameters in broader scenarios?\nHow is the performance and training efficiency affected by different hyperparameters?\nIs the effectiveness of the algorithm robust to different hyper parameters?\n\nLIMITATIONS:\n1. The proposed method highly relies on the premise that the training queries are used multiple times in RLVR, which might not hold when the base model or initial policy is strong, which usually trains less than one epoch to avoid overfitting or reward hacking.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRESO, at selective rollout method for reinforcement learning in LLM reasoning tasks. The key observation is that prompts yielding zero reward variance (uninformative) in one training epoch are likely to remain uninformative in subsequent epochs due to temporal consistency. GRESO leverages this by predicting and skipping such prompts before rollout using reward dynamics, reducing computational overhead. The method employs probabilistic filtering with adaptive exploration rates and batch sizing. Evaluations on math reasoning tasks (using 1.5B/7B models) report up to 2.4× rollout speedup and 2.0× total training speedup compared to Dynamic Sampling, without accuracy degradation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- The proposed method is clearly explained and the authors attempted to address the high computational cost issue of rollouts for LLMs.\n\n- The authors provide a practical solution with detailed implementation (adaptive batch sizing, dual exploration rates).\n\nWeaknesses:\n\n- The observation that zero-variance samples are not very helpful for the LLM finetuning is not novel. For example, [1] also utilized similar mechanism to filter the dataset.\n\n- Temporal correlation of sample utility is well-known in curriculum learning and NLP tasks [2,3]. GRESO applies this concept to RL rollouts but without significant algorithmic innovation.\n\n- This paper has a limited scope to math reasoning and small models (≤7B). Modern LLM RL (e.g., 70B+ models) that may face distinct challenges (e.g., prompt complexity, reward sparsity) are not addressed here. Speedup gains may not generalize.\nThere is no formal mathematical analysis of how skipping prompts affects convergence, reward shaping, or generalization beyond test accuracy.\n\nReferences:\n\n[1] Muennighoff, Niklas, et al. \"s1: Simple test-time scaling.\" arXiv preprint arXiv:2501.19393 (2025).\n\n[2] Zhou, Tianyi, Shengjie Wang, and Jeff Bilmes. \"Curriculum learning by optimizing learning dynamics.\" International Conference on Artificial Intelligence and Statistics. PMLR, 2021.\n\n[3] Bejan, Irina, Artem Sokolov, and Katja Filippova. \"Make Every Example Count: On the Stability and Utility of Self-Influence for Learning from Noisy NLP Datasets.\" Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023.\n\nQUESTIONS:\n- How does GRESO fundamentally differ from prior work on data pruning via training dynamics?\n\n- Can GRESO scale to models >30B parameters or non-mathematical tasks (e.g., dialogue, code generation)?\n\nLIMITATIONS:\nI would encourage the authors to discuss the potential negative societal impact of their work. For example, what if the prediction is not perfect? How to solve the out-of-distribution problem?", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces GRESO, a lightweight rollout filtering strategy for efficient reinforcement learning (RL) on large language models (LLMs). GRESO dynamically adjusts each prompt’s sampling probability to limit the frequency of zero-variance rollouts and thereby conserve consumption, all with few hyperparameters to tune. On mathematical reasoning tasks, it achieves substantial reductions in rollout cost without performance loss. Its key innovation lies in an automatic control of sampling probability to maintain a preset zero-variance target by monitoring the sampling history, ensuring both exploration and efficiency.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n\n1. The paper addresses an important challenge of rollout efficiency in RL on LLMs with a simple yet effective approach. \n2. The design of the dynamic control of exploration probability is insightful and well motivated. \n3. The manuscript is well writen and easy to follow, and the purpose of each component is convincingly justified. \n\nWeaknesses：\n\n1. The experiments on DeepSeek-R1-Distill-Qwen-1.5B seems problematic: an 8192 context length is clearly too short and likely incurs a high truncation rate during training. The authors should either include a more robust setting in the main result or explain their choice. \n2. Although the zero-variance target is chosen intuitively and leave the discussion to future work, a thorough ablation on this parameter would be highly informative and would significantly strengthen the paper. \n3. The ablation study of adaptive batch size is not entirely convincing. If the comparison is based on rollouts per step, then the optimal batch size would be 1 per GPU, which is obviously unreasonable. A direct comparison of actual rollout time savings with ABS should be provided. \n4. Several issues in the empirical results need further clarification: \n4.1 In Figure 6, the zero-variance ratio of easy problems fails to converge to the target, and the corresponding base exploration probability is bounded excessively high. This may result from too large a Δp; if so, the method should be revised to handle this case. \n4.2 In Figure 5(d), the rollout per step for GRESO shows large spikes after step 600, which seems abnormal given the continuous adaptation of batch size.\n4.3 The DS/GRESO rollout ratio of DM-Qwen2.5-1.5B is smaller than in Figure 5 (a). Does this indicate a performance drop when training steps are fixed? Including a step-to-accuracy curve would help address this concern.\n\nQUESTIONS:\nPlease see weaknesses.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-06"}