Title: Improving Data Efficiency for LLM Reinforcement Fine-tuning Through Difficulty-targeted Online Data Selection and Rollout Replay
Venue: NeurIPS 2025
Accepted: yes
Keywords: Data Efficiency

Online Data Selection

Experience Replay

Reinforcement Learning

Large Language Model
Abstract: Reinforcement learning (RL) has become an effective approach for fine-tuning large language models (LLMs), particularly to enhance their reasoning capabilities. However, RL fine-tuning remains highly resource-intensive, and existing work has largely overlooked the problem of data efficiency. In this paper, we propose two techniques to improve data efficiency in LLM RL fine-tuning: difficulty-targeted online data selection and rollout replay. We introduce the notion of adaptive difficulty to guide online data selection, prioritizing questions of moderate difficulty that are more likely to yield informative learning signals. To estimate adaptive difficulty efficiently, we develop an attention-based framework that requires rollouts for only a small reference set of questions. The adaptive difficulty of the remaining questions is then estimated based on their similarity to this set. To further reduce rollout cost, we introduce a rollout replay mechanism inspired by experience replay in traditional RL. This technique reuses recent rollouts, lowering per-step computation while maintaining stable updates. Experiments across 6 LLM-dataset combinations show that our method reduces RL fine-tuning time by 23% to 62% while reaching the same level of performance as the original GRPO algorithm. 
Our code repository is available at https://github.com/ASTRAL-Group/data-efficient-llm-rl/.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, the authors address the computational inefficiency of reinforcement learning fine-tuning for large language models by introducing two techniques: Difficulty-targeted Online Data Selection (DOTS) and Rollout Replay (RR). DOTS employs an attention-based adaptive difficulty prediction framework that efficiently selects training questions with moderate difficulty, providing the most informative learning signals by generating rollouts for only a small reference set. Simultaneously, RR reduces computational costs by reusing recent rollouts and generating new rollouts for only a fraction of each training batch, using a modified loss function to maintain training stability. Experimentally validated across six LLM-dataset combinations, their approach demonstrates remarkable performance, reducing RL fine-tuning time while preserving the original GRPO algorithm's performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Rigorous experimental methodology with comprehensive testing across multiple LLM models and datasets\n2. Theoretically grounded approach, including a formal theorem (Theorem 1) explaining the rationale behind difficulty-based sample selection.\n3. Demonstrates substantial time savings (25-65% reduction) across different model sizes and datasets.\n\nWeaknesses:\n1. No statistical significance tests due to high computational costs (acknowledged in the paper)\n2. Pre-trained embedding models have difficulty capturing fine-grained similarities in mathematical problems. The paper alleviates this problem by fine-tuning an MLP adapter, but the adapter may overfit and its generalization ability to problems in different fields is questionable.\n\nQUESTIONS:\n1. The theorem is based only on binary rewards. How to generalize it to more complex reward scenarios?\n2. Why did you choose a mathematical reasoning dataset?  Have you tried to verify it on non-mathematical fields?\n\nLIMITATIONS:\n1. Only validates mathematical reasoning datasets, generalization is unknown.\n2. Difficulty selection may inadvertently reinforce training data bias.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper aims to improve the data efficiency of reinforcement learning for LLM post-training. The authors recognized problems that not all training samples are used effectively to provide useful learning signals, and data from past iterations are simply discarded. Therefore, the authors proposed to techniques, difficulty-targeted online data selection (DOTS) and rollout replay (RR) to reduce the number of training steps and the computational costs per step. Specifically, DOTS introduces an “adaptive difficulty” metric that selects data of moderate difficulty to maximize the learning signal for each training step, while RR lowers the cost by reusing rollouts from recent training steps. With these two techniques combined, the paper demonstrates that the method reduces the RL training time by 25% to 65% percent without sacrificing performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. Recognizing the data efficiency in RL as a major problem to address, this is also a timely contribution.\n2. The results do show the great performance of the proposed method in terms of efficiency gain.\n3. The proposed method is shown to greatly improve the ratio of effective questions in the training, as shown in Figure 4.\n\nWeaknesses:\n\n1. The empirical results could lack credibility due to the limited number of runs, and some additional issues described in the Questions below.\n\nQUESTIONS:\n1. It is unclear from the paper whether the reference set’s rollouts are used in the training process other than the adaptive difficulty prediction.\n2. Relatedly, when the reference set is 256 and the training batch size is 512, substantial time is spent on the rollout of the reference set. Have you considered using those samples for training (especially when their difficulty is around 0.5), for replay, or are they just discarded?\n3. While there are high-level computational time comparisons, it is unclear how much time was taken for the additional difficulty prediction step. Can the authors present a breakdown of time for the overall process? Also, will the efficiency advantage (wall-clock time wise) still be present when there is only POTS (no RR)?\n4. The experimental results lack credibility and reproducibility. There is only one run reported, without error bars. The experimental run is also terminated early - the convergence behavior is not seen in the plots.\n5. Another questionable presentation of the result is that the ablation results in Figure 5 are shown on training step, while Figure 3 is shown on GPU time. I would like to see both as the x axis, for both overall result and ablation to assess reliably.\n6. Though AIME results are excluded, they could have been reported separately because it could be possible that the designed online data selection method does not help with learning extremely hard questions like AIME.\n\nLIMITATIONS:\nPlease discuss the limitations in the main paper instead of the appendix B.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new method for selecting LLM RL training problems via their difficulty level for the current policy. This is done by rolling out the policy on a small core-set of reference problems and estimating problem difficulty for the remainder of the train set using an attention based scoring. Selected rollouts are placed in a replay buffer and used to train the current policy (by additionally modifying the standard GRPO loss accomodate off-policy data). Problems are sampled by the replay buffere reweighted by their likelihood under the policy by which they were generated. Experiments show the method improves sample efficiency by 25%-65%.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n- The presentation and layout of the paper is clear and easy to follow.\n- The evaluation is done across multiple model scales (1.5B, 3B, 7B) and multiple datasets\n- The paper addresses a key bottleneck in the LLM RL training pipeline: rollout generation inefficiency.\n\n**Weaknesses:**\n\n- I'm a a bit skeptical of the adaptive difficulty assessment approach: many questions may be semantically similar in statement but may actually be quite different in difficultly (e.g. consider the simplicity of Goldbach's conjecture). The correlation between estimated difficulty and ground-truth also does not seem to be very strong.\n- In the theorem, it is assumed rewards are independent from the gradient ($\\nabla_\\theta log (o_i | q_i)$). Can the authors motivate why these assumptions are reasonable?\n\nQUESTIONS:\n- Why not at least also include reference solutions in the embedding adaptive diffuclty embeddings to better calibrate solution difficulty?\n- In a similar direction, why not include an oracle training baseline estimating problem difficulty by rolling out the model for every problem? While this may be costly, it seems possible to maintain slightly off-policy difficulty estimates by reusing training rollouts.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a method to improve data efficiency in reinforcement learning (RL) fine-tuning of large language models (LLMs), primarily through two techniques: Difficulty-targeted Online Data Selection (DOTS) and Rollout Replay (RR) . The authors validate the effectiveness of their approach both theoretically and experimentally, conducting extensive evaluations across multiple LLM-dataset combinations. The results show that the proposed method reduces training time by 25% to 65% while maintaining performance.\n\nOverall, the paper is well-structured, clearly written, and presents a thorough experimental design with strong practical value. Especially given the high computational cost of current RL training methods, this work offers an efficient and reproducible solution.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n--Clear problem definition:\nThe paper precisely identifies the core issue of low data efficiency in LLM reinforcement fine-tuning and quantifies the severity of the problem using computational cost analysis (e.g., 3,800 A100 GPU hours / $4,500). The motivation is clear and highly relevant to real-world applications.\n--Comprehensive experimental design:\nThe experiments cover six LLM-dataset combinations (ranging from 1.5B to 7B parameters) and are evaluated on widely used mathematical reasoning datasets such as MATH and DeepMath. Multiple evaluation metrics are employed, providing solid empirical support for the claims.\n--Strong theoretical foundation:\nAppendix provides a complete proof of Theorem 1 , which states that gradient magnitude is maximized when the reward success rate is 50%. This theorem underpins the difficulty selection strategy and is supported by rigorous mathematical derivation.\n\nWeaknesses:\n--Incremental innovation:\nWhile DOTS and RR are effective, they represent incremental improvements over existing methods. Difficulty-based curriculum learning has been studied extensively in RL, and the paper does not sufficiently clarify the fundamental differences between DOTS and classical curriculum learning approaches. Similarly, RR is essentially a variant of experience replay, adapted for GRPO, but its technical contribution remains a straightforward extension rather than a novel breakthrough.\n--Assumption mismatch in Theorem 1:\nTheoretical result Theorem 1 assumes independent and identically distributed (i.i.d.) Bernoulli rewards and that the policy gradient is independent of the reward distribution. However, in LLM generation tasks, token predictions are inherently sequential and interdependent—early errors can propagate and affect later tokens and final rewards. Thus, the i.i.d. assumption may not hold, potentially undermining the validity of the theorem in practical settings.\n--Missing implementation details of the prediction framework:\nThe lightweight MLP adapter used for adaptive difficulty prediction lacks detailed architectural description (e.g., number of layers, activation functions). Additionally, it is unclear how the training data for the adapter is constructed and whether it is independent of the RL training process.\n--Potential failure cases in difficulty prediction:\nThe prediction framework estimates adaptive difficulty via embedding similarity interpolation. However, in domains like mathematics, surface-level similarity does not necessarily imply similar difficulty levels. For example, \"taking derivatives\" and \"computing integrals\" may have similar embeddings but vastly different difficulties. Furthermore, the reference set (only 256 samples) may lack sufficient coverage, leading to unreliable predictions for unseen question types.\n\nQUESTIONS:\n--Enhance Novelty Explanation:\nClarify the core differences between DOTS and automatic curriculum learning methods, emphasizing the innovation of using an attention mechanism to achieve zero-shot difficulty transfer.\n--Supplement Empirical Validation of the Independence Assumption:\nConduct empirical tests of the independence assumption in Theorem 1, such as binning statistics of gradient norm distributions across different success rate intervals p .\n--Refine Method Details:\nClearly specify the architecture of the MLP prediction framework, and discuss how well the assumptions in Theorem 1 hold in practical policy gradient settings.\n--Error Analysis by Question Type and Dynamic Reference Set Expansion:\nBreak down prediction errors by question type, and explore mechanisms for dynamically expanding the reference set. Include additional experiments—such as evaluating on non-mathematical tasks (e.g., AlpacaEval)—to validate generalization.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}