Title: Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM

RLHF

Data Scaling
Abstract: Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity.
To mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach not only exhibits enhanced resistance to reward hacking, but also enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named \textbf{Pre-PPO}, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that \textbf{prioritizing mathematical and coding tasks during the early phases of RLHF training} significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths.
Through comprehensive experiments conducted across two model sizes, we validate the effectiveness and scalability of our proposed methods. Results show that RTV exhibits the strongest resistance to reward hacking, followed by GenRM with ground truth, and finally GenRM relying on SFT Best-of-N responses. Moreover, our proposed strategies enable the model to rapidly capture subtle task-specific distinctions, leading to substantial improvements in overall RLHF performance. This work underscores the importance of careful data construction and provides practical methodologies to overcome critical performance barriers in RLHF.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper explores the challenges of data scaling in RLHF, focusing on reward hacking and response diversity issues. The authors introduce a hybrid reward system combining Reasoning Task Verifiers and Generative Reward Models to enhance model robustness and prevent reward manipulation. They also propose the Pre-PPO method for prompt selection, targeting difficult prompts to improve sample efficiency. Through experiments on a smaller model up to 25B parameters and a larger model up to 150B parameters, the authors demonstrate significant performance improvements, particularly in mathematical and coding tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n\n**S1. High Practical Relevance:** The paper addresses highly relevant challenges for RLHF practitioners, particularly in terms of reward model robustness and response diversity. These are practical concerns for scaling RLHF to large models, making the research applicable in real-world scenarios.\n\n**S2. Innovative Approach to RLHF Data Construction:** Unlike much of the existing literature, which mainly addresses algorithmic challenges, this paper emphasizes the crucial role of training data in improving RLHF performance. The introduction of Pre-PPO and the hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM) is an innovative approach to mitigating reward hacking and improving response diversity.\n\n**Weakness:**\n\n**W1. Further Exploration Needed for Academic Originality and Depth:** The paper's academic originality could be further strengthened, as its proposed methods, such as task prioritization and prompt selection, while demonstrating performance improvements, closely resemble earlier curriculum-learning approaches, which structure training by sequencing tasks from simpler to more reasoning-intensive to gradually build the model’s capabilities. For instance, the task prioritization strategy bears conceptual similarities to DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025), which also prioritize reasoning-intensive tasks during the early training stage of the model to enhance reasoning abilities. However, the paper does not sufficiently articulate how its approach surpasses or diverges from such prior studies, which substantially weakens the work’s novelty. To enhance its academic contribution, the authors should explicitly highlight the specific differences between their method and works like DeepSeek-R1, such as unique task selection criteria or novel integration with RLHF, and support these distinctions with theoretical analysis or comparative experiments.\n\n**W2. Need for More Systematic Exploration in Method Design:** The current task priority method is overly specialized and lacks universality due to the absence of systematic experiments. The design of the task prioritization method could be further optimized, particularly in terms of experimental rigor and mechanistic insight. The authors did not explore alternative setups, such as placing mathematical and coding tasks in the later stages of training (a common practice in some pretraining approaches) or adjusting data proportions (e.g., reducing the weight of non-mathematical/coding data) to test their assumptions. This lack of exploration may hinder a comprehensive evaluation of the method’s effectiveness.\n\n**W3. Limited Reproducibility Due to Closed-Source Data and Models:** The use of closed-source data and models in this study significantly hampers its reproducibility, thereby limiting the academic impact and verifiability of the findings. While the decision to keep resources proprietary may be justified, it prevents external researchers from replicating or validating the results. To enhance the study’s credibility and applicability, the authors are encouraged to conduct supplementary experiments using open-source models, such as qwen or llama, and publicly available datasets, like MT-Bench, AlpacaEval or GSM8K, to verify their conclusions and demonstrate the method’s robustness across diverse settings.\n\n**W4. Writing flaws (A minor point):**\nSome writing flaws exist in the paper. For example, the abstract contains repetitive sentences, particularly \"we propose a novel prompt-selection method named Pre-PPO\" appears twice. This redundancy reduces the conciseness and professionalism of the abstract, which should be as concise as possible as it serves as the facade of the paper. It is recommended to check the writing flaws in your paper.\n\nQUESTIONS:\n**1. Regarding the revision of paper for clarity and conciseness:**\n\nCan you revise the abstract to eliminate redundancy, such as the repeated mention of \"we propose a novel prompt-selection method named Pre-PPO,\" and ensure it concisely conveys your contribution? Could you also address any additional writing flaws (e.g., unclear phrasing) identified in the abstract or elsewhere in the paper?\n\n**2. Regarding differentiation from existing work:**\n\nWould you be willing to conduct a detailed comparison showing how your task prioritization and prompt selection methods differ from or improve upon prior work, such as DeepSeek-R1? Could you include specific evidence (e.g., theoretical differences or empirical results) to highlight your approach’s novelty in the RLHF context?\n\n**3. Regarding systematic exploration of task prioritization design:**\n\nCan you provide experiments testing alternative task orderings (e.g., placing mathematical and coding tasks later in training) or varying data proportions to assess their impact on performance? Alternatively, could you offer a detailed justification for your current design without such experiments?\n\n**4. Regarding validation of reproducibility with open-source resources:**\nWould you consider validating your results using open-source models (e.g., Qwen, Llama) and datasets (e.g., MT-Bench, AlpacaEval, GSM8K) to address reproducibility concerns tied to closed-source resources? If not, can you propose an alternative plan to enhance verifiability?\n\n**Evaluation Criteria:** My assessment would improve to \"Accept\" (5) if the authors provide a revised abstract with enhanced clarity and no redundancy, clearly differentiate their method from prior work like DeepSeek-R1 with specific evidence, conduct or justify experiments exploring alternative task orderings or data proportions, and validate results using open-source resources or a robust reproducibility plan. The score would move to \"Borderline Accept\" (4) if the authors partially address these issues—e.g., by improving the abstract and providing some comparisons or limited experiments—but fail to fully resolve concerns about originality, systematic exploration, or reproducibility.\n\nLIMITATIONS:\nNo, the authors have not fully addressed the limitations of their work. Although the paper provides a strong theoretical foundation and presents promising experimental results, some key limitations are not adequately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
The paper presents a comprehensive study on the data scaling in the RLHF stage of LLMs. The authors study two types of rewards, i.e. GenRM and RTV (reasoning task verifier). The authors identify that simply scaling the number of prompts does not leads to better performance. Instead, the authors provide two useful strategies for data selection in RLHF:
1. Pre-PPO prompt selection. Only keep the lowest-scoring 10% prompts, which suffers less from reward hacking.
2. Prioritize math and code in early training.

The authors conduct extensive experiments on two kinds of MoE models (25B and 150B), showing that these strategies mitigate the reward hacking and the diversity deterioration problems, and effectively improve the model performance.

STRENGTHS AND WEAKNESSES:
Strength:
1. Data selection is a crucial problem that needs more research. The paper has strong motivation and addresses a core problem.
2. The experiments are very comprehensive and large-scale. The experiments need a lot of resources to be run. The information from this paper is valuable.
3. The strategies are simple but effective.

Weakness:
1. All models and benchmarks are internal, making external comparison difficult.
2. It would be good to show more details,  like models, verifiers, data.

QUESTIONS:
1. Does this study cover tasks that need long-form outputs, for example, tasks (e.g., AIME) that may need inference scaling to over 10k tokens responses? If not, what would be different if these were included?

2. What if the GenRM is re-trained for several iterations during the RL process? Will that alleviate the reward hacking problem? Will the current discoveries still hold?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the data scaling challenge in Reinforcement Learning from Human Feedback (RLHF), where increasing the amount of training data does not consistently lead to improved model performance. \nTo handle this issue, the authors propose the following contributions:\n1) Hybrid Reward System: combining the Bradley-Terry (BT) reward model, reasoning task verifiers (RTV) and the generative reward model (GenRM).\n2) Pre-PPO Prompt Selection: training with low-reward prompts to effectively mitigate the reward hacking issue\n3) Early Training Emphasis: empirical finding that prioritizing mathematical and coding tasks during the early phases of RLHF training improves final performance\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) The proposed methods are simple, easy to implement, and demonstrate clear effectiveness in the experimental results.\n2) Addressing the data scaling issue in RLHF is both important and relatively underexplored, making this work a valuable contribution to the field.\n\nWeaknesses:\n1) The paper does not clearly demonstrate the individual or combined advantages of using the BT reward model, GenRM, and RTV. An ablation study or comparative analysis would strengthen this claim.\n2) Key aspects of the experimental setup are under-specified. For example, the composition and evaluation methodology of the \"V1.0\" and \"V2.0\" datasets are not clearly described.\n3) While the empirical results are promising, the paper lacks theoretical analysis or intuition explaining why Pre-PPO prompt selection and early prioritization of math and coding tasks improve robustness against reward hacking.\n4) The writing could be improved for clarity and conciseness. For instance, the abstract contains redundant descriptions of the Pre-PPO method (e.g., line 14).\n5) While the integration of components is thoughtful, the overall novelty is limited, and the paper would benefit from a more refined presentation and deeper analysis.\n\nQUESTIONS:\n1) Could the authors provide per-task learning curves for the Early Training Emphasis strategy? It would be helpful to see how this strategy affects individual task performance over time.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses an understudied aspect of RLHF: the role of prompt-data construction and its scalability. The authors identify two bottlenecks preventing effective data scaling in RLHF—reward hacking and declining response diversity—and propose practical solutions. Their approach introduces a hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM), along with a novel Pre-PPO prompt selection strategy that prioritizes challenging prompts. Additionally, they demonstrate that prioritizing mathematical and coding tasks during early RLHF training stages significantly improves performance. The methods are validated on models of different scales (25B and 150B parameters), showing consistent improvements across multiple evaluation metrics.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n1. Novel Problem Focus: While most RLHF research concentrates on algorithmic improvements, this paper uniquely addresses data construction and scaling—a critical but overlooked aspect that directly impacts practical deployment.\n2. Strong Empirical Results: Consistent improvements across model scales (25B and 150B) and evaluation sets, with particularly impressive gains in STEM (+3.9) and coding (+3.2) tasks.\n3. Good Rigor: The paper provides valuable insights into fine-grained vs. coarse-grained response differences and their impact on RLHF effectiveness, backed by comprehensive experiments in the appendices. Extensive appendices with ablation studies, case studies, and detailed analyses strengthen the paper's contributions.\n\nWeaknesses\n\n1. Limited Reproducibility: Due to proprietary data and code restrictions, the results cannot be independently verified, which is a challenge. The paper relies heavily on internal evaluation sets (V1.0 and V2.0) without comparison to widely-used public benchmarks, making it difficult to contextualize the improvements.\n2. Theoretical Justification: While the empirical results are strong, the paper lacks theoretical analysis of why Pre-PPO selection and early task prioritization work, relying mainly on intuition and post-hoc explanations.\n\nQUESTIONS:\n1. Your method shows strongest improvements in mathematical and coding tasks. Have you tested whether the approach generalizes to more open-ended tasks like dialogue or creative writing where ground truth is less well-defined?\n2. Public Benchmark Validation: Why weren't standard public benchmarks (e.g., HumanEval, MATH, GSM8K) used for evaluation? This would help the community better understand the practical impact of your improvements.\n\nLIMITATIONS:\nYes, the authors adequately discuss limitations in Section 5, acknowledging the resource-intensive nature of their approach, computational overhead, and uncertainty about long-form Chain-of-Thought RLHF. However, they could expand on:\n\n- The potential bias introduced by focusing heavily on STEM tasks during early training\n- How the method might perform with different reward model architectures\n- The impact of their approach on model capabilities outside the evaluated domains", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}