Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
Authors: Wei Shen
Guanlin Liu
YuYue
Ruofei Zhu
Qingping Yang
Chao Xin
Lin Yan
Venue: NeurIPS 2025 poster ()
Keywords: LLM
RLHF
Data Scaling
Primary area: reinforcement_learning

Abstract: Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity.
To mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach not only exhibits enhanced resistance to reward hacking, but also enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named \textbf{Pre-PPO}, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that \textbf{prioritizing mathematical and coding tasks during the early phases of RLHF training} significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths.
Through comprehensive experiments conducted across two model sizes, we validate the effectiveness and scalability of our proposed methods. Results show that RTV exhibits the strongest resistance to reward hacking, followed by GenRM with ground truth, and finally GenRM relying on SFT Best-of-N responses. Moreover, our proposed strategies enable the model to rapidly capture subtle task-specific distinctions, leading to substantial improvements in overall RLHF performance. This work underscores the importance of careful data construction and provides practical methodologies to overcome critical performance barriers in RLHF.

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback
Wei Shen, Guanlin Liu, Yu Yue, Ruofei Zhu, Qingping Yang, Chao Xin, Lin Yan
ByteDance Seed
(shenwei.09H guanlin. 1iu, yueyu, zhuruofei.245 jiangyun.jvy jy, xinchap. neilJabytedance.. com
Abstract
Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused algorithmic sddvancements-suss as reducing computational overhead or strengthening reward models to mitigate reward hack- ing ing-the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing re- sponse diversity. To mitigate reward hacking, we introduce hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, order to ensure response diversity and en- hance learning effectiveness, we propose a novel prompt-selection method named Pre-PPO, explicitly identifying training prompts that are inherently challenging and thus less prone reward hacking. Furthermore, mathematical and coding tasks during the early phases of RLHF training sig- nificantly boosts performance, given that these tasks naturally encode fine- une-grained response distinctions and possess clearly defined ground truths. Through experi- ments conducted on both small and large models, we demonstrate the effectiveness and scalability of our proposed methods. Our approach exhibits robust general- ization capabilities, especially on challenging and out-of -iit--i--ittiuttioon tasks, while yielding significant improvements in mathemmatics-itteniiv (STEM) and coding domains. Moreover, the proposed strategies enable the model effectively capture subtle, task-specific distinctions during the RLHF process, substantially enhancing overall model performance. This work emphasizes the critical role of careful data construction and provides practical methodologies for addressing key performance bottlenecks in RLHF.
find that prioritizing
Introduction
Reinforcement Learning from Human Feedback (RLHF) is crucial technique for aligning large language models (LLMs) with human values and preferences [ 2 studies have focused on enhancing the efficiency and performance of RLHF at the algorithmic level. Methods proposed include dropping the critic model from PPO to reduce computational overhead and filtering noisy samples during the PPO sampling process achieve more efficient training and improved performance [9] Additionally, many papers [1O] [T [2] focus on leveraging the capability of the reward model to mitigate the reward hacking problem, thereby enhancing the
3 []. Recently, numerous
66 7 8
"Correspondigg Author. Email: guanlin. liu @ @eytedance.com
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

performance of RLHF. However, there have been few studies [3] focusing on the construction of RLHF data (i.e., training prompts) and its performance scaling based on these training prompts, despite extensive research on large-scale datasets during pretraining and supervised fine-tuning (SFT)
stages 14 15 [6].
In this paper, we find that simply scaling the number of training prompts during RLHF does not lead to performance improvements. Accordingly, we investigate the bottlenecks of data scaling in RLHF and propose novel methods for constructing training prompts, as well as strategies to enhance RLHF performance. We identify two primary obstacles to effective data scaling: reward hacking and the deterioration of model response diversity. To mitigate reward hacking, we develop a hybrid reward system that integrates Reasoning Task Verifiers (RTV) [17 [8] with Generative Reward Models (GenRM) 20), allowing model predictions to be validated against reliable ground-truit references. Additionally, we find that training models primarily on coarse-graines response differences rapidly diminishes response diversity, causing important fine-gran fine-grained distinctions among responses to be overlooked. While directly preventing this reduction in diversity during RLHF training remains challenging, we demonstrate that targeted prompt selection can facilitate the learning of subtle, fine- grained distinctions and make reward hacking more difficult. Specifically, we introduce an innovative Pre-PPO prompt selection method that emphasizes more challenging prompts and is inherently more robust to reward hacking. Furthermore, we find that initially prioritizing mathematical and coding tasks during RLHF training consistently yields superior results, these tasks naturally encourage fine-grained distinctions and provide safeguards against reward hacking due to their well-defing dell-defined ground truths.
Our main contributions are as follows:
Comprehensive Analysis of RLHF Scaling Bottlenecks: We systematically analyze the key factors that hinder RLHF performance scaling, identifying reward hacking and the deterioration of model response diversity as primary obstacles.
Joint Prompt Selection and Task Prioritization Strategy: We propose novel training strategy that combines Pre-PPO prompt selection method strategically chooses difficult and informative prompts-with with a curriculum that prioritizes mathematical and coding tasks in the early stages of RLHF training. This unified approach enables the model to acquire fine-grained distinctions while improving robustness reward hacking.
We conduct extensive experiments on two distinet model sizes, demonstrating the effectiveness and scalability of our proposed methods. Comprehen- sive ablation studies confirm that each technique individually contributes to RLHF performance improvements at different model scales.
Empirical Validation Across Model Scales:
2 Related Work
Reward hacking. Reward hacking occurs when an AI system exploits flaws in the reward function to maximize rewards without achieving the intended objectives. Consequently, the success of RLHF heavily depends on the quality of the reward model. Unfortunately, reward models often struggle to provide accurate scores due three main challenges: 1) mis-specified reward modeling in : 2) the presence of incorrect and ambiguous preferences in training datasets (23 24: and 3) poor generalization ability 25). These inaccuracies in reward modeling have been identified as major contributors to reward hacking and hallucination in LLMs 26 Recent work by Zhang et al. (20 introduced a generative reward model (GenRM) to validate model predictions against ground-truth responses, demonstrating greater resistance to reward hacking and has been adopted by statc-of-the- art LLMs such as DeepSeekV V3 [27]. Additionally, Deepseek- RI [18] developed reasoning task verifiers (RTV) that have proven effective in addressing reward hacking, particularly in mathematical, coding, and other reasoning tasks. While previous research has focused on improving the accuracy of reward models themselves,
representing human preferences [2] 22]
work takes different approach: we aim to design an effective RLHF data construction method under a robust reward system that combines both GenRM and RTV to mitigate the reward hacking problem.
RLHF data construction. There are few works that focus on how to construct RL data (i.e., RL training prompts) to enhance RLHF performance. Gao et al. (28 propose a principled data selection method for the DPO algorithm, where they find overly difficult data hinder alignment and filter out such challenging instances during DPO training. Additionally, Li et al. [29] introduce strategic

---

selection method to identify key training prompts from complete prompt set, achieving comparable RLHF performance while using only a subset of the data. While these methods demonstrate that careful dataset curation can match or exceed the performance of training complete datasets, there remains a significant gap understanding the factors that limit RI. data scaling in PPO-based training. Specifically, no existing work has systematically analyzed how to select and structure training prompts to substantially improve model performance when using the PPO algorithm.
RLHE performance scale analysis. Recent studies have increasingly focused on analyzing RLHF 24 30), particularly examining its generalization ability and response diversity. Kirk al. demonstrate that RLHF exhibits superior generalization compared to Supervised Fine-Tuning Tuning (SFT) on novel inputs, especially as the distribution shift between training and testing data increases. However, they also observe that RLHF significantly reduces output diversity compared to SFT across various metrics, suggesting a fundamental trade-off between generalization and diversity in current L.M fine-tuning approaches. Furthermore, several recent works 32 33
13
31
investigate how RLHF can effectively distill the best responses as evaluated by reward models, proposing various algorithms to enhance this distillation capability. In our study, similarly observe that the diminishment of response diversity impedes RLHF scaling, particularly when models attempl to learn coarse differences among responses. Additionally, we observe that RLHF performance is only comparable to the strategy of sampling five responses from the SFT model and then selecting the highest- one according to the reward model (i.e., SFT Bo5). This observation motivates further investigation into methods for enhancing the effectiveness of RLHF.
3 Approach
3.1 Framework Overview
GenRM with
Prompts
PPO
Responses
GenRM w/o GT
Reward
(a) HPO Framework
Codle and Math Prompts
Selected 10% Prompts
First Stage
Secondblage
Prompt Schedule Training Process
(C) Solect Prompts from Pro.
Figure 1: Overview of our RLHF framework. Our pipeline consists of two stages: (1) Reward Model Training, where we construct three complementary reward models: a Bradley-Terry (BT) model trained on human pairwise preferences; Generative Reward Model (GenRM) assigning explicit reward scores using either ground-truth (GT) responses or BT-best responses when GT unavailable; and Reasoning Task Verifiers (KTV) providing task-specific validation (such as code- execution sandboxes); and (2) Reinforcement Learning Optimization, where the language model is trained using PPO guided by GenRM and RTV feedback, and leveraging strategically selected prompts via our Pre-PPO method. Additionally, we explicitly prioritize matherattial and coding prompts during early training stages, subsequently incorporating general-domain tasks.
As shown in Figure T our RLHF pipeline consists of three main stages:
Initial Supervised Fine-tuning: We first fine-tune the pre-trained language model on human- written demonstrations to achieve basic instruction-followwee capabilities.
Reward Model Training:
We prepare three types of reward models: the Bradley-Terry Reward Model (BT Model), the Generative Reward Model (GenRM), and Reasoning Task Verifiers (RTV).
The BT model leams reward functions using pairwise comparison data, optimizing parameters

---

via maximum likelihood estimation to infer underlying reward scores from human preferences. The Generative Reward Model (GenRM) is trained using Pairwise Reward Modeling (pairRM), where the model learns from human preference judgments of paired outputs (34 Rather than assigning individual scores to each output, GenRM directly predicts a comparative score for each pair, optimizing these pairwise comparisons to align closely with human evaluations. To provide GenRM with a reliable ground truth for comparison, we collect explicit ground-truth answers for reasoning tasks. For other task types, utilize the trained BT reward model to select the best outcome from N candidate samples generated by the supervised fine-tuned (SFT) model. For RTV, construct series of specialized verifiers to directly validate the correctness of model responses for specific tasks. These include, for example, code sandboxes for programming tasks, which can execute and evaluate code outputs in real-time.
Reinforcement Learning Optimization: We leverage a combination of GenRM and RTV to provide comprehensive feedback for optimizing the language model through PPO. This process is guided by carefully curated training prompts and a well-crafted training strategy. The iterative nature of this approach progressively refines the model's outputs by maximizing predicted reward scores while ensuring minimal deviation from its original policy.
During RL training, we observe two key factors that hinder the performance scaling of RLHF: first, reward hacking. As shown in Figure 2 the overall performance of the model undergoing RLHF exhibits an initial increase followed by a decline during training. Specifically, abilities in mathematics, creative tasks, and instruction following all follow this pattern of improvement and subsequent deterioration. Our analysis reveals that reward hacking occurs across these tasks, where the model learns to generate responses containing certain syntactic patterns that artificially inflate reward scores rather than genuinely improving task performance; second, the deterioration of response diversity. During the RLHF process, we observe continuous decline in the entropy of model responses (illustrated in subfigure (a) of Figure 6 in the Appendix), indicating a decrease in response diversity.
Despite numerous efforts to address reward hacking and diminishing response diversity-including approaches such iterated RLHF (24 35 and reinforcement learning from pre-trained models [[8] 36 these challenges remain difficult to fully resolve. To mitigate these issues, we propose proactive prompt selection strategy combined with curriculum learning approach.
Training Process
39
38
38.5
38.6
38
37.8
37.9
Overall Test Score
3/
36
36
35.5 5
35
34
Six Million Prompts
Prompts
1000
2000
3000
4000
5000
Training Steps
Figure 2: Overall test scores from the initial run using an expanded dataset combining newly collected data (six million prompts) with the original dataset (one million prompts). Despite increasing dataset size substantially, RLHF did not yield improvements in performance. Additionally, the best performance was observed at around the 3,500-step mark, after which test scores gradually declined.

---

3.2 Pre-PPO for Training Prompts Selection
Initial PPO Experiment. In our first trial, we initially collected 5 million new training prompts covering various domains, such mathematics, coding, creative writing, and additional tasks. These new prompts were combined with the original I million prompts to train for the first trial. As illustrated in Figure 2 we observed that the RLHF performance did not improve despite the increase in the number of prompts. Consequently, we conclude that simply expanding the number of training prompts does not necessarily yield improved RL performance.
Reward Analysis of Newly Collected Prompts. We investigated why newly collected prompts did not improve RLHF performance by analyzing their reward scores. We found that approximately 90% of these prompts received reward scores greater than 0.5 on a scale from to 1 (see Appendix Figure In this scoring scheme, score of 0.5 indicates that the model's output is comparable to the reference, while scores above 0.5 suggest superior performance. Our GenRM trained to compare the model response with the ground truth in reasoning tasks and SFT Best-of-N responses in other tasks. Therefore, scores above 0.5 imply that the model-generated outputs were judged as superior to these presumed optimal responses. However, after careful manual inspection, we discovered that a substantial portion of these high-scoring outputs exhibited reward hacking behavior and were qualitatively worse than the original best-selected responses.
Selecting Prompts with Lower Reward Model Scores for RL Training. Given the observations above, we designed selection algorithm called Pre-PPO, which explicitly identifies prompts with lower reward model scores for use in the initial PPO experiment. These low-scoring prompts are both more challenging for the model to learn from and less susceptible to reward hacking. Finally, we combined these selected prompts with the original prompt dataset to retrain the RL model. Recognizing that reward model scores exhibit distinct distributions across different task domains, we normalize these scores within each domain before performing prompt selection.
3.3 staly-stage RLHF: Prioritizing Mathematical and Coding Tasks
In our initial trials, we observed that test scores for both coding and math tasks steadily improved throughout training. In contrast, other tasks without explicit ground-truth encountered reward hacking issues early in the RLHF process (see Appendix Figure 10 We attribute the improvement in coding and math tasks to their evaluation methods: specifically. these tasks are assessed by RTV and GenRM using ground-truth references, which makes them inherently more resistant reward hacking. Accordingly, we explicitly trained the RLHF model on math and coding prompts during the early stages. Subsequently, we combined these math and coding tasks with general-domain prompts to continue RLHF training. This approach can enhance performance on both coding and math tasks while preserving general capabilities.
Experiments Details
4.1 Experimental Setup
We describe our experimental setup as follows:
Models:
We conducted our experiments using two pre-trained Sexture-of-Exppoe (MoE) language models: a smaller model (approximately 25B total parameters) and larger model (approximately 150B total parameters), Due to the MoE architecture, only about 10% of the total parameters are activated per token.
Our original dataset consists of one million training prompts collected from publicly available sources and through human annotation. These prompts span diverse domains, including mathematics, coding, instruction-follot following, creative writing, logical reasoning, and other related tasks. To expand our dataset, we additionally collected five million new prompts primarily from open-source resources, primarily covering mathematics, coding, cstruction-followiin and creative writing tasks. (The detailed distribution of our training prompts is illustrated in Figure 9 in the Appendix.)
Prompts:
Evaluations: Given that external public benchmarks are susceptible to data contamination, we constructed an internal evaluation set that comprehensively covers multiple skill areas. This set
5

---

Method
LR
IF
STEM
Coding
NLP
Knowledge
CU
OOD
Overall
Initial Run-25B (V1.0)
27.1
34.8
49.3
51.6
37.0
40.0
39.0
37.7
Baseline-25B1 (V1.0)
26.4
35.1
48.8
50.9
24.8
36.1
40.6
40.5
37.7
Data Scale-25B (V1.0)
28.7
36.1
50.4
53.3
24.2
36,6
39.7
43.6
38.8
Improvement
+2.4
+1.1
+1.6
+2.4
-0.6
+0.6
-0.9
+3.1
+1.1
Baseline- 50B (V1.0)
37.3
46.3
55.5
45.7
58.4
54.5
49.7
Data Scale- 150B (V1.0)
39.6
46,0
56.5
58.7
44.9
47.9
59.6
55.6
50.8
Improvement
+2.2
-0.4
+0.9
+3.2
-0.8
+1.1
+1,2
+1.2
+1.1
Baseline (V2.0)
17.6
26.5
26.5
41.2
21.2
28.2
19.6
21.3
23.9
Data Scale-25B (V2.0)
19.9
27.3
29.5
42.3
21.8
28.9
20.2
21.7
25.1
Improvement
+2.3
+0.8
+3.0
+1.1
+0.6
+0.7
+0.8
+0.4
+1.2
Baseline- 1 150 (V2.0)
29.5
36.3
28.0
48.5
29.5
45.6
36.8
35.0
34.0
Data Scalc-150B (V2.0)
31.2
36.4
31.9
50.7
32.3
45.5
36.6
37.1
35.4
Improvement
+1.8
+0.1
+3.9
+2.1
+2.7
-0.1
-0.2
+2.1
+1.4
Table 1: We present performance comparison between our proposed method, termed "Data Scale,' and baseline method (PPO-based RLHF) on evaluation datasets V1.0 and V2.0. Results are reported across various abilities, including logical reasoning (LR), instruction- sorrucctoonfollowinng (IF), STEM tasks, coding, natural language processing (NLP), knowledge, contextual understanding (CU), and out- of-distribution generalization (OOD), Results highlighted in bold indicate statistically significant improvements.
consists of two main versions, V1.0 and V2.0, as well separate human evaluation subset. While V1.0 and V2.0 share some overlapping prompts, the newly introduced prompts in V2.0 are notably more challenging. Importantly, all prompts in V2.0 and the human evaluation set are not accessible to researchers at any stage during optimization, ensuring an unbiased and fair assessment. Model performance (human-based) evaluations, with the latter conducted using the newly introduced human evaluation subset. Furthermore, ensure fair comparison with external models and reproducibility of our method, we conduct comprehensive evaluations on a series of open-source benchmarks in Appendix B
assessed using both automated (LLM-basedd) and manual
Further details on experimental setups- -nceus-inc Experimental Details of Pre-PPO and Experimental Details of Prioritizing Mathematical and Coding Tasks-are are provided in the Appendix C.2
4.2 Experimental Results
Overall Evaluation Results. The experimental results presented in Table and Table the following key findings:
2 demonstrate
Overall Performance Improvement. Our proposed approach (combining Pre-PPO with prioritized mathematical and coding tasks) consistently and significantly outperforms the baseline method (PPO with the original dataset) across different model sizes and evaluation datasets.
Strong Generalization on More Challenging Test Sets. We evaluate checkpoints from both the baseline method and our proposed approach at every 100 training steps using TestSet V1.0, and select the best- pest-performing checkpoint from training steps up to 4000. Under this evaluation, our approach achieves noticeable improvement (+1.1) over the baseline on TestSet V1.0. Furthermore, when comparing the best checkpoints from each method on a more challenging TestSet V2.0, our approach yields even greater performance increase (+1.4). Given that TestSet V2.0 contains substantially more challenging prompts than TestSet V1.0, these results indicate that the proposed approach exhibits robust generalization capability, especially on harder, tiitof---itrbbttion tasks.
Significant Improvements in Mathematical and Coding Tasks. Our proposed approach notably enhances performance on mathematics-intenn intensive (STEM) and coding tasks. Specifically, we observe improvements of +3.9 points in STEM (Large, V2.0) and +3.2 points in coding (Large, V1.0), alongside consistent gains across other model sizes and datasets. We attribute these significant improvements to our strategic prioritization of mathematical reasoning and coding tasks during the early stages of RLHF training, which effectively strengthens the model's capabilities these specialized areas.

---

Method
Knowledge
STEM
IF
Creation
Coding
Overall
Lassle
63.3
76.7
46.7
52.1
67.2
64,4
Data Scrge-Large
66.1
80.6
48.3
54.6
71.0
67.6
Improvement
+2.8
+3.9
+1.6
+2.5
+3.8
p-value
0.41
0.04
0.39
0.09
0.12
0.01
Table 2: Performance comparison based on comprehensive human evaluations between our proposed
method, termed 'Data Scale' and the baseline method (PPO-based RLHF). Results are shown across multiple abilities, including Knowledge. STEM, Instruction-Folle Following (IF), Creation, Coding, and Overall performance. Improvements highlighted in bold indicate statistically significant differences. All metrics represent aggregated scores from human assessments.
Effect Model Performance (Higher is Better) Study: Comparison Training Strategies
39
27
Performancs
35
Raseline
Pre-PPO
Early Training Emphasis
500
1000
2000
3000
Training Steps
Figure 3: Ablation study on small-size model.
Case Study Comparisons. Based on feedback from human annotators (as shown in Table observe clear improvements across different task categories. For STEM-related tasks, annotators noted notable enhancements in logical reasoning and content richness, exemplified in Case F. the Appendix. Meanwhile, complex creative tasks exhibited moderate improvements. specifically in adhering secondary instructions and overall content quality, including enhanced literary style and increased richness, as illustrated in Case F.2 For coding tasks, annotators recognized distinct improvements in information accuracy and depth of content (Case F.3).
4.3 Ablation Studies
To investigate the impact of Pre-PPO and the early-stage emphasis on mathematical and coding tasks, independently compare the performance of each approach against baseline methods TestSet V1.0. Due to computational constraints, all subsequent experiments, except for those analyzing scaling trends with respect to model size, conducted exclusively using the small-sized model.
Prompt Selection with Pre-PPO. The experimental results depicted in Figure reveal that the Pre-PPO method achieves comparable performance to the baseline approach up to the 2000-step mark in the training process. Notably, Pre-PPO demonstrates continued improvement between 2000 and 4000 steps, whereas the baseline performance plateaus. This sustained enhancement suggests that the prompts selected through Pre-PPO are more resistant to "hacking". thereby fostering continued learning and ultimately boosting the effectiveness RLHF
Impact of Early Training Emphasis on Mathematical and Coding Tasks. As shown in Figure 3 early training emphasis on mathematical and coding tasks can improve the overall performance of RLHF. Furthermore, as shown in Figure early emphasis mathematical and coding tasks during training significantly enhances both the coding and STEM performance RLHF models. Notably, in the coding task, the early training emphasis method surpasses the bascline performance plateau
7

---

Method
Logical Reasoning
IF
STEM
Coding
NLP
Knowledge
CU
OOD
Overall
Baseline-Laage (V2.0)
29.5
36.3
28.0
48.5
29.5
45.6
36.8
35.0
34.0
Pre-PPO-Large (V2.0)
31.3
35.9
30.8
49.5
32.3
45.7
36.1
37.9
35.1
Improvement
+1.8
-0.4
+2.5
+1.0
+1.8
+1.1
-0.7
+2.9
+1.1
Data Scale-Large (V2.0)
31.2
36.4
31.9
50.7
32.3
45.5
36.6
37.1
35.4
Improvement Pre-PPO
-0.2
+0.5
+1.1
+1.2
+0.0
-0.2
+0.5
-0.8
+0.3
Table 3: Ablation Study: Performance Scaling of Pre-PPO and Early Training Emphasis Large
Language Models
Ablation Study: Effect on Model Performance (Higher Better)
Comparison on Coding Task
Comparison on Math Task
32.0
Performance Metric
51.0
Performance Metric
48.0
50.0
40.0
48.0
64.0
Coding
45.0
Early Training Emphasis-Codin
Early Training Emphasis-
46.0
40.0
700
2800
2300
2800
Training Steps
Training Steps
Figure Early emphasis on mathematical and coding tasks significantly improves RLHF performance in both coding and STEM areas on Testset- V1.0. Notably, the coding performance with this approach surpasses the baseline within just 1000 training steps.
early the 1000-step mark. This improvement demonstrates the efficacy of prioritizing these foundational skills in the initial stages of training, leading to accelerated learning and superior overall performance.
Performance Scaling Trends of Model Size. As shown in Table] our method demonstrates direct applicability to large-scale models, achieving significantly better performance than the baseline. This indicates positive scaling trend with respect to model size.
Effect Model Performance (Higher Better)
Data Scaling Pre-PPO
+10 percent new prompts
24.5
+20 percent new prompts
+50 percent new prompts
Performance Metric
Original Prompts
26.0
23.0
22.5
21.5
Training Steps
Figure 5: Impact of data scaling on Pre-PPO strategy performance. The graph shows the overall RLHF performance as the percentage of newly collected training data increases from 10% 20% and 50%.

---

Furthermore, we investigated the individual performance of Pre-PPO and early emphasis strategies on mathematical and coding tasks when applied large models. Due to computational constraints, we focused on applying the Pre-PPO strategy (using data selected from the small-size model) to the large model for dataset selection in our ablation study. As shown [ Pre-PPO yields a significant performance improvement in large models. Similarly, emphasizing mathematical and coding tasks early in the training process of large models results in substantial performance gains in both STEM and coding evaluations. However, this approach yields only marginal improvements in overall performance. Notably, directly scaling the hyperparameters from the small model to the large model does not necessarily yield optimal results; in future work, will further investigate how to properly scale hyperparameters to ensure even greater performance boosts.
Data Scaling in the Pre-PPO Strategy. We investigated the effect of data scaling within the Pre-PPO strategy, as shown in Figure 5 However, increasing the amount of newly collected training data from 10% to 20% or 50% leads to a noticeable degradation in overall RLHF performance. This nounter-intuitive result suggests that high-quality training prompts are scarce in real-world settings, and simply scaling the quantity of collected data does not necessarily lead to improvements. In future work, we will explore approaches to generating prompts directly from large language models (LLMs) themselves, which we consider a more promising direction than relying purely on real-world collections.
5
Discussion
Although we have demonstrated the effectiveness and positive scaling trend of our method, we aim to further explore the underlying mechanisms that contribute to its success. In Appendix [D] we design a series of experiments to seek understand why our approach enhances the performance of RLHF and how it breaks through two critical bottlenecks: reward hacking and the deterioration of model response diversity. We observed that early acquisition of fine-grained response differences enhances performance scaling ideally. Specifically, We find that explicitly prioritizing prompts exhibiting ene-grained response differennes-eeseccillly in mathematical and coding tasks supervised by reward models with ground-truth or verification feedback significantly enhances model performance and scalability. In contrast, prematurely introducing tasks dominated by coarse-grained variations (e.g.. creative writing) negatively affects learning effectiveness. This highlighted sensitivity underscores the importance of initially acquiring fine-grained response distinctions to mitigate adverse impacts from early exposure to coarse- gaasse-rrined patterns.
Our method requires the collection of large number of prompts paired with high-quality reference answers, which can be resource-intensive. Additionally, incorporating pre-PPO steps involves executing an initial PPO process filter and augment data, thereby increasing computational overhead during training. While emphasizing fine-grained response distinctions has shown promise, further research is necessary understand how best implement this in different contexts and tasks. Moreover, the impact of our approach on long-form Chain-of- Thought RLHF remains uncertain, as current analyses are limited to specific scenarios and may not directly transfer more complex, multi-step reasoning tasks. These factors could pose challenges for practical deployment and broader applicability. Please refer to Appendix for additional discussion.
Conclusion
In this paper, we explored two key bottlenecks preventing effective data scaling in Reinforcement Learning from Human Feedback (RLHF): reward hacking and declining diversity of model responses. To address these issues, we introduced hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM) with ground-truth supervision, significantly en- hancing resistance to reward hacking. Furthermore, we proposed novel Pre-PPO prompt selection strategy that prioritizes challenging prompts to help models better capture fine-graine response distinctions, and demonstrated the advantages of prioritizing mathematical and coding tasks early in the training process. Our results indicate that strategically curating training prompts markedly improves response diversity and overall RLHF performance and scalability. We hope our work pro- vides foundational insights for future research into optimized RI.FF data construction and stronger alignment methodologies.

---

References
[1] Sai Vemprala, Rogerio Bonatti, Arthur Bucker, and Ashish Kapoor. Chatgpt for robotics: Design principles and model abilities. Microsoft Auton. Syst. Robot. Res, 2:20, 2023.
[2] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.088 087744 2023.
[3] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.021555 2022.
[4] Al Anthropic. Introducing claude, 2023.
URL
https: / http://wwww..th..i...oc wwm/nemocoaeeeee
introducing- claude
[5] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- -ean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312. //805 2023.
[6] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300. 2024.
[7] Ziniu Li, Tian Xu, Yushun Zhang, Yang Yu, Ruoyu Sun, and Zhi-Quan Luo. Remax: A simple, effective, and efficient method for aligning large language models. arXiv preprint arXiv:2310. 10505, 2023.
[8] Yuan Wu et al. Back to basics: Revisiting reinforce style optimization for learning from human feedback in 1lm.. arXiv preprini arXi::2402.14744 2024.
[9] Wei Shen and Chuheng Zhang. Policy filtration in rlhf to fine-tune Ilm for code generation. arXiv preprint arXiv:2409.06957, 2024.
[10] Leo Gao, John Schulman, and Jacob Hilton. Scaling laws for reward model overoptimization.
In International Conference on Machine Learning, pages 10835-10866. PMLR, 2023.
[11] Minae Kwon, Sang Michael Xie, Kalesha Bullard, and Dorsa Sadigh. Reward design with language models. arXiv preprint 2:Xxx:330330000 2023.
[12] Binghai Wang. Rui Zheng, Lu Chen, Yan Liu, Shihan Dou, Caishuang Huang, Wei Shen, Senjie Jin, Enyu Zhou, Chenyu Shi, et al. Secrets of rihf in large language models part ii: Reward modeling. arXiv preprint arXiv:2401 200,000000000 2024.
[13] Zhenyu Hou, Pengfan Du, Yilin Niu, Zhengxiao Du, Aohan Zeng, Xiao Liu, Minlie Huang, Hongning Wang. Jie Tang, and Yuxiao Dong. Does rlhf scale? exploring the impacts from data, model, and method. arXiv preprint 1iki::2412.06000 2024.
[14] Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001. 003611 2020.
[15] Ziquan Liu, Yi Xu, Yuanhong Xu. Qi Qian, Hao Li, Xiangyang Ji, Antoni Chan, and Rong Jin. Improved fine-tuning by better leveraging pre- pee-training data. Advances in Neural Information Processing Systems, 25:32568-32553 2022.
[16] Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, et al. Lima: Less is more for alignment. Advances in Neural Information Processing Systems, 36:55006-55021, 2023.
[17] Duo Zhou, Junyu Zhang, Tao Feng, and Yifan Sun. A survey on alignment for large language model agents.
[18] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. arXiv preprint arXiv:2501.12948, 2025.
[19] Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, and Alon Albalak. Generative reward models. arXiv preprint 2.0.:22110.2232... 2024.
10

---

[20] Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative verifiers: Reward modeling as next-token prediction. arXiv preprint arXiv:2408. /5240, 2024.
[21] Nathan Lambert, Thomas Krendl Gilbert, and Tom Zick. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pages arXiv-2310, 2023.
[22] Silviu Pitis. Failure modes of learning reward models for Ilms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Baae -Bseed Learning, 2023.
[23] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedhack. Advances Neural Information Processing Systems, 35:27730-27744, 2022.
[24] Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, al. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:220..0586. 2022.
[25] Lev McKinney, Yawen Duan, David Krueger, and Adam Gleave. On the fragility of learned reward functions. arXiv preprint arXiv:2301.03652, 2023.
[26]
Adam Tauman Kalai and Santosh Vempala. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pages 160-171, 2024.
[27] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arXiv:2412.19437, 2024.
[28] Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, and Zhiqiang Xu. Principled data selection for alignment: The hidden risks of difficult examples, arXiv preprint arXiv:2502.09650, 2025.
[29] Xuefeng Li, Haoyang Zou, and Pengfei Liu. Limr: Less is more for rl scaling. arXiv preprint arXiv:2502. /8866 1886,
2025.
[30] Thomas Coste, Usman Anwar, Robert Kirk, and David Krueger. Reward model ensembles help mitigate overoptimization. arXiv preprini arXiv:2310.02745 2023.
[31] Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. Understanding the effects of rlhf on I1m generalisation and diversity. arXiv preprint arXiv:2310.06452 2023.
[32] Lin Gui, Cristina Gárbacea, and Victor Veitch. Bonbon alignment for large language models and the sweetness of best-of-n sampling. arXiv preprint arXiv:2406.000 2024.
[33] Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexan- dre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, et al. Bond: Aligning Ilms with best-of-n distillation. arXiv preprini arXiv:2407. 14622, 2024.
[34] Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Lim-blender: Ensembling large language models with pairwise ranking and generative fusion. arXiv preprint vrXiv:2306.02561, 2023.
[35] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Ane Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[36] Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.080733 2022.
11

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract, claim that we introduce hybrid reward system and a novel prompt-selection method named Pre-PPO, which enables the model to rapidly capture subtle task-specific distinctions, leading to substantial improvements in overall RLHF performance.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our paper in Section 5
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
12

---

Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: In Sections 3 and 4, have provided comprehensive and detailed descriptions of our proposed methods and clearly outlined our experimental setup.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
13

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification: Due to company privacy restrictions, the internal data and code utilized in this work cannot be openly shared. Thus, the paper does not provide open access to data and code or instructions for their reproduction in the supplemental materials. However, the methods and experimental setups are clearly and thoroughly described within the main text.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We show the details in Section 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: All experiments have been conducted at least three times to ensure statistical robustness.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
14

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question:
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [No]
Justification: Due to company confidentiality constraints, details regarding the computa- tional resources used for running experiments, such computing hardware types, memory specifications, and execution time, cannot be publicly disclosed.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the cttp:ppssccpppppcccccccc
NeurIPS Code of Ethics httpg//sss/ppppp.ccooooo////iiir
Answer: [Yes]
Justification: We have considered potential harms caused by the research process, societal impacts, and potential harmful consequences, as described in the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our paper addresses research field that not currently utilized for direct practical applications, thereby limiting its immediate social impact.
15

---

Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Not Applicable
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: In our paper, we cite all sources that inspired our work, and we provide links in our git repository to the codes from which we drew inspiration.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
16

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the datasets.paperssiii paperswi thcode. www.com//aaaa datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [Yes]
Justification: the paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
17

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does not involve LLMS as any important, original, non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy hatt:p//eearrpppcc///ooeeeeeeeee /ct/////oeeeeeccc///2/////. for what should or should not be described.
18

---

A The deterioration of model response diversity
During the RLHF training process, we observe continuous decline in the entropy of model responses (illustrated in subfigure (a) of Figure O indicating reduced response diversity. Such decline not only constrains the model's capability to produce varied and creative outputs but may also negatively impact its adaptability and generalization across diverse tasks and contexts. Additionally, we analyze the entropy across various task categories and observe that tasks associated with creative writing, role-play, and others supervised by GenRM without ground truth exhibit notably higher entropy than tasks involving mathematical, coding, and logical reasoning skills tasks that typically are supervised by GenRM with ground truth. We compare response entropy between the baseline and our proposed method in subfigures (b), (c), and (d), categorizing the results according to the reward model types: GenRM with ground truth, GenRM without ground truth, and RTV. We observe that, for tasks supervised by GenRM with ground truth or RTV, the response diversity using our method is lower than that of the baseline. In contrast, for tasks supervised by GenRM without ground truth, our method exhibits higher response entropy compared to the baseline. These observations indicate that our proposed method effectively guides the model to focus more explicitly on tasks supervised by RTV and GenRM M with ground truth, thus enabling the model to acquire more fine-grained response distinctions during RL.HF training.
Mean Entropy Training
Mean Entropy Training
0.80
0.18
Baselire GenRM, with
DataScale with G1
0.55
Entrpy
Mean Entrgy
0.14
0.50
0.45
0.10
0.06
0.35
0.04
0.02
250
500
750
1000
1250
1500
1750
0
250
500
750
1000
1250 1500
1750
Step
Step
(a) Response entropy change during the RL.FF training process.
(b) A comparison of response entropy changes during the RLHF training process, aggregated across tasks supervised by the 'GenRM with ground truth'.
Entrepy over Iraining
Mean Entropy over Training Steps
0.60
0.225
Basselnn GeentM GeniN__ithhouu withour_Gi SelaacaleeGenihee BatnRMMWttttt_tCC
Beseline-KIV DataScala-RIV
0.55
0.200
0.50
0.175
Wean untrogy
0.45
Mean Entropy
0.150
0.40
0.125
0.35
0.300
0.30
0.075
0.25
0.20
0.050
250
500
750
1000
1250
1500
1750
0
250
500
750
1000
1250
1500
1750
Step
Step
(c) A comparison of response entropy changes dur- ing the RLHF training process, aggregated across tasks supervised by the 'GenRM without ground truth'.
(d) A comparison of response entropy changes during the RLHIF training process, aggregated across tasks supervised by the RTV"
Figure 6: The comparison of response entropy change during the RLHF training process
B Experimental Results on Open-Source Benchmarks
In this section, we provide a performance comparison between the baseline model and our method. As shown in Table 4 we observe the following patterns:
19

---

Table 4: Performance comparison across different task series
Task Series
Sub-Task Name
Question Count
Baseline Score
Data Scale-
Large
AMC23
40
68.8
68.7
MATH Aligned
MATH500
500
88.2
87.8
AIME 2025
30
18.6
17.9
AIME (Open-Style)
AIME 2024
30
26.9
27.2
Coding Aligned
LiveCodeBench
131
30.3
36.2
MMLU
150
87.3
87.3
General Knowledge
MMLU pro
1000
79.6
80.1
GPQA diamond
198
59.6
60.1
Reasoning
SuperGPQA
250
52.8
54.0
Substantial gains in coding tasks: LiveCodeBench shows notable improvement from 30.3 to 36.2, representing a 19.5% relative increase.
Consistent performance in general knowledge: Both MMLU (87.3 - 87.3) and MMLU Pro (79.6 - 80.1) maintain slightly improve their scores, demonstrating the preservation of broad knowledge capabilities.
Marginal trade-offs in standard mathematical tasks: Minor decreases are observed in AMC23 (68.8 68.7) and MATH500 (88.2 87.8), though the differences remain within negligible ranges.
Comparable performance on challenging mathematical problems: AIME tasks show mixed but comparable results (AIME 2025: 18.6 17.9; AIME 2024: 26.9 27.2), indicating robust handling of complex mathematical reasoning.
Overall, our method demonstrates strong improvements in coding capabilities while maintaining competitive performance across mathematical and general knowledge tasks. The results suggest that our approach successfully enhances specialized skills without compromising the model's 's general capabilities, achieving a well-balanced performance profile across diverse task domains.
C Experimental Details
C.1 Reward Analysis of Newly Collected Prompts.
We investigated why newly collected prompts did not improve RLHF performance by analyzing their reward scores. As shown Figure 7 We found that approximately 90% these prompts received reward scores greater than 0.5 scale from 0 to 1. In this scoring scheme, score of 0.5 indicates that the model's output is comparable to the reference, while scores above 0.5 suggest superior performance. Our GenRM is trained to compare the model response with the ground truth in reasoning tasks and SFT Best-of-N responses in other tasks. Therefore, scores above 0.5 imply that the model-generated outputs were judged as superior these presumed optimal responses. However, after careful manual inspection, we discovered that a substantial portion of these high-scoring outputs exhibited reward hacking behavior and were qualitatively worse than the original best-selected responses. Moreover, we observed a direct correlation between the magnitude of the reward score and the severity and frequency of reward hacking instances. The higher the reward score, the more severe and frequent the reward hacking issue became. This finding reveals a critical limitation in our current reward model and underscores the need for more robust evaluation metries that can effectively distinguish between genuine improvements and instances of reward hacking.
C.2 Detail Experiment Setups
Experimental Details Pre-PPO: As shown Figure 2 we first combine the newly collected prompts with the original prompts to construct the training prompt set for the initial run. Then, as
20

---

RM Score Distribution
1.0
RM
RM Score on Small Size Model
3.0
RM Score on Large Size Mocill
0.0
0.2 0.4
0.6
0,8
1.0
Percentage
Figure 7: Distribution of reward scores for newly collected prompts. The x-axis shows the percentage of prompts. The y-axis represents the reward score range from 0 to with 0.5 indicating parity with the reference. Approximately 90% of prompts received scores above 0.5 for both small-size and large-size models, suggesting apparent superiority over reference outputs. However, manual inspection revealed that many high-scoring outputs exhibited reward hacking behavior and were qualitatively inferior to the original best-selected outcomes.
Training/math frac
Training/sandbox code frac
1.0
1.0
Sandbox Fraction
Praction
0.6
Fraction
0.6
0.2
0.2
100
150
250
600
100
150
250
600
Step
Step
Figure 8: The distribution of prompts across both math and coding task during the training phases
illustrated in Figure 7, we select only the bottom 10% of prompts based on their scores assigned by the generative reward model. This prompt selection process is conducted using the small-sized model. To reduce computational costs, we do not repeat this process on the large-sized model.
Experimental Details of Prioritizing Mathematical and Coding Tasks. Since performance coding tasks is measured via unit tests, which are more robust and less susceptible to reward hacking compared to math tasks, leverage this property by assigning higher proportion of coding-related prompts during the early stages of RLHF training. Specifically, we begin training exclusively on coding prompts, gradually introduce mathematical prompts, and ultimately utilize the complete mixed-domain training dataset. The distribution of math and coding prompts throughout the training process is presented in Figure g
C.3 Prompt Distribution
We collect approximately million diverse prompts from open-source resources to construct our RI. training prompt set. As illustrated in Figure 9 categorize these prompts into multiple task types
21

---

(e.g., math, knowledge, and creative writing). The relative proportions of each task category within the collected prompt dataset are presented the figure.
Frequency Distribution of Different Data Sources
sandbox_codé
36.2%
math
18.3%
knowledge
14.2%
Ability
NLP
11.6%
creation
9.6%
cosplay
5.9%
Other
4.2%
5
10
15
20
25
30
35
Percentage (%)
Figure 9: Prompts Distribution covering varies domains.
D Further Experimental Analysis
Although we have demonstrated the effectiveness and positive scaling trend of our method, we aim further explore the underlying mechanisms that contribute to success. Specifically, we seek to understand why our approach enhances the performance of RLHF and how it breaks through two critical bottlenecks: reward hacking and the deterioration of model response diversity.
Reward Hacking Problems Across Different Reward Models. Aside from perfect verifiers, any reward model used during RLHF can potentially be hacked. However, as shown in Figure TO observe that:
Effect Model Performence (Higher is Better) Further Analysis: Comparison of Roward Models
1.0
0.8
Performance
0.6
Verifiers
Cen RM with Ground Truth
SFT BoN
RM
2100
2800
Training Steps
Figure 10: Comparison of Reward Hacking Susceptibility and Performance Trends for RTV, GenRM, and BT Reward Models During RLHF Training
22

---

Effect on Model Performance (Higher is Better)
Compartson Coding Task
Comparison Math Task
$4.0
50.0
$2.0
Performance Metric
Performance Metric
08.0
50.0
46.0
01.0
45.0
Initial, Run-Codlngg Data_ Scale-Coding
Initial, Run-STEM Pata_Scale-STEM
700
1400
2100
2000
3500
0
700
1400
2200
2800
3500
Training Steps
Training Steps
Figure 11: Data Scale method boost both math and code performance.
For tasks evaluated using RTV, scores continued improve throughout the entire RLHF training period. This sustained improvement suggests that RTV provides robust and hack-resistant feedback.
When using GenRM with ground truth data, observed consistent score improvements up approximately the 2800 training step. This indicates that GenRM maintains its effectiveness feedback mechanism for a significant portion of the training process.
reward model (or GenRM utilizing responses selected by Best-of-N sampling (BoN) from the SFT model) showed improvements only up to the 2100 training step, after which the test scores began to decline. This downturn indicates that the BT reward model or GenRM with SFT BoN response might be more susceptible to issues such as overfitting or reward hacking in later stages of training.
contrast,
Accordingly, in our proposed approach, we increase the number of prompts allocated to RTV. supervised tasks and place an early emphasis on mathematical and coding tasks, supervised respec- tively by GenRM (with ground-truth data) and RTV. We anticipate that this strategy will enable the model to achieve optimal overall performance across various task types: those supervised by RTV, those supervised by GenRM with ground truth references, and those supervised by GenRM with SFT Best-of-N responses. This approach is expected to yield the best combined results, especially by allowing the model to reach peak performance tasks in the last category before ceward-hacking issues emerge.
Early Acquisition of Fine-Grained Response Differences Enhances Performance Scaling, Al- though the observed overall performance improvement can be partially explained by mitigating reward hacking issues associated with tasks supervised by GenRM with SFT Best-of-N responses, the specific performance boost in mathematical and coding tasks still merits further investigation. As math
illustrated in Figure 11 our 'Data Scale' method achieves substantially better performance and coding tasks compared to the initial run. Notably, this improvement occurs despite our method utilizing roughly the same number of prompts for mathematical and coding tasks in the initial run.
Accordingly, we first analyze the types of prompts filtered by the Pre-PPO strategy. To conduct this analysis, we collect five responses per prompt, compute the maximum edit distance among these responses, and then categorize the prompts into separate bins based on these maximum edit distances. Next, we calculate the average normalized reward model score for each bin. In our view, the edit distance between responses can reflect the granularity of their differences some extent-large edit distances indicate coarser-grained differences, whereas smaller distances suggest finer-grained distinctions. As illustrated Figure 12
we have the following observations and findings:
Prompts supervised by GenRM with ground truth (e.g., mathematical and logical tasks) and those supervised by GenRM without ground truth (e.g., creative writing and cosplay tasks) exhibit distinctly different trends in normalized reward-model scores as the edit distance varies. These trends highlight fundamental differences how the model learns across task types: for tasks supervised by GenRM without ground truth, the model readily captures coarse-grained
23

---

Comparison between GenRM Models under Different Conditions
0.600
GenRM with SFT BoN Creation, CosPlay)
GenRM with Ground Truth g.. Malth, Logic] Filter Threshold
0.575
Normalized RM Score
0.500
0.475
0.450
0.425
2
Bins of Maximum Response Edit Distances for Each Prompt
Figure 12: Comparison of Reward Model Scores across Different Edit Distance Bins for GenRM with and without Ground Truth.
differences; whereas for tasks supervised by GenRM with ground truth, the model shows greater sensitivity to fine-grained distinctions.
In the Pre-PPO strategy, we explicitly exclude prompts that exhibit fine-grained response differ- ences mathematical and logical tasks, as well those reflecting coarse-graid graiind differences in creative writing and cosplay tasks. A subsequent ablation study suggests that reintroducing the previously excluded mathematical and logical prompts still delivers marginal improvements in overall performance. This finding implies that learning coarse-grained patterns from creative writing and cosplay tasks negatively impacts the scalability of RLHF data.
We hypothesize that emphasizing mathematical and coding tasks during early training may also guide the model towards capturing fine- fine-grained distinctions first, thereby mitigating potential adverse effects from prematurely learning coarse-graiee beareegrrained patterns.
Furthermore, we analyze how reward score differences vary across prompt bins categorized by their maximum edit distances for different reward models. As shown in [3 both GenRM with ground truth and RTV assign larger score differences within bins corresponding to smaller edit distances. Conversely, GenRM without ground truth fails to produce meaningful score differences within these lower edit-distance bins. These results suggest that reward models leveraging ground truth data or verification feedback demonstrate stronger sensitivity fine-grained response variations compared to models trained without explicit ground truth supervision. Moreover, when directly comparing RTV and GenRM equipped with ground truth data, RTV exhibits consistently larger score differences at low edit distances, highlighting RTV's enhanced capability in capturing subtle response distinctions.
Deterioration of Model Response Diversity. Finally, compare the model response entropy between the baseline method and our proposed method. As illustrated in Figure the Appendix, the response entropy for creative writing and cosplay tasks is higher in our approach compared to the baseline. In contrast, our method achieves lower response entropy for mathematics, coding, and other reasoning tasks. These findings suggest that removing coarse- suuse-grained pattern prompts from tasks supervised by GenRM without ground truth alleviates the decline in model response diversity typically observed during RLHF This strategy thus enables the model to better capture fine- mne-grained differences in reasoning task responses, thereby enhancing the data-scaling effectiveness RLHF.
E Discussions
Q1: Do prompts with large edit distances negatively impact model performance, and should the model avoid learning from them?
24

---

Normalized Score Differences Across Bins
GenRM (No Ground Truth)
(With Ground Tuth))
0.8
Normelized Score Difference
0.2
Figure 13: Comparison of score differences across varying edit-distance bins for GenRM (with and without ground truth) and RTV. Scores provided by GenRM without ground truth align with the response edit distance, indicating that larger edit distances -hhunc which represent greater response differences correspondd to larger score differences. However, RTV and GenRM with ground truth do not exhibit this trend. This suggests that GenRM effectively detects large differences between responses but struggles to identify smaller differences.
A1: Actually, prompts with larger edit distances (coarse- (coarse-grained variations) and smaller edit distances (fine-grained variations) both contribute positively improving the model. However, prioritizing coarse-grained (large edit distance) data early in training can adversely affect the model's ability to effectively learn fine-grained (small edit distance) distinctions later. Ideally, want the model to thoroughly master fine-grained variations first before transitioning to learning from coarse-grained data.
Q2: The "O1 series" method introduces long chain-of-thouuht (CoT) responses, which theoreti- cally increases the edit distance of all response pairs. Why is this approach still effective?
A2: The "Ol series" can essentially be viewed as transforming all fine- gre-grained differences into sufficiently large, coarse-grained ones. By doing this, the model can more clearly categorize and gen- eralize variations across different granularities. Personally, this approach seems like a comprehensive solution for handling varying levels of granularity. However, we acknowledge that there might exist deeper insights or interpretations beyond our current understanding.
Q3: According to this analysis, should we first train our models on data with smaller edit dis- tance variations (fine-grained responses), and later data with larger edit distance variations (coarse-grained responses)?
A3: We haven't conducted experiments as detailed as this due to several practical considerations. Edit distance is merely a coarse proxy for defining granularity levels, and its computational overhead is quite high. Hence, it's suitable for exploratory understanding rather than practical training strategy. Nevertheless, we have performed a similar strategy experiment in which tasks with abundant fine-grained variations (such math and coding tasks) are trained first, followed by broader, coarsely- varied data later on. This approach demonstrated improved final performance. Importantly, this improvement depends heavily on the capacity of the verifier and GenRM with ground truth to accurately perceive fine-grained variations.
Q4: What practical insights does this analysis provide us?
A4: The concept "fine-graine grained control" was initially highlighted by Anthropic when introducing Constitutional AI (CAI) [36. The creation of CAI was inspired by the realization that methods such as Reinforcement Learning from AI Feedback (RLAIF) alone cannot directly capture preae-eref fine-grained distinctions. To address this limitation, Anthropic proposed CAI, which explicitly encourages generative reward models (Gen-RM) to become sensitive to subtle aspects of prompts and responses. For example, CAI aims to help models detect subtle prompt-level linguistic nuances-suu
25

---

as differences between requests for "critical reviews" versus just "reviews" as well as recognize sophisticated vocabulary that enhances response quality, exemplified by Claude frequentl using poetic phrases like "moonlight as in dream" in literary creation tasks. Therefore, moving forward, we can build upon these insights by carefully constructing CAI-style humanistic datasets aimed specifically training GenRMs without ground truth to perceive subtle, fine-graines distinctions. Subsequently, reinforcement learning (RL) techniques could then leverage these enhanced reward models, progressively improving their capability for tine-grained control over generated responses.
Case Study
We collect the human feedback compare the baseline and our model. However, due the complexity of the prompts, we only list shot case comparsion in our paper.
F1 Case 1 STEEM
We present a STEM case study demonstrating that the response of our model exhibits enhanced logical coherence (e.g. line 7-8 in the response of our model vs. line 12- 14 in the response of baseline model) ) and better informational density.
<-- PROMPT -->
It is known that the eye color of certain type of fruit fly, white and black, is pair of relative traits controlled by a pair of alleles on an autosome. Black is dominant over white, and the gene controlling black eyes is not expressed in female fruit flies. Therefore, male fruit flies have both black and white eyes, while female fruit flies only have white eyes. Additionally, there is a lethal homozygous dominance for the black eye gene At the same time, short wings and long wings are controlled by another pair alleles on an autosome, with long wings being dominant over short wings, and there is a lethal homozygous dominance for the long wing gene. A black-eyed, long-winged male is crossed with a white white-eyed, eyed, long-winged female, and it is found that the ratio of black to white male fruit flies in the F1 generation is 1:1. All long- fruit flies from Fi deduce the theoretically expected traits and ratios of the F2 male individuals.
then allowed to freely mate. Please
The line aipc_data_dit encode ('utf-8") encodes a string into a UT formatted byte sequence. What are the benefits of this?
RESPONSE OF OUR MODEL -->
1. Determine the parental genotypes**: Let the gene controlling eye color be represented by ((A) and (aa), and the gene controlling ving length by A(B)) and \(bl).
It is known that black eyes are dominant and homozygous dominant is lethal. Male fruit flies have both black and white eyes, while female fruit flies only have white eyes. A black-eyed male fruit fly has the genotype \(aa)) (since ((AA)) is lethal), and white-eyed female fruit fly could have the genotype ((aa)) or \aal). The F1 generation shows a 1:1 ratio of black (\(Aa))) to white-eyed (N(aa))) male fruit flies, indicating that the parental white-eyed female fruit fly has the genotype (aaa).
Since long wings are dominant over short wings and homozygous dominant for long wings is lethal, long winged
26

---

individuals have the genotype \(bb)) Therefore, the parental black wan-winged male fruit fly has the genotype \(aBbb\), and the white-eyed, long- female fruit fly has the genotype \aaabb)).
2. *aaaculate the genotypes and ratios of the F1 wing-winge fruit flies**:
For the wing length trait, the parental cross is (Bbb \ times Bb)). According offspring genotypes and ratios are \BB::bb:bb 1:2:1\)
the law of segregation, the
Since \(BB) is lethal, the F1 long- fruit flies have the genotype \(bb)) accounting for (2ffrcc(2)33)) of the surviving individuals, and short-winged fruit flies have the genotype \(bb)) accounting for ((1frac(1)33)) of the surviving individuals. Thus, all F1 long-winged fruit flies are \(Bb))
10 3. ** Calculate the genotypes and ratios of the F2 generation produced by free mating Fi**: F1 long winged fruit flies \(bb)) freely mate. According to the law of segregation, the gene frequency of \(V) (1)))))(1)22)) and the gene frequency of (bb) is \(Ifra {1H2)N) Using the Hardy-Neinberg law (A((p+q) p pq + q22), where \(p)) and \(g) are the frequencies of the alleles), the theoretical offspring genotypes and ratios are BBBB:Bb:bb:1:2:11 lethal, the actual ratios are \(Bb:bb 2:11), meaning long -winged fruit flies (\(Bb))) account for ((Afrac(2H3)) and short-winged fruit flies (N(bb))) account for \(ffrac (1H3)))
However, since \BBV) is
12
For eye color, the parental cross is ((Aa \times aa and the F1 genotypes and ratios individuals freely mate, with the gene frequency of \(A) being (1) and the gene frequency of \(a)) being (3)
\(Aa:aa 1:11) F1
The theoretical offspring genotypes and ratios are \(AA: 1:6:9\\) Since \(M\) is lethal, the actual ratios are \Maaa = 6:9 = 2:31), meaning \(Aa\) accounts for ((1frac(2M6))) and ((aa)) accounts for \(\ frac (3)(5))))
13
14 4. ** Calculate the ratios of different traits in F2 male individuals*#:
For the two pairs of relative traits, wing length and eye color, the traits
male individuals are calculated by combining the two according to the law of independent assortment.
Black- long-winged (\(Aabb)): \(ffrac(2M5) \times \
{3)) (rac{4))165))
Black-eyen short short-winged ()(aab))); [1ffra(2HH5 Atimes
(1)
(2rac
White eyed, long-winged (Y(aBb))): ;): \(3frac(3355 Atimes
frac frac{6)
White short-winged (\(aabb\)): (Afracc33)(5 Atimes
\frac (3) (raa
20
Therefore, the traits and ratios F2 male individuals are black-eyed long-winged black-eyed, short-winged white-eyed eyed,, ling-winged white eyed, short winged \( 4:2:6:31).
27

---

In conclusion, the theoretically expected traits and ratios of F2 male individuals are black eyed, long - winged : black eyed, short -winge white eyee,, long winged white-eyed, short-wingen \( 4:2:6:3\).
<-- RESPONSE OF BASELINE MODEL
This is a question about the laws of genetic inheritance and the derivation of trait ratios. The key to solving the problem lies in clarifying the dominant-recessil relationships
the genes, the lethality conditions, and deducing the parental genotypes based on the given mating combinations and offspring ratios. Then, using the method of calculating free mating, the trait ratios of the offspring can determined.
1. Determiining the parental genotypes for eye color**: It is given that black eyes (denoted ((A)) are dominant over white eyes (denoted as Ma))), and the gene controlling black eyes is not expressed in female fruit flies. Male fruit flies have either black or white eyes, with the homozygous dominant genotype for black eyes (\(AA \) being lethal.
black- male (V(A)_)) is crossed with white-eyed female. The \(F1)) generation of male fruit flies shows a black eyed (\(A\_)) \(1:1\) Based on the test cross results, it can be deduced that the bladk-eyed male parent has the genotype \(Aa)) for eye color, while the white-eyed female parent has the genotype \(aa))
white-eyed (Naa))) ratio of
2. Determinnig the parental genotypes for wing type**: It is given that long wings (denoted as (BB)) are dominant over short wings (denoted as \(b)), and the homozygous dominant genotype for long wings (XBB))) is lethal.
The parents are long winged male (\(B)_\) and long- winged female (A(B)_)) Since the homozygous dominant genotype for long wings is lethal, both parents must have the genotype \(Bb)) for wing type.
12
3. Deriving the complete parental genotypes and the \(11)) long winged fruit fly genotype ratios**: Combining the above analysis, the parental genotypes are (AaBb\) for the male and (aaBb\) for the female.
13
14
For wing type, the \(F1V) long-wing winged fruit flies have genotypes and ratios \(Bb:bb bb 2:11), meaning \(bb)) accounts for ()frrc((2)3)) and ((bb)) accounts for \(1 frac
15
16 4. Calculating the trait ratios for wing type in the A(F2)) generation*:; The ((F1) long- fruit flies mate freely. The frequency of the \BV) allele in the gametes produced by \ F1\) is ((1frac \times \1rac(1)(2 \frac(1)33)),, and the frequency of the \(b)) allele is \(1 (1) (3)
17
According to the Hardy - Weinberg law, the genotype frequencies in \(F2\) are: \(BB (1frac(1)(3)) \frac (1)())) (lethal), \(Bb \times \1rac(1H3) \times \frac
28

---

(2)(3) = (4)) and \(b - (2) = Afrac
(4)())) Thus, the ratio long- (\(B)) short-
winged (((bb)) individuals is ())frac(4) (fac 1:11).
20 5. Calcullting the trait ratios for eye color in the ((F2))
generation*::
21
The (F11) genotypes for eye color and their ratios are \
Aa:aa 1:11) The frequency of the \(A) allele in the gametes produced is (()frac(1)22) \times \frac(1) (1) and the frequency of the \(a)) allele is \(1 \frac(1)44} (frac
The genotype frequencies in \(F2V) are: ((AA (\frac {1>{4) (frac())(16))) (lethal), \(aa = 2 \times \frac [1}{4} \times (3rac(3)44) (frac(6)166)) and \aa () frac (3)(4)) \frac(9H166)) Thus, the ratio of black- eyed (\(Aa)) white- (a(a))) individuals is \(Afra 2:31).
25
6. ** Calculating the trait ratios for male individuals in the (F2V) generation**:
Since the inheritance of eye color and wing type are independent events, the ratio black-eyed, long-winged males in \(FV) is (2) \times (frac(1)(2) Afrac {1}{)}), black- short-winged males is (2) times \frac(1)2) \frac(1H6)), white-eyed, long-winged males is (3)frac(3)65) \times \frac(1)() (3rac and white-eyed, short-winged males is \(\frac(3)65 \times (1) (raa
Therefore, the traits and ratios of \(F2) male individuals are: eyec long- winged : black- slack-eyed short- winged white-eyer wingwd white-eyed short short-winged ((2:2:3:3))
28 In conclusion, the theoretical traits and ratios of ((22\) male individuals are black long-winged black-eyed short- winged : white-eyed long- winged white eyed short \
F.2 Case 2
Writing
We present a case study demonstrating how our model's responses exhibit superior linguistic elegance (e.g. title of essay) while meticulously adhering to nuanced instructions in prompts.
<-- PROMPT -->
You need complete the article creation according to the folloving requirements:
1: The content of the article requires rigorous structure and clear logic, with a strict argumentation process from raising the issue, analyzing the problem, to solving it. 2: The title of the article needs to be in professional style succinctly listing the main information of the event straightforvard news involved should
manner. Amounts of money reflected in the title.
3: Please strictly follow the requirements for output, the content must comply with the law, and it is prohibited evaluate the law and government law enforcement departments Adhere to the WeChat public account community norms, prohibit the appearance of violent, pornographic, and
29

---

sensitive vocabulary, and prohibit the output of any content unrelated the article. It is forbidden deviate from the core focus of the event, keeping the article content closely tied to hot events.
4: The article layout requires that the title and content, the first line outputs the title of the article (do not start with the word "title"), and the use of book title marks is prohibited. The second line starts with the specific content of the article. The full text should be around 2000 words. The article content must have subheadinge, and the subheadings should be expressed in straightforward manner with the use of book title marks prohibited in subheadings.
The theme is: "Urban Waste Sorting", this theme cannot be used the title, write article of about 2000 words
<-- RESPONSE OF OUR MODEL -->
Urban Waste Sorting: Current Status, Challenges, and Breakthrough Strategies
Exploring New Paths for Sustainable Development
**Significant Achievements in Waste Sorting Implementation** In recent years, with the gradual increase in environmental
awareness, urban waste sorting has become a focal topic of social concern. Many cities have actively responded to national calls, vigorously promoting waste sorting initiatives, and have achieved remarkable results improving environmental quality and enhancing resource utilization rates
Taking Shanghai as an example, since official
implementation of the waste sorting policy on July 1, 2019,
the government has invested approximately 20 billion yuan in infrastructure construction and public education campaigns. Today, Shanghai waste sorting efforts have yielded impressive outcomes. Neatly arranged sorting bins have become a common sight on the streets, while scenes residents conscientiously sorting and disposing of waste are widespread accuracy rate of waste sorting in Shanghai has significantly improved. The sorting and processing systems for dry, wet, recyclable, and hazardous waste are increasingly refined, with the resource utilization rate of waste exceeding 35%%. This has effectively reduced environmental pollution and laid a solid foundation for the city's s sustainable development.
communities. According to statistics, the
Beijing is not far behind. To advance waste sorting, the government has cumulatively invested around 15 billion yuan in building waste treatment facilities, purchasing sorting transport vehicles, and conducting promotional activities. Through a series of measures, Beijing waste sorting efforts have achieved phased success. More residents are becoming aware leading to a noticeable improvement in the city's environmental hygiene. Data shows that the recycling rate of domestic waste in Beijing has reached approximately 30V,, making a positive contribution to resource recycling.
and participating in waste sorting,
30

---

Challenges Facing Waste Sorting**
13 Despite notable achievements some cities, undeniable that the nationvide promotion of urban waste sorting still faces numerous severe challenges.
15
Residents' Awareness and Habits Need Improvement**
Some residents lack sufficient understanding of waste sorting and have inadequate environmental awareness and a sense of responsibility. Many find the process cumbersome, believing it increases their living and time costs, and are therefore reluctant long estabbistbl habits of improper waste disposal are difficult to change quickly, resulting in low accuracy rates in waste sorting. For instance, in some older neighborhoods, indiscriminate waste disposal remains common and recyclables are often mixed with other types of waste
participate actively. Additionally,
17
18 Complex and Confusing Sorting Standards**
Currently, waste sorting standards in China are not fully unified, with variations across different regions. This creates significant confusion for residents and workers. For example, some cities classify waste into four categories, while others use five or more. The lack of uniformity makes difficult for residents to grasp the correct sorting methods. Moreover, the specifics of waste sorting are often complex, with ambiguous boundaries between categories. For instance, whether large bones belong to kitchen waste or other waste is a common point of confusion for many residents.
20
Lagging Infrastructure Development**
22
The development of waste sorting infrastructure lags behind actual needs. On hand, waste collection containers often inadequately placed, insufficient in number, and poorly labeled.
some streets and communities, uneven distribution of bins makes waste disposal inconvenient for residents. On the other hand, transportation and processing facilities are also lacking. In some cities, waste transport vehicles leading to the remixing of different waste types during transportation, rendering previous efforts futile. Furthermore, waste processing capacity, especially for hazardous and kitchen waste, remains limited, with technological and operational capabilities needing improvement.
not support sorted collection,
Lack Effective Supervision Mechanisms** Supervision mechanisms for waste sorting still underdeveloped, lacking effective regulatory measures and enforcement tools. Currently, most cities rely community workers and volunteers for oversight, but limited manpower makes it difficult process comprehensively. Without clear penalties and strong enforcement for non non-compliance, some residents do not take waste sorting regulations seriously, leading to frequent violations.
monitor the entire waste sorting
26
31

---

** Seeking Breakthrough Solutions Through Collaborative Efforts
To further advance urban waste sorting and achieve sustainable development goals, businesses, and residents to work together and take effective measures to address current challenges.
essential for the government,
30 Strenggheniing Public Education**
The government should intensify efforts educate the public about waste sorting, using various channels and methods to disseminate knowledge and raise environmental awareness and responsibility. For example, diverse promotional campaigns be conducted through television, radio, newspapers, and online platforms. Workshops and training sessions can be organized in communities, schools, and workplaces.
Engaging and informative materials, such as brochures and public service announcements, can be distributed and broadcasted. Additionally, themed activities like waste sorting competitions and family events can encourage participation and help residents develop good sorting habits.
33
**Unffinn Sorting Standards**
The government should expedite the unification of waste sorting standards, creating scientific, reasonable, and easy-to- understand classification guide. Clear definitions and boundaries for each waste category should be established to reduce ambiguity. Simultaneously, efforts should be made to promote and train residents and workers on these standards to ensure accurate implementation. Drawing international best practices, China can refine its waste sorting standards and systems better suit local conditions.
36 Improving Infrastructure Development*
37
Increased investment in waste sorting infrastructure is crucial Governments should plan and deploy waste collection containers more rationally, ensuring even distribution and clear labeling. Upgrading waste transportation and processing facilities collection and treatment. Advanced transport vehicles equipped with sorting mechanisms should be procured, and modern waste treatment plants should be built to enhance processing capacity Encouraging private sector participation in infrastructure development can also diversify funding sources.
also essential to enable sorted
technological capabilities,
39 Establishing Robust Supervision Mechanisms**
comprehensive supervision mechanism for waste sorting must be established to oversee the entire process. Governments should formulate clear regulatory measures and penalty standards, enforcing stricter consequences for violations. Leveraging the efforts of community workers, volunteers, and residents, a reporting and reward system can be implemented encourage public participation in oversight. Additionally, technological solutions like surveillance cameras and smart bins enable real time monitoring and management.
32

---

**Promoting Industry Development*
The development of Waste sorting- related industries should be actively promoted to form complete industrial chain. Governments can introduce policies to encourage technological innovation and efficiency improvements in waste sorting. Support for waste recycling projects can enhance resource utilization. Furthermore, establishing industrial parks for waste sorting can cluster related businesses, fostering scalable and specialized development.
Urban waste sorting is long-term and challenging task, integral to sustainable urban development and residents
quality of life. Despite current difficulties, with determination and collaborative efforts, effective measures can be taken to overcome these challenges and achieve comprehensive progress in waste sorting Let us work together to build beautiful, green, and sustainable future for our cities.
<-- RESPONSE OF BASELINE MODEL --> "Challenges and Solutions in Urban Waste Sorting: A Multidimensional Analysis"
Current Status and Urgency of Urban Waste Sorting With the acceleration of urbanization, the amount of waste generated in cities has grown significantly. Statistics show that the total annual waste output in major cities has continued to rise in recent years, making waste management a critical issue for urban development. Traditional waste disposal methods, such as landfilling and incineration, not only consume large amounts
land resources but also pose environmental risks. Waste sorting, as the front-end process of waste management, plays a key role in achieving waste reduction, resource recovery, and harmless disposal. It can effectively improve recycling rates, minimize environmental damage, and lay the foundation for sustainable urban development, Horever, although waste sorting initiatives have been gradually implemented in many Chinese cities, the overall results remain uneven, with numerous challenges persisting. The urgency of advancing waste sorting efforts is undeniable.
Challenges in Implementing Waste Sorting Lack of Public Avareness and Habits
considerable portion of urban residents lack sufficient knowledge about waste sorting. Many are unsure how to properly categorize different types of vaste---such as distinguishing between hazardous, recyclable, and other waste. Even among those who understand the basics, deeply ingrained habits mixed waste disposal are difficult change in the short term. Some residents find sorting cumbersome and lack motivation to participate, leading careless disposal. These issues in avareness and behavior make it difficult source.
enforce effective waste sorting the
10
Inadequate Infrastructure
1a some cities, waste sorting facilities are insufficient. Trash bins are not set up according to strict sorting
33

---

standards, or their labels are unclear, making proper disposal confusing for residents. Additionally, the soott- then-mix'' phenomenon occurs during waste collection and transportatonn areas have sorted bins, but collection trucks mix all waste together, rendering residents' efforts meaningless and discouraging participation. Furthermore, terminal waste treatment facilities face shortcomings such as underdeveloped recycling systems and lack of specialized hazardous waste disposal methods, preventing sorted waste from being properly processed.
13
Weak Enforcement and Supervision of Regulations
1S
Although many regions have introduced waste sorting laws and regulations, enforcement remains inconsistent. Penalties for non non-come compoomli are often too lenient to serve as an effective deterrent. Due the vast urban areas and numerous waste generation points, regulatory bodies struggle to conduct comprehensive, real-time oversight, allowing violations to unchecked. Moreover, insufficient public avareness of these regulations weakens their guiding and binding effects.
16
Strategies for Salti-Stakeholdr Collaboration in Waste Sorting Enhancing Public Education and Awareness Governments and communities should intensify awareness
18
campaigns through diverse activities such as community lectures, themed exhibitions and online quizzes to educate residents on proper waste sorting. Tailored approaches should be used for different demographics integrating waste sorting into school curricula to instill environmental consciousness in youth, while using simple, accessible materials for elderly residents. Media platforms including TV, radio, social media, and short videos, should also be leveraged to disseminate information and foster culture of participation.
example,
20
Improving Infrastructure
22
At the disposal stage, clearly labeled and strategically placed sorting bins should provided to facilitate correct waste separation. Regular maintenance and upgrades of these facilities are essential. For collection and transportation, dedicated vehicles for different waste categories must deployed to prevent mixed collection. efficient waste transfer system should also be established On the processing end, investments should be made in recycling and hazardous waste treatment facilities, incorporating advanced technologies to enhance disposal efficiency.
23
Strengthening Regulation and Enforcement
25 Regulatory bodies must enhance daily supervision and enforcement, establishing performance based evaluation systems to reward success and address shortcomings. Violations should be penalized strictly to increase accountability. Public reporting mechanisms can also encourage residents monitor and report con-complaance fostering collective oversight. Additionally, continuous
34

---

legal education should ensure residents understand and adhere to waste sorting requirements.
27 Innovative Models for Sustainable Waste Sorting Introducing Market Mechanisms
Businesses should be encouraged to participate in the waste sorting industry chain through policy incentives and subsidies, attracting private investment in collection, transportation, and processing. For instance, specialized waste sorting service providers can offer comprehensive solutions to communities and institutions. A recycling industry system can also be developed to monetize recyclables, creating circular economy. Innovative business models, such as reward points or exchange programs can further motivate public participation.
30
31 Leveraging Technology
32
Smart waste sorting technologies -aahnollgies---ucch as AI-powered bins and recycling machines using facial recognition or QR codes--- can track and incentivize proper sorting through reward systems. Big data and IoT can enable real-time monitoring and analysis of waste generation and sorting rates, supporting data-driven decision making. Additionally, advancements like AI-based sorting robots can improve efficiency waste processing.
34 Conclusion
Urban waste sorting is systematic endeavor requiring changes in public habits, infrastructure upgrades, regulatory enforcement, and societal collaboration. Only through robust education, improved infrastructure, strict oversight and continuous innovation waste sorting, enhance environmental quality, and achieve sustainable development---reee environment for all residents.
cities effectively advance
better living
F.3
Case 3
Coding Knowledge
We present a case study demonstrating our model's enhanced response quality, characterized by better information accuracy and content richness, exemplified by the provided code sample (line 12-23).
PROMPT
The line aipc_data_dir encode (' l'utf -8 ) encodes string into -8 formatted byte sequence. What are the benefits this?
RESPONSE OF OUR MODEL
Encoding strings into UTF-8 byte sequences offers several key benefits in Python broader programming contexts:**
1. **Broad Compatibility UTF-8 supported character encoding standard. It is the most common encoding on the internet, with nearly all modern operating systems, progranming languages, and applications supporting it. This ensures reliable data transmission and processing across different systems, avoiding garbled text due to encoding incompatibiltties. For example, when transmitting text data in web applications, UTF-8 is the
widely accepted and
35

---

standard encoding, correctly interpreted by browsers, servers, and other components.
2. Variable - Encoding**: -8 uses variable -lngth encoding scheme, representing characters with 1 to 4 bytes. For ASCII characters (basic Latin letters, numbers, and symbols), UTF-8 uses single byte, maintaining
compatibility with ASCII. For mon-ASCII characters, it uses 2 to 4 bytes for storage and transmission, especially when the data contains many ASCII characters
needed. This efficiency makes UTF-8 ideal
3. ** Mulliilingual Support**: UTF-8 represent nearly all global writing systems, including letters, symbols, Chinese Japanese, Korean, and more. This makes it perfect for internationalized and multilingual applications Regardless of the language, UTF-8 provides unified encoding solution.
10 4. ** Backward Compatibility Since UTF -8 encodes ASCII characters identically ASCII, maintains backward compatibility with legacy bSceI systems. This allows seamless upgrades support broader character sets without disrupting existing infrastructure.
### Example Code:
13 python
14 Define string containing Chinese characters "hello, world"
15
bipc_data_dii,
Encode the string into - byte sequence 17 encoded_data aipc_data_dee encode ('utf-8') print (encoded_dataa))
18
19
20 Decode the UTF-8 byte sequence back string decoded_data encoded data. decode utfff")
21
22
print (decoded_data)
23
24
This example demonstrates encoding Chinese string into UTF-8 bytes and decoding it back, showcasing UTF -s reliability in handling multilingual text.
25
26
summary, UTF encoding ensuring smooth data exchange across systems. Whether for web development, text processing, or system interoperability, UTF-8 remains trusted standard.
robust universal choice,
<-- RESPONSE OF BASELINE MODEL -->
The primary benefits of encoding strings into UTF-8 byte sequences are as follows:
1. Universality and Compatibility
Cross pllafform Support: UTF-8
universal character encoding standard supported by nearly all modern operating systems, programming languages, and network protocols. Data encoded in -T can be properly processed across Windows, Linux, and macOS systems. For example, a text file stored with UTF -8 encoding on Linux server can be read on Windows client without garbled characters, as Windows systems equally support UTF-8 encoding.
36

---

Multilingual Capability: -8 can represent characters from nearly all writing systems worldwide, covering languages from different countries Chinese, Arabic, Japanese, regions. Whether it's English, others, UTF-8 can encode them all. This makes - highly practical for handling multilingual content, such in internationalized websites or software applications. For instance, a global e- commerce platform dealing with product descriptions and user comments in various languages can rely on UTF encoding to ensure proper storage and display of all text.
2. Network Transmission
10
Efficiency: - offers efficiency advantages in network transmission. For ASCII characters (basic English letters, numbers, and common symbols), UTF-8 encoding uses only 1 byte, identical additional overhead when transmitting English text. For non ASCII characters, UTF-8 uses to 4 bytes depending on the character. Compared to fixed-lenges encodings like UTF (which typically uses
traditional ASCII encoding, ensuring no
bytes for all characters), UTF -8 may occupy less space when transmitting text containing predominantly ASCII characters, thereby improving network efficiency. For example, when transmitting an English news article with minimal Chinese content, -T encoded text consumes less bandwidth than UTF-16 encoded text.
Protocol Support: Many network protocols, such HTTP, SKTP, and POP3, default to or recommend UTF-8 encoding. When transmitting web content via HTTP, using UTF ensures that all characters (including text in different languages and special symbols) browsers.
correctly parsed and displayed by
14 3. Data Storage
15
Space Efficiency: As mentioned earlier, UTF-8 uses byte for ASCII characters and typically 3 bytes for common Chinese characters. Compared which uses 4 bytes per character), - can significantly reduce storage space for text containing large amounts ASCII or commonly used characters. This is particularly important for databases and file systems storing substantial text data. For example, in a database table storing user comments, UTF-8 encoding can dramatically reduce storage requirements and lower costs compared to UCS -4 encoding.
fixed-length encodings like UCS-4
In Flexibility: UTF-8's variable- encoding provides great flexibility in storing different types of characters. It dynamically allocates bytes based on actual character requirements, meeting the need to represent all characters while minimizing space usage for common ones.
37


Reviews:
[{"review_id": "mIVm6ZhWhI", "reviewer": "Reviewer_PsAt", "summary": "This paper addresses an understudied aspect of RLHF: the role of prompt-data construction and its scalability. The authors identify two bottlenecks preventing effective data scaling in RLHF—reward hacking and declining response diversity—and propose practical solutions. Their approach introduces a hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM), along with a novel Pre-PPO prompt selection strategy that prioritizes challenging prompts. Additionally, they demonstrate that prioritizing mathematical and coding tasks during early RLHF training stages significantly improves performance. The methods are validated on models of different scales (25B and 150B parameters), showing consistent improvements across multiple evaluation metrics.", "questions": "1. Your method shows strongest improvements in mathematical and coding tasks. Have you tested whether the approach generalizes to more open-ended tasks like dialogue or creative writing where ground truth is less well-defined?\n2. Public Benchmark Validation: Why weren't standard public benchmarks (e.g., HumanEval, MATH, GSM8K) used for evaluation? This would help the community better understand the practical impact of your improvements.", "limitations": "Yes, the authors adequately discuss limitations in Section 5, acknowledging the resource-intensive nature of their approach, computational overhead, and uncertainty about long-form Chain-of-Thought RLHF. However, they could expand on:\n\n- The potential bias introduced by focusing heavily on STEM tasks during early training\n- How the method might perform with different reward model architectures\n- The impact of their approach on model capabilities outside the evaluated domains", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Novel Problem Focus: While most RLHF research concentrates on algorithmic improvements, this paper uniquely addresses data construction and scaling—a critical but overlooked aspect that directly impacts practical deployment.\n2. Strong Empirical Results: Consistent improvements across model scales (25B and 150B) and evaluation sets, with particularly impressive gains in STEM (+3.9) and coding (+3.2) tasks.\n3. Good Rigor: The paper provides valuable insights into fine-grained vs. coarse-grained response differences and their impact on RLHF effectiveness, backed by comprehensive experiments in the appendices. Extensive appendices with ablation studies, case studies, and detailed analyses strengthen the paper's contributions.\n\nWeaknesses\n\n1. Limited Reproducibility: Due to proprietary data and code restrictions, the results cannot be independently verified, which is a challenge. The paper relies heavily on internal evaluation sets (V1.0 and V2.0) without comparison to widely-used public benchmarks, making it difficult to contextualize the improvements.\n2. Theoretical Justification: While the empirical results are strong, the paper lacks theoretical analysis of why Pre-PPO selection and early task prioritization work, relying mainly on intuition and post-hoc explanations.", "was_revised": false, "final_justification": "", "rebuttal": "# Evaluation and Impact\n\nWe sincerely thank the reviewers for their insightful feedback and constructive suggestions. Below, we address the specific questions and concerns raised:\n\nGeneralization to Open-Ended Tasks like Dialogue or Creative Writing ：\nWe have conducted manual evaluations on creative writing tasks to assess the generalizability of our approach. The results indicate that our model maintains a consistent performance level comparable to the baselines. Similarly, in dialogue tasks, our model demonstrates performance that is on par with existing methods. These preliminary evaluations suggest that our approach is capable of handling more open-ended tasks effectively. However, we acknowledge that further extensive evaluations are necessary to thoroughly establish the generalization capabilities across diverse open-ended scenarios.\n\nAdditional Remarks:\nOur approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities.\n\nWe appreciate the reviewers' suggestions to explore the generalization of our approach further. Building on this feedback, we are committed to expanding our evaluations to encompass a wider range of tasks, including more complex and lengthy text generation scenarios. These future endeavors will aim to validate and potentially enhance the applicability of our method in various real-world applications.\n\n# Reproducibility and Use of Standard Public Benchmarks\n\nWe appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:\n\n- Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).\n- Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).\n- Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).\n- Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).\n\nHowever, we've identified limitations in many public benchmarks:\n- Potential IID issues, not fully reflecting real-world challenges.\n- Risk of data leakage due to widespread use.\n- Insufficient difficulty to comprehensively assess advanced models.\n\nFor instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.\n\nConsequently, we developed an internal evaluation suite with the following advantages:\n- Higher difficulty, better differentiating top-tier model capabilities.\n- Continuous iteration, preventing model overfitting.\n- Black-box testing, ensuring fair and authentic evaluation.\n\nWe believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.\n\nRegarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.\n\n| Task Series       | Sub-Task Name                              | Question Count | Baseline Score | Our Model Score |\n|-------------------|-------------------------------------------|----------------|----------------|-----------------|\n| MATH - Aligned    | AMC23 (AMC23fblshot_v2.rwq_score)         | 40             | 68.8           | 68.7            |\n| MATH - Aligned    | MATH500                                   | 500            | 88.2           | 87.8            |\n| AIME (Open-Style) | AIME 2025                                 | 30             | 18.6           | 17.9            |\n| AIME (Open-Style) | AIME 2024                                 | 30             | 26.9           | 27.2            |\n| Coding - Aligned  | LiveCodeBench                             | 131            | 30.3           | 36.2            |\n| General Knowledge | MMLU                                      | 150            | 87.3           | 87.3            |\n| General Knowledge | MMLU pro                                  | 1000           | 79.6           | 80.1            |\n| Reasoning         | GPQA diamond                              | 198            | 59.6           | 60.1            |\n| Reasoning         | SuperGPQA (superGPQAfrdom_1K_sample_2)    | 250            | 52.8           | 54.0            |\n\n# Theoretical Justification and Novelty\nWhile there isn't much theoretical analysis, in the \"Further Experimental Analysis\" appendix, we've made some relevant analyses and interpretations.\n1. Analyzing prompt filtering by Pre-PPO strategy\nWe collected five responses per prompt, calculated the maximum edit distance among them, and categorized prompts into bins. Then we computed the average normalized reward model score for each bin. Edit distance can reflect the granularity of response differences (larger for coarser-grained, smaller for finer-grained).\n2. Observations from the analysis\nTask-type differences: For tasks supervised by GenRM with ground truth (e.g., math and logic) and without ground truth (e.g., creative writing and cosplay), the normalized reward model scores show different trends as edit distance changes. The model captures coarse-grained differences easily for tasks without ground truth, and is more sensitive to fine - grained ones for tasks with ground truth.\nEffect of Pre-PPO exclusion: In the Pre-PPO strategy, we excluded certain prompts. An ablation study found that reintroducing previously excluded math and logic prompts improved overall performance marginally. This implies that learning coarse-grained patterns from creative writing and cosplay tasks can harm the scalability of RLHF data.\nHypothesis on early task emphasis: Emphasizing math and coding tasks in early training may help the model capture fine-grained distinctions first, reducing the negative impact of learning coarse-grained patterns too early.\n3. Reward model analysis\nAnalyzing reward score differences across prompt bins, we found that GenRM with ground truth and RTV assign larger score differences in bins with smaller edit distances. GenRM without ground truth fails to show meaningful score differences in these bins. RTV shows greater ability to capture subtle response distinctions compared to GenRM with ground truth at low edit distances. Overall, reward models with ground truth or verification feedback are more sensitive to fine-grained response variations."}, {"review_id": "l37YOWhco1", "reviewer": "Reviewer_V9Wr", "summary": "This paper investigates the data scaling challenge in Reinforcement Learning from Human Feedback (RLHF), where increasing the amount of training data does not consistently lead to improved model performance. \nTo handle this issue, the authors propose the following contributions:\n1) Hybrid Reward System: combining the Bradley-Terry (BT) reward model, reasoning task verifiers (RTV) and the generative reward model (GenRM).\n2) Pre-PPO Prompt Selection: training with low-reward prompts to effectively mitigate the reward hacking issue\n3) Early Training Emphasis: empirical finding that prioritizing mathematical and coding tasks during the early phases of RLHF training improves final performance", "questions": "1) Could the authors provide per-task learning curves for the Early Training Emphasis strategy? It would be helpful to see how this strategy affects individual task performance over time.", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n1) The proposed methods are simple, easy to implement, and demonstrate clear effectiveness in the experimental results.\n2) Addressing the data scaling issue in RLHF is both important and relatively underexplored, making this work a valuable contribution to the field.\n\nWeaknesses:\n1) The paper does not clearly demonstrate the individual or combined advantages of using the BT reward model, GenRM, and RTV. An ablation study or comparative analysis would strengthen this claim.\n2) Key aspects of the experimental setup are under-specified. For example, the composition and evaluation methodology of the \"V1.0\" and \"V2.0\" datasets are not clearly described.\n3) While the empirical results are promising, the paper lacks theoretical analysis or intuition explaining why Pre-PPO prompt selection and early prioritization of math and coding tasks improve robustness against reward hacking.\n4) The writing could be improved for clarity and conciseness. For instance, the abstract contains redundant descriptions of the Pre-PPO method (e.g., line 14).\n5) While the integration of components is thoughtful, the overall novelty is limited, and the paper would benefit from a more refined presentation and deeper analysis.", "was_revised": true, "final_justification": "The authors have added further analysis on the ablation study of the reward models and the Early Training Emphasis strategy, provided additional empirical results on PrePPO, and improved the overall writing.\nHowever, I still believe that the lack of theoretical support and novelty remains a concern.", "rebuttal": "# The Individual or Combined Advantages of Using the BT Reward Model, GenRM, and RTV\nWe appreciate the reviewer's concern regarding the demonstration of individual and combined advantages of the BT reward model, GenRM, and RTV. We would like to clarify that we have indeed conducted a comprehensive comparative analysis of these components in the Appendix under \"Further Experimental Analysis.\" This analysis provides substantial evidence for the effectiveness and distinctions between these methods.\n\nSpecifically, our analysis focuses on two critical aspects:\n\n1. Reward Hacking: \nWe demonstrate that RTV is significantly more robust against reward hacking compared to other methods. Our experiments show a clear hierarchy in performance:\n   RTV > GenRM with ground truth > GenRM without ground truth ≈ BT RM.\nThis hierarchy illustrates the superior ability of RTV to maintain integrity in reward signals, crucial for reliable model training.\n\n2. Fine-grained Difference Detection:\nOur analysis reveals that RTV excels in identifying subtle differences between responses, outperforming other methods. The performance ranking follows the same pattern:\n   RTV > GenRM with ground truth > GenRM without ground truth ≈ BT RM.\nThis capability is essential for nuanced evaluation and improvement of model outputs.\n\n\n# Test Dataset Details and Evaluation Methodology\n\nWe apologize if this crucial information was not sufficiently highlighted in the main text. We will consider emphasizing these findings more prominently in the final version to ensure their significance is not overlooked. We would like to offer additional information about our test datasets:\n\nThe test sets are independently created by evaluators and are divided into white-box and black-box test collections. The white-box tests are visible to algorithm developers, while the black-box tests are not. The domains covered in these datasets are already listed in the paper.\n\nFurthermore, these test sets are updated every 1-2 months. During each update, new challenging questions are added, and some simpler questions are replaced to maintain the overall difficulty level. This regular refresh ensures that the evaluation remains relevant and challenging over time.\n\nV1.0 and V2.0 refer to two consecutive versions of these test sets. This versioning reflects our commitment to continuous improvement and adaptation of our evaluation metrics.\n\nIn the final version of the paper, we will include a dedicated section that elaborates on these key aspects of our experimental setup, providing readers with a more comprehensive understanding of our evaluation process and the robustness of our results.\n\n# Theoretical Justification and Novelty\nWhile there isn't much theoretical analysis, in the \"Further Experimental Analysis\" appendix, we've made some relevant analyses and interpretations.\n1. Analyzing prompt filtering by Pre-PPO strategy\nWe collected five responses per prompt, calculated the maximum edit distance among them, and categorized prompts into bins. Then we computed the average normalized reward model score for each bin. Edit distance can reflect the granularity of response differences (larger for coarser-grained, smaller for finer-grained).\n2. Observations from the analysis\nTask-type differences: For tasks supervised by GenRM with ground truth (e.g., math and logic) and without ground truth (e.g., creative writing and cosplay), the normalized reward model scores show different trends as edit distance changes. The model captures coarse-grained differences easily for tasks without ground truth, and is more sensitive to fine - grained ones for tasks with ground truth.\nEffect of Pre-PPO exclusion: In the Pre-PPO strategy, we excluded certain prompts. An ablation study found that reintroducing previously excluded math and logic prompts improved overall performance marginally. This implies that learning coarse-grained patterns from creative writing and cosplay tasks can harm the scalability of RLHF data.\nHypothesis on early task emphasis: Emphasizing math and coding tasks in early training may help the model capture fine-grained distinctions first, reducing the negative impact of learning coarse-grained patterns too early.\n3. Reward model analysis\nAnalyzing reward score differences across prompt bins, we found that GenRM with ground truth and RTV assign larger score differences in bins with smaller edit distances. GenRM without ground truth fails to show meaningful score differences in these bins. RTV shows greater ability to capture subtle response distinctions compared to GenRM with ground truth at low edit distances. Overall, reward models with ground truth or verification feedback are more sensitive to fine-grained response variations.\n\n# Experimental Design and Clarity\n\nWe appreciate the reviewer's request for per-task learning curves for the Early Training Emphasis strategy. Due to our rebuttal policy, we are unable to include new figures in the PDF. However, readers can refer to Figure 4 in the original paper, specifically the first 1400 steps, to gain insights into the learning progression for math and coding tasks.\n\nIn addition to the information provided in the paper, we have conducted supplementary experiments regarding the order of training for code and math tasks. On our 25B model, we tested the following configurations:\n\n1. Training exclusively on coding data first\n2. Training exclusively on math data first\n3. Our current mixed approach\n\nResults on our internal test set V1 showed:\n- Coding first: 38.1\n- Math first: 37.8\n- Current mixed approach: 38.8\n\nThese results demonstrate that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.\n\nWe hope this additional information provides further clarity on the effects of task ordering in our training strategy. We appreciate the reviewer's suggestion and will consider generating more detailed per-task learning curves for future iterations of our work, as computational resources allow.\n\n\n# Writing and Presentation\nWe acknowledge the issues of redundancy and lack of clarity in our writing. We will promptly revise the abstract to eliminate redundancy and clearly convey our contributions. Moreover, we'll review and enhance the clarity and conciseness of other paper parts. Thank you for pointing these out. \n\nWe present the revised abstract below, incorporating feedback on clarity and scope:\n\n\"Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity. To mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named Pre-PPO, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that prioritizing mathematical and coding tasks during the early phases of RLHF training significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths. Through experiments conducted on both small and large models, we demonstrate the effectiveness and scalability of our proposed methods. Our approach exhibits robust generalization capabilities, especially on challenging and out-of-distribution tasks, while yielding significant improvements in mathematics-intensive (STEM) and coding domains. Moreover, the proposed strategies enable the model to effectively capture subtle, task-specific distinctions during the RLHF process, substantially enhancing overall model performance. This work emphasizes the critical role of careful data construction and provides practical methodologies for addressing key performance bottlenecks in RLHF.\"\n\n# Broader Impact\n## Coverage of Tasks Requiring Long-Form Outputs:\n\nWhile the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.\n\n## Additional Remarks:\n\nOur approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities."}, {"review_id": "NCX632G7AE", "reviewer": "Reviewer_TLTJ", "summary": "The paper presents a comprehensive study on the data scaling in the RLHF stage of LLMs. The authors study two types of rewards, i.e. GenRM and RTV (reasoning task verifier). The authors identify that simply scaling the number of prompts does not leads to better performance. Instead, the authors provide two useful strategies for data selection in RLHF:\n1. Pre-PPO prompt selection. Only keep the lowest-scoring 10% prompts, which suffers less from reward hacking.\n2. Prioritize math and code in early training.\n\nThe authors conduct extensive experiments on two kinds of MoE models (25B and 150B), showing that these strategies mitigate the reward hacking and the diversity deterioration problems, and effectively improve the model performance.", "questions": "1. Does this study cover tasks that need long-form outputs, for example, tasks (e.g., AIME) that may need inference scaling to over 10k tokens responses? If not, what would be different if these were included?\n\n2. What if the GenRM is re-trained for several iterations during the RL process? Will that alleviate the reward hacking problem? Will the current discoveries still hold?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strength:\n1. Data selection is a crucial problem that needs more research. The paper has strong motivation and addresses a core problem.\n2. The experiments are very comprehensive and large-scale. The experiments need a lot of resources to be run. The information from this paper is valuable.\n3. The strategies are simple but effective.\n\nWeakness:\n1. All models and benchmarks are internal, making external comparison difficult.\n2. It would be good to show more details,  like models, verifiers, data.", "was_revised": false, "final_justification": "", "rebuttal": "# Reproducibility and Use of Standard Public Benchmarks\n\nWe appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:\n\n- Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).\n- Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).\n- Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).\n- Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).\n\nHowever, we've identified limitations in many public benchmarks:\n- Potential IID issues, not fully reflecting real-world challenges.\n- Risk of data leakage due to widespread use.\n- Insufficient difficulty to comprehensively assess advanced models.\n\nFor instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.\n\nConsequently, we developed an internal evaluation suite with the following advantages:\n- Higher difficulty, better differentiating top-tier model capabilities.\n- Continuous iteration, preventing model overfitting.\n- Black-box testing, ensuring fair and authentic evaluation.\n\nWe believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.\n\nRegarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.\n\n| Task Series       | Sub-Task Name                              | Question Count | Baseline Score | Our Model Score |\n|-------------------|-------------------------------------------|----------------|----------------|-----------------|\n| MATH - Aligned    | AMC23 (AMC23fblshot_v2.rwq_score)         | 40             | 68.8           | 68.7            |\n| MATH - Aligned    | MATH500                                   | 500            | 88.2           | 87.8            |\n| AIME (Open-Style) | AIME 2025                                 | 30             | 18.6           | 17.9            |\n| AIME (Open-Style) | AIME 2024                                 | 30             | 26.9           | 27.2            |\n| Coding - Aligned  | LiveCodeBench                             | 131            | 30.3           | 36.2            |\n| General Knowledge | MMLU                                      | 150            | 87.3           | 87.3            |\n| General Knowledge | MMLU pro                                  | 1000           | 79.6           | 80.1            |\n| Reasoning         | GPQA diamond                              | 198            | 59.6           | 60.1            |\n| Reasoning         | SuperGPQA (superGPQAfrdom_1K_sample_2)    | 250            | 52.8           | 54.0            |\n\n#  Experimental Design and Clarity\n\nWe appreciate the request for more clarity on our experimental setup and exploration of alternative approaches. While computational constraints limited our ability to exhaustively test all possible configurations, we did conduct several key experiments to explore task ordering and data proportions:\n\n1. On our 25B model, we tested alternative task orderings by:\n   a) Training exclusively on coding data first\n   b) Training exclusively on math data first\n   c) Using our current mixed approach\n\nResults on our internal test set V1 showed:\n- Coding first: 38.1\n- Math first: 37.8\n- Current mixed approach: 38.8\n\nThis demonstrates that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.\n\n2. We also experimented with data composition by incorporating all verifiable reasoning data into the math dataset. This yielded nearly identical overall performance (50.6 vs 50.8), suggesting our current data balance is effective.\n\n3. While we didn't test placing math and coding at the very end of training due to computational constraints, we explored a similar concept in subsequent long-chain-of-thought reinforcement learning experiments. These showed comparable performance on open reasoning benchmarks.\n\nThese experiments provide evidence supporting our current task ordering and data balance. However, we acknowledge that further systematic exploration could yield additional insights. We're open to conducting more extensive experiments in future work, including generating per-task learning curves for our Early Training Emphasis strategy, as computational resources allow.\n\nIn addition, our subsequent paper explains more details on how we implement GenRM and the specifics of PPO training. Due to anonymity policy issues, we cannot share these details at present. The final version will reference this paper.\n\nQ：What if the GenRM is re-trained for several iterations during the RL process? Will that alleviate the reward hacking problem?\n\nA：We have indeed experimented with re-training GenRM for several iterations during the RL process. This approach showed some effectiveness in alleviating the reward hacking problem, and many of our current discoveries still hold true. However, we found that this method requires significant human resources and computational power. Due to these high costs, we ultimately decided against implementing it in large-scale updates and enterprise scenarios. While it offers benefits, the trade-off in terms of resource allocation and efficiency led us to explore other, more scalable solutions for our production environments.\n\n# Broader Impact\n## Coverage of Tasks Requiring Long-Form Outputs:\n\nWhile the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.\n\n## Additional Remarks:\n\nOur approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities."}, {"review_id": "RxDTa5VAb4", "reviewer": "Reviewer_UPWF", "summary": "This paper explores the challenges of data scaling in RLHF, focusing on reward hacking and response diversity issues. The authors introduce a hybrid reward system combining Reasoning Task Verifiers and Generative Reward Models to enhance model robustness and prevent reward manipulation. They also propose the Pre-PPO method for prompt selection, targeting difficult prompts to improve sample efficiency. Through experiments on a smaller model up to 25B parameters and a larger model up to 150B parameters, the authors demonstrate significant performance improvements, particularly in mathematical and coding tasks.", "questions": "**1. Regarding the revision of paper for clarity and conciseness:**\n\nCan you revise the abstract to eliminate redundancy, such as the repeated mention of \"we propose a novel prompt-selection method named Pre-PPO,\" and ensure it concisely conveys your contribution? Could you also address any additional writing flaws (e.g., unclear phrasing) identified in the abstract or elsewhere in the paper?\n\n**2. Regarding differentiation from existing work:**\n\nWould you be willing to conduct a detailed comparison showing how your task prioritization and prompt selection methods differ from or improve upon prior work, such as DeepSeek-R1? Could you include specific evidence (e.g., theoretical differences or empirical results) to highlight your approach’s novelty in the RLHF context?\n\n**3. Regarding systematic exploration of task prioritization design:**\n\nCan you provide experiments testing alternative task orderings (e.g., placing mathematical and coding tasks later in training) or varying data proportions to assess their impact on performance? Alternatively, could you offer a detailed justification for your current design without such experiments?\n\n**4. Regarding validation of reproducibility with open-source resources:**\nWould you consider validating your results using open-source models (e.g., Qwen, Llama) and datasets (e.g., MT-Bench, AlpacaEval, GSM8K) to address reproducibility concerns tied to closed-source resources? If not, can you propose an alternative plan to enhance verifiability?\n\n**Evaluation Criteria:** My assessment would improve to \"Accept\" (5) if the authors provide a revised abstract with enhanced clarity and no redundancy, clearly differentiate their method from prior work like DeepSeek-R1 with specific evidence, conduct or justify experiments exploring alternative task orderings or data proportions, and validate results using open-source resources or a robust reproducibility plan. The score would move to \"Borderline Accept\" (4) if the authors partially address these issues—e.g., by improving the abstract and providing some comparisons or limited experiments—but fail to fully resolve concerns about originality, systematic exploration, or reproducibility.", "limitations": "No, the authors have not fully addressed the limitations of their work. Although the paper provides a strong theoretical foundation and presents promising experimental results, some key limitations are not adequately discussed.", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strength:**\n\n**S1. High Practical Relevance:** The paper addresses highly relevant challenges for RLHF practitioners, particularly in terms of reward model robustness and response diversity. These are practical concerns for scaling RLHF to large models, making the research applicable in real-world scenarios.\n\n**S2. Innovative Approach to RLHF Data Construction:** Unlike much of the existing literature, which mainly addresses algorithmic challenges, this paper emphasizes the crucial role of training data in improving RLHF performance. The introduction of Pre-PPO and the hybrid reward system combining Reasoning Task Verifiers (RTV) and Generative Reward Models (GenRM) is an innovative approach to mitigating reward hacking and improving response diversity.\n\n**Weakness:**\n\n**W1. Further Exploration Needed for Academic Originality and Depth:** The paper's academic originality could be further strengthened, as its proposed methods, such as task prioritization and prompt selection, while demonstrating performance improvements, closely resemble earlier curriculum-learning approaches, which structure training by sequencing tasks from simpler to more reasoning-intensive to gradually build the model’s capabilities. For instance, the task prioritization strategy bears conceptual similarities to DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (2025), which also prioritize reasoning-intensive tasks during the early training stage of the model to enhance reasoning abilities. However, the paper does not sufficiently articulate how its approach surpasses or diverges from such prior studies, which substantially weakens the work’s novelty. To enhance its academic contribution, the authors should explicitly highlight the specific differences between their method and works like DeepSeek-R1, such as unique task selection criteria or novel integration with RLHF, and support these distinctions with theoretical analysis or comparative experiments.\n\n**W2. Need for More Systematic Exploration in Method Design:** The current task priority method is overly specialized and lacks universality due to the absence of systematic experiments. The design of the task prioritization method could be further optimized, particularly in terms of experimental rigor and mechanistic insight. The authors did not explore alternative setups, such as placing mathematical and coding tasks in the later stages of training (a common practice in some pretraining approaches) or adjusting data proportions (e.g., reducing the weight of non-mathematical/coding data) to test their assumptions. This lack of exploration may hinder a comprehensive evaluation of the method’s effectiveness.\n\n**W3. Limited Reproducibility Due to Closed-Source Data and Models:** The use of closed-source data and models in this study significantly hampers its reproducibility, thereby limiting the academic impact and verifiability of the findings. While the decision to keep resources proprietary may be justified, it prevents external researchers from replicating or validating the results. To enhance the study’s credibility and applicability, the authors are encouraged to conduct supplementary experiments using open-source models, such as qwen or llama, and publicly available datasets, like MT-Bench, AlpacaEval or GSM8K, to verify their conclusions and demonstrate the method’s robustness across diverse settings.\n\n**W4. Writing flaws (A minor point):**\nSome writing flaws exist in the paper. For example, the abstract contains repetitive sentences, particularly \"we propose a novel prompt-selection method named Pre-PPO\" appears twice. This redundancy reduces the conciseness and professionalism of the abstract, which should be as concise as possible as it serves as the facade of the paper. It is recommended to check the writing flaws in your paper.", "was_revised": true, "final_justification": "I have carefully reviewed the authors' rebuttal and appreciate the thoughtful efforts to address my concerns. However, I still believe that the differences in focus between long-form and short-form RL do not adequately address my concerns regarding the novelty of the proposed method. Furthermore, the issue of lacking open-source verification (models and evaluation benchmarks) persists, which limits the generalizability and reproducibility of the approach presented in this paper.", "rebuttal": "# Reproducibility and Use of Standard Public Benchmarks\n\nWe appreciate the importance of using standard public benchmarks and have indeed conducted evaluations on several open benchmarks. Our results show:\n\n- Significant improvement in coding tasks (e.g., LiveCodeBench: 30.3 -> 36.2).\n- Slight improvements or comparable performance in general knowledge tests (e.g., MMLU: 87.3 -> 87.3, MMLU pro: 79.6 -> 80.1).\n- Slightly lower but comparable performance on simple math tasks (e.g., AMC23: 68.8 -> 68.7, MATH500: 88.2 -> 87.8).\n- Comparable performance on high-difficulty math tasks (e.g., AIME: 18.6 -> 17.9, 26.9 -> 27.2).\n\nHowever, we've identified limitations in many public benchmarks:\n- Potential IID issues, not fully reflecting real-world challenges.\n- Risk of data leakage due to widespread use.\n- Insufficient difficulty to comprehensively assess advanced models.\n\nFor instance, our model can score nearly 90 points on some math benchmarks, but only around 30 points on our more challenging internal test sets.\n\nConsequently, we developed an internal evaluation suite with the following advantages:\n- Higher difficulty, better differentiating top-tier model capabilities.\n- Continuous iteration, preventing model overfitting.\n- Black-box testing, ensuring fair and authentic evaluation.\n\nWe believe combining public benchmarks with rigorous internal assessments provides a more comprehensive and reliable evaluation of model performance. We will continue to monitor and use public benchmarks while refining our internal evaluation system.\n\nRegarding the use of open-source models and datasets, we acknowledge their value in enhancing reproducibility. We are considering conducting supplementary experiments with these resources and will include them in future work to provide a more comprehensive comparison.\n\n| Task Series       | Sub-Task Name                              | Question Count | Baseline Score | Our Model Score |\n|-------------------|-------------------------------------------|----------------|----------------|-----------------|\n| MATH - Aligned    | AMC23 (AMC23fblshot_v2.rwq_score)         | 40             | 68.8           | 68.7            |\n| MATH - Aligned    | MATH500                                   | 500            | 88.2           | 87.8            |\n| AIME (Open-Style) | AIME 2025                                 | 30             | 18.6           | 17.9            |\n| AIME (Open-Style) | AIME 2024                                 | 30             | 26.9           | 27.2            |\n| Coding - Aligned  | LiveCodeBench                             | 131            | 30.3           | 36.2            |\n| General Knowledge | MMLU                                      | 150            | 87.3           | 87.3            |\n| General Knowledge | MMLU pro                                  | 1000           | 79.6           | 80.1            |\n| Reasoning         | GPQA diamond                              | 198            | 59.6           | 60.1            |\n| Reasoning         | SuperGPQA (superGPQAfrdom_1K_sample_2)    | 250            | 52.8           | 54.0            |\n\n# Systematic Exploration of Task Prioritization Design\n\nWe appreciate the reviewer's request for per-task learning curves for the Early Training Emphasis strategy. Due to our rebuttal policy, we are unable to include new figures in the PDF. However, readers can refer to Figure 4 in the original paper, specifically the first 1400 steps, to gain insights into the learning progression for math and coding tasks.\n\nIn addition to the information provided in the paper, we have conducted supplementary experiments regarding the order of training for code and math tasks. On our 25B model, we tested the following configurations:\n\n1. Training exclusively on coding data first\n2. Training exclusively on math data first\n3. Our current mixed approach\n\nResults on our internal test set V1 showed:\n- Coding first: 38.1\n- Math first: 37.8\n- Current mixed approach: 38.8\n\nThese results demonstrate that our current mixed approach outperforms exclusively prioritizing either coding or math tasks.\n\nWe hope this additional information provides further clarity on the effects of task ordering in our training strategy. We appreciate the reviewer's suggestion and will consider generating more detailed per-task learning curves for future iterations of our work, as computational resources allow.\n\n\n# Writing and Presentation\nWe acknowledge the issues of redundancy and lack of clarity in our writing. We will promptly revise the abstract to eliminate redundancy and clearly convey our contributions. Moreover, we'll review and enhance the clarity and conciseness of other paper parts. Thank you for pointing these out. \n\nWe present the revised abstract below, incorporating feedback on clarity and scope:\n\n\"Reinforcement Learning from Human Feedback (RLHF) is essential for aligning large language models (LLMs) with human preferences and values. While recent research has primarily focused on algorithmic advancements—such as reducing computational overhead or strengthening reward models to mitigate reward hacking—the critical role of prompt-data construction and its scalability has received comparatively less attention. In this paper, we address this gap by systematically exploring data-driven bottlenecks that currently hinder RLHF performance scaling, focusing specifically on the challenges posed by reward hacking and decreasing response diversity.\nTo mitigate reward hacking, we introduce a hybrid reward system combining reasoning task verifiers (RTV) and a generative reward model (GenRM). This approach enables accurate assessment of responses against clearly defined ground-truth solutions. Additionally, in order to ensure response diversity and enhance learning effectiveness, we propose a novel prompt-selection method named Pre-PPO, explicitly identifying training prompts that are inherently challenging and thus less prone to reward hacking. Furthermore, we find that prioritizing mathematical and coding tasks during the early phases of RLHF training significantly boosts performance, given that these tasks naturally encode fine-grained response distinctions and possess clearly defined ground truths.\nThrough experiments conducted on both small and large models, we demonstrate the effectiveness and scalability of our proposed methods. Our approach exhibits robust generalization capabilities, especially on challenging and out-of-distribution tasks, while yielding significant improvements in mathematics-intensive (STEM) and coding domains. Moreover, the proposed strategies enable the model to effectively capture subtle, task-specific distinctions during the RLHF process, substantially enhancing overall model performance. This work emphasizes the critical role of careful data construction and provides practical methodologies for addressing key performance bottlenecks in RLHF.\"\n\n# Broader Impact\n## Coverage of Tasks Requiring Long-Form Outputs:\n\nWhile the current study primarily focuses on tasks involving short-form outputs, we have initiated experiments targeting long-form content generation. In our forthcoming paper, we employ the same methodological framework to address long-text generation tasks. Initial findings are promising, indicating that our approach scales well to long-cot setting. We believe that integrating these additional evaluations will provide a more comprehensive understanding of the applicability and robustness of our method in generating extensive textual content.\n\n## Additional Remarks:\n\nOur approach shares similarities with concurrent work, specifically Deepseek-R1, in several aspects. Both methodologies involve initial training on verifiable tasks such as code and mathematics, followed by a preference learning phase. However, our research uniquely identifies and demonstrates the efficacy of this approach in the context of short-text generation. This distinction underscores the versatility and potential of our method across different text lengths and complexities."}]

Author rebuttal:

Meta-review: This paper investigates data scaling challenges in RLHF, proposing several data-centric strategies to mitigate reward hacking and improve performance. The primary contributions are a hybrid reward system, a "Pre-PPO" method for selecting challenging prompts, and a curriculum learning strategy that prioritizes math and coding tasks during early training phases, all validated on large-scale internal models.

The problem has practical relevance, and the results are convincing but reviewers raised significant concerns about the lack of reproducibility due to the use of proprietary models and internal benchmarks. Other critical issues included the limited novelty, with methods closely resembling concurrent work and prior curriculum learning approaches. The author's rebuttal provided some results on public benchmarks.

For the final version, I recommend including additional results on public benchmarks. Additionally, a thorough discussion and empirical comparison to differentiate their work from prior and concurrent approaches, particularly regarding curriculum strategies in RLHF, would be good. Overall, given strong empirical and large-scale results, I am leaning towards accepting this paper.
Decision: Accept (poster)