Policy Filtration for RLHF to Mitigate Noise in Reward Models
Authors: Chuheng Zhang
Wei Shen
Li Zhao
Xuyun Zhang
Xiaolong Xu
Wanchun Dou
Jiang Bian
Venue: ICML 2025 poster (2025)
Keywords: AI
LLM
RLHF
Primary area: applications->language_speech_and_dialog
TL;DR: We design a method to filter samples during RLHF to mitigate the inaccuracy of reward model.

Abstract: While direct policy optimization methods exist, pioneering LLMs are fine-tuned with reinforcement learning from human feedback (RLHF) to generate better responses under the supervision of a reward model learned from preference data. One major challenge of RLHF is the inaccuracy of the intermediate reward model, especially in the tasks that requires complex reasoning for the reward model to score a response. We find that the reliability of the reward model varies across responses assigned with different rewards. This motivates us to filter the samples whose rewards may be unreliable to improve the signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a proper policy filtering strategy, we use the coefficient of determination ($R^2$) between the rewards and actual scores on filtered samples as the metrics to help us find promising strategies since it measures how well the rewards filtered by PF-PPO indicate real performance. We provide extensive experiments to validate the effectiveness of PF-PPO in code generation and math reasoning tasks. In code generation, PF-PPO achieves the state-of-the-art performance of 7-billion-parameter models on HumanEval (+7.9%), MBPP (+0.7%), and LeetCode Contest (+10.0%) which is a more challenging benchmark created by us. In math reasoning, PF-PPO yields performance increase using different reward models and benchmarks (Ape210K and CMATH).

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Chuheng Zhang
Wei Shen *2
Li Zhao'
Xuyun Zhang?
Xiaolong Xu"
Wanchun Dou
Jiang Bian'
Abstract
1.0
While direct policy optimization methods exist, pioneering LLMs are fine-tuned with reinforce- ment learning from human feedback (RLHF) to generate better responses under the supervision of a reward model learned from preference data. One major challenge RLHF is the inaccuracy of the intermediate reward model, especially in the tasks that requires complex reasoning for the reward model to score response. We find that the reliability of the reward model varies across responses assigned with different rewards. This motivates us to filter the samples whose rewards may be unreliable to improve the signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF- PPO). To choose proper policy filtering strategy, use the coefficient of determination (R²) be- tween the rewards and actual scores filtered samples as the metrics to help us find promis- ing strategies since it measures how well the re- wards filtered by PF-PPO indicate real perfor- mance. We provide extensive experiments to val- idate the effectiveness of PF-PPO in code gener- ation and math reasoning tasks. In code gener- ation, PF-PPO achieves the state-of-the-an per- formance of 7-billion-parann parameeeer models on Hu- manEval (+7.9%), MBPP (+0.7%) and LeetCode Contest (+10.0%) which is a more challenging benchmark created by us. In math reasoning. PF-PPO yields performance increase using dif- ferent reward models and benchmarks (Ape210K and CMATH). Code is available on https:// git com/
Actual Score Score
0.6
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Reward
Figure I. The reward model can be inaccurate, i.e. the actual score of the response does not align well with the reward given by the reward model. However, the reward model in specific regions (e.g.- when it gives rewards higher than 0.8) is more reliable, i.e., the responses with similar rewards result consistent performance. We use a fine-tuned policy generate 10 responses for each of 164 prompts in HumanEval dataset and use reward model trained with the common recipe generate their rewards. We group the responses with similar rewards and calculate the average of their actual scores (i.c., the average correctness), indicating each group by one point. To evaluate the reliability of the reward model, repeat the process ten times corresponding to the ten lines.
1. Introduction
Reinforcement Learning from Human Feedback (RLHF) is key technique to align large language models (LLMs) with human values and preferences (Christiano et al., 2017: Ziegler al., 2019; Ouyang al., 2022). RLHF has been proven be essential process for LLMs to produce helpful, harmless, and honest responses (Bai et al., 2022). Despite various non-RL algorithms such as DPO (Rafailov et al., 2024) are proposed, state-of-the-art applications such as ChatGPT/GPT-4 (OpenAI, 2023), Claude (Anthropic, 2023), and Gemini (Team et al., 2023) adopt the RL algo- rithm (c.g.. PPO) for policy optimization. The key challenge of RLHF is the inaccuracy of the intermediate reward model. While there are researchers investigate how to learn reliable reward models (see e.g., Wang et al., 2024), we focus on how to learn better policy under the guidance of such inaccurate reward models.
Eqqull contribution 'Microsoft Research 'Independent Re- searcher 'Macquarie University "Nanjing University of Informa- tion Science and Technology State Key Laboratory for Novel Software Technology, Nanjing University. Correspondence to: Chuheng Zhang <zhangccuheng @23@@ive.com> Wei Shen <shenweit9117 126.com...
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
We observe that, though the reward model gives inaccurate

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
rewards in general, it can be more reliable in specific regions (e.g., when it gives high rewards) than the others. The observation is based on the simple experiment: We use policy model fine-tuned for code generation to generate a set of responses for prompts in the HumanFval dataset. Later, we score these responses using a reward model trained with the common recipe (see Ouyang et al., 2022, and also Section 2) and compare them with the actual scores. We find that, across different sets of samples, the reward model is more reliable when it gives high or low rewards than when it gives moderate rewards (cf. Figure 1). This property also holds on other datasets and tasks and see Appendix A for more experiment results and further discussion. Considering that RLHF updates the policy solely based on the reward signal, this observation motivates us to filter out the samples with possibly unreliable rewards aiming improve RLHF by increasing the signal-to-noise ratio training samples.
2. Related Work
Limitation of reward model. The outcome of RLHF highly relies on the quality of the reward model. Unfortunately, the reward model can hardly provide accurate scores due to 1) the mis-specified reward modeling to represent human preferences (Lambert et al., 2023; Pitis, 2023); 2) the pres- ence of incorrect and ambiguous preferences in the dataset (Ouyang et al., 2022; Bai et al., 2022), and 3) the poor gen- eralization ability of the reward model (McKinney et al., 2023). The inaceuracy of reward model attributed as one major cause of reward hacking and hallucination in LLMs (Kalai & Vempala, 2024). While there previous papers try to improve the accuracy of the reward model itself (Wang et al., 2024; Coste et al., 2023; Zhang et al., 2024), the ob- jective of our paper is to design a better RLHF algorithm in the face of inaccurate reward models. Moreover, Bai et al. (2022) also mentioned that using the output of the reward model directly in the RLHF process may not be a good choice. A possible solution is penalize the outputs with low rewards more to improve the worst-case responses but they did not further implement this.
Based on this motivation, we propose simple modification to the standard PPO-based RLHF algorithm (Ouyang et al., 2022), resulting in Policy Filtration for PPO (PF-PPO). As in standard PPO, we generate N samples for each prompt and score these samples using the reward model. Then, we use filtered subset of these samples in PF-PPO for sub- sequent policy training. We design filtering strategies to improve the reliability of the reward model on the filtered samples by maximizing the coefficient of determination (R²) between the rewards and actual scores on these fil- tered samples. We show that the rewards of these filtered samples are more accurate, thus providing better training signal and improving the performance of the policy. Our method is also connected with reject sampling that filters out responses with low rewards during inference to yield better response. Reject sampling is a simple but surprisingly strong inference-time strategy, whereas we adopt similar filtration an RL algorithm.
Reject sampling. Reject sampling (or best-of-N sampling) is a popular and effective inference-time strategy to enhance the response of an LLM by generating N responses and select the best one according to reward model (Nakano et al., 2021; Cobbe et al., 2021). This trick can yield good responses while keeping a tight KL constraint to the original policy. Inspired by its effectiveness in inference, researchers also try to involve this trick in policy optimization. For example, RAFT (Dong et al., 2023), BOND (Sessa et al., 2024) and vBoN (Amini et al., 2024) learn policy that distills the best-of-N policy using supervised fine-tuning losses. In boarder sense, the rank information of the N samples can also be leveraged. For example, RRHF (Yuan et al., 2023) and PRO (Song et al., 2024) train the policy using the combination of ranking loss and a SFT loss (w.r.t. the best response) based on N responses for each prompt. However, these algorithms do not adopt an elaborate RL algorithm, while state-of-the-ae language models adopts RL. algorithms in alignment, benefiting from the generalization power of the reward model especially in reasoning tasks (Ivi- son et al., 2024). Unlike these algorithms, we adopt the idea of reject sampling in the sampling phase of an RL algorithm instead of using supervised learning losses.
Empirically, we show that PF-PPO can improve the per- formance of LLMs on the tasks where the complex logic makes the reward model inaccurate in general. We con- duct extensive ablation studies to validate the design of our algorithm. In code generation, we illustrate the effective- ness of our algorithm by fine-tuning LLMs that achieves new sota on HvmanEval and MBPP benchmarks across 7-billion -aalimon- LLMs. We also create the LeetCode Contest benchmark that includes competition-leve coding tasks for human experts and observe that PF-PPO results in even more significant improvement on this challenging benchmark. In math reasoning, we demonstrate that PF- PPO can improve the performance across different types of reward models.
RLHF algorithms in the face of inaccurate reward mod- els. One key challenge in RLHF the inaceuracy of reward model, which can lead to reward over-optimization (Gao et al., 2023; Skalse et al., 2022; Chaudhari et al., 2024). Op- timization with policy constraint (e.g., KL divergence be- tween the target policy and the reference policy) is remedy frequently used in not only RL-based algorithms (Ouyang et al., 2022; Wu et al., 2023; Zhu et al., 2023) but also di-
2

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
rect policy optimization algorithms (Rafailov et al., 2024; Zhao al., 2023; Liu al., 2023). Going beyond policy constraint, Moskovitz al. (2023) only maximize rewards up to a threshold to avoid excessive deviation from pre- trained policy. In this paper, we not only rely on the policy constraint to optimize in the face of inaccurate rewards but also try to avoid using samples with unreliable rewards.
based on collected preference data DHF == {(e, y"; y specifying that y is a preferred response to than y The reward model is initialized by x-ST with additional out- put layer. A preference model links the reward model with the preference data, and Bradley-Tery model (Bradley & Terry, 1952) is a common choice:
PPy" > s'cc) = (Rs(V - Ro(v"c)). (1)
3. Preliminary
where a is the sigmoid function. The learning objective of reward model is to maximize the log-probability on preference data:
Notations. We use (a, b) to denote the set {a, a + 1,... b)) and use [6] as the shorthand for [1, bj. We use @ to denote the concatenation on tokens, and use Tazb as the shorthand for the concatenation (Ie @ Ta+1 @ ....... @ m1).. We use C4 and y; to indicate the i-th token in the context e (including task instruction, prompt, inputs, etc.) and the response y respectively.
max E ((G. [lg P(y > 3r/c]]-
(2)
RL fine-tuning. In this stage. we fine-tune the language model ιTy to maximize the rewards given by the reward model with policy constraint. The optimization problem is formulated as
MDP formulation. We adopt a Markov decision pro- cess (MDP) formulation for RLHE. Specifically, language generation is formulated as an MDP M = AS P. R) with states s E S. actions a E A, transition probabili- ties P E AS))SAA and the next-state-base
max By [roluie) -Dxx 7 SFT (+cc)
The second term prevents the learned policy deviating too much from the SFT model, and this is popular technique to alleviate reward over-optimization (Jaques et al., 2019; Stiennon et al., 2020).
reward fune- tion R: : S [0, 1]. Given context e with T. to- kens, on each step t E [Te + 1,T],, the language model (s)a)ss)) selects token ag = Dt-T, based on the state St : (c1:T + -1:--T-1). Then, the language model enters the next state 8t+1 == (CI.T. @ ),) until the language model completes the response TT For simplicity, we will also use contextual prnextuul-bandi-sttyye notations, e.g., we de- note the language generation process as y (-le))
PPO. Proximal policy optimization (PPO) (Schulman et al., 2017) is an RL algorithm that uses a clipped version of the policy gradient for more conservative and stable learning. It becomes a standard algorithm for RL fine-tuning in RLHF that optimizes the modified (cumulative) reward
RLHE. Reinforcement learning with human feedback (RLHF) is an important process to address objective mis- match between the next-token- prediction cex--token-reeditiion objective in pre- training and our expectation of LLMs to follow the instruc- tions and assist humans to complete various tasks. We briefly review the pipeline of RLHF.
(o(c)))
Σ T=
I log To(yelc @ 91:1-1)
(4)
- -log π SFT (v. @ pha-11)
where the reward model gives sparse rewards and the policy constraint yields dense rewards. PPO is an on on-policy algo- rithm where the policy gradient estimated based on the samples collected by the current policy TT.
Supervised fine-tuning. In the supervised fine-tuning (SFT) phase, a pre-trained LLM is fine-tuned with high- quality supervised dataset collected for specific down- stream tasks. Typically, the LLM is fine-tuned with a maximum likelihood loss, and we denote the output of this phase as SST, While subsequent RLHF procedure is necessary for training high-quality LLMs,, this phase alone can also yield an LLM that reasonably follows hu- man instructions (see e.g., Longpre et al., 2023).
Algorithm Proximal policy optimization (PPO)
for iteration = 1, 2....... do
Fill the buffer B with samples collected by the current language model Ta
Update To using PPO w.r.t. the cumulative reward defined in Equation (4) based on B end for
Reward model learning. In the reward model learning phase, we learn a reward model (vle) E 1] param- eterized by e that scores the response y to the context
4. Methods
Our method is motivated by the observation that the reward model is more reliable for the responses assigned with high- /low rewards (cf. Figure 1). Consequently, we conjecture
We fix the index of the terminal state to be the maximum length T. To adapt responses of different lengths, left pad the context c.
3

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Algorithm 2 Policy Filtration Proximal policy Optimization (PF-PPO)
Policy SFT
No filter
BoN
BR
BW
0.886
0.454
0.922
0.952
for iteration = 1,2, do
Middle RLHF
0.907
0.389
0.935
0.956
Fill the buffer B with samples collected by the current
Final RLHF
0.876
0.431
0.916
0.946
language model fe
Table 1. The coefficient of determination (R²) of the unfiltered policy no and different filtered policies po between the rewards given by the reward model and the actual scores. This metries correlates well with the final performance (see Section 5) and helps us to determine the weight vector (or the poliicy filtering strategy) in our algorithm PF-PPO. We choose the SFT policy and the middle/final RLHF policy as the unfiltered policy mo respectively.
Update To using PPO w.r.t. the cumulative reward defined in Equation (4) based on B
end for
that, if we wrap the policy with proper filtration during policy optimization of RLHF, the reward model can avoid yielding unreliable rewards and thus give better signal to guide policy learning.
Policy filtration. Given an unfiltered policy model The (vlcc) that generates responses y to the context c, we denote the (y(c)). We consider family of policy filtration, from which we can sample re- sponses to the context e as follows: We first sample N re- sponses from (+[- and rank them by the reward model Rs,
by the filtered policies can be maximized. To measure this accuracy, we choose a simple heuristic, the coefficient of de- termination (aka R-squared or R²) (Draper, 1998) between the rewards and the actual scores of the responses generated by the policy. R² measures how well the actual scores can be predicted by the rewards with a linear model. Specifi- cally, given set of responses {(c, y13} sampled from the filtered policy yı ~ puo()cc), we can collect the correspond- ing reward R, := (g(ye (c.) and the actual score s;. Then, we fit linear model f to predict the actual score based on the reward and denote the predicted score as 8; = [(Ra). Σ, &.)?
corresponding filtered policy
obtaining y1, :.. VN with (y)lcc 2 ... 2 Rø(Uv (c). Then, given a weight vector w = (wi,,... WN) satis- fying EieNy W/= = 1, we sample a one-hot vector z from the categorical distribution parameter- ized by w such that Plzz- = 1] = wt-- At last, the filtered policy 150 (-cc) yields the response selected by z following y -i-N 2:3/i.
The R-squared is calculated as 1 - (. (s;-8)2 where s is the average of actual scores. Since PF-PPO optimizes the policy based on the rewards on these responses, how well these rewards indicate the actual performance is closely re- lated to the final performance of our algorithm. We find R² well correlates with the final performance and can imply the level of reward over-optimization of the subsequent RLHF algorithm, therefore serving as a useful metrics to determine the weight vector used in PF-PPO.
We can define several filtered policies under this family. Specifically, we obtain the best-of-N (BoN), best-random (BR), and best-worst (BW) filtered policy by setting the weight vector as follows:
w BoN = (1,0,..,,))
BBR
=
2' 22N
1) -1)"
2(N -
(5)
To select weight vector, we first checkpoint three poli- cies Ty collected from different stages of a standard RL.FF process and collect responses using filtered policies μig in combination with different policy filtering strategies. Then, we group the responses with similar rewards, record the average actual score and reward for each group, and calcu- late the R2 by treating each group sample point. We exam how different policy filtering strategies can improve the reliability of the rewards on the responses generated by the corresponding filtered policies.
w BBW
12:00 0, ..... 1 2
1
Training objective. Since our target is to learn good filtered policy (40, we consider the follow objective:
max B. [ro(vlc)- (DK
SFT
(-1=)
(6)
In practice, use the samples collected by the unfiltered pol-
icy ng as if they were collected by 1i in the original PPO
We present the results in Table 1. We observe that best- random (BR) and best-worst (BW) can improve the reli- ability of the given reward model on sampled responses compared with unfiltered policy. The BoN strategy does not improve the R², which indicates that learning BoN filtered policy may not result in good performance in RL, although learning for a best-of-N N policy using supervised learning presents good performance (Sessa et al., 2024).
algorithm. This leads Policy Filtration Proximal Policy Optimization (PF-PPO) listed in Algorithm 2, which is algorithm that only modifies the sampling process of PPO.
Weight choice. By defining different weight vectors w, we can obtain different policy filtering strategies for PF-PPO. Our objective is to choose a weight vector w such that the accuracy of the reward model on the responses generated
4

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
5. Experiments
reasoning tasks. Ape210K contains 210k diverse math prob- lems and we use separate split of 5k problems for evalua- tion, following the practice in Zhao et al. (2020). CMATH contains 1.7k math word problems sourcing from actual Chinese workbooks and exams. We check the conrectness of the answers using the automatic evaluation scripts provided in Zhou et al. (2024), To train models for math reasoning, we use the training split of 200k problems from Ape210K.
5.1. Benchmarks
We conduct experiments on two tasks where the quality of LLM responses can be precisely measured, code generation and math reasoning. Specifically, we evaluate the algorithms using the following benchmarks.
HumanEval benchmark and MBPP benchmark. Hu- manEval (Chen et al., 2021) and MBPP (Austin et al., 2021) are two popular benchmarks for evaluating code I.MMs. Hu- manEval includes 164 Python problems, each of which is associated with multiple test cases used to assess the cor- rectness of generated code in a zero-shot setting. Similarly, MBPP includes 378 problems.
5.2. Implementation Details
We use deepseek-6.777 6.7B (Guo et al., 2024) and Qwenl.5- 7B (Team, 2024) as our base model for code generation and math reasoning respectively. In the SFT phase, we train on the SFT dataset for 5 epochs with the learning rate x 10-5, resulting in the SFT policy. In the reward model training phase, we follow Ouyang et al. (2022) and train on our reward model dataset for 1 epoch with the learning rate x 10- 3_ In the PPO phase, we adopt the training tricks from the blog (Shen al., 2024). Specifically, we adopt re- ward normalization and advantage normalization for stable training. In addition,
To train models for these two benchmarks, we select data from 75k
(Wei et al., 2023b) and 55k evol-codealpaca-vl (Luo et al., 2023) to construct the SFT dataset, the reward model dataset, and the PPO query dataset. For SFT, we use all the 130k training samples from Magicoder-OSS-instricc and evol-codealpaca-vl. For reward modeling, we curate 7k prompts from these 130k samples and generate five responses using the SFT model for each prompt. Following the methodology in Pal et al. (2024), we select two responses with the maximum edit distance to create response pairs for each prompt. We use these 7k prompts with generated response pairs as the reward model dataset. For policy optimization, we curate 3k prompts from the 130k samples as the PPO query dataset.
set the learning rate for the policy network as 5 x 10-7 and learning rate for the value network as 9 x 10-6. In the PPO algorithm, we collect responses for the context in the PPO query dataset and iterate through this dataset for 5 iterations (enough for convergence) and select the best checkpoints on evaluation set as the outcome policy. For each collected context-respe reesposse pair, we use it to accumulate loss and gradient for 3 times on average. We use full parameter fine-tuning in all the phases. We provide the source code for all experiments in the supplementary.
LeetCode contest benchmark. To evaluate code LLMs on more challenging problems, we construct the LeetCode Contest benchmark. This benchmark includes competition- level problems designed for human, and therefore is more challenging since it requires human-level problem under- standing and code generation skills. In this benchmark, we collect 160 problems from LeetCode weekly contests from July 2022 to January 2024. For each problem, we in- clude 100 test cases to ensure the generated code is assessed thoroughly.
5.3. Experiment Results on Code Generation
Baselines. For code generation, we compare different vari- ants of PF-PPO with reinforcement learning algorithms, supervised fine-tuning methods, and direct policy optimiza- tion methods. We use greedy decoding during inference and pass@1 1 (Chen et al., 2021) as the performance met- rics. For fair comparison between different baselines, we re- re-implement these baselines with the same code base and the same datasets. We also use the same reward model and the same SFT policy if applicable.
To train models for this benchmark, we construct LeetCode training datasets comprising 1,000 problems collected from the LeetCode website. For SFT, use self-generated cor- rect answers to create the SFT dataset following the method- ology in Setlur et al. (2024). For reward modeling, we generate five responses using the SFT model for each of the 400 curated prompts and selected two responses for each prompt following the similar procedure as above. For policy optimization, we used the full 1,000 prompts as our PPO query dataset train the code LLM.
Supervised fine-tuning. Starting from deepseek-6.77 we first fine-tune this policy on the SFT dataset. Other algo- rithms learn based on this SFT policy. RAFT (Dong et al., 2023) and BOND (Sessa et al., 2024) train the policy to fit the best-of-N (BoN) responses or the BoN policy via dif- ferent supervised learning losses. RAFT maximizes the log- probability of the BoN response, whereas BOND minimizes a combination of the forward and backward KI. divergence w.r.t. the BoN policy. We set the coefficient to combine these two loss terms as PROND 1.0. BOND is an iterative algorithm to fit the BoN policy based on the policy of the
Ape210K and CMATH benchmarks. Ape210K (Zhao et al., 2020) and CMATH (Wei et al., 2023a) are two popu- lar Chinese benchmarks for elementary-schoolleey math
5

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Family
Method
HvmanEval
MBPP
LeetCode
Supervised Fine-Tuning
SFT
74.2
70.8
15.2
RAFT (Dong et al., 2023)
76.9
71.3
17.8
BOND (Sessa et al., 2024)
80.8
75.2
30.0
Direct Policy Optimization
DPO (Rafailov et al., 2024)
78.4
73.7
23.0
IPO (Azar et al., 2024)
78.2
72.9
23.2
KTT (Ethayarajh al., 2024)
77.9
72.5
22,4
Ieraativ-DPOO (Pang al., 2024)
78.1
74.8
23.8
Reinforcement Learning
PPO-S (Hu et al., 2024)
78.1
73.8
25.2
PPO-M (cf. Shao et al., 2024)
80.2
75.0
29.8
PF-PPO (BoN)
75.8
71.7
16.8
PF-PPO (BR)
82.9
75.9
33.0
PF-PPO (BW)
82.4
76.2
30.4
SOTA (7B models)
Magicoder (Wei et al., 2023b)
76.8
75.7
Table The performance different algorithms on Code Generation. We compare pass@1 PF-PPO (our algorithm) against baseline methods. For each benchmark, we select the best score across 5 epochs for cach method. The highest and the second highest scores on each benchmark are highlighted bold underline respectively. experiments are based on the same code base for fair comparison, except for the scores reported by Magicoder which is the best 7B model so far.
last iteration, and train the policy for 4 iterations.
comparing with PPO-M can help distinguish the effect of collecting multiple responses and that of filtering collected responses. For fair comparison, we ensure that the computa- tional costs of PF-PPO for each iteration is no larger than those of PPO-M and PPO-S, and we refer the readers to Appendix for the detailed analysis on the computational efficiency of PPO-S. PPO-M, and PF-PPO.
Direct policy optimization. To implement direct policy opti- mization methods, we use our reward model dataset as the preference dataset required in these methods. We imple- ment DPO (Rafailov al., 2024), IPO (Azar et al., 2024), KTO (Ethayarajh et al., 2024), and iterative DPO (Pang al., 2024). For iterative DPO, we train the DPO model for three iterations. For each iteration, construct the prefer- ence dataset as follows: The prompts are sampled from the reward model dataset and responses are generated by the trained DPO model from the previous iteration (if exists) or the previous SFT phase.
Experiment results. We present the pass @ 1 results of dif- ferent methods on the three benchmarks in Table 2. The experiment results show that PF-PPO (BR) and PF-PPO ()) obtain the highest scores on these benchmarks, indi- cating the effectiveness of our method. Furthermore, we have the following observations:
Reinforcement Learning. For standard RLHF, we use the implementation from OpenRLHF (Hu et al., 2024), which incorporates several advanced PPO training techniques and has demonstrates strong performance on various bench- marks. We denote this baseline as PPO-S. For our method PF-PPO, we implement three variants (BoN, BR, and BW) as introduced the previous section. Since PF-PPO col- lects multiple responses for given prompt/context, we in- troduce another baseline called PPO-M (PPO with multiple responses) that uses all the N responses for training without filtering? The effective difference between PPO-S and PPO- M is that the buffer B in PPO-M contains more samples with the same context c but with different responses which may provide detailed token-level instruction by comparing the responses corresponding to the same context. Therefore,
IPO and KTO (improved versions of DPO) do not out- perform DPO when trained on properly selected datasets. This indicates that appropriate dataset construction can address the weaknesses of DPO found in previous papers, enabling DPO to achieve performance comparable to its improved versions.
PPO-based algorithms outperform SFT-based and DPO- based algorithms in general demonstrating that PPO is superior to these algorithms on reasoning tasks. We spec- ulate that the good performance of PPO may stem from the generalization ability of the reward model and the value network used in PPO, which can be used to trans- form trajectory-level reward modeling to token-wise ad- vantages and thus provides more fine-grained guidance. Moreover, the gap between PPO-based algorithms and the others becomes larger on the more challenging LectCode benchmark, which further highlights the advantage of RL.
?PPO-M can also be regarded as integrating GRPO (Shao et al., 2024) into PPO, which has been adopted by Deepseek- V2 (Zhu al., 2024) and Qwen2 (Yang et al., 2024).
6

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Train and Eval Rewards over Checkpoints
Pass@1 on the HumaanEval Benchmark
83
Pass@g of (T-PPO(BR)
Pass@1 of PPO-M
Passge1 of PPO-S
Reward
79
Eval Reward of PF-PPO(BR)
Train Reward of PF (FOIF))
78
Eval Reward of PPO-M
Train Reward of PPO-M
Reward
Train Reward of PPO-5
3
2
4
8
1
4
5
7
3
Checkpoint
Checkpoint
Figure 2. Left: The training and evaluation reward of PPO-S, PPO-M, and FP-PPO on HumanEval. The training reward and the evaluation reward are evaluated the samples generated the filtered policy yso and the unfiltered policy mo respectively. Right: The pass @ of PPO-S, PPO-M, and PF-PPO on the HumanEval benchmark.
on complex reasoning tasks
Tg. We observe that both the training reward and evaluation reward of PPO-M and PF-PPO surpass those of PPO-S. This indicates that sampling multiple responses from context enhances the performance of the RLHF method, consistent with the findings in Shao et al. (2024). Morcover, in terms of optimizing reward for the same given reward model, FP- PPO achieves higher equal reward compared with PPO- S and PPO-M, which indicates that the approximation made in the FP-PPO (i.c., optimizing the unfiltered policy Ta as if it were the filtered policy (4g) does not induce negative effect on its capability of optimizing the reward.
BOND achieves the highest score among the baseline methods. It demonstrates that iterative best-of-N (BoN) distillation is an effective alignment approach. We spec- ulate that BOND also benefits from its ability to reduce learning on samples with unreliable rewards by selecting the best candidate from of N samples.
Motivated by the good performance of BOND, we imple- ment PF-PPO (BoN) natural attempt to apply BoN to RL-based algorithm. However, PF-PPO (BoN) results in poor performance. This indicates that compared with SFT methods that only need good samples, bad samples for the contrastive learning purposes are also important for RL-based methods. This explains the reason why PF-PPO (BR) and PF-PPO (BW) outperform PF-PPO (BoN).
We also show the pass@) results of different algorithms in Figure 2 (right). We observe that, while PF-PPO achieves similar reward to that of PPO-M, the pass @ result of PF-PPO exceeds that of PPO-M significantly. This results from the fact that PF-PPO optimizes on the reliable region of the reward model and thus alleviate the reward over- optimization issue.
PF-PPO (BR) and PF-PPO (BW) outperform the others with larger gap challenging LeetCode tasks. We find that the accuracy of the reward model decreases on this benchmark since is more difficult for the reward model response is better than another,
5.4. Alternative Policy Filtering Strategies
distinguish whether especially when both responses contain errors. This de- creases the reliability of the reward model in the moderate reward region (cf. Figure 1). Consequently, PF-PPO (BR) and PF-PPO (BW) can improve the performance in these complex reasoning tasks by avoiding learning on unreliable rewards.
PF-PPO modifies the sampling procedure of standard PPO by sampling N responses and randomly filtering responses based on their ranks. In this part, consider other alter- natives to filter by threshold or doin-weight the responses with unreliable rewards in the sampling procedure.
Filtering based on reward thresholds. Given a reward model, we can filter the responses based on their rewards using specified threshold. This results in three strate- gies, PPO-top that only keeps the top samples whose rewards exceeding a certain threshold, PPO-top-randon that keeps also keeps random samples with 50% proba- bility, and PPO-top-botton that keeps top samples and bottom samples whose rewards are below another spec-
Training curves. To provide comprehensive view of the three algorithms, we show the details of the training process.
We first present the training curves of PPO-S, PPO-M, and PP-PPO in Figure (left). The training rewards are evalu- ated on the samples collected by the filtered policy μe and the evaluation rewards are calculated on the unfiltered policy

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Policy filtering strategies
pass @ I on
pass @ @ on MBPP
Rº based on SFT policy
Ape210K
CMATH
Human Eval
PPO-S + ORM
84.1
92.3
PF-PPO + ORM
86.2
95.1
PPO
78.1
73.8
0.782
PPO-S Oracle
82.1
90.8
PPO-M
80.8
75.0
0.886
PE-PPO (BoN)
75.8
71.7
0.454
PF-PPO + Oracle
83.8
91.2
PE-PPO (BR)
82.9
75.9
0.841
PPO-S + CRM
83.9
93.1
PF-PPO (BW)
82.4
76.2
0,952
PF-PPO + CRM
84.3
94.2
PPO-top
80.5
71.2
0,621
PPO-top-rand
81.9
75.3
0.889
Table 4. Comparison between PF-PPO and PPO-S on two math benchmarks (Ape210K and CMATH) using three different reward functions (the original reward model, the oracle model, and the combined reward model). Better results for each reward model is highlighted in bold.
PPO-top-bott
81.7
75.4
0,927
PPO-pow-1
81.0
74.2
0.926
PPO-pow-2
81.3
75.4
0.939
PPO-pow-3 81.9
76.5
0.946
Table The comparison on the pass@1 results of different policy filtering strategies HvmanEval and their corresponding It based on the SFT policy. The background are colored based on their values with blue and red indicating the minimum and the maximum respectively.
Ra can serve as a tool to predict the performance of dif- ferent policy filtering strategies. Second, different policy filtering strategies (except for BoN versions) improve the performance of the base PPO algorithms. This indicates that filtering samples with unreliable rewards can increase the signal-to-noise ratio of the reward model feedback and thus improve the performance. Third, PF-PPO strategies (which are rank-based) outperforms other strategies (which are threshold- -ased or reweighting (weighhing-based).. This may due to the fact that rank-based strategies are more robust to the reward distribution the given reward model.
ified threshold. These strategies be regarded the threshold version of PF-PPO (BoN), PF-PPO (BR) and PF-PPO (BW) respectively. The thresholds are tuned coarsely to achieve good results on a separate validation set.
Filtering based on reward reweighting, Compared with the above strategies that use thresholds, we consider a softer version that adjusts the sample weights based on their rewards, aiming at down-weight the samples with moderate and possibly unreliable rewards. Specifically, we increase the sample weight of the responses with rewards in the reliable region and decrease the sample weight otherwise. To achieve this goal, given reward model Rs that returns rewards in the range [],, we assign the weight for the sample (c, y) proportional to (ylc)) and collect samples with these weights from the buffer to train the policy network and the value network. We denote these strategies PPO-pow-k.
Discussion. The performance of different policy filtering strategies may vary across different tasks, different reward models, and different base models. Therefore, although we find that PF-PPO (BR) and PF-PPO (BW) are the best strategies in our setting, other policy filtering strategies may be a better choice in other settings. Therefore, a more practical procedure should first calculate the R2 using the given reward model and the corresponding SFT policy on the specific task and select candidate policy filtering strategies. Note that R is not perfect tool to select policy filtering strategies and we leave seeking for better predictive metrics as future research direction.
question then arises: how to choose policy filtering strategy from these strategies? To answer this question, propose to calculate the R between the rewards and the actual scores on the samples collected by different strategies, and then choose strategy with good results on this metrics. We can use the SFT policy as the unfiltered policy and calculate R² as described Section 4. Since the SFT policy obtained prior to the PPO training phase, this metric can be used to predict the results of different filtering strategies before actually conduct costly PPO training.
5.5. Experiment Results on Math Reasoning
To evaluate the effectiveness of PF-PPO in other domains and different types of reward models, we applied PF-PPO to solve math problems. We consider three types of reward models: the original reward model (ORM) that is trained on preference datasets using a Bradley-Terry model (Bradley & Terry, 1952), an oracle model (Oracle) that extracts the final answer from the response and compares it with the ground truth, and a combined reward model (CRM) that integrates the above two models, similar to the approach used in Qwen-Math (Yang et al., 2024). We compare PP- PPO (BR) to PPO-S using these reward models.
We compare theses strategies on HumanEval and present the performance of different policy filtering strategies and their corresponding R² in Table 3. We make the follow- ing observations: First, the R² of different strategies posi- tively correlate with their performance in general, indicating
We can observe that PF-PPO consistently outperforms the
8

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Task
Evaluation Set Size
BO1 Accuracy (%)
BO5 Accuracy (%)
Logic Reasoning
1,203
48.9 (+2.3)
63.8 (+2.8)
Math
1,759
69.7 (+1.1)
79.9 (+2.3)
Code
3,933
55.8 (-0.2)
67.4 (+0.1)
STEM
4,466
54.7 (-0.1)
63.1 (+0.1)
Complex Tasks
2,990
9.5 (+1.0)
14.9 (+0.6)
Instruction Following
1,525
49.6 (+1.7)
59.8 (+1.8)
Knowledge
775
47.3 (+1.9)
58.3 (+1.8)
Language Understanding
680
63.8 (+1.6)
68.4 (43.8)
Table 5. Improvement of PF-PPO compared with PPO-S wide range of tasks. We present the best-of-1 (BO1) and best-of-5 (BO5) accuracies for PF-PPO the accuracy improvement PF-PPO compared to PPO-S. results with significant improvement are highlighted bold.
PPO algorithm on these benchmarks across different reward models. In addition, the experiment results indicate that even if we can have access to the ground truth, using the oracle as the reward function does not perform as well as using a reward model (either the original reward model or the combined model). This finding is consistent with experiment results in Qwen-Math (Yang et al., 2024) and Deepseek-Man (Shao et al., 2024).
baselines. Additionally, analyze PF-PPO, standard PPO, and PPO with multiple responses in details and show that filtering samples with unreliable rewards improve the performance of the outcome policy.
Acknowledgment
The work was partially supported by The State Key Labora- tory of Novel Software Technology (KFKT2024A03).
5.6. Experiment Results on Wider Range of Tasks
Impact Statement
To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks in- cluded logic reasoning. math, code generation, STEM prob- lems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evalu- ation sets and verifiers to assess response correctness. We consider the multi-task scenario where one model trained to complete various tasks. We present the results (accuracy improvement over vanilla PPO) in Table 5. We highlight statistically significant changes (exceeding +0.5%, based on test case counts) in bold. These results demonstrate PF-PPO's consistent effectiveness across tasks.
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
References
Amini, A., Vieira, T., and Cotterell, R. Variational best-of-n alignment. arXiv preprint 20d::2200700005, 2024.
Anthropic, A.
Introducing claude, 2023,
URL
pttps://wwerr www.ww.nttrrr.cccooo httpp/c.cccccccsssss/s.o cntroducing-cllude.
6. Conclusion
Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D.,, Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 2021.
In this paper, we propose new reinforcement learning with human feedback (RLHF) method, Policy Filtration for Proximal Policy Optimization (PF-PPO), aimed at miti- gating the adverse effects of reward noise. When training the reward model using the Bradlcy-Terry approach, the reward signal is generally more reliable the high or low reward regions but less reliable in the moderate reward regions. Mo- tivated by this observation, we adopt rank-based method selectively use sample from these reliable regions more in PPO to improve the quality of the signal provided by the reward model. We conduct comprehensive experiments to demonstrate that PF-PPO consistently outperforms existing
Azar, M. G., Guo. D.D Piot. B.. Munos, R.. Rowland, M., Valko, M., and Calandriello, D. A general theoretical paradigm to understand learning from human preferences. International Conference on Artificial Intelligence and Statistics, pp. 4447-4455. PMLR, 2024.
Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., Das- Sarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., al. Training a helpful and harmless assistant with rein-

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
forcement learning from human feedback. arXiv preprint wrXii:2204.05586,, 2022.
Ivison, H., Wang, Y., Liu, J., Wu, Z., Pyatkin, V., Lam- bert, N., Smith, N. A., Choi, Y., and Hajishirzi, H. Unpacking dpo and ppo: Disentangling best practices for learning from preference feedback. arXiv preprint arXiv:2406.094: 2024.
Bradley, R. A. and Tenry, M. E. Rank analysis of incom- plete block designs: I. the method of paired comparisons. Biometrika, 324334)) 1952.
Jaques, N., Ghandeharioun, A., Shen, J. H., Ferguson, C., Lapedriza, A., Jones, N., Gu, S., and Picard, R. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv: 000:::907000 9007000556, 2019.
Chaudhari, S., Aggarwal, P., Murahari, V., Rajpurohit, T.,
Kalyan, A., Narasimhan, K., Deshpande, A., and da Silva, B. C. Rihf deciphered: critical analysis of reinforce- ment learning from human feedback for Ilms. arXiv preprint arXiv:2404.005555 2024.
Kalai, A. T. and Vempala, S. S. Calibrated language models must hallucinate. In Proceedings of the 56th Annual ACM Symposium on Theory of Computing, pp. 160-171, 2024.
Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. O., Kaplan, J., Edwards, H., Burda, Y., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code. arXiv preprint vrXiv:2107.03374 2021.
Lambert, N., Krendl Gilbert, T., and Zick, T. The history and risks of reinforcement learning and human feedback. arXiv e-prints, pp. arXiv-2310, 2023.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. Advances in neural information pro- cessing systems, 30, 2017.
Liu, T., Zhao, Y., Joshi, R., Khalman, M., Saleh, M., Liu, P. J., and Liu, J. Statistical rejection sam- pling improves preference optimization. arXiv preprint arXiv: Nwww:3399.666 2023.
Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., et al. Training verifiers to solve math word problems. arXiv preprint a
Longpre, S., Hou, L., Vu, T., Webson, A., Chung, H. W., Tay, Y., Zhou, D., Le, Q. V., Zoph, B., Wei, J., al. The flan collection: Designing data and methods for effec- tive instruction tuning. In International Conference Machine Learning, pp.
10. 12.1100.444 /4168, 2021.
Coste, T., Anwar, U., Kirk, R., and Krueger, Reward model ensembles help mitigate overoptimization. arXiv preprint arXiv:2310.0274. 2023.
PMLR, 2023.
Luo, Z., Xu, C., Zhao, P., Sun, Q., Geng, X., Hu, W., Tao, C., Ma, J., Lin, Q. and Jiang, D. Wizardcoder: Empowering code large language models with evol-instruet. arXiv preprint 230.::2300.88666 2023.
Dong, H., Xiong, W., Goyal, D., Zhang, Y., Chow, W., Pan, R., Diao, S., Zhang, J., Shum, K., and Zhang, T. Raft: Re- ward ranked finetuning for generative foundation model alignment. arXiv preprint arXiv:2304.06767, 2023.
McKinney, L., Duan, Y., Krueger, D., and Gleave, A. On the fragility of learned reward functions. arXiv preprint arXiv: 003652.00..... 2023.
Draper, N. Applied regression analysis. McGraw-Hill. Inc, 1998.
Moskovitz, T., Singh, A. K., Strouse, D., Sandholm, T., Salakhutdinov, R., Dragan, A. D., and McAleer, S. Con- fronting reward model overoptimization with constrained rlhf. arXiv preprint arXiv:2310.043733 2023.
Ethayarajh, K., Xu, W., Muennighoff, N., Jurafsky, D., and Kiela, D. Kto: Model alignment as prospect theoretic optimization. arXiv preprint arXiv:2402.013306 2024.
Gao, L., Schulman, J., and Hilton, J. Scaling laws for reward model overoptimization. International Conference on Machine Learning, pp. 10835-10866. PMLR, 2023.
Nakano, R., Hilton, J., Balaji. S., Wu, J., Ouyang, L., Kim, C., Hesse, C., Jain, S., Kosaraju, V., Saunders, W. et al.
Webgpt: Browser-assiste question anthlornnnnwweiing with hu- man feedback. arXiv preprint arXiv:21 2021.
Guo, D., Zhu, Q. Yang, D., Xie, Z., Dong, K., Zhang, W., Chen, G., Bi, X., Wu, Y., Li, Y., et al. Deepseek-coder: When the large language model meets programming the rise of code intelligence. arXiv preprint arXiv:2401.14133 2024.
OpenAL. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 2023.
Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744 2022.
Hu, J., Wu, x., Wang, W., Zhang, D., Cao, Y., et al. Open- rlhf: An casy-to-use, scalable and high- biighpeeffrmaacc rlhf framework, arXiv preprint arXiv:2405. 143, 2024.
10

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Pal, A., Karkhanis, D., Dooley, S., Roberts, M., Naidu, S., and White, C. Smaug: Fixing failure modes of pref- erence optimisation with dpo-positive. arXiv preprint arXiv:2402.13228, 2024.
P. F. Learning summarize with human feedback. Ad- vances in Neural Information Processing Systems, 33: 3008-3021, 2020.
Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
Pang, R. Y., Yuan, W., Cho, k., He, H., Sukhbaatar, S., and Weston, J. Iterative reasoning preference optimization. arXiv preprint arXiv:2404. 19733, 2024.
Pitis, S. Failure modes of learning reward models for ilms and other sequence models. In ICML 2023 Workshop The Many Facets of Preference-Bace Based Learning, 2023.
Team, Q Introducing qwen1.5, February 2024. URL https: http://wwwllm.....ta... www.ttuu....bblloo ho/blog///////oo /qwen1. 5/.
Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model secretly a reward model. Ad- vances in Neural Information Processing Systems, 36, 2024.
Wang, B., Zheng, R., Chen, I.,, Liu, Y., Dou, S., Huang, C., Shen, W., Jin, S., Zhou, E., Shi, C., et al. Secrets of rlhf in large language models part ii: Reward modeling. arXiv preprint 30Xxx:2441.06000 2024.
Wei, T., Luan, J., Liu, W., Dong, S., and Wang, B. Cmath: Can your language model pass chinese elementary school math test? arXiv preprint arXiv:2306. 16636, 2023a.
Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. Proximal policy optimization algorithms. arXiv preprint (CGiv/7700.06444 2017.
Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magi- coder: Source code is all you need. arXiv preprint arXh::2312.02120 2023b.
Sessa, P. G., Dadashi, R., Hussenot, L., Ferret, J., Vieil- lard, N., Ramé, A., Shariari, B., Perrin, S., Friesen, A., Cideron, G., et al. Bond: Aligning Ilms with best-of-n distillation. arXiv preprint arXiv:2407.14622 2024.
Wu, T., Zhu, B., Zhang, R., Wen, Z., Ramchandran, K., and Jiao, J. Pairwise proximal policy optimization: Harness- ing relative feedback for I1m alignment. arXiv preprint arXiv:2310.00211 2023.
Setlur, A., Garg, S., Geng, X., Garg, N., Smith, V., and Kumar, A. RI on incorrect synthetic data scales the effi- ciency of Ilm math reasoning by fight-fold. arXiv preprint arXiv:2406. 14532, 2024.
Yang, A., Yang, B., Hui, B., Zheng, B., Yu, B., Zhou, C., Li, C., Li, C., Liu, D., Huang, F,, al. Qwen2 technical report. arXiv preprint arXiv:2407.10671, 2024.
Shao, Z., Wang, P., Zhu, Q.. Xu, R., Song, J., Zhang, M., Li, Y., Wu, Y., and Guo, D. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024.
Yuan, Z., Yuan, H., Tan, C., Wang, W., Huang, S., and Huang, F. Rrhf: Rank responses to align language mod- els with human feedback without tears. arXiv preprint arXiv:2304.0530. 2023.
Shen, W. Hu, J., Zhao, P., He, X., and Chen, L. Advanced tricks for training large lan- guage models with proximal policy optimization. https: dot.iinkddd..nniio.... httte//bb22211111140324444322332 9433149144...611119333... 2024. Notion Blog.
Zhang, S., Chen, Z., Chen, S., Shen, Y., Sun, Z., and Gan, C. Improving reinforcement learning from human feedback with efficient reward model ensemble. arXiv preprint arXiv:2401.16633 2024.
Skalse, J., Howe, N., Krasheninnikov, D., and Krueger, D.
Defining and characterizing reward gaming. Advances in Neural Information Processing Systems, 35:9460-9471, 2022.
Zhao, W., Shang, M., Liu, Y., Wang, L., and Liu, J. Ape210k: large-scale and template-rich dataset of math word problems. arXiv preprint vxXiv:2009.11500 2020.
Song, F., Yu, B., Li, M., Yu, H., Huang, F., Li, Y., and Wang, H. Preference ranking optimization for human alignment. In Proceedings of the AAAI Conference on Artificial In- felligence, volume 38, pp. 18990-18998, 2024.
Zhao, Y., Joshi, R., Liu, T., Khalman, M., Saleh, M., and Liu, P. Slic-hf: Sequence likelihood calibration with human feedback. arXiv preprint arXiv:2305. /0425, 2023.
Zhou, J., Jiang, C., Shen, W., Zhou, X., and He, X. Leverag- ing web-crawled data for high high-tuality fine-tuning, arXiv preprint arXiv:2408. 080003, 2024.
Stiennon, N., Ouyang, L., Wu. J., Ziegler, D., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano,

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
Zhu, B., Sharma, H., Frujeri, F. V., Dong, S., Zhu, C., Jordan, M. I., and Jiao, Fine-tuning language models with advantage-induced policy alignment. arXiv preprint 2tt.:230000223.11 2023.
Zhu, Q. Guo, D., Shao, Z., Yang, D., Wang, P., Xu, R., Wu, Y., Li, Y., Gao, H., Ma, S., al. Deepseek-coder- v2: Breaking the barrier of closed-source models in code intelligence. arXiv preprint arXivv2406. //931, 2024.
Ziegler, D. M., Stiennon, N., Wu, J., Brown, T. B., Radford, A., Amodei, D., Christiano, and Irving, G. Fine-tuning language models from human preferences. arXiv preprint arXiv: /909//999000 08593, 2019.
12

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
A. Reward Model
The design of our algorithm motivated by the observation that the reward model less reliable when it yields moderate rewards. To provide more evidence that this property is universal across broader range of benchmarks, we also analyze the reward function on different benchmarks of code generation (MBPP and Lee(Code) and math reasoning (Ape210K (Zhao al., 2020) and CMATH (Wei al., 2023a)). We repeat the process in Figure 1 on these benchmarks and plot the figures in Figure 3 and Figure 4. Note that we train different reward functions based on the datasets from these two benchmarks. We observe that the property holds on these four additional benchmarks across different tasks, indicating this property may extend to broader fields.
Intuitively, this property should universal a broader range of tasks, e.g., on Helpfulness and Harmlessness tasks (Bai al., 2022). For code generation tasks, is quite common that some samples (e.g.. the response matches the known correct answer or the response contains an obvious error) are easier evaluate than others (e.g., the response tries solve the problem by novel approach). Therefore, those samples that are hard evaluate by human should also be hard instances for the reward model.
13

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
0.8
0.6
Actual Score
0.4
0.2
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Reward
The actual scores the reward values for the reward model evaluated on MBPP
0.9
0.8
Actual Score
0.7
0.6
0.5
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Reward
(b) The actual scores the reward values for the reward model evaluated LeetCode
Figure 3. We provide additional evidence that the reward model less reliable when yields moderate rewards than when it yields high or low rewards. We conduct the same statistics as in Figure but on different benchmarks. Specifically, the reward models for the MBPP and LeetCode benchmarks
trained separately using the corresponding datasets for these benchmarks. The MBPP and LeetCode benchmarks contains 378 and 1570 prompts respectively and we collect 10 responses for each prompt using fine-tuned policy. We group the responses with similar rewards and calculate the average of their actual scores (i.e., the average correctness). indicating each group by one point. To evaluate the reliability of the reward model, we repeat the pracess ten times resulting in ten lines.
14

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
1.0
0.8
Actual Score
0.6
0.4
0.2
0.0
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Reward
(a) The actual scores the reward values for the reward model evaluated Ape210k
1.0
0.8
Actual Score
0.6
0.4
0.2
-1.00
-0.75
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
Reward
The actual scores vs. the reward values for the reward model evaluated on CMATH
Figure 4. We provide additional evidence that the reward model is less reliable when yields moderate rewards than when yields high low rewards. We conduct the same statistics as in Figure but on different benchmarks. Specifically, the reward models for the Ape210k and CMATH benchmarks each prompt in the dataset using fine-tuned policy. We group the responses with similar rewards and calculate the average of their actual scores (i.e., the average correctness), indicating each group by one point. To evaluate the reliability of the reward model, repeat the process ten times resulting in ten lines.
trained separately using the corresponding datasets for these benchmarks. We collect 10 responses for
15

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
B. Qualitative results
In this section, we provide qualitative results on how responses with high/middle/Iow rewards look like and why responses with middle rewards are unreliable; and 2) the qualitative difference between the code generated by the PF-PPO policy and the standard PPO (PPO-S) policy.
B.1. Analysis on the the responses associated with different rewards
We present a prompt along with several responses, including correct response but assigned with low reward, an incorrect response but assigned with high reward, an incorrect response with low reward, and correct response with a high reward. The prompt describes coding problem that requires to convert fractions to decimals.
We have the following findings:
For the correct response but assigned with a low reward, the generated code is less clear and harder to read. For example, the code may mix several steps into one line.
For the incorrect response but assigned with high reward, the response incorrectly mixes two correct approaches. This mistake can hardly be identified by the reward model (and even GPT-4).
For the incorrect response assigned with low reward, the response contains obvious mistake which is easily detected by the reward model.
We also provide detailed analysis into the solutions to this problem. The given prompt is a coding task to convert fraction to decimal.
Given two integers representing the numerator and denominator of fraction, return
the fraction string format.
If the Fractional part is repeating, enclose the repeating part in parentheses.
3
If multiple answers are possible, return any of ther.
It is guaranteed that the length of the answer string is less than 104 for all the given inputs.
6
Please complete the following python code precisely:
" 'pyython
class Solution:
def FractionToDecimal (self, numerator: int, denominator: int) -> str:
10
This is correct response with high reward. This solution is thorough with clear variable names and properly structured steps. It is easier to read due to breaking down steps explicitly such as calculating the integer part and handling the remainder.
To solve the problem of converting a fraction to its decimal representation, we can follow these steps:
1. **Determine the sign of the resultant decimal**: both the numerator and the denominator are negative, the result should be positive. Otherwise, the result will have Leading minus sign it and only is either the numerator or the denominator is negative.
2. **Convert the numerator and denominator to positive values*x: Since the division process involves integer division and the decimal part requires modulus operation both numerator and denominator should be made positive for easier computation.
**Calculato the integer part**: The integer part is the result of integer division 10
of the numerator by the donominator.
11
4. **Manage the remainder4:: As perform integer division, the remainder will be used form the fractional part. This remainder will be treated the nurerator for the next step.
16

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
12
13
5. **Use a dictionary to manage repetition*:: To detect recurring decimals, we need to keep track of all remainders we have seen and their respective positions in the decimal expansion. If remainder repeats, it indicates recurring decimal.
14
15
6. **Build the result string**: Start with the integer part, then append decimal point. Then, use loop to compute each digit of the fractional part. Use the dictionary to detect and mark any recurring decimals.
16
17
7. **Return the result**: Finally, return the constructed string for the result.
18
19
Below is the complete Pythen code for the "Fractioncclees method:
20
21
.* "python
22
class Solution:
23
def FractionToDecime (self, numerator: int, denominator: int) -> str: if numerator == 0; return *0"
24
25
26
27
Determine the sign of the result result_sign
28
if (numerator < 0) (denominator < 0) else
29
30
Make both numerator and denominator pesitive numerator abs (numerator) denominator abs (dencminator)
31
32
33
34
Calculate the integer part of the result integer_part numerator // denominator remainder numerator denominator
35
36
37
38
If there's remainder, return the integer part if remainder == 0:
39
40
return result_sign str (intoger_part)
41
42
Prepare to build the fractional part result [] result.appent (*.") start with deciral point
43
44
45
remainder_man _mp [}
keep track of remainders and their positions
46
47
Process each place value in the fractional part position 0
48
49
while remainder not in remainder_map. (eriinder_mppremmiineeeeee
50
position
51
remainder == 10
52
result_digit = remainder // denominator
53
result. (str (result_digit))
54
remainder $= deneminator
55
if remainder == 0;
56
break
57
position +-
58
59
If there's s a repeat, insert parentheses if remainder in remainder_mapa
60
61
tepeat_stare = remainder map [emainder] result.insert (repeat_stee start 1, "(") result.append (")
62
64
65
return result_sign str (integer_part) + "" join (result)
66
67
68
This code correctly handles edge cases like zero denominator, negative numbers,
repeating decimals, and ensures that the result is less than 10"4 characters long
17

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
This is correct response but assigned with low reward. Compared with the previous response with high reward, this response mixes multiple operations in one line, making it harder understand (e.g., Line 34).
To solve the problem of converting fraction to its decimal representation, we can use the following approach:
1. Handle the sign of the result.
2. Calculate the integral part of the result, which is the part before the decimal peint.
5
3. Handle the case where the fractional part is repeating.
6 4. Calculate the fractional part and handle cycles.
Here's a the completed Python code:
10
" 'pyython
11
class Solution:
12
def fract (self, numerator: int, denominator: int) -> str: # Handle sign of result
13
14
sign = f-r if numerater denominator 0 else numerator, denominater abs (numerator), abs (denominator)
15
16
17
Calculate integral part integral part = numerator // denominator
18
19
20
Calculate fractional part remainder - numerator & denominator
22
if remainder =- 0;
23
return sign str (integral_ part)
24
25
Use a dictionary to track remainders and their positions in the fractional part to detect cycles rec__nermind = (}
26
27
28
result []
29
index =
30
31
while remai remainder not in seen_renn rree_rrmmindee inders: seen remainders remainder index result. append
32
33
append (str tremainder 10 // deneminator)
34
remainder (remainder 10) dencminator if remainder == 0;
35
36
break
37
index +=
38
39
there's s a cycle, insert the parentheses
40
if remainder in reen_remainders index = recainersss [remainder)
41
42
result.insez (index, (()
43
apsunt.append
45
return sign str (integral_part) + f.r + "..ooin
46
47
48
*** Explanation:
49
50
We first handle the signs of the numerator and denominator.
51
We then calculate the integral part of the result by integer division.
52 If the remainder zere, the fraction
an integer and we return as a string. to keep track of the positions where each remainder appears, allowing us to detect cycles.
53 We use a dictionary "seen_remainder
54 If cycle is detected, insert parentheses at the start of the repeating part. string and return it.
55 Finally, assemole the result
This is an incorrect response but assigned with a high reward. Line 32, the decimal point is added to the result list but later counted when getting wrapped by the parentheses, leading to the wrong format. This is mixture of two
18

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
correct approaches, that adds the decimal points result but an offset for this (cf. Line 44 and Line 62 in the first response) and one that outputs the decimal point separately (ef. Line 45 in the second response).
solve the problem of converting fraction to decimal string, need to handle the following cases:
3 1. +*Intogers4:; If the numerator and denominater are both integers, the decimal representation is straightforward: divide the numerator by the denominator and convert the result to string.
2. **Decimal Representaticnve: If the division produces a remainder, it indicates a repeating decimal pattern. Use dictionary to keep track of the remainders and their corresponding positions in the resultant decimal string. Whenever remainder repeats, enclose the repeating part in parentheses.
7
3. **Hlandling Negative Numbers**; Handle both pesitive and negative numbers appropristely.
Here is the complete code for the solution: 10
11 'python
12
class Solution:
13
def FractioniooDecial (self, numerator: int, denominators int) -> str: Handle sign of the result
14
15
sign f-r if numerater * denomi nator 0 else numerator, denominater abs (numerator), abs (denominatoz)
16
17
18
Calculate integral part integral_ part = numerator // denominator numerator $= deneminator
19
20
21
22
Handle zero remainder
23
numerator 0:
24
return sign + str (integrral_par)))
25
26
Prepare 1ist to stere the decimal digits result []
27
28
Dictionary to keep track of remainders and their positions Remainder_ne - (}
29
30
31
Initialize the decimal part
32
result. result.append
33
position =
34
35
Loop until the remainder repeats or becomes zero
36
while numerator
37
If the remainder has already appeared before, it's a repeating part if numerator in remainder_map:
38
39
result.insert. (remainder map [numerator), (() result.append(")
40
break
42
43
Record the position of the current remainder cemaiindr__mmnnnmeereee numerator 10
44
position
45
46
result. (str (numerator // denominator))
47
numerator $= denominator
48
position +=
49
50
return sign
str (intogral " ".join (result)
51
52
53
#*# Explanation:
54
We use a dictionary 'remainder_mape track the position each remainder seen.
This helps us identify the repeating part of the decimal if it exists.
19

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
55 loop continues until the remainder becomes zero or repeats itself. 56 If the remainder repeats, we insert the opening '(' immediately before the repeated part, and close it with ")
57 The result is assembled from the integral part, the decimal point, and the decimal digits or parenthesized pattern.
This is an incorrect response with a low reward. In Line 59-61, the program contains an obvious emoor that it cannot handle the scenario where there exists decimal part but does not contain any repeating part.
2
To solve the problem of converting fraction to its decimal representation, need to consider the following steps:
5 1. **Hanndlig ing žero Denominaterwy If the denominator is zero, it's impossible to divide, so we should return "NaN" (Not Number) or handle it according to the problem constraints.
2. **ilandling Negative Numbers**: If either the numerator or denominator is negative, the sign of the result should be negative.
3. Calalclate the Integral Part**: point, which can be calculated by integer division.
The integral part is the part before the decinal.
10
11
4. **Caiculate the Fractional Part*x:
12
Use dictionary to keep track of the positions where the remainder started repeating.
13
Divide the remainder by the denominator repeatedly, building the fractional part digit by digit.
14
If the remainder repeats, it indicates recurring decimal, and should insert parentheses mark it.
15
16
Here's 3 the complete code implementation:
17
18
'' ''yython
19
class Solution:
20
def fractionToDecime (self, numerator: int, denominator: int) -> str: # Handle zero denomi nator
21
22
if denominator =- 0: "NaN"
23
24
25
Handle signs result_sigg '-r if (numerator < 0) (denominator < 0) else
26
27
numerator, denominater abs (numerator), abs (denominator)
28
29
Calculate the integral part integral_part = numerator // denominator numerator $= deneminator
30
31
32
33
Initialize the fractional part 1ist and dictionary to track remainders Eractional_part = []
34
35
remainder_inen
[]
36
37
Continue while there remainder position -
38
39
while numerator > 0:
40
# If the remainder has already occurred before, it's s repeating decimal if numerator in remainder index: break
41
42
43
44
Record this remainder's position remaindes gemainder_inder [numerator] positien
45
46
47
Cal colculate the next digit
20

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
48
numerator 10 fractional parrttattt.. append rstr (numerator // denominator) numerator $- deneminator
49
50
51
position += 1
52
53
Build the result if not fractional_part:
54
55
# No repeating part, return integral part
56
return result_sign str(entggral_partt
57
else:
58
Repeating part detected
59
aapeating_stare = rema inde_iindee inde_iindex(ummrrttoy
60
return result_sign str (integrall_arrt) + ',' ...jin (fractior _p repeating startt)
(' " "".join part (wepearnggsttarteeeeeto ()
62
63
64
Explanation:
65
66
Integral Part*:: We calculate the integral part as 'numerator // denominator',
67
-Non-rapatting Part**: Re use a dictionary to track positions of zemainders, and if the same remainder appears again, we know the fraction is repeating.
68
insert parentheses around Repeating Part*:: When repeating part is detected, the repeating part.
69
* Edge Cases**: We handle edge cases such as negative numbers, zero denominator, and ensuring the result is within the constraints of the problem.
70
71
This implementation correctly handles both whole number and repeating decimal results adhering to the problem statement's requirements.
B.2. Analysis on the responses generated by PF-PPO compared with standard PPO (PPO-S)
We compare the answers from PF-PPO and PPO-S respectively for the prompts, and conclude that the answer from the PF-PPO policy is more inclined to follow a standard approach and the response is more concise (or shorter), making it easier to understand and implement.
Specifically, we present their responses for the prompt that request the agent to write a code to find all safe nodes a directed graph. The two responses given by PF-PPO and PPO-S are both correct. However, the difference is that PF-PPO adopts the deep first search (DFS) method while PPO-S adopts the topological sorting approach. The logic is simpler for DFS and the implementation is easier, making it easier understand. Both approaches have roughly equivalent time and space costs, but the DFS method is slightly more space efficient.
These findings suggest an advantage in terms of readability and implementation simplicity when using the PF-PPO policy (e.g., using recursion instead using a queue to track BFS).
Prompt (find all safe nodes in a directed graph):
There is a directed graph of n nodes with each node labeled from to n 1. The
graph is represented by 0-indexed 2D integer array graph where graph [i] is integer array of nodes adjacent node i, meaning there is an edge from node to each node in graph[i].
2 node is a terminal node if there are no outgeing ig edges. A node is safe node if every possible path starting from that node leads to a terminal node (or another safe node).
Return an array containing all the safe nodes of the graph. The answer should be scrted in ascending order.
4
Please complete the follewing python code preciscly:
'pyythnn
class Solution:
def SeaantualSafelie (self, graph: List [List[int]]] -> dist(int]]
9
21

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
The response from PF-PPO:
solve the problem of finding all safe nodes in directed graph, we can use topological sorting approach. The idea 1s to explore the graph using depth-first search (DFS) and maintain a status for cach node indicating whether it is safe or not. Here's the step-by-step breakdown of the solution:
1. **Initialize Status**:
will use status array to track the safety of each node means the node is not visited yet, 1 means it is a safe node, and -1 means it is currently in the recursion stack (i.e., being processed in the current DFS path).
2. **DFS Function**:
8
For each node, if it's safe or not is already determined (i.e., status is not 0)
return the status.
Set the status to -1 indicate that
currently processing it.
10
For each neighbor of the current node, call DFS. If any neighbor returns false,
it indicates that this path has a cycle and the node is not safe.
11
If no cyele is found for all neighbors, mark the node as safe and return true.
12
13
3. Iterate over all nodes**: Starting from each node that is visited, perform
DFS.
14
15
4. **Collect Results**: All nodes that return true from DFS are sate nodes. Collect these node indices and return them in ascending order.
16
17
Here's s the complete Python code for the solution:
18
19
' 'pyython
20
from typing import List
22
class Solution:
23
def eventualSafeNodeer (self, graph: List (List -> [int]: len (graph)
24
25
status: for unvisited, for safe, -1 for processing status [0] + n
26
27
28
def dfs (node):
29
If the node already processed (i.e., marked safe unsafe), return its status
30
if [aatus/nnde]!!! 0: return status/nodel ==
31
32
33
Mark the node as being processed status [nodc] = -1
34
35
36
Visit all neighbors
37
for neighbor in [radh[node]:
38
any neighber unsafe or in cycle, the node unsafe if not dís (neighbor):
39
40
return False
41
42
neighbors are safe, mark the node safe and return true status [node] -
43
44
return True
45
46
Try DFS from each node safe_nodes []
47
48
for in rango if dfs iife_nodes.append
49
50
51
52
return sa sade_nodes
22

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
53
54
55
444 Explanation:
56
Toooloogical Sorting+:: We are using DFS to check if path exists from node to terminal node. If cycle detected, the node is not safe.
57
**DS Function=s: The 'dfs" function determines if node safe by checking all of its neighbors recursively. If any neighbor is unsafe or in a cycle, the node is unsafe.
58
Status Tracking**; The status array is used to track if node is safe not, and helps avoid redundant computations.
59
Result collection++: All nodes that return Truee' from the 'ds function are safe nodes, and are added to the result list.
60
61
This approach efficiently identifies safe nodes by using topological sorting approach and depth-first search approach to detect cycles.
The response from PPO-S:
2
To solve this problem, depth-first search (DF3) approach to determine the terminal and safe nodes in the graph. The topological sort ensures that we process nodes the order of their cut-degres, which is suitable for di rected acyclic graphs (DAGs), where no cycles can form.
tred[topologicaal sort approach) combined with
4
The idea is to traverse the graph using DFS, keeping track of the current path during traversal to detect cycles. Nodes that do not lead to cycle are considered terminal, and nodes that are part of cycle or are part of path that leads to cycle are considered unsafe.
Here's the Python implementation of the solutien:
9
'pyythnn
10
from typing import List
from collections import defaultdict, deque
12
13
class Solution:
14
def eventualSa Saanttuaaaaaeoeeeee feNodes (self, graph: (Listt()))))) -> list Initialize dictienary store the cut-degree of each node out__degree out degree defaultdict (int)
15
16
17
# Initialize a dicticnary to store the dependencies (incoming edges) of each node
18
dependencies = defaultdict (list)
19
20
5 = len (graph) # Build the dependencies and out-degres dictionaries for in range (n)
21
22
23
out_degree(u] len (graph)[[)
24
for in graphlul: dependencies [v] aap.ledd append
25
26
27
Initialize queue for BFS and set to keep track of safe nodes queue deque ()
28
29
safe_nodes set ()
30
31
Add all nodes with outgoing edges (terminals) to the queue for in range (n):
32
33
if out_ácgree [u]
34
(u)
35
36
BFS to find safe nodes
37
while queue:
38
u queue.
23

---

Policy Filtration for RLHF to Mitigate Noise in Reward Models
39
safe_nodes add (u)
40
For each node v that depends on u, decrement its out-degree If v's out-degree becomes zero, it becomes terminal for in dependencies(ul: out [ut_degree(v] if out out__egree [v] 0: qucuc. appendd)
41
42
43
44
45
46
47
Sort and return safe nodes return sorted (safe_ncdes)
48
49
50
51
*** Explanation:
52
1. #xDependencieswe We build 'dependencies' dictionary, where each key is node and the values are a list of nodes that depend on the key. This is used to track back edges.
53
2. ++uttdegree+*: We also keep track of the cut-degree of each node. This is used to terminal (L.e., have outgoing edges).
determine which nodes
54
3. Topological Sort (A: We start by adding all nodes with no outgoing edges ( terminals) to the queue. We then process each node from this queue. For each node we mark it as safe, reduce the out-degree of its dependencies, and if any of those dependencies become terminals (i.e., have no outgoing edges), they are added to the queue. This ensures process all nodes that part of cyeles and prevent future revisits to nodes involved such cycles.
55
4. Resultt**: The safe nodes are collected and sorted and returned as the final result.
56
57
This approach ensures that we correctly identify terminal nodes and use them to establish safe nodes, as well as handle cycles the graph.
C. Discussion on the computational costs
PPO-S, PPO-M, and PF-PPO all collect different number of responses per query and train using different number of samples. For clarity, we list the computational complexity of these algorithms in Table Note that, for all algorithms, we select the best checkpoint on the evaluation and report the performance of this checkpoint. Combining the results in Table 6 and Figure 2, the same, and the only difference is that PPO-M is more likely to learn from different responses with the same query in the same batch or adjacent batches, which improves the performance. Second, the computational complexity of PF-PPO is less than that of PPO-S and PPO-M, while PF-PPO outperforms them. This indicates the effectiveness of our method.
draw the following conclusions: First, the total computational complexity of PPO-S and PPO-M is almost
PPO-S
PPO-M
PF-PPO (BR /W)
Queries sampled per iteration Responses sampled per query
5n
1
5
5
#Query-response pairs per iteration
5n
5n
5n
Reward model forward pass per iteration
5n
5m
5n
Critic forward&backwaat pass per iteration
5nm
5nmm
2mm
Policy forward&&acke pass per iteration
5nm
5mm
2nm
HumanEval
100%
+2.69%
+6.15% / +5.51
MBPP
100%
+1.63%
+2.85% / +3.25%
LeetCode
100%
+1 +18.25%
+30.95% / +20.63%
Table Comparison of computational complexity and the performance PPO-S, PPO-M, and PF-PPO. We of queries in the PPO query dataset, and use m to denote the number of PPO epochs (i.c., cach query-response pair is used to accumulate loss and gradient for m times on average). PPO-M and PF-PPO collect 5 responses per query, and PF-PPO select out of the N = 5 responses (on average) for network update. We also show the performance improvement of PPO-M and PE-PPO based on PPO-S.
denote the number
24


Reviews:
[{"review_id": "sVD4bEGhh0", "reviewer": "Reviewer_Dkmi", "summary": "The authors introduce Policy Filtration for Proximal Policy Optimization (PF-PPO). Their key insight is that the reward signal is more useful in cases of high-reward or low-reward and design an algorithm around exploiting this by filtering the samples used in PPO based on their quality as measured by some heuristic (R-squared). They also construct the LeetCode Contest benchmark, consisting of 160 weekly LeetCode problems from July 2022 to Janurary 2024\n\nThey evaluate their algorithm on the HumanEval, MBPP, Ape210K and CMATH benchmarks as well as a LeetCode benchmark, testing against a range of SFT, DPO and PPO methods, demonstrating strong results over their chosen baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer Rkmi\n\nWe thank the reviewer for the careful investigation and fruitful discussion.\n \n## Generalization to Messy Reward Scenarios\n \nWhile our experiments focus on code and math tasks with verifiable rewards, we acknowledge the importance of evaluating PF-PPO in scenarios with noisier or subjective rewards (e.g., safety, helpfulness). Our core insight—filtering ambiguous samples to retain high/low-reward extremes—is task-agnostic and relies only on the relative reliability of reward signals. Moreover, our additional experiment results shown in the post “Response to All Reviewers” should also support this claim since the tasks such as language understanding and instruction following are associated with different levels of reward accuracy (e.g., involving subjective rewards).\n\n# Response to All Reviewers\n\nWe sincerely thank the reviewers for their constructive feedback and insightful suggestions. \n\nOne major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\\pm 0.5$, based on test case counts) are **bolded**. These results demonstrate PF-PPO’s consistent effectiveness across tasks.\n\n| Task (Evaluation Set Size)   \t| BO1 Accuracy (%) | BO5 Accuracy (%) |\n|----------------------------------|------------------|------------------|\n| Logic Reasoning (1203)       \t| 48.9 (**+2.3**)  | 63.8 (**+2.8**)  |\n| Math (1759)                  \t| 69.7 (**+1.1**)  | 79.9 (**+2.3**)  |\n| Code (3933)                  \t| 55.8 (-0.2)  \t| 67.4 (+0.1)  \t|\n| STEM (4466)                  \t| 54.7 (-0.1)  \t| 63.1 (+0.1)  \t|\n| Complex Tasks (2990)         \t| 9.5 (**+1.0**)   \t| 14.9 (**+0.6**)  \t|\n| Instruction Following (1525) \t| 49.6 (**+1.7**)  | 59.8 (**+1.8**)  |\n| Knowledge (775)              \t| 47.3 (**+1.9**)  | 58.3 (**+1.8**)  |\n| Language Understanding (680) \t| 63.8 (**+1.6**)  | 68.4 (**+3.8**)  |\n\nWe thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.", "claims_and_evidence": "The primary claims of the paper are the following:\n-The reward model used in RLHF is primarily more reliable when it comes to exemplar samples (samples given a high score/reward). \n-Because of the former, a model trained with RLHF with filtering such that the primary samples used are exemplar positive or negative samples should yield higher performance.\n\nThese claims are supposed by the following: \n-across a large number of experiments, (10 outputs for 160 prompts across 10 trials) the authors show that the actual reward aligns with the predicted most closely as both the predicted score gets higher.\n-The authors demonstrate PF-PPO outperforms the other training methods.", "methods_and_evaluation_criteria": "The authors apply three different filtration methods, 'best-of-N' (BoN), always selecting the top response; 'best-of-random' (BR), 50% chance to select the best response, 50% chance to select any other policy; and 'best-worst' (BW) a coin flip on whether the best or the worst policy is selected. They use the R-squared between the predicted and the actual score and show a high linear correlation between BR and BW, even higher than the R-squared with no filter.\n\nThe benchmarks evaluated on consisted of a range of Math and Coding benchmarks. These are fitting evaluation benchmarks as both Math and Coding reward models tend to be easier to train due to how verifiable the rewards are. This attribute makes these benchmarks ideal for the authors method, as a messy reward signal could call into question the source of a decreased performance.", "theoretical_claims": "No proofs were presented in the paper. The correctness of theoretical claims was provided by the authors (see R-squared verification above).", "experimental_designs_or_analyses": "I believe the experimental design with respect to the baselines and evaluation benchmarks is sound (Please see 'Methods And Evaluation Criteria')\n\nI agree with the range of methods covered as baselines. They all focus around training an LLM to be better at a specific domain, and I do not believe other zero-shot or in-context learning examples are relevant. The authors primary contribution is an improvement over PPO and this is compared against. They also demonstrate that their approach is able to raise vanilla PPO over other methods such as DPO or BOND-SFT in, for example, HumanEval.\n\nThese results also correlate with their R-squared results with BR and BW improving performance while BoN harms performance.", "relation_to_broader_scientific_literature": "The key contributions of this paper are in their insight of the tendency of reward models to be most accurate in the extrema of the reward distribution and filtration method they developed for applying this to PPO along with their LeetCode benchmark.\n\nBoth are very relevant to RLHF and the latter will be a good, new environment for evaluating the new reasoning models.", "essential_references_not_discussed": "None that I am aware of.", "other_strengths_and_weaknesses": "Strengths:\n-The method is relatively straight-forward conceptually but the authors both verify its correctness and show that leveraging it can lead to increases in performance in relevant benchmarks.\n-The authors verifiably demonstrate the correlation between the reward model predictions and the actual rewards in cases of high/low reward.\n-A good range of benchmarks are selected and the author's method demonstrates gains over all of them when expected\n-The authors also introduce a new benchmark for evaluating their method\n\nWeaknesses:\n-While testing was done primarily in benchmarks with easily verifiable rewards, it was not studied what happens when this is not the case", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Did the authors consider using a benchmark with less verifiable rewards as an ablation? In such a case where the policy filtration may be less accurate in some cases, how does this performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mznWCfhO8C", "reviewer": "Reviewer_DFFh", "summary": "This paper introduces a novel method, Policy Filtration for Proximal Policy Optimization (PF-PPO), to address the challenge of reward model inaccuracy in Reinforcement Learning from Human Feedback (RLHF). The authors propose a filtering mechanism to select samples with more reliable rewards, thereby improving the signal-to-noise ratio during policy learning. Motivated by empirical observations on the reward models, PF-PPO achieves this goal by excluding samples with moderate rewards. The method is validated on code generation and math reasoning tasks, demonstrating significant performance improvements.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response to Reviewer DFFh\n \nWe thank the reviewer for the positive and constructive feedback.\n \n## W1: Theoretical Motivation\n\nWhile PF-PPO is empirically driven, we ground our approach in the signal-to-noise ratio principle: ambiguous samples (mid-reward) introduce conflicting gradients during PPO updates, destabilizing training. By filtering these, PF-PPO reduces variance in policy updates. We will formalize this intuition in future work by analyzing gradient variance under different filtration strategies.\n \n## W2: Dependency on Reward Model Quality\n \nWe believe that our algorithm will yield improvement over different rewards (either generated by reward models or provided by oracles) since filtering out less reliable samples should help across different settings. One example is that, even with perfect rewards (Table 4), PF-PPO improves performance by filtering suboptimal but correct samples (e.g., verbose code), allowing the policy to prioritize concise, high-quality solutions (see Appendix B). Please also refer to the new set of our experiment results shown in the post “Response to All Reviewers” which indicate that our method is robust to rewards for different fields.\n\n# Response to All Reviewers\n\nWe sincerely thank the reviewers for their constructive feedback and insightful suggestions. \n\nOne major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\\pm 0.5$, based on test case counts) are **bolded**. These results demonstrate PF-PPO’s consistent effectiveness across tasks.\n\n| Task (Evaluation Set Size)   \t| BO1 Accuracy (%) | BO5 Accuracy (%) |\n|----------------------------------|------------------|------------------|\n| Logic Reasoning (1203)       \t| 48.9 (**+2.3**)  | 63.8 (**+2.8**)  |\n| Math (1759)                  \t| 69.7 (**+1.1**)  | 79.9 (**+2.3**)  |\n| Code (3933)                  \t| 55.8 (-0.2)  \t| 67.4 (+0.1)  \t|\n| STEM (4466)                  \t| 54.7 (-0.1)  \t| 63.1 (+0.1)  \t|\n| Complex Tasks (2990)         \t| 9.5 (**+1.0**)   \t| 14.9 (**+0.6**)  \t|\n| Instruction Following (1525) \t| 49.6 (**+1.7**)  | 59.8 (**+1.8**)  |\n| Knowledge (775)              \t| 47.3 (**+1.9**)  | 58.3 (**+1.8**)  |\n| Language Understanding (680) \t| 63.8 (**+1.6**)  | 68.4 (**+3.8**)  |\n\nWe thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.", "claims_and_evidence": "The claims are supported by provided evidence.\n1  The authors claim that the reward model is less reliable for responses with moderate rewards compared to those with high or low rewards. This claim is supported by empirical evidence from experiments on code and math tasks where the reward model's reliability is analyzed across different reward regions. \n2 The authors also claim that the filtering strategies can be selected by measuring the R2 between the true rewards and the predicted rewards on the filtered samples. This claim is supported by monitoring this measure and the final performance across different strategies.", "methods_and_evaluation_criteria": "The proposed methods (PPO plus a sample filter) make sense in general, and the evaluation criteria (pass@1 and accuracy) follow the standard for the code and math tasks.", "theoretical_claims": "The paper does not present explicit theoretical claims.", "experimental_designs_or_analyses": "I have checked the experimental designs and gone through the analysis on the experimental results and computational costs.", "relation_to_broader_scientific_literature": "The paper builds on the foundation of RLHF and addresses the critical issue of reward model inaccuracy. It relates to prior work on improving reward model accuracy and methods to mitigate reward over-optimization. The proposed method follows a different path to address the issue of reward model inaccuracy, i.e., to analyze which samples are associated with inaccurate rewards and filter them out. This path is interesting but seems too empirical.", "essential_references_not_discussed": "The paper includes a comprehensive survey of related literature. Although Deepseek-math and Qwen-math employ a combination of a dense reward model and a rule-based reward model, claiming advantages for this configuration, I want to highlight the ongoing debate spurred by the success of DeepSeek-R1. Specifically, the debate centers on whether code or math tasks truly require a learned reward model. By leveraging long-chain-of-thought (CoT) reasoning, the DeepSeek team employs only the sparse true reward for these tasks. It would greatly benefit the authors to include a discussion on this debate in their paper.", "other_strengths_and_weaknesses": "S1 The idea of filtering samples based on reward reliability is easy to implement to address the reward inaccuracy issue issue in RLHF.\nS2 The authors provide extensive experiments on multiple benchmarks and different tasks to demonstrate the effectiveness of their method.\nS3 The findings are applicable to a wide range of tasks, including code generation and math reasoning.\n\nW1 This paper is mostly motivated by empirical observations and lacks theoretical analysis on why it works.\nW2 The effectiveness of PF-PPO is highly dependent on the quality or the property of the reward model. Although the paper provides experimental results to demonstrate the property applies to reward models trained in different domains, these is no strong evidence for the effectiveness of this method in other scenarios, e.g., where the reward model is poorly trained or biased.", "other_comments_or_suggestions": "See the questions below.", "questions_for_authors": "1 The method is proposed to address the inaccurate reward issue. Why this method also improves the results when the reward is given by the oracle (cf. Table 4)?\n2 For the computational costs, the authors state that PF-PPO and standard PPO and evaluated in a fair manner using approximately the same number of samples (including the filtered samples). Since the performance is reported as the best score during the whole training process, how does PF-PPO converges compared to standard PPO? It is possible PF-PPO converges slowly and therefore corresponds to a poorer computational efficiency.\n3 Are there other metrics besides R2 that could be used to select filtering strategies? How do these metrics compare in terms of predicting performance?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pFw0U4QeLN", "reviewer": "Reviewer_2Fwc", "summary": "This paper introduces Policy Filtration for Proximal Policy Optimization (PF-PPO), a reinforcement learning from human feedback (RLHF) method that addresses reward model noise by selectively training on samples where rewards are most reliable. Observing that reward models are more accurate for extreme (high/low) rewards than moderate ones, PF-PPO filters responses using strategies like Best-Random (BR) and Best-Worst (BW), guided by the coefficient of determination to optimize reward-actual score alignment. Evaluated on code generation (HumanEval, MBPP, LeetCode) and math reasoning (Ape210K, CMATH) tasks, PF-PPO is claimed to achieve state-of-the-art performance for 7B models, outperforming standard PPO and other baselines by reducing reward over-optimization and enhancing training efficiency through noise mitigation.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response for Reviewer 2Fwc\n\nWe appreciate the reviewer’s questions.\n\n##  Why does BW perform the best?\nBW selects extreme samples with high/low rewards, which are most reliably aligned with actual scores (Fig. 1). Mid-reward samples often mix correct/incorrect elements (e.g., non-standard solutions with partial correctness), confusing the policy. BW maximizes the proportion of high-confidence samples, and thus improves learning efficiency.\n \n## Why using the coefficient of determination (R2), why it measures the reliability, and how to calculate it in the experiments.\nR² measures how well the reward model ranks samples by their true scores/quality. The filtration strategy with high R² ensures remaining samples provide reliable learning signals (i.e., those rewards can well indicate the true quality of the responses). \nCalculation steps:\n   - Generate multiple responses per prompt using the SFT policy.  \n   - Filter responses using the given strategy (e.g., BW).  \n   - Compute R² between reward scores and ground-truth scores.  \n \n## Is the reward model $r_\\phi$ mentioned in line 144 the same as the reward model $R_\\phi$?\n\nThis was a typo, and it should be $R_\\phi$ in line 144. We will correct this in the revised manuscript. The reward model in line 144 should be $R_\\phi$ (not $r_\\phi$). This will be corrected in the manuscript.\n\n# Response to All Reviewers\n\nWe sincerely thank the reviewers for their constructive feedback and insightful suggestions. \n\nOne major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\\pm 0.5$, based on test case counts) are **bolded**. These results demonstrate PF-PPO’s consistent effectiveness across tasks.\n\n| Task (Evaluation Set Size)   \t| BO1 Accuracy (%) | BO5 Accuracy (%) |\n|----------------------------------|------------------|------------------|\n| Logic Reasoning (1203)       \t| 48.9 (**+2.3**)  | 63.8 (**+2.8**)  |\n| Math (1759)                  \t| 69.7 (**+1.1**)  | 79.9 (**+2.3**)  |\n| Code (3933)                  \t| 55.8 (-0.2)  \t| 67.4 (+0.1)  \t|\n| STEM (4466)                  \t| 54.7 (-0.1)  \t| 63.1 (+0.1)  \t|\n| Complex Tasks (2990)         \t| 9.5 (**+1.0**)   \t| 14.9 (**+0.6**)  \t|\n| Instruction Following (1525) \t| 49.6 (**+1.7**)  | 59.8 (**+1.8**)  |\n| Knowledge (775)              \t| 47.3 (**+1.9**)  | 58.3 (**+1.8**)  |\n| Language Understanding (680) \t| 63.8 (**+1.6**)  | 68.4 (**+3.8**)  |\n\nWe thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.", "claims_and_evidence": "It is unclear to me why BW performs the best.\n\nIs reward model $r_{\\phi}$ mentioned in line 144 the same as the reward model $R_{\\phi}$?", "methods_and_evaluation_criteria": "While $R^2$ correlates with performance, the causal link between it and policy improvement is assumed rather than proven. It seems to be calculated using SFT policy responses, and its generalization to RL-trained policies is not rigorously validated.", "theoretical_claims": "No theoretical claims. Providing some fundamental theoretical results would be better.", "experimental_designs_or_analyses": "More detailed discussion on experiment results would be better for me to understand the insights of the results. For example, why the current design works and what are the implications.", "relation_to_broader_scientific_literature": "Limited new contributions", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "It is unclear why using the coefficient of determination ($R^2$), how to calculate it in the experiments, and why it measures the reliability.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sypopHvP6G", "reviewer": "Reviewer_kkSM", "summary": "This paper finds that the reliability of the reward model varies across responses assigned with different rewards. Motivated by this fact, this paper considers filtering the samples whose rewards may be unreliable to improve the signal-to-noise ratio during policy learning, resulting in Policy Filtration for Proximal Policy Optimization (PF-PPO). To choose a proper policy filtering strategy, the authors use the coefficient of determination between the rewards and actual scores on filtered samples as the metrics to help us find promising strategies since it measures how well the rewards filtered by PF-PPO indicate real performance. The authors provide extensive experiments to validate the effectiveness of PF-PPO in code generation and math reasoning tasks. In code generation, PF-PPO achieves the state-of-the-art performance of 7-billion-parameter models on HumanEval (+7.9%), MBPP (+0.7%), and LeetCode Contest (+10.0%) which is a newly-created and more challenging benchmark. In math reasoning, PF-PPO yields performance increase using different reward models and benchmarks (Ape210K and CMATH).", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "# Author Response for Reviewer kkSM\n\nWe thank the reviewer for highlighting these points.\n\n## Novelty of this paper\nWhile simple, PF-PPO’s contribution lies in its *universal effectiveness* for RLHF noise mitigation. Key innovations:  \n1. **Empirical Validation:** Extensive experiments across domains (see extended results) demonstrate consistent improvements.  \n2. **Practical Impact:** Addressing reward noise—a critical but underexplored RLHF challenge—via a task-agnostic approach.  \nWe also refer the reviewer to check the new experiment results on various tasks (in “Response to All Reviewers”) and take our effort to demonstrate universal effectiveness of our method into account. \n\n## Lack of theoretical derivation\nThough heuristic, PF-PPO is motivated by empirical evidence that mid-reward samples harm convergence (likely due to conflicting gradients). Future work will formalize this via gradient variance analysis.  \n \n## Why sample quality matters in PPO?\nPPO relies on accurate advantage estimates. Noisy rewards distort advantage estimates, leading to suboptimal updates. Ablations (e.g., PF-PPO vs PPO-M) show that filtering unreliable samples stabilizes training by ensuring gradients derive from high-confidence data.\n\n# Response to All Reviewers\n\nWe sincerely thank the reviewers for their constructive feedback and insightful suggestions. \n\nOne major advantage of our method is its simplicity, universality, and effectiveness. To further validate the broader effectiveness of our method, we conducted experiments across diverse domains using Doubao-25k (policy and reward model backbone). Tasks included logic reasoning, math, code generation, STEM problems, complex tasks, instruction following, knowledge QA, and language understanding. Each task has distinct evaluation sets and verifiers to assess response correctness. Results (accuracy improvement over vanilla PPO) are shown below. Statistically significant changes (exceeding $\\pm 0.5$, based on test case counts) are **bolded**. These results demonstrate PF-PPO’s consistent effectiveness across tasks.\n\n| Task (Evaluation Set Size)   \t| BO1 Accuracy (%) | BO5 Accuracy (%) |\n|----------------------------------|------------------|------------------|\n| Logic Reasoning (1203)       \t| 48.9 (**+2.3**)  | 63.8 (**+2.8**)  |\n| Math (1759)                  \t| 69.7 (**+1.1**)  | 79.9 (**+2.3**)  |\n| Code (3933)                  \t| 55.8 (-0.2)  \t| 67.4 (+0.1)  \t|\n| STEM (4466)                  \t| 54.7 (-0.1)  \t| 63.1 (+0.1)  \t|\n| Complex Tasks (2990)         \t| 9.5 (**+1.0**)   \t| 14.9 (**+0.6**)  \t|\n| Instruction Following (1525) \t| 49.6 (**+1.7**)  | 59.8 (**+1.8**)  |\n| Knowledge (775)              \t| 47.3 (**+1.9**)  | 58.3 (**+1.8**)  |\n| Language Understanding (680) \t| 63.8 (**+1.6**)  | 68.4 (**+3.8**)  |\n\nWe thank the reviewers for their rigorous feedback, which has strengthened our analysis. The revised manuscript will incorporate these responses, clarify methodological details, and emphasize PF-PPO’s broader applicability.", "claims_and_evidence": "The claims made in this paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria (e.g., benchmark datasets) make sense.", "theoretical_claims": "There is no theoretical result in this paper.", "experimental_designs_or_analyses": "The experiments look reasonable.", "relation_to_broader_scientific_literature": "This paper is relevant to the literature.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "Strengths:\n\n1. This paper studies a well-motivated and important problem in the RLHF and LLM community, i.e., mitigating noise in reward models for RLHF. This topic and the experiments on math reasoning and code generation is timely.\n2. The authors conduct experiments on multiple LLM benchmarks, including HumanEval, MBPP, LeetCode Contest, Ape210K and CMATH, to evaluate their approach, and show an improved performance compared to the baselines.\n3. This paper is well-written and clearly organized.\n\nWeaknesses:\n\n1. The proposed Policy Filtration PPO approach seems very straightforward. The main idea of this approach is to generate multiple responses, use a weight vector (assign more weight to the response that has a higher reward) to combine them to generate a new response, and adopts this combined response in LLM finetuning. Is there any other novelty in this approach?\n2. This Policy Filtration PPO approach seems very heuristic. There is no theoretical result, or theoretical derivation that guides the design of this approach.\n3. The authors should explain more on the reason why this Policy Filtration PPO approach achieves better performance in experiments, except the simple idea of “generating multiple candidate responses and choosing the best one”. For example, why is the quality of samples (i.e., responses $y$) important for algorithm PPO? Why improving the quality of samples in algorithm PPO can achieve good performance?", "other_comments_or_suggestions": "Please see the weaknesses above.", "questions_for_authors": "Please see the weaknesses above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes policy Filtration to deal with noise in reward models. Reviewers are split in their final recommendations, with the main concern being the lack of theoretical support and deeper analysis. Through discussions, reviewers who are on the fence, are fine with a more positive outcome on the paper given the strong empirical evidence. The insight on the tendency of reward models to be most accurate in the extreme and the new evaluation benchmarks contribute to the LLM fine-tuning literature.
Decision: Accept (poster)